Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a method for representation of an observational context by optimization of a contrastive learning objective. In contrast to (conditional) neural process modeling, this avoids specification of a likelihood, which may be difficult and ultimately unnecessary in tasks that only require prediction of a downstream label. Experimental results illustrate the advantages of this method over related neural process models on such tasks.
main_review: This paper suggests a contrastive learning alternative to likelihood-based methods for training neural processes, which are themselves a recent attempt to overcome the computational issues known to hinder inference in stochastic (Gaussian) process models. In tasks that depend on some downstream property of the prediction, rather than the prediction itself, reconstruction is unnecessary and likelihood-free methods may be used for learning the context representation that features in the neural approximation to the posterior. Here, the authors propose a contrastive learning objective based on InfoNCE.

The paper is clear and detailed, including careful documentation of experimental data, preprocessing, and hyperparameters in the Appendix. The synthetic experiment offers a clear illustration of the relative advantage over (conditional) neural processes when the likelihood is misspecified. Experiments on image and dynamical process data show that the framework can be tailored to address a broader range of tasks (targeted and untargeted, i.e. conditional or not on the prediction at a given index) than FCLR and MetaCDE, two recent proposals for contrastive learning.

My main concern is that the stochastic process perspective taken in the paper seems only tenuously related to the actual training and prediction schemes detailed in Sec 3.1-3.3. In practice, the method learns an embedding for observational contexts via minimization of a contrastive loss, and this embedding is subsequently applied deterministically at test time to generate a fixed-dimensional feature representation for a linear model of the downstream task. The only notion of a predictive distribution that remains is from this linear modeling step (Appendix D, p. 2); there is no posterior uncertainty over contexts. These issues undermine the framing of this method and its comparison to (conditional) neural processes, which intentionally retain an emphasis on quantifying uncertainty. 

limitations_and_societal_impact: The authors acknowledge some limitations in terms of their model's inability to reconstruct data in the observation space, in contrast to other neural process methods. Per my concern above, I would like to understand whether the limitations are in fact more broad from the standpoint of modeling with stochastic processes - in particular, whether the method does not actually provide tools for quanitfying posterior uncertainty over contexts or downstream predictions.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposed a framework for learning contrastive representation of stochastic processes (CRESP) and designed two variants of models to tackle targeted and untargeted downstream tasks (e.g. label prediction) in a semi-supervised setting.  The models were evaluated on downstream predictive tasks of three stochastic processes and outperform previous contrastive learning methods.
main_review: The structure of this paper is clear and easy to read. The research in this paper is very important for the comparative learning of stochastic processes. Overall, it is an ok paper but not enough for acceptance.

This core idea of models proposed in this paper is very similar as the idea in paper  SimCLR (Chen et al. 2020). The novelty of this paper is low.

These figures lack some necessary variables. For example Figure 1 should also show \hat{z}, z*, and z’.
limitations_and_societal_impact: A limitation has already been mentioned in this paper, the models proposed in this paper aim to learn the representations from stochastic processes and further be used in the downstream tasks. It cannot predict observation from stochastic processes. A net for reconstruction from c to x may be helpful to improve the learned representation.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a meta-learning model that represents stochastic processes through a contrastive representation. The authors mainly addressed the unified framework which could handle the two types of tasks; 1) given a set of data and target covariate (they called it the targeted task) and 2) estimating the label for the set of data (called untargeted task). To do that, they designed the specific encoder for the targeted task and by applying a self-attention mechanism on their encoder, they improved the performance for the tasks. They validated their model on interesting tasks like bi-modal sinusoidal regression and semi-supervised learning. I think that the architectural novelty of their model is the unified design for targeted/untargeted tasks and self-attention layers. However, I don't think their contribution is not limited to them, they validated/analyzed their model and other meta-learning models trained through the contrastive losses (e.g., FCLR and MetaCDE) for a variety of interesting tasks, which is one of the strong points on this paper I thought.
main_review: When I first review this paper, I felt it is quite similar to FCLR. I focused to find the difference between FCLR and CReSP, and as I understand, the main differences are 1) targeted encoder for the targeted task and 2) self-attention on the encoder. The authors explained it as CReSP is the unified framework for meta-learning with contrastive loss. It could be seen as incremental work, but they validated their model and previous works on several interesting tasks including targeted and untargeted tasks. They pretty well analyzed the results (e.g., for the ShapeNet task, why the performances of CReSP and FCLR decrease when the number of given views increases), and their model outperforms previous models, so I think that we cannot say easily it is just incremental work. Overall writing is good to read but the notation for covariate $\xi$, observation $x$ and label $y$ caused some confusion for me, because usually, NPs papers use $x$ as the covariate and $y$ for observation. If one of your contributions is the unified framework for meta-learning model with contrastive loss, then using the below storyline as main could make this paper more understandable. 

Using contrastive loss for meta-learning is very beneficial -> FCLR was proposed ->  but it cannot cover targeted task well, so we proposed unified framework including the FCLR

Questions / comments
- Line 103-4, you mentioned NP model is the model with additional latent variable on top of CNP, but it is not correct. CNP samples the functional latent when decoding with each target query, which causes inconsistant uncertainty, so the authors updated their CNP to sample the functional latent before decoding to resolve the inconsistant uncertainty problem. GQN that is origin of CNP (CNP is generalized version of GQN) is also inconsistant for uncertain knowledge, so Consistant GQN (Ananya et al., 2019) was proposed.
- Line 203-7, what is the meaning of equation in line 204? self-attention? From my perspective, the equation causes not necessary confusion. Why did you use Image Transformer, not general Transformer? The performance when using Image Transformer was better?
- For Sinusoids task, how did you use ACNP for untargeted task? Did you summarize the representations for the context set?
- In Fig. 3, second image in second row is not distorted.

limitations_and_societal_impact: Their model has been validated for quite simple labels like parameters of sinusoids or one of 13 class labels. However, those models with contrastive loss are more powerful for visually complicated tasks like 3D map world model for agent. For those tasks, the representations $c$ are required to contain enough knowledge for environment. I am not sure that CReSP can cover those complex tasks also.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: My understanding of the paper is that authors propose to introduce contrastive methods (Van den Oord 2019, Chen 2020) to improve the learning of stochastic processes with NN architectures. Particularly, they identify the main points where the Neural Process (NP+CNP) family of models fails, e.g. due to high-dimensionality or complex likelihood densities. The advantage of using these contrastive representations for stochastic processes is that it makes the task likelihood-free, performing better when the aforementioned problems appear.
main_review: **Originality and significance:** 

It can be clearly seen that the main 3 references for this work are:

- Garnelo (2018) / conditional neural processes
- Van den Oord (2019) / contrastive representations
- Chen (2020) / contrastive learning

The contribution is also clear to me: to introduce the concept of contrastive learning to improve the learning mechanism used in NP+CNPs. Despite the work is a mix of the previous references, I see that it is definitely interesting and it might have a future impact. However, I think that the ideas are not that well developed through the paper (e.g. the likelihood-free inference and its importance for the CNP models)… particularly due to a lack of clarity. That is the main flaw, in my opinion.

- The training method in Eq. (6) is super similar to Chen (2020) as authors recognise. What does attention/aggregation steps imply in this particular scenario? 

**Quality and clarity:** 

- I find the choice of notation a bit counterintuitive, mainly for those familiar with stochastic processes or just the functional expression y=f(x). For instance, the use \xi, x for the context/dataset, where \xi are the covariates and x the output of the function is difficult to follow, at least for me. In the original NP and CNP papers, the usual notation {x,y} were used. I do not see the reason why this is not preserved. (Sometimes the notation is changed due to other variables are preferred to have other names).

- The paper is well motivated on stochastic processes, that’s great, and it is also well written until L110 approx. Later things are more difficult to follow. In particular I see that the loss of contrastive learning, in Eq. (3) and so on and the use of the likelihood ratio is not clearly explained. (See Section 2.1 of Chen (2020) as an example.) This is important, since contrastive learning is the key contribution, and the reader should understand very well what authors are introducing in the neural/stochastic process mechanism.

- Targeted tasks/untargeted tasks are not motivated, so it is difficult to figure out an example. Until L137 it is +/- clear what the elements of the model are.. context data: {\xi, x}, representation c, code z… but where does y come from?

- From a technical point of view, the paper is relevant. However, I am not fully convinced by the experiments. For example, the section 5.1. with the sinusoids and the multimodality problem: using such a large architecture for learning the parameters {\alpha, \phi} seems a bit problematic. Could be this problem easily solved by a simpler stochastic process? i.e. GP or similar? To me, is a bit weird to see that the CNP model fails here, particularly, if one looks to the original paper. This is because of the multivariate and multimodal likelihood problem, I guess.

**Questions and other comments:**

- What does a “negative sample” z_1, z_2, … (L175) mean?
- L185: more more
- L174: how is this projection done?

**References:**

Van der Oord et al. 2019, “Representation Learning with Contrastive Predictive Coding”. ArXiv
T. Chen et al. 2020, "A Simple Framework for Contrastive Learning of Visual Representations" ICML 2020
M. Garnelo et al. 2018, “Conditional Neural Processes”, ICML 2018

limitations_and_societal_impact: Limitations and societal impact are correctly addressed. Authors indicate that the method is not able to directly predict on the observation space as one could desire. Moreover, they indicate the advantages of using the model for better understanding of 3D scenes, what might imply some risks as well.
review_title: null

Meta-review: One of the reviews lacks thoroughness - I'm happy to disregard this. 

The work proposes meta learning through neural processes and contrastive learning. There is some disagreement about the level of contribution here, with several reviewers pointing out that the FCLR method also contributes contrastive learning, and another reviewer pointing out that this is not so incremental because of extensions in several directions. 

One reviewer raises a strong technical objection, that the "stochastic process perspective taken in the paper seems only tenuously related to the actual training and prediction schemes". The authors refute - "using a deterministic embedding of context data that is sampled from a stochastic process is (not) invalid". the authors agree to amend the work to improve clarity, and the reviewer conceded that clarifying this point in the manuscript would make for a strong contribution. 

One reviewer raised issues with the experiments - that they do illuminate the method or support the proposed ideas. Yet I'm inclined to agree with another reviewer who found the detailed analysis, descriptions, and hyperparameters in the supplementary material. I also approve of the experiment to show limitations of the method in learning the strongly multimodal case. 

Overall, I think the authors have enough feedback from the reviewers to make some tweaks to some of the explanations in the paper, making this a contribution to the NeurIPS community. 
Decision: Accept (Poster)