On Contrastive Representations of Stochastic Processes
Authors: Emile Mathieu
Adam Foster
Yee Whye Teh
Venue: NeurIPS 2021 Poster (2021)
Keywords: contrastive learning
neural processes
representation learning

Abstract: Learning representations of stochastic processes is an emerging problem in machine learning with applications from meta-learning to physical object models to time series. Typical methods rely on exact reconstruction of observations, but this approach breaks down as observations become high-dimensional or noise distributions become complex. To address this, we propose a unifying framework for learning contrastive representations of stochastic processes (CReSP) that does away with exact reconstruction. We dissect potential use cases for stochastic process representations, and propose methods that accommodate each. Empirically, we show that our methods are effective for learning representations of periodic functions, 3D objects and dynamical processes. Our methods tolerate noisy high-dimensional observations better than traditional approaches, and the learned representations transfer to a range of downstream tasks.

On Contrastive Representations of Stochastic Processes
Emile Mathieu Adam Foster Yee Whye Tehi-l
{emile mathieu, adam. foster, y t.w.tetssstats. ox. ac. uk, Department of Statistics, University of Oxford, United Kingdom DeepMind, United Kingdom
Abstract
Learning representations of stochastic processes is an emerging problem in ma- chine learning with applications from meta-learning to physical object models time series. Typical methods rely exact reconstruction of observations, but this approach breaks down as observations become high-dimensional or noise distributions become complex. To address this, propose unifying framework for learning contrastive representations of stochastic processes (CRESP) that does away with exact reconstruction. We dissect potential cases for stochastic process representations, and propose methods that accommodate each. Empir- ically, show that methods effective for learning representations of periodic functions, 3D objects and dynamical processes. Our methods tolerate noisy high-dimensional observations better than traditional approaches, and the learned representations transfer to a range of downstream tasks.
Introduction
The stochastic process (Doob, 1953; Parzen, 1999)
Table I: Example stochastic processes with covariate space X and observation space
powerful mathematical abstraction used in biology (Bressloff, 2014), chemistry (van Kampen, 1992), physics (Jacobs, 2010), finance (Steele, 2012) and other fields, The simplest incarnation of stochas- tie process random function R R, such Gaussian Process (MacKay, 2003), that can be used describe real-valued signal indexed by time or space. Extending to random functions from IR to another space, stochastic processes can model time- dependent phenomena like qucuing (Grimmett and Stirzaker, 2020) and diffusion (Itô et al., 2012). In
X
y
Illustration
fonetioon
R
in-li
R3
meta-learning, the stochastic process can be used to describe few-shot learning tasks mappings from im- ages to class labels (Vinyals et al., 201 image completion tasks mappings from pixel locations to RGB values (Garnelo et al., 201 8a). In computer vision, 2D views of 3D objects can be seen as obser- vations of stochastic process indexed by the space of possible viewpoints (Eslami et al., 2018; Milden- hall et al., 2020). Videos can be
TD dbject
SE(3)
Images
R
Images
samples from
a Lime-indexed stochastic process with 2D image observations (Zelnik-Manor and Irani, 2001).
Equua contribution. Author ordering determined by coin flip.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Machine learning algorithms that operate on data generated from stochastic processes are therefore in high demand. We assume that we have access to only a small set of covariate- obbervvttooononin pairs {1 9:) y+){=1) } from different realizations of the underlying stochastic process. This might correspond to few views of a 3D object, or a few snapshots of a dynamical system evolving in time. Whilst conventional deep learning thrives when there is large quantity of i.i.d. data available (Lake et al., 2017), allowing us to learn a fresh model for each realization of the stochastic process, when the context size is small it makes sense to use data from other realizations to build up prior knowledge about the domain which can aid learning on new realizations (Reed et al., 2018; Garnelo et al., 80).
Traditional methods for learning from stochastic processes, including the Gaussian Process family (MacKay, 2003; Rasmussen, 2003) and the Neural Process family (Garnelo al., 201 b; Eslami et al., 2018), learn to reconstruct realization of the process from given context. That is, given a context set ((r; yi ==1) these methods provide predictive distribution q (y" xx", (2;; y:) y:) for the observation that would be obtained from this realization of the process at any target covariate x² These methods use an explicit likelihood for q. typically a Gaussian distribution. Whilst this can work well when y² is low-dimenssonal and unimodal, it is a restrictive assumption. For example, when p |x" (a:;, y:) samples a high-dimensional image with colour distortion, traditional methods must learn to perform conditional image generation, notably challenging task (van den Oord et al., 2016; Chrysos and Panagakis, 2021).
In this paper, we do away with the explicit likelihood requirement for learning from stochastic processes. Our first insight is that, for a range of important downstream tasks, exact reconstruction is not necessary to obtain good performance. Indeed, whilst y may be high-dimensional, the downstream target label or feature may be simpler. We consider two distinct settings for & E L. The first is a downstream task that depends on the covariate x E X, formally a second process X' + L that covaries with the first. For example, {(x) could represent a class label or annotation for each video frame. The second is a downstream task that depends on the entire process realization, such single label for a 3D object. In both cases, we assume that we have limited labelled data, so we are in semi-supervised setting (Zhu, 2005).
To solve problems of this nature, we propose a general framework for Contrastive Representations of Stochastic Processes (CRESP). At its core. CRESP consists of flexible encoder network architecture for contexts (ax; y: yr) that unites transformer encoders of sets (Vaswani et al., 2017; Parmar et al., 2018) with convolutional encoders (LeCun et al., 1989) for observations that are images. To account for the two kinds of downstream task that may of interest, we propose a targeted variant of CRESP that learns representations depending the context and target covariate x", and an untargeted variant that learns one representation of the context. To train our encoder, we take our inspiration from recent advances in contrastive learning (Bachman et al., 2019; Chen et al., 2020) which have so far focused on representations of single observations, typically images. We define a variant of the InfoNCE objective (van den Oord et al., 2018) for contexts sampled from stochastic processes, allowing us to avoid training objectives that necessitate exact reconstruction. Rather than attempting pixel-perfect reconstruction, then, CRESP solves self-supervised task in representation space.
The CRESP framework unifies and extends recent work, building on function contrastive learning (FCLR) (Gondal et al., 2021) by considering targeted as well as untargeted representations and using self-attention in place of mean-pool aggregation. We develop on noise contrastive meta-learning (Ton et al., 2021) by focusing on downstream tasks rather than multi-modal reconstruction, replacing conditional mean embeddings with neural representations and using a simpler training objective.
We evaluate CRESP on sinusoidal functions, 3D objects, and dynamical processes with high- dimensional observations. We empirically show that our methods can handle high- nigh-dimensional observations with naturalistic distortion, unlike explicit likelihood methods, and our representations lead to improved data efficiency compared to supervised learning. CRESP performs well on a range of downstream tasks, both targeted and untargeted, outperforming existing methods across the board. Our code is publicly available at github. com/ae teee-fosterrrreeee
2 Background
Stochastic Processes
Stochastic Processes (SPs) are probabilistic objects defined as family of random variables indexed by a covariate space X. For each x E X, there a corresponding random variable y|æ e y living in the observation space. For example, x might represent a pose and y
2

---

photograph of an underlying object take from pose æ (see Tab. 1). We assume that there is realization F sampled from prior p(F). and that the random variable y/ 2: a sample from 2). Thus, for each æ E X, F defines conditional distribution p(y] p(y)F, æ). We assume that observations are independent conditional on the realization F. Hence, the joint distribution of multiple observations at locations 11.C from one realization of the stochastic process with prior p( F) is
p(yr.cla = f ((FF) II P(yi dF. -
Conversely, assuming exchangeability and consistency, the Kolmogorov Extension Theorem guaran- tees that the joint distribution takes the form (1) (Oksendal, 2003; Garnelo et al., 2018b).
Neural Processes The neural process (NP) and conditional neural process (CNP) are closely related models that learn representations of data generated by a stochastic process (SP) The training objective for the NP and CNP is inspired by the posterior predictive distribution for SPs: given context (1 y { the observation at the target covariate x' has the distribution
, (2), vi) = ff p(F((E), vi) (F,""FF,"" dF.
ply wx",
The CNP learns a neural approximation q (g"(z", (ee,)))))))) ply" x²)) to equation (2), where Ei (3;, yt) is a permutation-invariin
context representation and p(-lc, x)) is an explicit likelihood. Conventionally, is a Gaussian with mean and variance given by a neural network applied to c, x. The CNP model is then trained by maximum likelihood. In the NP model, an additional latent variable u is used to represent uncertainty in the process realization, more closely mimicking (2).
explicit likelihood. Indeed, requiring log q (y"|=", (zx, y0 to be large requires the model to successfully reconstruct y* based on the context, similarly to the reconstruction term in variational autoencoders (Kingma and Welling, 2014). Furthermore, the NP objective cannot be increased by extracting additional features from the context unless the predictive part of the model, the part mapping from (c, a²) to mean and variance, is powerful enough to use them.
A significant limitation, common to the NP family, is the reliance
Contrastive Learning and Likelihood-free Inference Contrastive learning has enjoyed recent success in learning representations of high-dimensional data (van den Oord et al., 2018; Bachman et al., 2019; He et al., 2020; Chen et al., 2020), and is deeply connected to likelihood-free inference (Gutmann and Hyvärinen, 2010; van den Oord et al., 2018; Durkan et al., 2020). In its simplest form, suppose we have distribution p(y. y²), for example y and y could be differently augmented versions of the same image. Rather than fitting a model to predict y given y. which would necessitate nigh-dimensional reconstruction, contrastive learning methods can be seen as learning the likelihood- ratio r (y) = p(y' lv)/p(y'). To achieve this, contrastive methods encode y: y to deterministic embeddings 2, z', and consider additional 'negative' samples 2/; .... x= which are the embeddings of other independent samples of p(yy) (for example, taken from the same training batch as y, y'). The InfoNCE training loss (van den Oord et al., 2018) is then given by
2(z,2')
CInfONCE -E log s(z, x²) + Σx 8(2, x44) C
log K.
for similarity score 8 > O. Informally, InfoNCE is minimized when 2 is more similar to the 'positive" sample- than it is to the negative samples z',-.., z'x-1 that are independent of z. Formally, Eq. (3) is the multi-class cross-entropy loss arising from classifying the positive sample correctly. can be shown that the optimal similarity score is proportional the true likelihood ratio (van den Oord et al., 2018; Durkan et al., 2020). A key feature of InfoNCE is that learns about the predictive density p(y' (y) indirectly, rather than by attempting direct reconstruction.
3 Method
Given data ((x,, ) sampled from a realization of a stochastic process, one potential task is to make predictions about how observations will look another z""this is the task that is solved
Note that neither the neural process (NP) nor the conditional neural process (CNP) is formally stochastic processs (SPs) they do not satisfy the consistency property.
3

---

by the NP family. However, in practice the inference that we want to make from the context data could be different. For instance, rather than predicting nigh-dimensional observation at a future time or another location, we could be interested in inferring some low-dimensional feature of that observation-
two objects have collided at that point in time, or if an object can be seen from given pose. Even more simply, we might be solely interested in classifying the context, deciding what object is being viewed, for example. Such downstream tasks provide a justification for learning representations of stochastic processes that are not designed to facilitate predictive reconstruction of the process at some x². We break downstream tasks for stochastic processes into two categories.
Targeted and untargeted tasks A targeted task is one in which the label &' depends on æ, as well as the underlying realization of the process F. This means that we augment the stochastic process of Sec. 2 by introducing a conditional distribution p(&)F, x). The goal is to infer the predictive density p (f*)a", (æ., 11). . An untargeted task associates one label y with the entire realization F via a conditional distribution F). The aim is to infer the conditional distribution p (4) y:),
Representation learning We assume a semi-supervised (Zhu, 2005) setting, with unlabelled con- texts for large number of realizations of the stochastic process, but few labelled realizations. To make best use of this unlabelled data, we learn representations of contexts, and then fit a downstream model on top of fixed representations. In the stochastic process context, we have the requirement for a representation learning approach that can transfer to both targeted and untargeted downstream tasks. We therefore propose general framework to learn contrastive representations of stochastic processes (CRESP). Our framework consists of a flexible encoder architecture that processes the context { y and x dependent head for targeted tasks. This means CRESP can encode data from stochastic processes in two ways: 1) targeted representation that depends on the context {(x, 3 ) and some target location x², being a predictive representation for the process at this covariate, suitable for targeted downstream tasks; or 2) single untargeted representation of the that summarizes the entire realization F, suitable for untargeted tasks.
context ((z,, yi )
3.1 Training
We have unlabelled data ((x;, )=)) that is generated from the stochastic process (1), but unlike the Neural Process family, we do not wish to place an explicit likelihood on the observation space V. Instead, we adopt a contrastive self-supervised learning approach (van den Oord et al., 2018; Bachman et al., 2019; Chen et al., 2020) to training. Whilst we adopt subtly different training schemes for the targeted and untargeted cases, the broad strokes are the same. Given a mini-batch of contexts samples from different realizations of the underlying stochastic process, create predictive and ground truth representations from each. We then use representations from other observations in the same mini-batch as negative samples in InfoNCE-style (van den Oord et al., 2018) training loss. This can be seen as learning an untormalized likelihood ratio. Taking gradients through this loss function allows us update CRESP network by gradient descent (Robbins and Monro, 1951). We now describe the key differences between the targeted and untargeted cases.
Targeted CReSP This setting is closer in spirit to the CNP. Rather than making a direct estimate of the posterior predictive p(y" [x=", (a.,, )1) for each value of æ", we instead attempt learn the following likelihood-ratio
ryy" (Z,, (e2,,)0))) (2)) p(y lx=" (æi, VI 21) p(y*)
(4)
where p(y*) is the marginal distribution of observations from different realizations of the process and different covariates. To estimate this ratio with contrastive learning, we first randomly separate the context {(,,, 3 into a training context {(0,, ( and target (æ* yy") We then process {(x,, 3 C-1 x²) and y" separately with an encoder network, yielding respectively a predictive representation è and a target representation c*. This encoder network is described in detail in the following Sec. 3.2. These representations are further projected into a low-dimensional space Z using a shallow MLP, referred as Projection head on Fig. 1, giving & and zt. We create negative samples z',,.., z'x 1 defined as samples coming from other realisations of the stochastic process, from representations obtained from the other observations of the batch. This means that we are

---

3/1
1/1
3/1
not
Oba,
Encoding
Enoockig
Encoding
Attention
Batch dim
Attortan
Atlention
Batchh dim
Agpeigasion
Target head
Appregataon
Asprogution
Representation
Representations
Projectio
Projection
Proyectoon
Projectio
Projection
Proection
Serilaity
Sicalinit
Figure 1: CRESP architecture with contrastive loss. [Left] Targeted, [Right] Untargeted.
drawing negative samples via the distribution p(y") as required for (4). We then form the contrastive loss
s(2', 2)
targetod = E
log
log K
(5)
s(2
Σx 5
with 2) exp (2 8/T||= |||
By minimizing this loss, ensure that the predicted representation is closer to the representation of the true outcome than representations other random outcomes. The optimal value of s(2", is proportional the likelihood ratio r(y" (a=", (æ;, y:))
For the untargeted version, simply require representation of each context; x" no longer plays a role. The key idea here is that, without estimating a likelihood ratio in y space, can contrastive methods encourage two representations formed from the same realization of the stochastic process to be more similar than representations formed from different realizations. To achieve this, we randomly split the whole context ((v,, yt) into two training contexts {<x,, y: ( } and {(x y0 J. with an equal split C1 C2 = C/2 being our standard approach. We encode both with an encoder network, giving two representations c', further projected into lower-dimensional representations z, z' as the targeted case. We also take samples 2i, z'K using other representations in the same training mini-batch.
Untargeted CReSP
negative
s(2, x²)
C
Cuntargeted
log s(2, x²)
log K.
(6)
Σx s(z, 2
This training method is closer in spirit SimCLR (Chen al., 2020), but here include attention and aggregation steps to combine the distinct elements of the context.
3.2 Representation
The core of our architecture is flexible encoder of a context {(x,, y:) as illustrated in Fig.
Covariate and observation preprocessing We begin by applying separate networks to the covari- ate (a)) and observation 9obs (y) of each pair (ær y) of the context. When observations are high-dimensional, such as images, this step is crucial because we can use existing well-developed vision architectures such CNNs (LeCun et al., 1989) and ResNets (He al., 2016) to extract image features. For covariates that are angles, we use Random Fourier Features (Rahimi and Recht, 2008).
Pair encoding We then combine separate encodings of æ. y into a single representation for the pair. We concatenate the individual representations and pass them through simple neural network, i.e. Gene (x, y) = Gene ([goov (x), Gobs (otss()))))))))) (y)]). In practice, we found that gated architecture works well.
5

---

Attention & Aggregation We apply self-attention (Vaswani et al., 2017) over the C' different encodings of the context {geme yi fi=1- We found transformer attention (Parmar et al., 2018) to perform best. We then pool the C'' reweighted representations to yield single representation = Ei Jene (2;; Yi). For targeted representations, we concatenate and æ", then pass them through a target head yielding é = h(x", c), the predictive representation at z².
3.3 Transfer to downstream tasks
We have outlined the unsupervised part of CRESP- way to learn representation of a context sampled from a stochastic process without explicit reconstruction. We now return to our core motivation for such representations, which is to use them solve downstream task, either targeted or untargeted. This will be particularly useful in a semi-supervised setting, in which labelled data for the downstream task is limited compared to the unlabelled data used for unsupervised training of the CRESP encoder. Our general approach to both targeted and untargeted downstream tasks is to fit linear models on the context representations of the labelled training set, and use these to predict labels on new, unseen realizations of the stochastic process, following the precedent in contrastive learning (Hjelm et al., 2019; Kolesnikov et al., 2019). We do not use fine-tuning.
For targeted tasks, we assume that we have labelled data from n realizations of the stochastic process that takes the form of unlabelled context (zzeg y:r) along with labelled pair (arj. f7) for each j = 1,...,,, n. Here, Ej is the label at location x= for realization j. To fit a downstream classifier using CRESP representations with this labelled dataset, we first process each (ætj) )o)) along with the covariate 23 through a targeted CRESP encoder to produce ēj. This allows us to form a training dataset (cj: J of representation, label pairs which we then use to train our downstream classifier. At test time, given test context (ar;: =() we can predict the unknown label at any x' by forming the corresponding targeted representation with the CRESP network, and then feeding this into the linear classifier. This is akin to zero-shot learning (Xian et al., 2018).
For untargeted tasks, the downstream model is simpler. Given labelled data consisting of contexts (xig yis);=1 with label fj for j I,..., nt we can use the untargeted CRESP encoder to produce a training dataset (cg, as before. Actually, targeled CRESP can also be used obtain untargeted representations Cj- without applying the target head. We then use this to train the linear classifier. At test time, we predict labels for contexts from new, unseen realizations of the stochastic process.
4 Related work
Neural process family Neural Processes (Gamelo et al., 2018b) and Conditional Neural Processes (Eslami et al., 2018; Garnelo et al., 2018a) are closely related methods that create a representation of an stochastic process realization by aggregating representations of a context. Unlike CRESP, NPs are generative models that uses an explicit likelihood, generally a fully factorized Gaussian, to estimate the posterior predictive distribution. Attentive (Conditional) Neural Processes (Kim et al., 2019, A(C)NP) introduced both self-attention and cross-attention into the NP family. The primary distinction between this family and CRESP is the explicit likelihood that is used for reconstruction. As the most comparable method to CRESP, we focus on the (A)CNP in the experiments.
SimCLR family Recent popular methods in contrastive learning (van den Oord et al., 2018; Bachman et al., 2019; Tian al., 2020; Chen et al., 2020) create neural representations of single objects, typically images, that are approximately invariant to a range of transformations such as random colour distortion. Like CRESP. many of these approaches use the InfoNCE objective to train encoders. What distinguishes CRESP from conventional contrastive learning methods is that it provides representations of realizations of stochastic processes, rather than of individual images. Thus, standard contrastive learning solves strictly less general problem than CRESP in which the covariate absent. Standard contrastive encoders do not aggregate multiple context, although simpler feature averaging (Foster et al., 2020) has been applied successfully.
pairs of
Function contrastive learning In their recent paper, Gondal et al. (2021) considered function contrastive learning (FCLR) which uses self-supervised objective to learn representations of functions. FCLR fits naturally into the CRESP framework as an untargeted approach that uses

---

mean mean-poollee poolinn in place of our attention aggregation. Conceptually, then, FCLR does not take account of targeted tasks, nor does propose method for targeted representation learning.
Noise contrastive meta-learning Ton et al. (2021) proposed an approach for conditional density estimation in meta-leamnn reta-learning, motivated by multi-modal reconstruction. Like targeted CRESP, their method targets the unnormalized likelihood ratio (4). They use a noise contrastive (Gutmann and Hyvärinen, 2010) training objective with an explicitly defined 'fake' distribution that is different from the CRESP training objective. Their primary method, MetaCDE, uses conditional mean embeddings to aggregate representations, unlike our attentive aggregation. This means that, when using as baseline within our framework, MetaCDE does not form fixed-dimensional representation of contexts, and so cannot applied to untargeted tasks. They also proposed MetaNN, a purely neural version of their main approach.
5
Experiments
We consider three different stochastic processes and downstream tasks which possess high- dimensional observations complex noise distributions: 1) inferring parameters of periodic functions, 2) classifying 3D objects and 3) predicting collisions in dynamical process. We compare several models summarized in Tab. 2 learn representations of these stochastic processes. All models share the same core encoder architecture. Please refer to Appendix D for full experimental details.
Table 2: Comparison of models used in least experiment in Sec. 5.
Criteria
CNP
ACNP
FCLR
MetaCDE
Targeted CRESP
Untargeted CRESP
Targeted
No
No
No
Yes
Yes
No
Reconstruction
Yes
Yes
No
No
No
No
Attention
No
Yes
No
No
Yes
Yes
5.1
Sinusoids
We first aim demonstrate that reconstruction-bee based methods like CNPs cannot cope well with bi-modal noise process since their Gaussian likelihood assumption renders them misspecified. We focus on synthetic dataset of sinusoidal functions with both the observations and the covariates living in IR, i.e. X' IR and y = RR. We sample one dimensional functions F p(F) such that F(x) a sin (2 T x + pp) with random amplitude a ([0.5, 2.0)), phase şp 2((0, r)) and period T 8. We break the uni-modality by assuming bi-modal likelihood: F()) 0.5 Fx(z) (y) + 0.5 +o( (see Fig. 2a). Context points x E are uniformly sampled in [-5,5], We consider the untargeted downstream task of recovering the functions parameters (a, and consequently put to the test our untargeted CRESP model along with FCLR and ACNP. We train all models for 200 epochs, varying the distance between modes and the number of training context points. We observe from Fig. 2b that for high intermodal distance, the ACNP is unable to accurately
Downstream MSE
Downstream MSE
Samples
ACNP
10-2
Density
10
CReSP
FCLR
10
CR&SP
10
0
10-3
10-1
-5
5
10
2
5
10
20
50
X
Distance between modes
Training context points
(a) Stochastic process sample
(b) CRESP vs ACNP
(c) Effect of self-attention
Figure 2: We use CRESP along with ACNP and FCLR recover sinusoid parameters with bi-modal likelihood. In each setting, we used 20 test views to form representations of the entire training set and fitted a linear classifier to predict the function parameters. Encoders and decoder are MLPs. (a) Visualization of conditional likelihood F F, æ). (b)(c) Shaded areas represent 95% confidence interval calculated using separately trained networks. We use the shorthand untargeted. In (b) we used 10 training views and in (c) the distance between the modes is set to 2.
7

---

Test accuracy (%))
86
86
80
accuracy
accuracy
70
84
84
V CReSP
CNP
U CReSP
FCLR
60
ACNP
82
FCLR
82
CReSP
T CReSP
distortion on 0.5
1.0 Colour distortion strength
No distortion 0.5
80
10
1.5
Colour distortion strength
Number training views
(a) CRESP reconstructive
(b) Contrastive methods
(c) CRESP vs FCLR
Figure 4: We compare CRESP with various baseline methods. In each case, we use 10 test views to form representations of the entire training set and fitted linear classifier to predict ShapeNet object labels. Encoder networks were lightweight CNNs. In (a)(b) we used 3 training views, (c) we used distortion strength 1. We present the test accuracy =1 s.e. and we use the shorthand U = untargeted, T= targeted in figure legends.
recover the true parameters as opposed to CRESP, which is more robust to this bi-modal noise even for distant modes. Additionally, we see in Fig. 2c that self-attention is crucial to accurately recover the sinusoids parameters, as the MSE several order of magnitude lower for CR&SP than for FCLR. We also see that CRESP is able to utilize a larger context better than ACNP.
5.2 ShapeNet
We apply CRESP to ShapeNet (Chang et al., 2015), standard dataset in the field of 3D object represen- tations. Each 3D object can be seen as a realization of stochastic process with covariates æ represent- ing viewpoints. We sample random viewpoints in- volving both orientation and proximity the object, with observations y being 64 x 64 images taken from that point. We also apply randomized colour distortion as a noise process on the 2D images (see Fig. 3). As the likelihood of this noise process not known in closed from, this should present partic- ular challenge to explicit likelihood driven models. The downstream task for ShapeNet is 13-way ob- ject classification which associates a single label with each realization- an untargeted task.
Figure The ShapeNet dataset can be seen as a stochastic process: the covariate æ is the viewpoint and the observation y is an image of the object from that viewpoint. [Topp] We il- lustrate an object viewed from random view- points. [Bottom] We show varying strengths of colour distortion applied to the same obser- vation, the lefthand column is no distortion.
CRESP outperforms reconstructive models Since the CNP learns by exact reconstruction of ob- servations, we would expect to struggle with high-dimensional image observations, and particularly suffer as we introduce colour distortion, which is a highly non- non-Gause Gaussian noise process. To verify this, we trained CNP and ACNP models, along with an attentive untargeted CRESP model which we would expect to perform well on this task. We used the same CNN observation processing network for each method, and additional CNN decoder for the CNP and ACNP. Fig. 4a shows that CRESP significantly outperforms both the CNP and ACNP, with reconstructive methods faring worse as the level of colour distortion is increased; CRESP actually benefits from mild distortion.
CRESP outperforms previous contrastive methods We next compare different contrastive ap- proachs along two axes: targeted vs untargeted, and attentive vs pool aggregation. This allows a comparison with FCLR (Gondal et al., 2021), which an untargeted pool-based method. Fig. 4b shows that no contrastive approach performs as badly as the reconstructive methods. Untargeted CRESP performs best, while the targeted method does less well on this untargeted downstream task. With our CNN encoders and matched architecture for fair comparison, FCLR does about as well as attentive targeted CRESP and worse than the untargeted counterpart. To further examine the benefits of the attention mechanism used in CRESP, we vary the number of views used during training, focusing on untargeted methods. Fig. 4c shows that as we increase the number of training views, the attentive method outperforms the non-attentive FCLR by an increasing margin. This indicates that careful aggregation and weighting of different views of each object essential for learning the best representations. The degradation in the performance of FCLR as more training
8

---

views are used is likely due to weaker training signal for the encoder as the self-supervised task becomes easier, this phenomenon also explains why CRESP slightly decreases in performance from 6 to 12 training views.
CRESP benefits from improved label efficiency We compare CRESP with semi-supervised learning that does not use any pre-training, but instead trains the entire architecture on the labelled dataset. Fig. 5a we see that pre-training with CRESP can outperform supervised learning on the same fixed dataset at every label fraction including 100%. Another axis of variation in the stochastic process setting is the number of views aggregated at test time. In Fig. 5b, we see that performance increases across the board as make more views available to form test representations, but that CRESP performs best in all cases.
Test accuracy (86
80
Test accuracy
60
Supervised
Supervised
40
CResP
U CReSP
10
10-1
10
10
10
Fraction labels available
Number of test views
(a) Semi-supervised evaluation
(b) Test views
Figure 5: CRESP for semi-supervised learning. We re-trained the final linear classifiers with different quantities of labelled data and number of test views, supervised learning trained the entire encoder architecture on the same labelled datasets. (a) We used 10 test views, (b) We used 100% of labels. Other settings were as in Fig. 4.
5.3 Snooker dynamical process over images
We now focus on the setting where downstream tasks depend on the covariate x=, i.c. targeted downstream tasks, In particular, we consider a dynamical system that renders 2D images of two objects with constant velocities and evolving through time as illustrated in Fig. 6a. The objects are constrained in 1 box and collisions are assumed to result in perfect reflection. The observation space y is consequently the space 28 28 RGB images, whilst the covariate space is IR representing time. We consider the downstream task of predicting whether the two objects are overlapping at given time x² = L or not. This experiment aims to reproduce, in a stripped-down manner, the real world problem of collision detection. Even though the object's position can be expressed in closed-form, it is non trivial predict the 2D image collection of snapshots. We expect targeted CRESP to be particularly well-suited for such task since the model is learning to form and match a targeted representation to the representation of the ground truth observation thorough the unsupervised task.
(a) 2D images associated with target times.
0.0
0.2
0.4
0.6
0.8
1.0
1.0
Probability
0.5
Ground
Prediction
0.0
(b) Probability of overlap.
assess the capacity of targeted CRESP to smoothly predict whether the objects are overlapping at given time xx given context set of size [Top] 2D images associated to t E j0, 1]. [Bottom] Confidence interval is computed over 50 random contexts and trained models.
Figure
specific time given
CRESP outperforms reconstructive and previous contrastive methods Alongside targeted CRESP, we consider the CNP, FCLR and MetaCDE models. They are trained for 200 epochs, with contexts of 5 randomly sampled pairs {y, = F(x;). x; ((0. 1])} The encoder is ResNet18 (He et al., 2016). We found that self-attention did not seem to help any method for this task, so we report un-attentive models. Both CNP and FCLR learn untargeted representations during the unsupervised task. We thus feed the downstream linear classifier with the concatenation (c,x*), Conversely, targeted CRESP and MetaCDE directly produce a targeted representation li(m"" (see See. 3.1). The downstream classifier simply rely on predict the overlap label L* We

---

consequently expect such a targeted representation è to be better correlated with the downstream label than untargeted representations c.
We observe from Tab. that targeted CRESP significantly outperforms both likelihood-hasee and previous contrastive methods, though MetaCDE outperforms both untargeted methods (CNP and FCLR). This highlights the need for the targeted contrastive loss from Eq. (5) along with a flexible target head learn targeted representations. Additionally, we observe that in the absence of noise process, CNP performs as well as FCLR. We further investigate the quality of the learned targeted representations. To do so, given fixed context we make an overlap prediction at different points in time as shown in Fig. 6b. We observe that targeted CRESP has successfully learned to smoothly predict the overlap label, but also be uncertain when the overlap ambiguous. Thus CRESP can successfully interpolate and extrapolate the semantic feature of interest (overlap) without reconstruction.
Table 3:
We examine how well learned representations can predict whether the two snooker balls overlap at randomly sampled test times. 95% confidence intervals were computed over runs.
CNP
FCLR
Targeted CRESP
MetaCDE
Accuracy (%)
85.3
85.640.3
96.8+0.1
87.7
6 Discussion
Limitations
Our method directly learns representations from stochastic processes, without per- forming reconstruction on the observations, thus if one requires prediction in the observation space y then our method cannot be directly applied. Whilst our method is tailor made for a setting of limited labelled data, we require access to a large quantity of unlabelled data to train our encoder network.
In this work, we do place uncertainty over context representations. Learning stochastic embed- dings would have the primary benefit of producing correlated predictions at two or more covariates, similarly NPs. As there is no trivial nor unique way extend the InfoNCE loss to deal with distributions (e.g. Wu and Goodman, 2020), we leave such an extension of our method to future work.
Future applications One potential use of CRESP is to generate representations that can be used for reinforcement learning, following the approach of Eslami et al. (2018). One of the key differences between real environments and toy environments is the presence of high-dimensional observations with naturalistic noise. This is case where the contrastive approach can bring edge because naturalistic noise significantly damages explicit likelihood methods, but CRESP continues to perform well with more distortion.
Conclusion In this work, we introduced framework for learning contrastive representation of stochastic processes (CRESP). We proposed two variants of method specifically designed to effectively tackle targeted and untargeted downstream tasks. By doing away with exact reconstruction, CRESP directly works in the representation space, bypassing any challenge due high dimensional and multimodal data reconstruction. We empirically demonstrated that our methods are effective for dealing with multi-modal and naturalistic noise processes, and outperform previous contrastive methods for this domain on range of downstream tasks.
Acknowledgments
We would like to thank Yann Dubois and Jef Ton for valuable discussions. We also thank Hyunjik Kim, Neil Band and Lewis Smith for providing feedback on earlier versions of the paper. EMM research leading to these results received funding from the European Research Council under the European Union's s Seventh Framework Programme (FP7/2007- 2013) ERC grant agreement no. 617071 1 and he acknowledges Microsoft Research and EPSRC for funding EM's s studentship. AF gratefully acknowledges funding from EPSRC grant no. EP/N50971//L.
10

---

References
Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. (rXivv1607006450
arXiv preprint
Bachman, P., Hjelm, R. D., and Buchwalter, W. (2019). Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15535-15545.
Bressloff, P. C. (2014). Stochastic processes in cell biology, volume 41. Springer.
Chang, A. X., Funkhouser, T., Guibas, L.,, Hanrahan, P., Huang, Q.. Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L.,, and Yu, F. (2015). ShapeNet: Information-Rich 3D Model [cs].
Repository. arXiv:1512.0300
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). simple framework for contrastive learning of visual representations. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Cho, K., Merrienboer, B., Gülgehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Moschitti, A., Pang, B., and Daelemans, W., editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL., pages 1724-1734 ACL.
Choy, C. B., Xu, D., Gwak, J., Chen, K., and Savarese, S. (2016). 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV).
Chrysos, G. and Panagakis. Y. (2021). Cope: Conditional image generation using polynomial expansions. arXiv preprint arXiv:2104.06 05.0777..
Doob, J. L. (1953). Stochastic processes, volume 10. John Wiley & Sons, New York. MR 15,445b. Zbl 0053.26802.
Durkan, C., Murray, I., and Papamakarios, G. (2020). On contrastive learning for likelihood- inference. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 88 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 2771 1781-.781. PMLR.
Eslami, S. M. A., Jimenez Rezende, D., Besse, F,, Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, Gregor, K., Reichert, D. P., Buesing, L.., Weber, T., Vinyals, O., Rosenbaum, D., Rabinowitz, N., King, H., Hillier, C., Botvinick, M., Wierstra, D., Kavukcuoglu, K., and Hassabis, D. (2018). Neural scene representation and rendering. Science, 360(6394):1204- 1210.
Foster, A., Pukdee, R., and Rainforth, T. (2020). Improving transformation invariance in contrastive representation learning. arXiv preprint arXiv:2010.09575.
Garnel,, M.. Rosenbaum, D.. Maddison, C., Ramalho, T., Saxton, D.. Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. M. A. (2018a). Conditional neural processes. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1703- PMLR.
Garnelo, M., Schwarz, J.. Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. (2018b), Neural processes. arXiv preprint arXiv: 1807.01672
Gondal, M. W., Joshi, S., Rahaman, N., Bauer, S., Wuthrich, M., and Schölkopf, (2021). Function contrastive learning of transferable meta-representattios. In Meila, M. and Zhang, T., editors, Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pages 3755-3765. PMLR.

---

Grimmett, G. and Stirzaker, D. (2020). Probability and random processes. Oxford university press.
Gutmann, M. and Hyvärinen, A. (2010). Noiseaconerastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778.
Hjelm, R. D., Fedorov, A.,
S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. (2019). Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations.
Hochreiter, and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 988) 1780.
Ito, K., Henry Jr, P., al. (2012). Diffiusion processes and their sample paths. Springer Science Business Media.
Jacobs, K. (2010). Stochastic processes for physicists: understanding noisy systems. Cambridge University Press.
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W. (2019). Attentive neural processes. In International Conference on Learning Representations.
Kingma, D. P. and Welling, M. (2014). Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Kolesnikov, A., Zhai, X., and Beyer, L. (2019). Revisiting self-supervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1920-1929.
Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. (2019). Quantifying the carbon emissions of machine learning. arxiv preprint arXiv: 191.0:991000070000
Lake, B. M.. Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. (2017). Building machines that learn and think like people. Behavioral and brain sciences, 40.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551.
Liu, D. C. and Nocedal, J. (1989). On the limited memory bfgs method for large scale optimization. Math. Program., 45(1-3):503-528
MacKay, J. (2003), Information theory, inference and learning algorithms. Cambridge university press.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, T., Ramamoorthi, R., and Ng, R. (2020).
Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405 421. Springer.
Øksendal, B. (2003). Stochastic differential equations. In Stochastic differential equations, pages 65-84. Springer.
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. (2018). Image transformer. In International Conference on Machine Learning, pages 4055-4064, PMLR.
Parzen, E. (1999). Stochastic processes. SIAM.
12

---

Paszke, A., Gross, S., Chintala, S.. Chanan, G.. Yang, E.. DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017). Automatic differentiation in PyTorch. In NIPS-W.
Radford, A., Metz, L., and Chintala, S. (2016). Unsupervised representation learning with deep convolutional generative adversarial networks. In Bengio, Y. and LeCun, Y., editors, 4th Inter- national Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Rahimi, A. and Recht, B. (2008). Random features for large-scale kernel machines. In Platt, J., Koller, D., Singer, Y., and Roweis, S., editors, Advances in Neural Information Processing Systems, volume 20, Curran Associates, Inc.
Rasmussen, C. E. (2003). Gaussian processes in machine learning. In Summer school on machine learning, pages 63-71. Springer.
Reed, S., Chen, Y., Paine, T., van den Oord, A., Eslami, S. M. A., Rezende, D., Vinyals, O., and de Freitas, N. (2018). Few-shot autoregressive density estimation: Towards learning learn distributions. In International Conference on Learning Representations.
Robbins, H. and Monro. S. (1951). A stochastic approximation method. The annals of mathematical statistics, pages 400-407.
Steele, J. M. (2012). Stochastic calculus and financial applications, volume 45. Springer Science & Business Media.
Tian, Y., Krishnan, D., and Isola, P. (2020). Contrastive Multiview Coding. 1rXiv:190..0558 [cs].
Ton, J.-F., CHAN, L.., Whye Teh, Y., and Sejdinovic, D. (2021). Noise contrastive meta-learning for conditional density estimation using kernel mean embeddings. In Banerjee, A. and Fukumizu, K., editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1099-1107, PMLR.
van den Oord, A., Kalchbrenner, N., Espeholt, L., kavukcuoglu, k., Vinyals, O., and Graves, A. (2016). Conditional image generation with PixeCCNN decoders. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint weXivv1807.03744
van Kampen, N. G. (1992). Stochastic processes in physics and chemistry, volume 1, Elsevier.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Vinyals, O., Blundell, C., Lillicrap, T., kavukcuoglu, k., and Wierstra, D. (2016). Matching networks for one shot learning. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc.
Wu, M. and Goodman, N. (2020). simple framework for uncertainty in contrastive learning. arXiv preprint arXiv:2010.02038.
Xian, Y., Lampert, C. H., Schiele, B., and Akata, Z. (2018). Zero-shot learning- comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9))2251-2265
Zeinik-Manor, L. and Irani, M. (2001). Event-based analysis of video. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2, pages II-II. IEEE.
Zhu, X. (2005). Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madiion.
13


Reviews:
[{"review_id": "arQ6DU3tvWh", "reviewer": "Reviewer_jJb1", "summary": "The authors propose a method for representation of an observational context by optimization of a contrastive learning objective. In contrast to (conditional) neural process modeling, this avoids specification of a likelihood, which may be difficult and ultimately unnecessary in tasks that only require prediction of a downstream label. Experimental results illustrate the advantages of this method over related neural process models on such tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper suggests a contrastive learning alternative to likelihood-based methods for training neural processes, which are themselves a recent attempt to overcome the computational issues known to hinder inference in stochastic (Gaussian) process models. In tasks that depend on some downstream property of the prediction, rather than the prediction itself, reconstruction is unnecessary and likelihood-free methods may be used for learning the context representation that features in the neural approximation to the posterior. Here, the authors propose a contrastive learning objective based on InfoNCE.\n\nThe paper is clear and detailed, including careful documentation of experimental data, preprocessing, and hyperparameters in the Appendix. The synthetic experiment offers a clear illustration of the relative advantage over (conditional) neural processes when the likelihood is misspecified. Experiments on image and dynamical process data show that the framework can be tailored to address a broader range of tasks (targeted and untargeted, i.e. conditional or not on the prediction at a given index) than FCLR and MetaCDE, two recent proposals for contrastive learning.\n\nMy main concern is that the stochastic process perspective taken in the paper seems only tenuously related to the actual training and prediction schemes detailed in Sec 3.1-3.3. In practice, the method learns an embedding for observational contexts via minimization of a contrastive loss, and this embedding is subsequently applied deterministically at test time to generate a fixed-dimensional feature representation for a linear model of the downstream task. The only notion of a predictive distribution that remains is from this linear modeling step (Appendix D, p. 2); there is no posterior uncertainty over contexts. These issues undermine the framing of this method and its comparison to (conditional) neural processes, which intentionally retain an emphasis on quantifying uncertainty. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors acknowledge some limitations in terms of their model's inability to reconstruct data in the observation space, in contrast to other neural process methods. Per my concern above, I would like to understand whether the limitations are in fact more broad from the standpoint of modeling with stochastic processes - in particular, whether the method does not actually provide tools for quanitfying posterior uncertainty over contexts or downstream predictions.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "WHOtRtmtOed", "reviewer": "Reviewer_S13o", "summary": "This paper proposed a framework for learning contrastive representation of stochastic processes (CRESP) and designed two variants of models to tackle targeted and untargeted downstream tasks (e.g. label prediction) in a semi-supervised setting.  The models were evaluated on downstream predictive tasks of three stochastic processes and outperform previous contrastive learning methods.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The structure of this paper is clear and easy to read. The research in this paper is very important for the comparative learning of stochastic processes. Overall, it is an ok paper but not enough for acceptance.\n\nThis core idea of models proposed in this paper is very similar as the idea in paper  SimCLR (Chen et al. 2020). The novelty of this paper is low.\n\nThese figures lack some necessary variables. For example Figure 1 should also show \\hat{z}, z*, and z’.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A limitation has already been mentioned in this paper, the models proposed in this paper aim to learn the representations from stochastic processes and further be used in the downstream tasks. It cannot predict observation from stochastic processes. A net for reconstruction from c to x may be helpful to improve the learned representation.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "MjM2SqIcv3-", "reviewer": "Reviewer_z4gr", "summary": "My understanding of the paper is that authors propose to introduce contrastive methods (Van den Oord 2019, Chen 2020) to improve the learning of stochastic processes with NN architectures. Particularly, they identify the main points where the Neural Process (NP+CNP) family of models fails, e.g. due to high-dimensionality or complex likelihood densities. The advantage of using these contrastive representations for stochastic processes is that it makes the task likelihood-free, performing better when the aforementioned problems appear.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality and significance:** \n\nIt can be clearly seen that the main 3 references for this work are:\n\n- Garnelo (2018) / conditional neural processes\n- Van den Oord (2019) / contrastive representations\n- Chen (2020) / contrastive learning\n\nThe contribution is also clear to me: to introduce the concept of contrastive learning to improve the learning mechanism used in NP+CNPs. Despite the work is a mix of the previous references, I see that it is definitely interesting and it might have a future impact. However, I think that the ideas are not that well developed through the paper (e.g. the likelihood-free inference and its importance for the CNP models)… particularly due to a lack of clarity. That is the main flaw, in my opinion.\n\n- The training method in Eq. (6) is super similar to Chen (2020) as authors recognise. What does attention/aggregation steps imply in this particular scenario? \n\n**Quality and clarity:** \n\n- I find the choice of notation a bit counterintuitive, mainly for those familiar with stochastic processes or just the functional expression y=f(x). For instance, the use \\xi, x for the context/dataset, where \\xi are the covariates and x the output of the function is difficult to follow, at least for me. In the original NP and CNP papers, the usual notation {x,y} were used. I do not see the reason why this is not preserved. (Sometimes the notation is changed due to other variables are preferred to have other names).\n\n- The paper is well motivated on stochastic processes, that’s great, and it is also well written until L110 approx. Later things are more difficult to follow. In particular I see that the loss of contrastive learning, in Eq. (3) and so on and the use of the likelihood ratio is not clearly explained. (See Section 2.1 of Chen (2020) as an example.) This is important, since contrastive learning is the key contribution, and the reader should understand very well what authors are introducing in the neural/stochastic process mechanism.\n\n- Targeted tasks/untargeted tasks are not motivated, so it is difficult to figure out an example. Until L137 it is +/- clear what the elements of the model are.. context data: {\\xi, x}, representation c, code z… but where does y come from?\n\n- From a technical point of view, the paper is relevant. However, I am not fully convinced by the experiments. For example, the section 5.1. with the sinusoids and the multimodality problem: using such a large architecture for learning the parameters {\\alpha, \\phi} seems a bit problematic. Could be this problem easily solved by a simpler stochastic process? i.e. GP or similar? To me, is a bit weird to see that the CNP model fails here, particularly, if one looks to the original paper. This is because of the multivariate and multimodal likelihood problem, I guess.\n\n**Questions and other comments:**\n\n- What does a “negative sample” z_1, z_2, … (L175) mean?\n- L185: more more\n- L174: how is this projection done?\n\n**References:**\n\nVan der Oord et al. 2019, “Representation Learning with Contrastive Predictive Coding”. ArXiv\nT. Chen et al. 2020, \"A Simple Framework for Contrastive Learning of Visual Representations\" ICML 2020\nM. Garnelo et al. 2018, “Conditional Neural Processes”, ICML 2018\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are correctly addressed. Authors indicate that the method is not able to directly predict on the observation space as one could desire. Moreover, they indicate the advantages of using the model for better understanding of 3D scenes, what might imply some risks as well.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3,5"}, "extra_text": {}}, {"review_id": "DuHtgzaSQt", "reviewer": "Reviewer_oGtj", "summary": "This paper proposes a meta-learning model that represents stochastic processes through a contrastive representation. The authors mainly addressed the unified framework which could handle the two types of tasks; 1) given a set of data and target covariate (they called it the targeted task) and 2) estimating the label for the set of data (called untargeted task). To do that, they designed the specific encoder for the targeted task and by applying a self-attention mechanism on their encoder, they improved the performance for the tasks. They validated their model on interesting tasks like bi-modal sinusoidal regression and semi-supervised learning. I think that the architectural novelty of their model is the unified design for targeted/untargeted tasks and self-attention layers. However, I don't think their contribution is not limited to them, they validated/analyzed their model and other meta-learning models trained through the contrastive losses (e.g., FCLR and MetaCDE) for a variety of interesting tasks, which is one of the strong points on this paper I thought.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "When I first review this paper, I felt it is quite similar to FCLR. I focused to find the difference between FCLR and CReSP, and as I understand, the main differences are 1) targeted encoder for the targeted task and 2) self-attention on the encoder. The authors explained it as CReSP is the unified framework for meta-learning with contrastive loss. It could be seen as incremental work, but they validated their model and previous works on several interesting tasks including targeted and untargeted tasks. They pretty well analyzed the results (e.g., for the ShapeNet task, why the performances of CReSP and FCLR decrease when the number of given views increases), and their model outperforms previous models, so I think that we cannot say easily it is just incremental work. Overall writing is good to read but the notation for covariate $\\xi$, observation $x$ and label $y$ caused some confusion for me, because usually, NPs papers use $x$ as the covariate and $y$ for observation. If one of your contributions is the unified framework for meta-learning model with contrastive loss, then using the below storyline as main could make this paper more understandable. \n\nUsing contrastive loss for meta-learning is very beneficial -> FCLR was proposed ->  but it cannot cover targeted task well, so we proposed unified framework including the FCLR\n\nQuestions / comments\n- Line 103-4, you mentioned NP model is the model with additional latent variable on top of CNP, but it is not correct. CNP samples the functional latent when decoding with each target query, which causes inconsistant uncertainty, so the authors updated their CNP to sample the functional latent before decoding to resolve the inconsistant uncertainty problem. GQN that is origin of CNP (CNP is generalized version of GQN) is also inconsistant for uncertain knowledge, so Consistant GQN (Ananya et al., 2019) was proposed.\n- Line 203-7, what is the meaning of equation in line 204? self-attention? From my perspective, the equation causes not necessary confusion. Why did you use Image Transformer, not general Transformer? The performance when using Image Transformer was better?\n- For Sinusoids task, how did you use ACNP for untargeted task? Did you summarize the representations for the context set?\n- In Fig. 3, second image in second row is not distorted.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Their model has been validated for quite simple labels like parameters of sinusoids or one of 13 class labels. However, those models with contrastive loss are more powerful for visually complicated tasks like 3D map world model for agent. For those tasks, the representations $c$ are required to contain enough knowledge for environment. I am not sure that CReSP can cover those complex tasks also.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9 Hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: One of the reviews lacks thoroughness - I'm happy to disregard this. 

The work proposes meta learning through neural processes and contrastive learning. There is some disagreement about the level of contribution here, with several reviewers pointing out that the FCLR method also contributes contrastive learning, and another reviewer pointing out that this is not so incremental because of extensions in several directions. 

One reviewer raises a strong technical objection, that the "stochastic process perspective taken in the paper seems only tenuously related to the actual training and prediction schemes". The authors refute - "using a deterministic embedding of context data that is sampled from a stochastic process is (not) invalid". the authors agree to amend the work to improve clarity, and the reviewer conceded that clarifying this point in the manuscript would make for a strong contribution. 

One reviewer raised issues with the experiments - that they do illuminate the method or support the proposed ideas. Yet I'm inclined to agree with another reviewer who found the detailed analysis, descriptions, and hyperparameters in the supplementary material. I also approve of the experiment to show limitations of the method in learning the strongly multimodal case. 

Overall, I think the authors have enough feedback from the reviewers to make some tweaks to some of the explanations in the paper, making this a contribution to the NeurIPS community. 
Decision: Accept (Poster)