Function Contrastive Learning of Transferable Representations
Authors: Muhammad Waleed Gondal
Shruti Joshi
Nasim Rahaman
Stefan Bauer
Manuel Wuthrich
Bernhard Schölkopf
Venue:  (2021)
Keywords: Representations Learning
Few-shot Learning
Contrastive Learning

Abstract: Few-shot-learning seeks to find models that are capable of fast-adaptation to novel tasks which are not encountered during training. Unlike typical few-shot learning algorithms, we propose a contrastive learning method which is not trained to solve a set of tasks, but rather attempts to find a good representation of the underlying data-generating processes (\emph{functions}). This allows for finding representations which are useful for an entire series of tasks sharing the same function. In particular, our training scheme is driven by the self-supervision signal indicating whether two sets of samples stem from the same underlying function. Our experiments on a number of synthetic and real-world datasets show that the representations we obtain can outperform strong baselines in terms of downstream performance and noise robustness, even when these baselines are trained in an end-to-end manner.

Under review as a conference paper at ICLR 2021
FUNCTION CONTRASTIVE LEARNING OF TRANSFER- ABLE REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot-learning seeks to find models that are capable of fast-adaptation to novel tasks which are not encountered during training. Unlike typical few-shot learning algorithms, we propose contrastive learning method which is not trained to solve set of tasks, but rather attempts to find good representation of the underlying data- generating processes (functions). This allows for finding representations which are useful for an entire series of tasks sharing the same function. In particular, our training scheme driven by the self-supeni signal indicating whether two sets of samples stem from the same underlying function. Our experiments on a number of synthetic and real-world datasets show that the representations we obtain can outperform strong baselines in terms of downstream performance and noise robustness, even when these baselines are trained in an end-to-end manner.
INTRODUCTION
The ability to learn new concepts from only a few examples is a salient characteristic of intelligent behaviour. Nevertheless, contemporary machine learning models consume copious amounts of data to learn even seemingly basic concepts. The mitigation of this issue is the ambition of the few-shot learning framework, wherein fundamental objective of different problems (Bengio et al. 2019). In this work, we propose a self-supervised method for learning such representations by leveraging the framework of contrastive learning.
learn representations that apply to variety
We consider setting very similar to the one in Neural Processes (NPs) Garnelo et al. 201 Kim
al. 2019):: The goal is to solve some task related to an unknown function after observing just few input-output examples O' = {(x,, y:) Ji- For instance, the task may consist of predicting the function value y at unseen locations x, or it may be classify images after observing only a few pixels (in that case x is the pixel location and y is the pixel value). To solve such task, the example dataset Of needs to be encoded into some representation of the underlying function f. Finding a good representation of function which facilitates solving a wide range of tasks, sharing the same function, is the object of the present paper.
Most existing methods approach this problem by optimizing representations in terms of reconstruction i... prediction of function values y at unseen locations x (see e.g. NPs (Garnelo et al.) 2018a b Kim ett al., 2019) and Generative Query Networks (GQNs) (Eslami et al. 2018)). A problem with this objective is that can cause the model to waste its capacity on reconstructing unimportant features, such as static backgrounds, while ignoring the visually small but important details in its learned representation (Anand et al. 22019: Kipf et al., 2019). For instance, in order to manipulate a small object in complex scene, the model's ability to infer the object's 's shape carries more importance than inferring its color or reconstructing the static background.
To address this issue, we propose an approach which contrasts functions, rather than attempting to reconstruct them. The key idea is that two sets of examples of the same function should have similar latent representations, while the representations of different functions should be easily distinguishable. To this end, we propose novel contrastive learning framework which learns by contrasting sets of input-output pairs (partial observations) of different functions. We show that this self-supervised training signal allows the model to meta-learn task-agnostic, low-dimensional representations of functions which are not only robust to noise but can also be reliably used for a variety of few- shot downstream prediction tasks defined those functions. To evaluate the effectiveness of the proposed method, we conduct comprehensive experiments on diverse downstream problems including classification, regression, parameter identification, scene understanding and reinforcement learning.

---

Under review as a conference paper at ICLR 2021
N
X
32
go
go
Figure 1: FCRL for superviee scene representation learning: The representations are learned by mapping the context points of each scene closer in the latent space while separating from the other seenes, The context points correspond to the tuples of camera viewpoints I; E X' := IR and
the images taken from those viewpoints y/ E y C Rd Seen here, TriFinger (Wüthrich et al. 2020
We consider different datasets, ranging from simple ID and 2D) regression functions to the challenging simulated and real-world scenes. In particular, find that a downstream predictor trained with our (pre-trained) representations performs comparable or better than related methods on these tasks, including the ones wherein the predictor is trained jointly with the representation.
Contributions.
Our key contribution is the insight that the good representation of an underlying data-generating function can be learned by aligning the representations of the samples that stem from it. This perspective of learning functions' representations different from the typical regression methods and is new to our knowledge.
We propose novel contrastive learning framework, Function Contrastive Representation Learning (FCRL), to learn such representation function, using only its few observed samples.
With experiments diverse datasets, show that the functions' representations learned by FCRL are not only robust to noise in inputs but also transfer well to multiple downstream problems.
2
PRELIMINARIES
2. PROBLEM SETTING
Consider data- data-generating function f:X y with X Rd and y C Rd:
y f=) +5: E {~NN00,
(I)
where { E~ {~N(00,0²) is Gaussian noise. Let O = ((1,, i be set of few observed samples of a function f, referred to as the context set and T = {TT) be a set of unknown downstream tasks that can be defined on f. The downstream tasks T can take the form of classification, regression and/or reinforcement learning problems and the targets in each task Tj vary accordingly. Our goal to learn the representation representation retains the maximum useful information about f and can interchangeably be used for multiple downstream tasks T, defined on the same function (without requiring retraining).
given function / from only the context set Of, such that the
Few-shot Interpretation. Note that the downstream tasks T are defined directly on the function's representation. Since the representation inferred from few observations Of, the overall down- stream problem becomes few-shot prediction problem. In the following, regard the downstream problems as few-shot problems, unless stated otherwise.
2.2 BACKGROUND
While there exist a variety of methods (Xu et al. 20199; Finn et al. 2017) that take different approaches towards few-shot learning for functions, a class of methods that is particularly relevant to our setting that of conditional neural processes (CNPs) and NPs (Garnelo al. 2018a b).
Conditional Neural Processes (CNPs), The key proposal in CNPs (applied to few-shot learning) is to express distribution predictor functions given context set. To this end, they first encode the context Of into individual representations r; he (I; y.) Vi E [N], where he a neural network. The representations then aggregated via mean-pooling operation into fixed size
2

---

Under review as a conference paper at ICLR 2021
vector r 1/N(r1 - -2 + r.N. The idea is that captures all the relevant information about from the context dataset Of; accordingly, the predictive distribution is approximated by maximizing the conditional likelihood of the target distribution p(y" x²", Of). where Ti = {( ) the target set of size M.
N+M is
Generative Query Networks (GQN). GQN (Eslami et al. 2018 can be seen as an extension of NPs for learning 3D scenes representations. The context dataset OJ in GQN consists of tuples of camera viewpoints in 3D space (2t) and the images taken from those viewpoints ())). Like NPs, GQN learns infer the latent representation of the scene (a function) by conditioning on the aggregated context and maximizing the likelihood of generating the target image corresponding to a target viewpoint.
2.3 MOTIVATION AND INTUITION
Learning about generaeeng functions is ubiquitous in machine learning. Whether it is about learning scenes as deterministic functions that map camera viewpoints to the images (Eslami et al. 2018) or digital soil mapping where one is given set of soil samples taken from some regions, and asked to predict the nature of soil in another region (Minasny & McBratney 2016). In practice, we only observe a few views Of of such functions and the goal is to predict certain properties regarding those functions. More importantly, the observed samples often do not follow any order, noisy and exhibit sampling biases, i.c. some samples are more informative than others. For instance, in learning scene representation from multiple views, a single sampled view can have some objects occluded.
An ideal representation of a function, therefore, should not only be invariant to the permutations in the order of the observed samples, but also to noise and sampling of partial views O², Such an invariance to input order and sampling is important, as at test time we do not know the input order of the samples or how informative certain sample is. In this work, we seek to learn such invariant representations of functions from only their few observations Of, The inductive bias pertaining to permutation invariance is introduced via -ecooooompo encoder architecture (Zaheer et al.. 2017). Whereas, invariance to the sampling of partial views is enforced via the contrastive objective. The idea to align the representations of different samples stemming from the same function such that they are invariant to where they are sampled.
FUNCTION-CONTR.CO
REPRESENTATION LEARNING (FCRL)
We take the perspective here that the sets of context points Of provide a partial observation of an underlying function f. Our goal is to find an encoder fie.0) which maps such partial observations to low-dimensional representations of the underlying function. The key idea is that good encoder Ito.0) should map different observations of the same function to be close in the latent space, such that they can easily be identified among observations of different functions. This motivates the contrastive-learingg objective which we will detail in the following.
Encoder Structure. Since the inputs to the encoder (( are sets, it needs to be permutation invariant with respect to input order and able to process inputs of varying sizes. We enforce this permutation invariance in ((o,4) via sum-decomposition, proposed by (Zaheer et al., 2017p. We first average-pool the point-wise transformations Of to get the encoded representations.
N
Σ
ha( v.) V(r, E O
where (.) is the encoder network. We then obtain a nonlinear projection of this encoded repre- sentation (().o) (Of)= = (r)). Note that the function Ps can be any nonlinear function. We use an MI.P with one hidden layer which also acts as the projection head for learning the representation. Similar to (Chen et al. 2020)), we found that it is beneficial to define the contrastive objective on these projected representations (r) than directly on the encoded representations More details can be found in our ablation study Appendix[A.I
Encoder Training, At training time, we are provided with the few observations 0 of K functions. Each observation is a set of N examples O* = {xx,, y²) ) To ensure that different observations of will now formulate a contrastive-leaniin
the same functions objective. To apply contrastive learning. we need to create different observations of the same function I by splitting each example set O into J subsets of size N/J. Defining Lj == (Gi-1)N/J,....NN
mapped to similar representations,
3

---

Under review as a conference paper at ICLR 2021
hence obtain split of Oh into J/ disjoint subsets of equal size
O = = with Of, n no, 0 iif
where each subset 0f is a partial observation of the underlying function k. We need at least two distinct observations (J 2 2) per function for contrastive learning. Hence, is a can vary in the range of [2, ] while ensuring the even division of N. Its value empirically chosen based on the data domain. E.g. in ID and 2D regression, the number of examples per observation is relatively large, since a single context point does not provide much information about the underlying function, whereas in scenes, few (or even one) images provide enough information. We can now formulate the contrastive learning objective as follows:
and
K Σ
exp
(ii
9 a) Ko (Ot:) (1e. /T)
L(ø, Φ 0l.
log
(4)
Σ exp
sim ( (Ot,) 91 (O /T)
where sim(a, = function and predicts for each observation O which of the J observations it was jointly drawn with, by assigning high scores to the pairs jointly drawn and low to other pairs. In other words, the objective function in Equation (4) enables learning the differences between the distributions of K underlying functions by discriminating between pairs of observations coming from the same function k, called positives, and observations stemming from different functions, called negatives. The is temperature parameter which scales the scores, returned by the inner product. Similar to SimCLR (Chen et al. 2020), we find that temperature adjustment is important for learning good representations. We treat it as a hyperparameter and perform an ablation study in Appendix A. The overall FCRL algorithm is deseribed in Algorithm
TTb is the cosine similarity measure. Intuitively, it acts as discriminatory
Note that even when J/ > 2, we only consider two positives at a time for computing the contrastive loss i.c. for each view, have only one positive match. We then compute the contrastive loss for all the pairwise combinations of the positives and then sum them up. For instance, when J=3, we have &C2 combinations of positives. The term Eigizgo in Equation (4) refers to that summation over all the combinations.
3.1
APPLICATION TO DOWNSTREAM TASKS
Once representation learning using FCRL has been completed, he is fixed and can now be used for any few-shot downstream prediction task T = {Tr)T= that can be defined on the underlying data-generating function. To solve a particular downstream prediction problem, parametric decoder Pv (-Ir) which conditions on the learned representation r. Specifically, the decoder trying to predict in the given task. Depending on the nature of the downstream prediction task, the conditional distributions and the associated objectives can be defined in different ways.
only optimize
maps the representations learned in the previous step the variable
4 EXPERIMENTS
To illustrate the benefits of learning function representations without explicit decoder, consider four different experimental settings. In all the experiments, we first learn the encoder, and then fix it and optimize decoders for the specific downstream problems in the second stage.
Baselines. We compare the downstream predictive performance of FCRL based representations with the representations learned by the closest task-oriented, meta-leariing baselines. For a fair comparison, all the baselines and FCRL have the same encoding architecture. For instance, for 1D and 2D regression functions, we considered CNPs and NPs as the baselines which have the same contextual embedding framework but optimize directly for the predictive distribution p(z(ar). Similarly for scene datasets, we took GQN as the baseline which explicitly learns to reconstruct scenes using a limited number of context sets.
4.1 1-D SINUSOIDAL FUNCTIONS
In the first experiment, we consider a distribution over sinusoidal functions, proposed by (Fim et al. 2017). The amplitude and the phase of the functions are sampled uniformly from [0.1, 0.5
4

---

Under review as a conference paper at ICLR 2021
MMM
Aup
FORL M
Papp
P
Pe
ONP
ho
(b)
Figure 2: (a) Qualitative comparison of different methods on 5-shot regression task on three different sinusoid waves. The decoder trained on FCRL's representation is able predict the correct form of the waves. (b) shows how we adapt the representation for the downstream few-shot regression task.
and [0, æ respectively. To generate a context set of a particular function f. the x-coordinates are uniformly sampled from [-5.0, 5.0] and then f is applied to obtain the y-coordinates. The dataset contains 20, 000 training. 1000 validation and 1000 test functions. Once the representation is learned, define downstream tasks on the learned representation Tin {T T spp),, where T far corresponds to few-shot regression task and yspi corresponds to few-shot parameter identification task. We use the same training data to train both the encoder in the representations learning phase and the decoders in the downstream transfer learning phase.
Representation Learning Stage. For training the encoder I(a.o), we have dataset O (O*) ) at our disposition, where each & corresponds to a function f&x which has been sampled as described above. Each of these individual datasets O { {< y 1/ consists of N input-output pairs from that particular function fk. such that y fx (z We fix the maximum number context sets to 20 and the number of examples N is chosen randomly in 20) for each k. The encoder I(. is then trained as described in Section by splitting each context set O into two observations i.e. J 2.
4.1.1 DOWNSTREAM TASKS ON 1-D SINUSOIDAL FUNCTIONS
After training the encoder ((),)), we discard the projection head Ps and use the trained encoder htp extract the representations. The decoders for the downstream tasks trained as following.
Few-shot Regression (FSR). FSR for 1-D functions
well-studied problem in meta-learning
frameworks (Garnelo al. 2018a Finn al., 20177: Kim et al., 2019: Xu al. 2019). For each sampled function fk. the target data is of the form 7F for = { Vi ) where M is the number of unseen target samples of function k. The number M is chosen randomly in [0, 20 N]. The goal is to predict the function values f (x=T) at unseen inputs n², given some context O² The predictive
distribution for f is (v lot" O4) and the decoder p&p is trained to optimize the following conditional log probability
min B E [log Pally 1 ( r']]]] N+M
where r is the encoded representation of fu. obtained by the context set O and the trained encoder he, following Equation (2). Figure 2 how the downstream decoder Po is trained on r* The decoder Pe is multi-layer perceptron with two hidden layers and is trained with the same training data as h4. The decoder learns to output the Gaussian mean and variance of y to quantify uncertainty in the point estimates. The qualitative results on test sets in Figure 2 a) show that our model is able to quickly adapt with only 5 context points. We compare our method with CNP and NP and show that the predictions of our method are closer to the groundtruth, even though the encoder and decoder in both CNP and NP are explicitly trained to directly maximize the log likelihood to fit the wave.
Few-shot Parameter Identification (FSPI). The goal here is to identify the amplitude and phase of the sampled sine wave ft. For each ft. the target data is of the form T fsp:= (vamp Jyhase where Vimp and Uphase are the amplitude and phase values of fg- We train a linear decoder Pv on top the representations by maximizing the likelihood of the sine wave parameters. The predictive distribution is Po. (T).ps 10y) and the objective is:
min E ( (log Pe (o (r
(6)
5

---

Under review as a conference paper at ICLR 2021
Few-shot Regression
Few-shot Parameter Identification
Models
5-shot
20-shot
5-shot
20-shot
NP
0.310 + 0.05
0.218 + 0.02
0.0087 + 0.0007
0.0037 + 0.0005
CNP
0.265 + 0.03
0.149 + 0.02
0.0096 + 0.0007
0.0049 + 0.0011
FCRL
0.172 + 0.04
0.100 + 0.02
0.0078 + 0.0004
0.0032 + 0.0002
Table 1: Mean squared error for all the target points in 5 and 20 shot regression and parameter identification tasks on test sinusoid functions. The reported values are the mean and standard deviation of three independent runs.
where r is the encoded representation of fx, obtained by the context set O and the trained encoder he, following Equation (2). Similar to FSR, we use the same training data. In Table we report the mean squared error for three independent runs, averaged across all the test tasks for 5-shots and 20-shots FSR and FSPI. In both prediction tasks, the decoders trained on FCRL representations outperform CNP and NP. More details on the experiment are given in Appendix G
4.2 MODELING IMAGES AS 2-D FUNCTIONS
In the second experiment, we consider a harder regression dataset where images are modelled as 2-D functions. This form of image modeling has been considered in the context of image completion by (Garnelo et al. [20 Sa bj Gordon et al., 22200 Kim et al.) (219). More specifically, the 2-D pixel coordinates of image are considered the inputs x; while the corresponding pixel intensities the outputs yi- We consider images of MNIST hand-written digits (LeCun et al. 1998) where d; are the normalized pixel coordinates [0, 1J2 and y, the grayscale pixel intensities (0, 1J. Similar to the previous section, we first describe how we learn the representations, and then we formulate two downstream prediction tasks on the learned representation: TaD = {T Tccce), where Tysci corresponds to few-shot image completion and T Jace corresponds to few-shot content classification task. The training data consists of 60, 000 MNIST training samples whereas the validation is done with the 10, 000 test samples. We use the same data to train both the encoder in the representations learning phase and the decoders the downstream transfer learning phase.
Representation Learning Stage. For training the encoder (d),)) we use the same training procedure as used for 1-D functions. We have context set O (O*) where each l corresponds an MNIST digit. Here, each individual context set O+ ((xr, yz N corresponds to N sampled pixels, where x; is the pixel location and is the corresponding pixel's intensity. We fix the maximum number of context sets 200 and the number of examples N is chosen randomly in [2, 200] for each k. The encoder I(o is then trained as described in Section by splitting each context set O into 40 observations i.e. J = 40.
K k=1 of K functions
disposition,
4.2.1 DOWNSTREAM TASKS ON 2-D FUNCTIONS
Again, after training the encoder ((..)). we discard the projection head ps and use the trained encoder h& to extract the representations. The decoders for the downstream tasks are trained as following.
Few-shot Image Completion (FSIC). For few-shot image completion, we train a decoder Pv to maximize the conditional likelihood of the target pixel values The / ((x, of
M ) where M is the number of unseen target pixels of function fs- The number M is chosen randomly in [0, 200 N
The conditional likelihood objective is the same as for FSR in ID sinusoid functions i.e. Equation (
The decoder is an MLP with two hidden layers as in CNP and NP. We use the same dataset to train as was used to train the encoder.
Qualitative results of FSIC on test images are shown in Figure 3a). It can be seen that the decoder trained on FCRL representations is able to predict the pixel intensities even when the number of context points is as low as 50, i.c., approximately 6% of the image. We compare its performance against CNP which uses the same architecture. It is important to note that the feature extraction via FCRL is not explicitly carried out to optimize for this pixel-based reconstruction, and the decoder has to rely on the signal in the pre- trairra representation for this complex image completion task. In other words, no gradient flows from the decoder to the encoder, and the former is trained independently of the latter. On the contrary, CNP jointly optimizes both encoder and decoder parameters for predicting

---

Under review as a conference paper at ICLR 2021
Shots
8
3
9
8
7
Contat
0.6
3
9 9
3
2
7
ONF
Dist.
FORL
2
3
8
7
9
200
400
300
6.00
703
Number Content Points
(a)
(b)
Figure 3: (a) Qualitative comparison of CNP and FCRL based few-shot image completion. The context shown in the second row where target pixels are colored blue. FCRL appears to be slightly better at predicting the correct form of digit in low-shot scenario of 50 context points only. (b) Quantitative evaluation of the models terms of digit classification from the fixed number of context points (varying along the x-axis). FCRL achieves substantially higher accuracy.
the pixels values. Despite the fact, it appears that the FCRL-based reconstruction is slightly more accurate when using only 50 context points.
Few-shot Content Classification (FSCC), To test how much semantic knowledge each model captures in its representations, we devised a few-shot content classification task 7 Jsec- The goal is to predict the class of each MNIST digit, given its few randomly sampled pixel values as context Ok The lack of spatial structure in the context points makes it challenging problem. We consider the representations of all the functions trained in the previous section and train a linear decoder on top to maximize the likelihood of the corresponding class labels T fsoc (yome i.e.:
min E (x llg Pay (y hot Ir*)]
(7)
We the same data was used for training the encoders. F shows the performance of three independently trained models for each method on MNIST validation set. FCRL has been able recognize the semantic labels the images much better than the competing methods for different number of fixed context points. This probably also explains why it is able to guess the correct form of digits in FSIC, albeit little blurry.
4.3 REPRESENTING SCENES AS FUNCTIONS
Like (Eslami et al. 2018), we represent scenes deterministic functions which map camera view- points to images. A specific scene (i.e. function) is given by the underlying factor of variations (e.g. the position of an object in the scene). To learn scene representations, datasets: the real-world robotics dataset, MPI3D (Gondal et af., 2019), and robotics simulation dataset (RLSCenes), which we created for the present work. Both datasets contain multiple views for each scene. In the MPI3D dataset, the multi-view setting is formulated by considering the images corresponding the camera heightt factor views, hence, there are three views per scene. RLScenes comprises of 36 views per scene, corresponding to 36 uniformly distributed 3D viewpoints. More details on the datasets are provided Appendix
two different
The purpose of using two different datasets for scenes is to test the generalization capability of the learned representations two different types of downstream tasks. The set of downstream tasks on MPI3D correspond to scene understanding. Tmpian = [Tx [e where u corresponds to a factor of variation. On the other hand, RI Scenes is used to test the generalization of the learned representations to reinforcement learning downstream task.
We use the same setting for learning the representations on both datasets. Each function fú corresponds to scene here, and randomly draw N tuples of (viewpoints, views) as context sets 0 = {(>} ,/) where 2; is the 3D camera viewpoints and Y: is the corresponding image, taken from that viewpoints. We fix the maximum number of context sets to in MPI3D dataset and 20 in RLScenes. The number of tuples drawn, N, is then chosen randomly in [2, 3] for MPI3D and [2, 20) for RCScees.. For more details, Appendix
Scenes' Representation Learning Stage.
7

---

Under review as a conference paper at ICLR 2021
Background Color
Object Color
Object Shape
Object Size
DOF
2nd DDF
1180
Robe Validatin Acc.
0.99
0.29
3.187
8.03
100
3.0
FDRL
KCN
PCRL
FCM
453
FCAL
FORL
0.94
8:05
GON
Figure 4: Quantitative Comparison of FCRL and GQN on MPI3D downstream classification tasks.
4.3.1 DOWNSTREAM TASKS ON SCENES
After training the encoder ((a,0), we discard the projection head Ps and use the trained encoder hop to extract the representations r and train decoders for downstream problems.
Scene Understanding MP13D Dataset. In MPI3D, each scene described by 6 factors of variations. Therefore for each sampled scene fx. we have 6 different tasks Tmpi3D = ]e=> Each task T comprises of a discrete label for a factor of variation i.e. T = {vr}. Hence, we train linear decoders to maximize the likelihood of the labels corresponding objective as in Equation (7). Note that we use only one image to infer the representation r. Figure shows the linear probes validation performance for five independently trained models on both GQN learned representations and FCRL learned representations. It can be seen that the representations learned by FCRL consistently outperform GQN for identifying all the factors of variations scenes.
the factors, using the same
Reinforcement Learning on RLScenes Dataset. For RLScenes, instead of training linear decoders for downstream tasks, we train an RL agent to learn a control policy for the robot finger. The goal for the agent is to locate the object in the arena, reach it, and stay close to it for the remainder of the episode. We use the Soft-Actor-Criee Critic (SAC) algorithm (Haarnoja et al., 2018) using the implementation from (Raffin et al. 2019) to learn single MLP policy for all the joints of the robot, and predict their positions by having access only to the representation of the scene. These representations inferred from single view. The positions of the robot-finger and the object are reset at the beginning of every episode. should be noted that unlike previous experiments, the downstream policy not trained with the same dataset as was used train the encoder. We again use the representations trained with GQN as our baselines and evaluate the performance on five independent runs. The results are shown in Figure 5(a). It can be seen that the RL agent, learned with FCRL representation clearly outperforms GQN- based RL agent in terms of learning data-efficient policy, obtaining convergence level control performance with approximately times fewer interactions with the environment.
-1.2
0.8
MNIST Probe Validation Acc.
Episode Mean Rewards
0.7
-1.6
0.6
Method
NP
0.5
CNP
FCRL
-2.0
FCRL
0.3
-2.2
15K
0.25
Number of Episodes
Noise Std.
(a)
(b)
Figure (a) Comparison between GQN and FCRL on learning data-efficient control policy
for object reaching downstream task. FCRL based representations clearly outperform GQN's representations. (b) Quantitative comparison for noise robustness on MNIST content classification downstream task. The representations learned with FCRL are much robust noise than GQN's.
8

---

Under review as a conference paper at ICLR 2021
4.4 ROBUSTNESS TO NOISE CORRUPTIONS
So far in our experiments, we have considered the functions to be noiseless. However, in real-world settings, the functions are often corrupted with noise which is manifested in their outputs. In this section, we investigate how robust different representations learning algorithms are to Gaussian noise with varying levels of noise corruption. We train all the models on the noisy data and evaluate the quality of the learned representation on downstream tasks, as defined above. We find that the representations learned by FCRL are far more robust to such noise corruptions than the competing methods Figure(5ib) In fact, the performance of CNPs, NPs and GQNs drop so much that downstream tasks achieve random accuracy. This might be because the representations are learned by optimizing to reconstruct the noisy outputs where signal noise ratio is very low. On the other hand, FCRL learns by contrasting the set of examples, extracting the invariant features, and getting rid of any non-correlated noise in the input. See Appendix B for noise robustness study on MP13D dataset.
RELATED WORK
Meta-Learning. Supervised meta-learning can be broadly classified into two main categories.
The first category considers the learning algorithm to be an optimizer and meta-lcarning is about optimizing that optimizer, for e.g., gradient-based methods (Ravi & Larochelle, [2066] Finn et al.)
2017. Li et al.] 2017: Lee et al. (019)) and metric-learning based methods (Vinyals et al. 2016 et al.]| 2019 Qiao et al.] 2019). The second category is the family of Neural Processes (NP) (Garnelo et al. 20 8a Kim et al. 2019 Eslami et al. 2018) which draw inspirations from Gaussian Processes (GPs). These methods use data-specific priors in order to adapt to new task at test time while using only a simple macoder-decode architecture. However, they approximate the distribution over tasks in terms of their predictive distributions which does not incentivize NP to fully utilize the information in the data-specific priors. Our method draws inspiration from this simple, yet elegant framework. However, our proposed method extracts the maximum information from the context which is shown to be useful for solving not just one task, but multiple downstream tasks.
Snell et al. [ Sung et [0.8]201:|AAl
Self-Supervised Learning. Self-supervised learning methods aim learn the meaningful representa- tions of the data by performing some pretext learning tasks (Zhang et al., 2017 Doersch et al. (2015).
These methods have recently received huge attention (Tian et al.) [0e9] et al. [0188] Bachman et al. 2 Chen et al. 2220; He et al. 2219)) mainly owing their success to noise contrastive learning objectives (Gutmann & Hyvärinen 2010). At the same time, different explanations have recently come out to explain the success of such methods for e.g. from both empirical perspective (Tian et al., 2020 [Tschannen et al., 2019)) and theoretical perspective (Wang & Isola 2020 AAror et al. (2119). The goal of these methods has mostly been to extract useful, low-dimensional representation of the data while using downstream performance as a proxy to evaluate the quality of the representation. In this work, take inspiration from these methods and propose self- super-sueee learning method which meta-learn the representation of the functions. Instead of using randomly augmented views of the data points, our self-supervised loss uses partially observed views, sampled from the underlying functions. More recently, (Doersch et al. 2020) pointed out the generalization issue in meta-learning algorithms, namely the supervision collapse, and proposed self-supervised pre-training address this issue. Similar setup is used by (Medina et al. 2020) for prototypical networks (Snell et al.) 2017)) However, these methods study generalization tasks defined on new datasets. In contrast to these methods, we study generalization in terms of downstream tasks, defined on functions.
6 CONCLUSION
In this work, we proposed a novel self-supervised representations learning algorithm for few-shot learning problems. We deviate from the commonly-used task-specific training routines in meta- learning frameworks and propose to learn the representations of the relevant functions independently of the prediction task. Experiments on widely different datasets and the related set of downstream few-shot prediction tasks show the effectiveness of our method. The flexibility to reuse the same representation for different task distributions defined over functions brings us one step closer towards learning generic meta-learning framework. In the future, plan to adapt this framework to tackle multiple challenging few-shot problems such as object detection, segmentation, visual question answering etc. all of them using the same generic representations of the geteraine processes.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Kelsey R Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning, arXiv preprint wrXiv:192..0455, 2019.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Mare-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems, pp. 8766-8779, 2019.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: 1rXiv:9902.09229, 2019.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15509-15511 2019.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms, 2019.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL httph: wandb com/ Software available from wandb.com.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet: org 2016-2019.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformenss spatially-aware few-shot transfer. arXiv preprint arXiv:2007. 11498, 2020.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 16046994):1244-1220, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp.
JMLR. org, 2017.
Marta Garnelo, Dan Rosenbaum, Chris J Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Eslami. Conditional neural processes. arXiv preprint 10XXv:1807.0161.1 2018a.
Marta Gareelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Tch. Neural processes. arXiv preprint arXiv:1807.01622. 2018b.
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Berhhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. In Advances in Neural Information Processing Systems, pp. 15740-15751, 2019.
Jonathan Gordon, Wessel Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. 2020.
10

---

Under review as a conference paper at ICLR 2021
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. pp. 297-304, 2010.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprini arXiv:1801.0122) 2018.
Charles Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, et al. Array program- ming with numpy. Nature, 585(7825):357- 362,, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 2019.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint wrXiv:1808.06670 2018.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9()): 90-95, 2007.
Shruti Joshi, Felix Widmaier, Vaibhav Agrawal, and Manuel Wüthrich. httpt:://ggttuiiittt com/
open-dynamic-rooeee http://wwnn..ooooooooaaaaaatta/a finger. s simulation 2020.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint 1901.00001.057600 2019.
Thomas Kipf, Elise van der Pol, and Max Welling, Contrastive learning of structured world models. arXiv preprint arXiv:1911.122
2019.
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter notebooks-a publishing format for reproducible computational workflows. In ELPUB, pp. 87-90, 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2324 1998.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint 1707 2017.
Carlos Medina, Aroou Devos, and Matthias Grossglauser. Self-supervised prototypical transfer learning for few-shot classification. arXiv preprint arXiv:2006. 1325, 2020.
Budiman Minasny and Alex B McBratney. Digital soil mapping: A brief history and some lessons. Geoderma, 264:301-311, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.0330 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http::::pppppeeeee www.pesesseerrirsscoomcm 9015- 1115-pytorch-ee an wwwpriitieeeeeeeeeeeeeeee orrprpacttiitttteeeeeeeeoererrer deep- pyt.orrcaaaa--mprraateee-e---ee-
high
Library.
pdd

---

Under review as a conference paper at ICLR 2021
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830. 2011.
Limeng Qiao, Yemin Shi, Jia Li, Yaowei Wang, Tiejun Huang, and Yonghong Tian. Transductive episodic-wise adaptive metric for few-shot learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3603-3612, 2019.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave Anssi Kanervisto, and Noah Dormann. Stable baselines3, https: http://////tttcbcoooooo/o RW/staable 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, In Advances in neural information processing systems, pp. 4077-4087, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 19X,::1906.05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning, arXiv preprint arXiv:2005. 10243, 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13623 2019.
Guido Van Rossum and Fred L. Drake. Python Reference Manual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. arXiv preprini urXiv:2005.10242 2020.
Michael Waskom, Olga Botvinník, Drew O'Kane, Paul Hobson, Saulius Lukauskas, David C Gemperline, Tom Augspurger, Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven, Julian de Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram, Tal Yarkoni, Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, Chris Fonnesbeck, Antony Lee, and Adel Qalieh. mwaskom/seabomm: v0.8.1 (september 2017), September 2017. URL stttps://doi.. 0tt1:/000002022222od...333
Manuel Wüthrich, Felix Widmaier, Felix Grimminger, Joel Akpo, Shruti Joshi, Vaibhav Agrawal, Bilal Hammoud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, et al. Trifinger: An open-source robot for learning dexterity. arXiv preprint arXiv:2008. 20t.::0000033996,, 2020.
Jin Xu, learning with iterative functional updates. arXiv preprint arXiv: 2019.
Ton, Hyunjik Kim, Adam R Kosiorek, and Yee Whye Teh. Metafun: Meta-
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391-3401,201
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
12

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 ABLATION STUDY
Role of Number of Observations (J) The number of observations corresponds the number of partial observations that we have of a functions f*. Ideally, we only need two such observations to learn the representations via contrastive objective. However, it has been shown that having more positive pairs result in learning better representations (Chen et al. 2020 Tian et al., 2019). It also corresponds to reducing the amount of Bayesian surprise in the models, as more observations from the same function results in reducing the entropy.
Temperature 0.007
Temperature 0.07
Temperature = 1.0
0.8
MNIST Frobe Valliaation Acc.
0.7
0.6
0.5
Cntlic
bilnear
0.3
nonlnncor
100
40
20
40
20
Num, of Views
Num. of Views ())
Num. Views ())
Figure Ablation for observations (J), critics and temperature for learning FCRL on MNIST
validation dataset.
should be noted that in our setting, the number of observed context sets N do not necessarily correspond to the number of observations J. This is because for some datasets, we aggregate the context points to get one partial observation. This is important for the simple 1D and 2D regression functions where single context point does not provide much information, hence the individual partial observations need more than one context sets. In our setting, we treat as a hyperparameter whose optimal value varies depending the function space. For instance, the MPI3D scene dataset has only three views per scene, therefore J can not be greater than 3 and we keep it fixed. For ID and 2D regression functions. observe that for a fixed number of context points N, the optimal number of observations varies. For understanding the role of better in these experiments, we perform the ablation study MNIST2D validation dataset where three models with different seeds are trained on the the varying number of J while N is fixed to 200, Figure 7
It can be seen that varying the number of observations J contributes to very small changes. However, its values seems to be highly dependent on the critic and the temperature parameter as well. For MNIST2D, we fixed J = 10. For training on RLScene dataset, we fixed the maximum number of context points be N 20 and found the optimal number of partial observations to be 4.
Role of Crities and Temperature 7. We regard the discriminative scoring functions, including the projection heads as critics. The simplest critic function does not contain any projection layer, regarded as dot product critic, where the contrastive objective defined directly the representations returned by the base encoder hg. However, recently the role of critics in learning better representations has been explored (Oord et al. 2018 Chen al. 2020). Building on these findings, we evaluate the role played by different critics in learning the functions representations. Figure 7 shows the ablation for three different crities on MNIST2D validation dataset. It can be seen that the performance of critics is also linked with the temperature parameter
Such hyperparameter grid search (done for MNIST2D) is very expensive for the ablation studies on the bigger datasets i.e., the MPI3D and RLScenes datasets. We therefore performed a random sweep of 80 models with randomly selected syper-paramette values for critic and temperature MPI3D dataset. We did not find any pattern for the effect of temperature 7 on the downstream tasks, however the pattern emerged for the class of critics. Figure shows the ablation for critics MPI31>
13

---

Under review as a conference paper at ICLR 2021
dataset. Again, it can be seen that the nonlinear critic performs good in extracting features which useful for the downstream classification tasks. Because this trend across two different datasets, we therefore performed all our experiments with nonlinear critic. The project head in nonlinear critics is defined as an MI.P with one hidden layer and batch normalization between.
1.0
67
0.5
0.9
0.8
08
Obyyct Stape
Baskground
Objectt
0.8
0.6
0.5
04
0.98
10
0.6
0.90
0.5
2ne
0.6
Objjett
0.3
64
02
Crec
Cittic
Crbo
Figure 7: Ablation for crities on MPI3D validation dataset.
B ROBUSTNESS TO NOISE
Functions with additive noise have been well-studied, however, the contemporary literature meta- learning mostly considers them to be noise free. In this work, we explore whether the learned representations the functions are prone to the additive Gaussian noise. We consider the form of the function given in Equation ( and vary the standard deviation of the added noise. It can be seen that with the increased level of noise the features in the image start to diminish, shown in the Figure & GQN approach the learning problem by reconstructing these noisy images where the signal is aiready very weak. On the other hand, FCRL Iearns to contrast the scene representations with other scenes without requiring any reconstruction the pixel space. This helps it in extracting invariant features in the views of a scene, thus getting rid of any non-correlated noise in the input.
In addition to the analysis on MNIST 2D regression task Figure we test the performance of these representations learning algorithm on MP13D factors identification task in Figure [ It can be seen that FCRL representations can recover the information about the data factors, even in the extreme case where the noise level is very high (standard deviation of 0.2). Whereas, GQN performs very poorly such that the downstream probes achieve the random accuracy.
C PSEUDOCODE FOR SCRL ALGORITHM
For the sake of completeness, we provide the pseudocode for FCRL representations learning algo- rithm:
14

---

Under review as a conference paper at ICLR 2021
o == 0.0
o 0.1
0.2
Figure 8: MPI3D dataset with the varying level of additive noise.
9.6
0.
1.6
0.3
005
4.2
332 0.24
0.55
0:08 03.00 2.28
0.20
200
559
835
848
2.20
0:28
0.35
0:38
4.20
0:22
069
0:32
00.0
6:00 0.12
8.15
0.00
Figure 9: Quantitative comparison for noise robustness MP13D downstream tasks.
Algorithm 1: FCRL algorithm.
input: batch size K, dataset of functions with input-output pairs Of = (ar, y). maximum number context points M, constant T, J, networks has Pa- for sampled minibatch of K functions {Or ( do draw number N from range [2, M]
define observation size
ty := (() - 1)N/J. ..... jNJ Vj 1,
for all le E (1...., K) do
draw N context points O* {xx, 9: )
# split context into disjoint observations
O =f with O no 0 if +
for i E (1,..., J) and j E (1, (1, Jo
=x= Pol To,,
m=
ho (O,,
10tg1
2)= Pol 10,11
Em=1
ho (O) m) ))
8 2.T /
*
Z li
end for
end for
L (0) o, 6,0kKKK----- Σ
Eigresss log 5x exp a/ ///
explet,/7
update networks h4 and ps to minimize L. end for return base encoder network (.)
15

---

Under review as a conference paper at ICLR 2021
D ESTIMATING DENSITY RATIOS CORRESPONDING TO THE FUNCTIONS
The contrastive objective in Equation in essence, tries solve classification problem i.e. to identify whether the given observation O' comes from the function f or not. The supervision signal is provided by taking another observation O from the same function f' as an anchor (a target label), thus making a self-supervised method. This self supervised, fiew-classification task, for function f', leads to the estimation of density ratios between the joint distribution of observations p(O², O²)i) and their product of marginals p(O (O))))(((9))) This joint distribution turn corresponds to the joint distribution of the onput-output pairs of the function p(z,, y/t). This way of learning function's 's distribution is different from the typical regression objectives, which Iearn about a given function f by trying to approximate the predictive distribution p(y|zz)
By assuming the universal function approximation capability of I(o.0), and the availability of infinitely many functions f ~ p(f) with fixed number of context points N cach, the model posterior learned by the optimal classifier corresponding to Equation (4) would be equal to the true posterior given by Bayes rule.
~ P(f) A KE {l...,K)
(8)
0 ~ P(O)S*) vk {} i i~K(
(9)
(10)
f= 8"
(11)
Ô ~ P(OIf)
(12)
Pi .) P(O Oli)p(t)
(13)
Σ, E, P(OU O(i)p(i)
OOO 1L
(14)
Σ, P(O), 01))P() ((O))))(())
P( (()
(15)
Σ,
ROOL MO)MOO
RIO .1)) p(j)
The posterior probability for a function f is proportional to the class-conditions probability density function p(O't Ôli), which shows the probability of observing the pair (O', Ô) from function fe. The optimal classifier would then be proportional to the density ratio given below
exp(simn (0,))) p(O.)p(()
(16)
Similar analysis has been shown by the (Oord al.) 2018 for showing the mutual information perspective associated with self-supervised contrastive objective (infoNCE). The joint distribution over the pair observations correspond to the distribution of the underlying function f Thus, given some observation of a function, an optimal classifier would attempt at estimating the true density of the underlying function.
E SCENES' DATASETS
MPI3D Dataset. The MPI3D dataset (Gondal et al., 2019) is introduced to study transfer in unsupervised representations learning algorithms. The dataset comes in three different formats, varying in the levels of realism i.e. real-world, simulated-realistie and simulated-toy. Each dataset contains 1, 036, 800 images of robotic manipulator each, encompassing seven different factors of variations i.e., object colors (6 values), object shapes (6 values), object sizes (2 values), camera heights (3 values), background colors (3 values), rotation along first degree of freedom ((40 values)) and second degree of freedom ((40 values)). Thus, each image represents a unique combination of all the factors. See Figure 10 a) to sample of the dataset.
16

---

Under review as a conference paper at ICLR 2021
70
230
200
200T
(b)
Figure 10: Datasets for Scenes Representation Learning (a) MPI3D Gondal al. 2019) has three camera viewpoints, with images robotics arm manipulating an object. (b) RLScenes has 36 possible camera viewpoints for capturing arena consisting of robot finger and object.
In this work, consider the real-world version of the dataset. The multi-view setting is formulated by considering the images of a scene captured by three different cameras, placed at different heights. This effectively gives us 345, 600 scenes with three views each. We split the dataset into training and validation chunks, where the training dataset contains 310, 000 scenes and the validation dataset contains the rest 35, 600 scenes, approximately 10% of the dataset.
RLScenes. The RLScenes dataset generated in simulation using (Joshi et al.) 2020) for a single 3-DOF robotic manipulator in 3D environment. The dataset consists of 40, 288 scenes, cach scene parametrized by: object colors (one of 4), robot tip colours (one of 3), robot positions (uniformly sampled from the range of feasible joint values), and object positions (uniformly sampled within an arena bounded by high boundary as seen in Figure T0P. Each scene consists 36 views, corresponding to the uniformly distributed camera viewpoints along ring of fixed radius and fixed height, defined above the environment, As can be seen in Figure 10] the robot finger might not be visible completely in all the views, the object might be occluded in some view. The 36 views help by capturing a 360 deg holistic perspective of the environment. First a configuration
the above scene parameters is selected and displayed in the scene, then the camera is revolved along the ring to capture its multiple views. For learning the scene representations via both FCRL and GQN, we split the dataset into 35000 training and 5288 validation points.
F DETAILS OF EXPERIMENTS ON SCENE REPRESENTATION LEARNING
For learning the scene representations for both MPI3D dataset and RL Scenes, we used similar base encoder architecture. More specifically, adapted the "pool" architecture provided in GQN (Eslami et al., 2018), as it has been regarded to exhibit better view-invariance, factorization and compositional characteristies as per the original paper. We further augmented this architecture with nutch-oormalizttio. The architecture we use is as in Figure
9N
%
%
252
Hirk
%
Ostpel
Pesture See
54
155 32
25t 28
325
285
25
25
155
Figure 11: Network Architecture used for learning representations for the scenes' datasets.
F.1 LEARNING SCENES FOR MPI3D DATASET.
Since we have three view per each scene in MPI3D dataset, therefore restricted in defining the number of context points and the number of views. In all our experiments the number context points N is fixed three while the number of views is set to two. In contrast the experiments
17

---

Under review as a conference paper at ICLR 2021
for regression datasets where more datapoints per each view resulted in better representations, the restriction to a single image view in MPI3D dataset did not hurt the representation quality, measured only one
in terms of downstream performance. In the downstream experiments MPI3D, image to train and validate the probes. Due to the limitation of available views, we could not measure the effect of varying the number of views on the downstream performance.
Even though no explicit structural constraint was imposed for learning the representation. The FCRL algorithm implicitly figures out the commonality between the factors in different scenes. We visualize these latent clusterings in the Figure [2] We plot the 2D TSNE embeddings of the 128D representations inferred by the model. Thereafter, to visualize the structure corresponding to each factor, we only vary one factor and fix the rest of them except for first degree of freedom and second degree of freedom factors. A clear structure can be seen in the learned representations.
Implementation Details. We use the GQNs 'pool' architecture with batch normalization as encoder. As mentioned the ablation study, we did random sweep over the range of hyperparameters and selected the best performing model. Further details on the hyperparameters is provided in Table 2
F.2 LEARNING SCENES FOR RLSCENES DATASET.
To train the FCRL encoder, randomly sample the number of views from each scene to lie within the range [2, 20]: upper-bounded by 20 to restrict the maximum number of images per scene to be the same as that used in (Eslami al., 2018), and lower bounded by in case just the one view is not from a suitable angle. So, here the maximum number of context points N is 20. The number of subsets J
to 8. In the downstream reinforcement learning task, we use only one image train the policy network, as is the usual practice, and the same as (Eslami et al. 2018) 8). We kept the joint ranges from which joint positions are uniformly sampled to randomly reset the robot at the beginning of every episode while training the policy network to be the same as the ranges used for sampling the robot position while generating the dataset to train the FCRL encoder. These joint ranges are selected so to ensure that there more scenes in which the robot finger is visible. However, in order to not constrain the agent's s exploration, we let the action space for training the reaching agent to be less constrained, and be able to explore the entire range from -pi to pi. So, effectively, the space seen by the robot during the training of the representations is a subspace of that seen while inferring the representations from the environment used for this downstream reaching task. Interestingly, the inferred representations can also work effectively on unseen robot configurations as demonstrated by the success of the reacher.
Implementation Details. Similar to the encoder training for MPI3D scenes, learned the en- coder for RLScenes. However, since the downstream tasks is a reinforcement learning task, it was hard judge the quality of representations. Therefore, we took some insights from the MPI3D experiments and selected the model, trained with hyperparameters, which performed the best on the RL downstream tasks. Further details on the hyperparameters provided in Table
G DETAILS OF EXPERIMENTS ON ID FUNCTIONS
Implementation Details. We used the same encoder architecture for our method and the baselines (Garnelo et al. 2018a by in all experiments. For 1D and 2D functions, the data fed in the form of input-output pairs (T. y), where x and are 1D values. We use MLPs with two hidden layer to encode the representations of these inputs. The number of hidden units in each layer d = 50. All MLPs have relu non-linearities except the final layer, which has no non-linearity. Encoder: Input(2) - x (FC(50), ReLU)- + FC(5)).
While learning the representations of sinusoid functions with FCRL, we also scale the output scores with temperature to be 0.07. We used the following ppremmerreeter settings to train an encoder with FCRL..
Downstream Tasks. To learn the subsequent task-specific decoders on the representations, adapted the same data processing pipeline as above. For 1D functions, we train decoders for two different tasks: few-shot regression and few-shot parameter identification. The decoders for each task
18

---

Under review as a conference paper at ICLR 2021
Pursle Sea Green Salmin
grean
rec
Background Color
olive
Objects
100
(b)
15
Whish
Small
Cubc
Cylinder
10
Large
Hesagonal
Objects Shape
Pyramid
Sphere
-10
-20
-30
(d)
1st DOF
2nd DOF
(c)
Figure 12: 2D-TSNE projections of 128 dimensional representations learned by FCRL on the MPI3D dataset. Each individual plot exhibits the latent structure corresponding to the factor mentioned above. The plot for each factor is generated by varying the factor of interest and keeping the rest of the factors fixed, except the factors of the first degree of freedom and the second degree freedom.
19

---

Under review as a conference paper at ICLR 2021
Table 2: Hyperparameters Settings for Scene Representation Learning Experiments.
Parameter
Values
Parameter
Values
Batch size
64
Batch size
32
Representation dimension
128
Representation dimension
256
Temperature: 7
0.88
Temperature: 7
0.46
Number of subsets: JJ
Number of subsets: J
8
Max number of context points: N
Max number of context points:
20
Epochs
100
Epochs
100
Critic
Nonlinear
Critic
Nonlinear
Objective
NCE
Objective
NCE
Optimizer
Adam
Optimizer
Adam
Adam: betal
0.9
Adam: betal
0,9
Adam: beta2
0.999
Adam: beta2
0.999
Adam: epsilon
1e-8
Adam: epsilon
1e-8
Adam: learning rate
0.0005
Adam: learning rate
0.0005
Learning Rate Scheduler
Cosine
Learning Rate Scheduler
Cosine
Number of workers
10
Number of workers
10
Batch normalization
Yes
Batch normalization
Yes
(a) Hyperparameters to train FCRL based encoder the MPI3D Dataset.
(b) Hyperparameters to train FCRL based encoder on the RLScenes Dataset.
Table 3: Hyperparameters Settings for Sinusoid Experiments.
Parameter
Values
Parameter
Values
Batch size
256
Batch size
256
Latent space dimension
50
Epochs
30
Temperature: 7
0.07
Critic
Nonlinear
Number of subsets: J
Optimizer
Adam
Max number of context points: N Epochs
20
Adam: betal
0.9
30
Adam: beta2
0.999
Critic
Nonlinear Adam
Adam: epsilon
le-8
Optimizer
Adam: learning rate
0.001
Adam: betal
0.9
Learning Rate Scheduler
Cosine
Adam: beta2
0.999
Adam: epsilon
1e-8
Adam: learning rate
0.0003
(b) Hyperparameters to train FSR Decoder on FCRL learned representations.
Learning Rate Scheduler
Cosine
(a) Hyperparameters to train FCRL based encoder for
ID sinusoid functions.
20

---

Under review as a conference paper at ICLR 2021
Predictions
Ground Truth
Context
FCRL
25-
00
26-
CNP
25-
NP
Figure 13: Additional results on 5-shot sinusoid regression. Each column corresponds to a different sinusoid function where only 5 context points are given. The predictions of the decoder trained on FCRL based encoder are closer to the groundtruth.
2
FORL
0-
-2-
CNP
0-
2-
25-
NP
00
-2
Figure 14: Additional results on 20-shot sinusoid regression. Each column corresponds to a different sinusoid function where only 20 context points are given. The predictions of the decoder trained FCRL based encoder are comparable to CNP and better than NP.
are trained with the same training dataset as was used to train the encoders. The training procedure for both downstream tasks on sinusoid functions follows
For Few-Shor Regression (FSR), we use an MLP architecture with two hidden layers. The same architecture used in CNP (Garnelo al. (018a), however in CNP the decoder and encoder are trained jointly. All the baselines and our model are trained for the same number of iterations. We used slightly higher learning rate to train the decoder the training converges quite easily. FSR Decoder:
Input(50) + 2 x (FC(50), ReLU) FC(1).
For Few-Shot Parameter Identification (FSPI), train linear decoder without any activation layers on the representations learned via FCRL and the baseline methods. The decoder is trained for only epoch.
FSPI Decoder: Input(50) - FC(1).
Additional Results. In Figure 13 and Figure 14 we provide additional results on 5-shot regression on test sets and compare the results with CNP and NP. The curves generated by the decoder using FCRL learned representations are closer to the groundtruth. The difference evident in 5-shot experiments which supports the quantitative results in Table
21

---

Under review as a conference paper at ICLR 2021
4
2
7
3
GT
Context
2
3
9
7
CNP
FCRL
4
2
4
9
6
1
3
Figure 15: Additional results 50-shot mnist image completion. The context is shown in the second
row where target pixels are colored blue. Predictions made by decoder trained on FCRL based
encoder are slightly better than the CNP in terms of guessing the correct form of digits.
633 5
S
4
GT
Context
2
5
GNP
6
3
2
9
FCRL
Figure 16: Additional results on 200-shot mnist image completion. The context is shown in the second row where target pixels are colored blue. Predictions made by decoder trained on FCRL based encoder are comparable to CNP.
H DETAILS OF EXPERIMENTS ON 2D FUNCTIONS
Implementation Details. In this experiment, we treat MNIST images as 2D functions. We adapt the architectures of encoders and decoders from the previous ID experiments. However, due to the increased complexity of the function distributions we increase the number of hidden units of MLP to d = 128. Moreover, the input x is 2D as it corresponds to the cartesian coordinates of an image. The hyperparameter settings train FCRL based encoder on such 2D function is given in Table 4
H.1 DOWNSTREAM TASKS.
We consider two different downstream tasks for the representations learned on 2D functions: few-shot image completion and few-shot content classification. A separate decoder is trained for both of these tasks.
For Few-Shot Image Completion (FSIC), an MLP based decoder with two hidden layers. The decoder is trained on the same training data for the same number of iterations. Details are given in Table (b).
22

---

Under review as a conference paper at ICLR 2021
Table 4: Hyperparameters Settings for MNIST as 2D Functions Experiment.
Parameter
Values
Parameter
Values
Batch size
16
Batch size
16
Latent space dimension
128
Epochs
100
Temperature: 7
0.007
Critic
Nonlincar
Number of subsets: J
40
Optimizer
Adam
Max number of context points: N Epochs
200
Adam: betal
0.9
100
Adam: beta2
0.999
Critic
Nonlinear
Adam: epsilon
1e-8
Optimizer
Adam
Adam: learning rate
0.001
Adam: betal
0.9
Learning Rate Scheduler
Cosine
Adam: beta2
0.999
Adam: epsilon
1e-8
Adam: learning rate
0.0006
(b) Hyperparameters to train Few-Shot Im- age Completion (FSIC) Decoder trained on FCRL learned representations.
Learning Rate Scheduler
Cosine
(a) Hyperparameters train FCRL based encoder for 2D functions.
For Few-Shot Content Classification (FSCC), we train a linear regression on top of the representations obtained by both FCRL and the baselines. The decoder is trained for only one epoch.
we provide additional results for 50-shot and 200-shot image completion. We can see that the the results from FCRL based decoder consistently perform better than CNP in low-shot scenario of 50 context points. In 200-shot scenario, the results look comparable to CNP.
Additional Results. Figure 15 and Figure 16
I SOFTWARE PROGRAMS USED
The following software programs were used for the work done in this paper: PyTorch (Pazkke et al. 2019), Weights and Biases (Biewald (020), PyBullet (Coumans & Bail 2016-2019) Stable Baselines (Raffin et al. (0019). Gym (Brockman al. (016)) Jupyter Notebook (Kluyver et al.
2016
Matplotlib (Hunter) 2007). Seaborn (Waskom et al. (017)). Scikit-learn (Pedregosa et al.
201
NumPy (Harris et al. 2020), Python3
(Van Rossum & Drake 2009).
23


Reviews:
[{"review_id": "zkHX1k7kKC1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "As far as I can tell, this paper proposes to use contrastive learning to solve few-shot learning -- not just few-shot learning, but meta \"task-wise prediction\" as well (predict some properties of the task itself). The method learns representation of each task, by forcing random instantiations of the same tasks to have similar representations. The method tests on sine-function fitting + param prediction, image (MNIST) inpainting + classification, and (robot) scene understanding + RL (both are meta \"task-wise prediction\").\n\nPro:\n- Method seems reasonable and improves performance over compared methods (NP and CNP).\n- Highlights an interesting point of view (which I think is relatively novel -- it's glossed over in the NP paper) that treats location-value prediction for pixels in different images as different tasks to test meta-learning algorithms. This change of view encourages the analogy and idea of using e.g. self-supervised learning or inpainting methods for meta-learning.\n\nCon:\n- The goal and motivation of this paper is lost to me. The abstract and intro does not inform readers of the goal of the paper and how it can be used in applications (such as ImageNet meta-learning). It seems to be all of (1) few-shot learning, (2) extension of existing works NP & CNP, and (3) learning representations for down-stream tasks. But it seems to be suboptimal for all of these goals.\n  (1) If this is few-shot learning, then it would make sense to compare to other few-shot methods, e.g. ones using meta-learning.\n  (2) If this is an extension of NP and CNP, then this method has no grounding probabilistically. It cannot provide estimates such as P(x|y), for example, which is the point of these methods. It is not very interesting that non-generative methods can outperform NP/CNP which are generative models.\n  (3) If this is learning representations for down-stream tasks, the the down-stream-focus of the experiment setup violates the \"few-shot\" part since only the original formulation (e.g. image inpainting) is few-shot. For down-stream tasks (e.g. image classification) this is rather a \"sparse input\" problem. Despite the method being about learning representations, it does not compare to regular methods that treat images as images, not a set of (location,value) tuples. For example, a regular CNN may do better than the results reported in Figure 5.\n  (4) If this is a method that can do both few-shot and learning representations, it should be still compared to regular few-shot and representation learning methods, especially because there are so many papers that do few-shot and representation learning, independently or combined.\n- The motivation is quite similar to self-supervised learning, when considering down-stream applications where we treat each \"task\" as samples again.\n- The statistical significance is confusing. For example, Table 1 has 0.0087±0.0007 and 0.0078±0.0004 being significantly different with 3 runs, which does not sound plausible.\n\nOther minor issues:\n- The method assumes each task is different. If there are many duplicated tasks (e.g. very similar images) then making each task representation different from all others would be suboptimal.\n- In eq. (5), it seems the input for downstream tasks include the pixels/view-images, so it will be unclear if the knowledge is from X or $h_\\Phi(D^T)$.\n\nSummary:\nAlthough viewing location-value prediction for pixels in different images as different tasks for meta-learning is really interesting, the paper fails to motivate its goal, and by extension, designing convincing experimental comparisons. I will be reserving judgement and lean towards rejection.\n\nUpdate post-rebuttal:\nThe rebuttal clarified the motivation, but has yet to address the flaws that was associated with the choice of motivation and positioning among related work. Unfortunately that means I will not be changing my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "q3BfNybccin", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel training approach that uses a self-supervised signal to learn whether two sets of data are generated from the same function.  The goal is to align representations of different samples generated from the same function for use in few shot learning.\n\nBackground section should be expanded to provide a better understanding of the contrastive learning space.  In general section 2 needs to be improved.  The problem is not clear in this section as the main problem of few-shot learning is not clearly set up as an intuitive result of the approach of aligning representations of different samples that stem from the same function.\n\nThe writing in Section 3 needs to be improved.  Particularly the \"Encoder Training\" section as it is a little hard to follow.  There are K functions but the observations from the K functions are split into J sets of size N/J.  Is this just having N total observations that stem from K functions where positive pairs are pairs of observations from the same function and negative pairs are from different functions?  Or is this a method to sample a set of partial observations that are used in Equation 2 to generate the representation?  I am assuming the latter but this is not clear.\n\nIn Equation 4, when J > 2 then doesn't this become a multi-label problem where there are multiple pairs of observations from the same function?  I am confused on this equation in how it will prevent different positive pairs from competing with eachother.\n\nI am not sure what the main contribution here is.  The paper describes a method for using contrastive learning to align representations of observations from the same generator.  The learning approach is not novel but the applications are interesting and the idea of aligning representations of observations from the same generator is good.\n\nrecommendation and reasoning\n\nThis paper is slightly below the acceptance threshold.  The general approach is clear and the evaluations show good performance but the writing needs improvement and the novelty is lacking.\n\nI have read the author responses but my initial review remains unchanged.  The paper needs to be greatly improved in terms of clarity and the responses from the authors help but I think it is better off being resubmitted to another conference with heavy revisions. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "lufJOKVRvG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper works on the problem of few-shot learning. They propose to use contrastive learning to obtain good representations for different mapping functions. Experiments on 1-D, 2-D image and scene representation show certain advantages.\n\n[Strengths]\n- The idea of using contrastive learning to obtain good representations for different mapping functions is interesting and novel.\n- The writing is in general clear and there are also a lot of experimental results.\n\n[Weakness]\n- There is a strong assumption that the mapping function during the few-shot task is one of the functions during training. Is this practical?\n- The contrast learning only encourages those function representations to be different and it might not be able to learn the underlying function characteristics. Why can it help during down-stream few-shot tasks? In other words, I could use one-hot vector to represent each function. I could give the function index to the down-stream task. Is it helpful?\n- Although contrast learning is hot, why must use contrast learning to learn function representation? You know the function labels, so you could also use triplet loss during the representation learning.\n- Although the authors consider quite a few downstream tasks, all the comparisons are quite preliminary. It would be more convincing to show that in at least one task your approach can outperform SOTA or your function representation can improve SOTA.\n- Figure 3(a) shows a better result with 50 shots, but with 200 shots. Why? \n- There is no explanation for Fig. 2(b). It seems you use N samples to identify the function and the rest N-M samples to train the decoder. Is it correct? What's a good N here? What is the \"S\" operation?\n- Eq. (4) does not look correct.\n- Is Section 3.1 complete or not ? The organization is a bit wired.\n\n---\nUpdate:\n- I think the idea of this paper is very interesting. The authors' responses also address most of my concerns. The remaining issues are: 1) the writing needs to be improved to make the paper easy to read; 2) I am still not so convinced with the experiments. It would be more convincing if the authors can demonstrate this on few-shot image classification to improve SOTA, for which the joint (x,y) distribution might be too big at the image level. On the other hand, I am not so familiar with this topic. If such experiments are quite normal in this area, then I am fine to increase the rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VUCTVAbvmP_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to find a good representation of the underlying data generating function (data distribution) via contrastive learning. In contrast to existing works on applying contrastive learning to learn sample-wise representation, this idea is novel.\n\nPros:\nThe insight that \"two sets of examples of the same function should have similar latent representations, while the representations of different functions should be easily distinguishable\" is interesting. This can also relate to learning from a sets of tasks in meta-learning, where most tasks are related but some tasks are out of the distribution.\n\nCons:\n\nOn page 3, the illustration on training is like meta-learning. Is it? If so, please rewrite it using meta-learning notations. If not, please describe the difference with meta-learning. \n\nAlso, sentences under equation 2 use the terms \"projected representations\" and \"encoded representations\", what do they refer to? Can you use notations? \n\nThe paper provides a number of experiments on diverse data sets. These are valuable. However, as most of these tasks (especially Sec 4.2 and 4.3) are formulated into few-shot problems by authors, can you provide mathematical problem formulation at the beginning? It is hard to get why it is few-shot and what is the target. \n\nOverall, I find some clarifications are required. The current version is not clear enough to convey its importance. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the authors provided extensive responses to the reviewers and most of the reviewers did a good job of accounting for the author responses the final ratings for this paper was unanimously 5s -- all marginally below acceptance. The paper's positioning, writing were identified as key issues that remained to be addressed. The AC recommends rejection.
Decision: Reject