A Self-Supervised Framework for Function Learning and Extrapolation
Authors: Simon Segert
Jonathan Cohen
Venue: Accepted by TMLR ()

Abstract: Understanding how agents learn to generalize — and, in particular, to extrapolate — in
high-dimensional, naturalistic environments remains a challenge for both machine learning
and the study of biological agents. One approach to this has been the use of function
learning paradigms, which allow agents’ empirical patterns of generalization for smooth
scalar functions to be described precisely. However, to date, such work has not succeeded
in identifying mechanisms that acquire the kinds of general purpose representations over
which function learning can operate to exhibit the patterns of generalization observed in
human empirical studies. Here, we present a framework for how a learner may acquire
such representations, that then support generalization-and extrapolation in particular-in a few-shot fashion in the domain of scalar function learning. Taking inspiration from a classic theory of visual processing, we
construct a self-supervised encoder that implements the basic inductive bias of invariance
under topological distortions. We show the resulting representations outperform those from
other models for unsupervised time series learning in several downstream function learning
tasks, including extrapolation.

Published in Transactions on Machine Learning Research (07/2022)
A Self-Supervised Framework for Function Learning and Ex- trapolation
Simon N. Segert
sregert@princcetn.edd
Princeton Neuroscience Institute
Princeton University
Jonathan D. Cohen
jde@princeton.com edu
Princeton Neuroscience Insitute
Princeton University
Reviewed on OpenReview:
Abstract
Understanding how agents learn to generalize - and, in particular, to extrapolate in high-dimensional, naturalistic environments remains a challenge for both machine learning and the study of biological agents. One approach to this has been the use of function learning paradigms, which allow agents' empirical patterns of generalization for smooth scalar functions to be described precisely. However, to date, such work has not succeeded in identifying mechanisms that acquire the kinds of general purpose representations over which function learning can operate to exhibit the patterns of generalization observed in human empirical studies. Here, we present a framework for how a learner may acquire such representations, that then support generalization lew-shot fashion in the domain of scalar function learning. Taking inspiration from construct self-supervised encoder that implements
and extrapolation in particular in
classic theory of visual processing, the basic inductive bias of invariance under topological distortions. We show the resulting representations outperform those from other models for unsupervised time series learning in several downstream function learning tasks, including extrapolation.
1 Introduction
A key feature of an intelligent agent is the ability to recognize and extrapolate a variety of abstract patterns that commonly occur in the world. Here, we focus tractable but still highly general special case of such patterns, that take the form of one-dimensional smooth functions. From formal perspective, the space of all such functions is vast (Reed and Simon 1980). necessitating the use of inductive biases for making useful inferences. Thus, while this setting does not encompass all possible kinds of structures that can be generalized or extrapolated, it a sufficiently rich space that insights gained here are likely to shed light on the ability of biological and artificial agents to generalize more broadly. At the same time, the abstract structure of this space is relatively simple and well-understood, thus making it amenable to precisc analysis and interpretable experimental manipulations.
A further virtue of the space of functions is the existence of detailed experimental data from humans in this domain, thus facilitating a direct comparison of models with natural agents. Indeed, over the past few decades, the empirical studies of function learning in humans has catalogued the forms of several such commonly applied biases, including associative similarity, rule based categorization (McDaniel and Busemeyer 2005). bias towards positive lincar forms (Kwantes and Neal 2006) and compositional construction from small number of basis elements (Schulz et al. (017). Taken together, such results describe a class of "intuitive functions," " which functions that people appear readily able to recognize and
1

---

Published in Transactions on Machine Learning Research (07/2022)
While it is generally accepted that efficient generalization implies the existence of some previous expectations about the structure of the space of functions, what is not obvious is how such expectations are acquired; that is, what mechanisms are capable of learning and encoding abstract structure through unsupervised or self-supervised experience, in such a way that features relovant to any particular task may be casily "read out" as required. Here, we propose to address these challenges by adapting and extending the general framework of the field of self-supervised learning (Chen et al. 2020 He et al., 2020). The framework consists of two components: a "slow" encoder that learns general purpose representations of one-dimensional functions using standard self-super learning algorithm, and collection of "fast" heads, which can rapidly adapt to different function learning paradigms, based on a small amount of task-specific annotated data, using simple form of supervised learning (linear or logistic regression). The heads are trained on top of the representations learned previously by the encoder, allowing the model to make use of its general knowledge to rapidly adapt. to the particular task demands.
While efforts have taken this general approach (Chen et al. 2020 He et al. 2020 , none to our knowledge
have specifically considered the domain of function learning with intuitive functions that is, ones that people have been empirically observed to use (DeLosh et al. 1997 McDaniel and Busemeyer 2005 Schulz al. Our approach is further distinguished in the design of the encoder used for self-superviser learning. For this, we treat a scalar function as a (typically very short) time series. The crucial feature of our encoder is a novel family of augmentations of time series, derived from the theory and phenomenology of topological visual processing (Zeeman 1965 Chen 2005). This theory holds that the visual system invariant to certain kinds of local topological distortions of stimuli, distortions that we design our augmentations to mimic. We hypothesize. that such distortions reflect commonly occurring structure in the world, that may in turn have been discovered by the brain, cither through evolution or carly development, and used as basis for generalization. Following this idea, we train sell-supervised objective that tries to enforce invariance across these augmentations, adapting the framework of Chen et al. (2020).
2017
We demonstrate that our choice of encoder and training procodure learns representations that perform better on a collection of downstream function learning and generalizaton tasks than do comparison models for learning and/or representing time series. This should be of particular interest to the field of semi-supervised learning, since works in that field have not yet systematically analyzed time scries that correspond to intuitive functions. Moreover, we directly compare the generalization patterns of the model with those of humans asked to perform a multiple-choice extrapolation paradigm modeled after an empirical study by Schulz et al. 2017). We find that the model exhibits a qualitatively similar bias as people in this setting, namely, a greater accuracy on functions that are compositionally structured. This should also be of interest to psychologists, since it suggests that behavioral biases function learning may arise as consequences of more general representation-leariigg procedure.
2
Background
2.1
Contrastive Learning
Here we provide a brief summary of the elements of contrastive lcarning that are necessary to define our encoder. This is adapted from Chen et. al. (2020) and van den Oord el. al. (2018). The basic assumption is that we are provided with set of positive pairs (I;, -,),==1.--. N, which are taken inputs that we wish to consider similar to each other. All other pairs of inputs are considered as negative pairs, which the objective will attempt to push apart in the latent space. For convenience, we will treat the input as a single flattened dataset of size 2N, in which the positive pairs are those of the form (wi, ui+ & N. Let fo : 1 + R and 9s : =R= + 5m be two parametric families of functions (e.g. neural networks). Here mi the dimensionnlity of the inputs vi, while ni and ny are arbitrary, and 5ma-1 denotes the hypersphere consisting of all vectors in _ of unit norm. The objective is
2N
max 8,0
< (go o fo)( (go fo)(H++)) > -LSE (go o fo)( (gs fo)(u,)
(1)
2

---

Published in Transactions on Machine Learning Research (07 (202)
Here, 0 is a hyperparameter, and LSE denotes the logsumexp function LSET T log E, cault. The brackets < ; are the Euclidean dot product. After optimizing this objective, we discard the function and take the encoder to be the function f.
Informally, the first term of the objective function acts to push positive pairs together in the latent space, since is maximized when both elements the pair have equal representations. Conversely, the second term monotonically increasing
acts to push apart all other pairs of inputs. This is because the logsumexp function of each of its imputs; therefore it will be minimized when all of the pairwise similarities
small
possible. A more precise analysis of properties of this objective may be found in Wang and Isola (020
2.2 A Generative model of Intuitive Functions
To define a generative model for reference curves that plausibly resemble the distribution encountered by people, we adapt the generative process proposed Schulz et al. (2017). This generative model uses the
formalism of Gaussian processes Rasmussen and Williams 2006); we provide further general background in the Appendix.
Schulz el al. (0177) define the Compositional Graminr by starting from three basic Gaussian Process kernels:
Kiinear (I, y) (r-0:) (yy (1)
Krdf (I, y) = Oge (a-w)
K Kperiodte (I. y) Oge (212(2)00=11))//0
where 0g are hyperparameters. addition, the authors include in the CG ten kernels which are defined using pointwise sums and products of these above three. We refer to the Appendix for a more detailed description.
A natural point of comparison is the Spectral Mixture (SM) kernel (Wilson and Adams 2013), which is a flexible non-parametric kernel family defined by the formula
K Kiic (x, v)
Σ
wae 2n (xr-)))) cos(2 -y)))
Schulz al. (2017) demonstrated that people learn curves generated from the CG more easily than ones generated from the SM Therefore the family of kernels in the CG are good candidates for generating curves that are both naturalistic and are easily recognized by people.
Lastly, it is important to note that, due to the nature of continuous space, in practice it is necessary to represent functions by their values on a finite set Iι <... < xx of ordered sample points. In our case, we take the points to be evenly-spaced, and use the same set of points for every function. Thus any function {(r,, yt)) may be treated as a time scries and vice versa In what follows we will use the terms "curve," " "function" and "time series" interchangeably, with the understanding that the points xę remain fixed across all functions. Also, since the positions of the 2,'s are the same for all functions, omit them from explicit notation, and use y to denote the vector with components { that defines a function.
3 A Contrastive Encoder for Intuitive Functions
To define the encoder, following Section 2.1 we need to specify the architecture and the family of positive pairs. For the encoder architecture, we simply take f to be a foodforward network of several 1D convolutions, and g to be an MLP with a single hidden layer. We set 712 = n1 128 and 7 .5. For the class of augmentations,
take inspiration from the field of topological visual perception (Chen 2005 Zeeman 1965), which posits
that the visual system maintains invariance to local topological distortions (or "tolerances") of stimuli in order to facilitate global processing.
In function learning, the x-axis does not necessarily correspond to time. The same true of "time series," " despite the name: it is merely an ordered list of numbers.
3

---

Published in Transactions on Machine Learning Research (07 (07/2022)
In our case, consider 1-dimensional functions rather than 2-dimensional images, but similar principles apply. We propose lamily of transformations that implements localized topological distortions to the function, together with several basic global distortions: (1) random vertical reflection, (2) random jittered upsampling and (3) random rescaling. We denote these respectively by stochastic transformations Tı, T2, Ts, which we describe more detail below.
The first transformation, that accommodates vertical reflection, defined by (iyy with 50 percent probability and Ti (ÿ) elaborate. To evaluate (y)), we first select random interval [a, [πı, We then randomly select points 2 <... < o in [a, bj. These points are not required to be uniformly spaced. We generate them by sampling uniformly and independently at random from (a, b] and then sorting the samples, and then define (c) m²)²
otherwise. The second, that accommodates horizontal bending, is the most
T2),; -, E, Cie 20 Yj where 1/C, E,
2n In other words, the values of (v) are given by The effect is three-fold: since the points
Gaussian Kernel Density Estimator (KDE) at the points xt lie in a proper sub-interval of (a. b), this crops a portion of the function and up-samples to the original resolution. Secondly, bocause the points af are not evenly spacod, some inhomogencous horizontal stretching or coutraction introduced. Thirdly, the nature of the Gaussian KDE means that the augmented functions are smoothed with respect to the originals. Finally, we apply Σ3 that accommodates vertical rescaling, For this, choose random interval (a, b) [o, 1] and then apply affine transformation such that the maximum value of the new function is b and the minimum a. More explicitly, (v)s);
11-min;22 Vi + a. a) max, min, 2; Since this is applied last, the resulting functions always take values in the interval [o, 1]. Therefore the positive pairs lake the form (T3T2Ti (y), TyT2Ti (v)) where y is a function. We reiterate that T; are stochastic transformations, so despite the notational appearance, the two functions comprising a given positive pair will not be equal, since they are generated using two separate evaluations of stochastic transformation. In our experiments the function y itself is generated from the Compositional Grammar over intuitive functions described in Section 2.2 An illustration of the augmentations provided in Figure 1.Furthermore, we provide ablation studies the effect of each augmentation individually in the Appendix.
only
transforms
all trtansffrms
W
0.2
50
100
0.5
300
50
100
025
30
100
54
300
130
50
100
Figure 1: Illssratioon of the augmentations. Each plot consists of two functions comprising posilive pair. In the four plots on the left, only the horizontal stretch transformation T/ is applied. In the four plots on the right, all three transformations are applied.
4

---

Published in Transactions on Machine Learning Research (07/2022)
4 Data description and Encoder training
To evaluate the ability of the encoder to learn a representation of intuitive functions, we generated and trained it on two types of functions: one generated from the family of 13 kernels defined by the CG (see Section 2.2 and the other (used as control) from non-compositional SM Kernel, for a total of 14 kernels. As noted above, Schulz et al. (0177) showed that human completions are closer to those generated by the CG than by the SM. We included the SM in our generative distribution to allow a similar comparison. Each function was generated by first sampling one of these 14 kernels, then sampling any hyperparameters of that kernel, and finally sampling from the resulting covariance matrix evaluated on T = 100 evenly horizontally spaced points. We sampled from the SM kernel 50 percent of the time, and each of the remaining 13 CG kernels 3.85(= 50/13) percent of the time. Therefore, any differences in the representations between the SM and the CG cannot be ascribod to data availability. We normalized all functions to lie in the interval [0, 1]. All encoders were trained using a batch size of 512, with an Adam oplimizer with learning rate of .001 and weight decay of 10-6. All encoders were exposed to 500,000 curves during training. We trained three copies of cach encoder using random initializations and averaged results over these copics.
As described above, our model consists of a combination of a rastivevive loss function, and a convolutional encoder architecture. In addition, we considered eight comparison models, six of which are encoding models trained using different objectives than the contrastive loss, and two of which are architectural ablations trained using the same contrastive loss, but with non-convolutional encoder architectures. Of the first six, four were unsupervised time series models: Triplet Loss (tloss) (Jean- Yves et al. 2019). Temporal Neighborhood coding (tnc) (Tonekaboni et al. 2021). Contrastive Predictive Coding (cpc) van den Oord et al. 2018 and
Conditional Neural Processes (cnp) Gaarnelo et al. 2018a). We also lested a Variational Autoencoder (vae)
(Knggma and Welling 2014) as an example of an unsupervised algorithm that has been successful in other domains, but does not exploit any structure particular to time scries. Finally, we inchuded a bascline encoder ("raw") that simply copies the raw input. To control for the latent space capacity, all encoders except for the baseline had representations of equal dimensionality (128).
The first of the two architectural ablations consisted of replacing the convolutional encoder with multi-layer perceptron. We denole this by "contrastive-nlpp" In the second, we replaced the convolutional encoder with the same permutation-invariaa encoder as was used in the CNP model, which we denote by "contrastive- perm-inv". In this encoder, the representation of the function {(z, b:), takes the form n - M ML yi) Garnelo et al., 2018a b). This encoder thus permutation invariant in the sense that the representation of the function does not depend on the ordering of its constituent x-y observations. For both of these ablations, the architectures were constrained to have approximately the same number of parameters as the convolutional encoder. Further implementational details of all comparison models are provided in the Appendix.
5 Results on Downstream Classification and Extrapolation tasks
To evaluate the quality of the Icarned reprosentations, we adapted three function learning paradigns that are either directly translated from or inspired by paradigms from studies of human performance: (1) kernel classification, (2) multiple choice extrapolation, and (3) freeform extrapolation. The first one corresponds directly to the standard paradigm for unsupervised learning evaluation in computer vision (Chen et al.) 2020), in which an unsupervised algorithm is trained on a dataset for which ground truth annotations are available, and then a supervised classifier such as a logistic regression is fit on top of the frozen representations. Although to our knowledge this has not been used dircctly in the analysis of empirical results concerning human function learning, it may be regarded as an abstraction of the experiments from Leon- Vill-rillagaa and Lucas 2019), which showed that people's completions depend on their judgements about the category to which a function belongs, suggesting that people make use of categories when judging functions. The two extrapolation tasks are drawn directly from Schulz et al. 2017 A version of the third task also appears in Wilson et al. (015), however using a different generative process for the probe curves.
For cach task, we define a head that transforms the encoder representations to a task-specific output, and train the head on small amount of labeled data. In all cases when training the heads, the weights of the encoder are frozen.
5

---

Published in Transactions on Machine Learning Research (07/2022)
Table 1: Accuracy on the categorization task, as a function of the number of training examples per category. Chance performance is 7.14 pereent
3
10
30
100
300
contrastive
40.951 1.83
55.27 1.41
64.81 1.80
72.00
1.52
76.23
1.31
enp
15.07+ 1.28
19.25+
1.35
22.69 0.96
26.19
0.95
28.10+ 0.73
epc
25.061 1.29
35.45 1.57
46.381 1.14
54.941 1.18
58.481 1.04
raw
11.86# 1.47
15.54 2.02
14.27# 1.96
15.73# 1.51
14.25# 1.64
I-loss
30.414: 1.73
41.31 1.89
52.16+ 1.20
59.78+ 1.24
63.35 1.19
tnc 23.15:t: 1.14
31.22+ 1.09
38.55 1.23
44.85+ 1.08
49.16 1.20
vae
9.27 1.13
12.77 1.65
21.51 1.89
29.17 1.56
33.74 1.50
contrastive- -etessttive-eee-inn perm-inv
16,24 1.48
22.89+ 1.43
27.09 0.88
29.37 0.70
31.75# 0.87
contrastivo-mlp
33.58 2.05
46.08+ 1.37
53.96+ 0.90
57.79+ 0.67
60.30 0.65
In addition, we note that several of the tasks required that we compute the posterior mean with respect to Gaussian kernel in order to construct the training data, which required that we initially fit the kernel hyperparameters. For example, in the multiple choice task, to construct two candidate completions we took the posterior mean of the prompt curve with respect to both the SM and the CG kernels, which required we first fit the hyperparameters for those kernels. The fitting of GP kernel hyperparameters is known to suffer from under-fitting and instability problems (see Wilson et al. 2015 including the supplementary material). To address this, we fixed the hyperparameter values of each kernel class, and evaluated the downstream performance of all encoders on curves generated with this fixed set of hyperparameters. We repeated this times using different random choices of hyperparameters each time, and averaged the results. This ensured that the hyperparameter values were correctly specified within each task, and that results were not influenced by the imperfections of any particular hyperparameter optimization procedure.
We trained three copies of each of the six encoders (contrastive encoder and five comparisons) using different initializations, and all reported results were averaged over the 3 copies and 10 hyperparameter choices, for a total of 30 measurements. The error bars are 95 percent confidence intervals of the standard error of the mean over those measurements.
5.1 Kernel classification
Here, the task to predict which of the 14 kernels used to generate given function. The head simply lincar layer + softmax, the outputs of which were interpreted as the probabilities of cach class. Thus it is equivalent to 14-way logistic regression on the encoder representations. In all cases, we fit the head using the SGDClassifier class from scikit-learn. Additionally, we separately chose an L2 penalty for each head using cross validation. We report the accuracy of each such classifier on a collection of 2800 held-out curves (200 per class). As shown in Table [ the contrastive encoder is able to attain approximately 55 percent accuracy using only 10 labeled examples per class, which improves to approximately 75 percent when using 300 examples per class, improving upon the second-best model (t-loss) by around 10 percentage points.
5.2 Multiple Choice Extrapolation
In the multiple choice completion paradigm, the models were presented with a prompt curve y E 1880 as well as several candidate completions curves y e 100 with the properties that y/ = yy for j <= 80 and were required to select the correct completion. Following Schulz et. al., we constructed the candidate completion curves by computing the posterior mean with respect either to the SM kernel, or the best-fitting CG kernel, with the correct answer corresponding to which of these two kernels used to generate the prompt curve.

---

Published in Transactions on Machine Learning Research (07 2022))
1/1
M
MM
Figure 2: example multiple choice completion problem. The prompt curve on the left. The compositional completion is in the middle and the mixture completion is the right. In this case, the correct answer the compositional completion. The coloring of the candidate curves is for visual aid only.
Table 2:
Performance on the Multiple Choice completion task, as function of the number of training examples per category. Chance performance is 50 percent.
3
10
30
100
300
contrastive
67.74 2.48
73.68
1.69
78.09
1.47
79.07
1.75
80.90
1.93
enp epc
59.30 2.35
59.801 2.35
59.72+
2.10
60.04 2.14
61.561
2.30
62.80 2.60
67.55
2.26
70.55
1.93
71.58
1.97
74.48 1.81
raw
51.644 2.83
54.80 2.13
54.28+ 2.20
54.68+
1.85
56.964 2.22
t-loss
60.18 1.44
62.741 1.62
65.121 1.57
65.31 1.55
67.761 1.73
tnc
58.381 2.46
63.65 1.89
68.91 1.64
70.05 1.68
72.251 1.99
vae
52.48: 0.74
53.03 0.83
53.324 0.85
53.57 0.84
55.00 1.29
contrastive-perr-inn
60.38+ 2.70
60.634
2.67
60.68+ 2.68
62.30+ 2.58
63.32+ 2.54
contrastive-mlp
67.051 3.00
67.494 2.56
70.00 2.85
71.71 2.50
72.79 2.24
The training data for the head consisted of 50 percent prompt curves sampled from the SM and 50 percent curves sampled from the CG.
Since this task required comparing the prompt curve to cach of the candidate curves, we used quadratic decision rule for the head. Let by denote the encoder representation of the prompt (upsampled to 100 points prior to being fed into the encoder), and h4 > 0 denote the representations the choices. The head linearly projected these vectors into a lower-dimensional space, and chose between the alternatives using a dot product this space. That is, we fit a model of the form Ps o wlwh,,whh) where w is a linear projection from the encoder space to 32 and Pi, 1,2 the choice probabilities. All heads were trained on cross-entropy loss using the Adam optimizer with learning rate of .01. We report the accuracy on collection of 400 held-out curves (200 per class). In this case, see from Table 2 an improvement in accuracy of approximately 5 to 10 percent compared with the second best model. Interestingly, the rank ordering of the models also differs compared to the categorization task: here epc and tne both outperform t-loss, and the vae generally matches performance of the raw encoder baseline.
5.3 Freeform extrapolation
In this lask, for given function y R2000 the model was presented with initial portion y1:80 and required to make prediction E 20 that extended it for fixed sized window (length 20). Performance measured by Sūm 100), for some choice Sim of similarity function. It has been argued that, due to its high-dimensional and underconstrained nature, this task provides a more rigorous test of extrapolation than do discrete categorization tasks and that, in an empirical setting, it may provide finer-graindd insights into peoples' inductive biases DDLossh et al. 1997 However, may be unreasonable to expect that an algorithm
More explicitly, generate the CG completion, computed the likelihood of the prompt with respect to each of the CG kernela, and then took the posterior mean of the kernel that attained the highest likelihond, mimicing the procedure Schulz al.

---

Published in Transactions on Machine Learning Research (07/2022)
trained without any predictive experience can exhibit a reasonable ability to perform free-form extrapolation. Here, we tested the hypothesis that this capacity can arise from modest amounts of supervised (predictive) training based on categorization judgements among function representations acquired in a self-supervised manner from the contrastive learning mechanism described above. To test this, we implemented a simple form of curriculum learning.
In the first phase of the curriculum, we trained a logistic regressor on the encoder features learned during unsupervised training, to predict the generative kernel of input function, exactly in the categorization task from Section 5.1 Here we presented the regressors with 300 functions and corresponding category annotations from cach kernel. In the second phase of the curriculum, we present a small number of functions y* with no label annotations. We then fit a simple class-dependent forecasting model of the form:
5 w ct + WO + Σ (w + wy)vi
(2)
where & E (1,2,..., 14) denotes the kernel class of the function ye predicted by the logistic regressor. Here L is a hyperparameter that controls the autoregressive time lag. We set L= 20 in all cases, The parameters JusJJssss JosisLısmmu are weights that are fit using least-squares, When given function y to extrapolate at test, we first estimated the class & E (1,2,.... 14] of y using the logistic regressor and encoder features. We then forecast it using the autoregression weights {w, + w Jugisss.
We compared the results of this procedure with two controls. The first was an Ideal Observer model that was given access to the true underlying Gaussian kernel used to generate each function, information to which the other models were not privy. This model forocast given function by computing the posterior mcan with respect to the kernel on which it was trained. Since this model used Bayesian inference on the exact underlying distribution over functions, it represented the best performance that any model could attain. We refer to this as the "GPIO" (Gaussian process Ideal Obsorver) model. The second control was a simple auloregression model, that removed the calegorization slep in order lo evaluate ils contribution to the forecast. quality. It used an unconditional forecasting model of the form u ~ wp + Σソ= wyy"", that ignored any category structure. The autoregression model was trained on exactly the same number of functions as the other forecasting models (with the number of curves used to train the logistic regressor included in this count..
We evaluated the extrapolation performance of each model using the Pearson correlation coefficient and L2 distance (see the Appendix for results of L2 distance) between the actual and predicted curves. We report the average values for 4200 held-out curves (300 per class). The results, shown in Table g are similar to those for the catcgorization task. All models substantially outperformed the autoregression bascline, indicating that even imperfect category information is helpful for extrapolation. The contrastive model performed better than any other model except the GPIO model. Several example extrapolations from the contrastive model are shown in Figure
6 Comparison with human data
The multiple choice completion task from Section 5.2 was modeled after Experiment in Schulz et al. 2017 An intriguing result of that experiment was that people were more likely to select the CG completion than the SM completion. We asked whether any of the models shares this property. To do this, we measured the difference accuracy when the prompt curve was sampled from the CG compared to when it was sampled from the SM. More precisely, let, {ve} denote collection of prompt curves, {yea) the corresponding completions generated by the CG, and {y'M J the completions generated by the SM. Furthermore, define 2 to be a binary variable that indicates whether y was sampled from the CG or from the SM. In our design, half of the prompt curves were sampled from the CG, meaning that ž; assumes each of the two values with 50 percent probability. For a given model, the choice probabilities ((roo p'MM) are given as in Section 5.2. Then the accuracy difference defined by
Ance B, CG) -B.
=M SM)
(3)
8

---

Published in Transactions on Machine Learning Research (07 (07/202) 2022)
10
00000000000
34-
Gho
predicus
53-
02-
30
53
107
100
MM
28-
5.-
10c
95
202
100
1.6
Freeform completions generated by the contrastive model, using the maximal amount of training data. The GPIO completions also shown for comparison.
Figure 3:
Table 3: Results on the freeform task, as function of the number of samples per category used to train the regression (not counting the functions used to train the kernel classifier). Values are the Pearson correlation of the predicted to the true completion. The value for the GPTO model is 83.70+ 1.78.
10
30
100
autoregression
18.48 4.46
18.484 4.48
18.474 4.52
18.504: 4.47
18.664 4.47
contrastive
30.15# 2.36
49.05 2.00
60.781 1.59
63.34 1.39
63.91 1.43
enp
24.29# 2.07
33.124 1.98
40.91 1.44
43.60 1.33
45.37 1.29
cpc
25.13# 2.83
42.32 2.61
51.04 2.39
53.06# 1.92
54.791 1.92
19.89 4.20
23.134 4.81
27.02+ 5.82
28.184 5.81
29.141 5.89
t-loss
29.45# 3.28
44.48 2.74
54.624 2.00
56.87 1.93
58.101 1.95
tnc
26.72 3.22
39.11 2.84
48.30 1.81
51.52+ 1.60
52.78 1.74
vae
27.36# 3.09
33.51 2.53
37.18 2.67
39.27 2.51
40.89 2.39
cnuurastive-eerm-iin
22.51 2.65
32.33 2.10
39.53+ 1.52
42.57+ 1.49
44.04 1.33
contrastive-mil
26.721 2.21
45.45.1 2.41
56.191 1.55
58.47 1.65
59.64 1.35
A short calculation shows that Ance is directly related to the model's propensity to favor the CG completion over the SM completion:
1 (Pcgl= = SM)
Ex(Poa)
E. CG)
(4)
2
2 E(Pcalz CG) , E,(psmnle = SM))
(5)
+ 2 acc
(6)
2
In other words, Dace is positive exactly when the CG completion is chosen more often than the SM completion. Note that an unbiased model would have (,(ra) Pi(#: CG) = and thus Apec 0. Note also that,

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Table 4: Value of Pace on the multiple choice task, as a function of number of labeled training samples per category, The corresponding value for people approximately 39, We do not bold the highest number because, unlike in the other tables, the values here do not correspond to normative performance metric.
3
10
30
100
300
contrastive cnp cpc raw t-loss
19.82+ 5.88
21.49+ 5.26
20.53+ 4.87
18.57+ 4.84
19.39+ 4.81
0.73# 5.39
5.25+ 3.01
6.61 4.35
8.17#
3.00
9.61# 3.32
-3.924 7.59
1.50 5.50
4.734 5.60
6.384 4.43
10.334 4.02
5.19 5.36
2.25= 3.79
3.87
5.09 1.81
7.92 2.61
0.57 5.04
5.46 4.37
8.24 3.95
10.30 3.44
12.72 3.75
tnc vae
-3.23 6.13
4.70 4.07
6.42+ 2.81
8.18 3.60
9.55= 3.97
2.01 1.34
2.62+ 1.25
3.12+ 1.42
3.30 1.41
4.19+ 1.79
contrastive-perm-int
1.95 5.01
4.60= 2.88
5.98 4.05
8.17# 3.20
9.74 3.55
contrastivo-mlp
16.06 3.74
16.56 3.35
16.14 4.01
16.48 4.30
18.324 3.84
described in Section 4, all models were trained on an equal proportion of curves from the SM and CG, so any resulting bias cannot be due to differential data availability between the two classes of curves.
We see from Table 4 that all models had at least weak form of the CG bias, in that they attained higher accuracy the multiple choice task when the prompt sampled from the CG. The contrastive model had the highest value of this bias, albeit the error bars overlap with the values for t-loss and contrastive-mlp.
Crucially, however, even the baseline "raw" model showed a significant positive bias, thus indicating that the observed biases may be due to statistical properties of the curves themselves, independent of the properties of the learned representations of the models. The contrastive and contrastive-mlp were the only models that attained Aace value significantly higher than that of raw, thus indicating that the observed bias for these models is partially due to the properties of their representations.
However, all of these biases are smaller quantitatively than reported in Schulz et al. (217)). There it was found that people attain an accuracy of 32 percent when the prompt curve is from the SM (that is, they choose the CG completion 68 percent of the time), while they attain an accuracy of at least 71 percent when the prompt curve is from the CG. We say *al least", because in that experiment, there were act uallally three choices presented to the participants in the CG case: the CG completion, the SM completion, and an additional distractor completion. Thus can estimate that, for people, the accuracy difference is given by
A people R0C
2 39
7
Related Work
7.1 Contrastive Learning and Time Series Representation Learning
The idea of learning representations by maximizing an information theoretic criterion can be traced at least back to Linsker's InfoMAX [Linsker 1988) principle, in which it was shown that certain properties of neurons in visual cortex could be replicated by training the encoder to maximize mutual information between the input: and the encoder representation. This principle was subsequently extended to the problem of unsupervised deconvolution of time series (Bell and Sejnowski, 1995) by extraction of independent components. A network that learns by instead trying to maximize representational similarity between two different parts of the same input, presaging the modern approach to contrastive learning, was introduced by Becker and Hinton (1922). In a similar spirit, the BCM learning rule (Bienenstock et al., 1982), introduced as a model of synaplic plasticity in the visual cortex, can be shown to be equivalent to projecting the data onto subspaces that are "maximally discriminative" (Intrator and Cooper 1992), and thus, most likely to be useful for downstream classification tasks. Rather than trying optimize the mutual information directly, however, most modern
implementations of this idea use a form of the InfoNCE loss, introduced by van den Oord et al. 2018 There
10

---

Published in Transactions on Machine Learning Research (07/2022)
it was shown that this objective is tractable lower bound to the mutual information criterion, which can be difficult to estimate directly. This loss is also strongly reminiscent of the older technique of Contrastive Hebbian learning (Hinton 1989), insofar as both involve computing average network activations over a set of "positive pairs" of inputs as well as over a set of "negative pairs", and try to maximize the difference between the two averages. The authors incorporated this objective their Contrastive Predictive Coding model (CPC), in which a recurrent encoder trained to predict its own future outputs. This basic loss function has been adapted and modified in several ways. In Chen et al. (2020) it is used in tandem with siamese network architecture, as described in Section 2.1 while Aberdam et al. (2020) extends this setup to a seq-to-seq objective and Li et al.] (2020) integrates the contrastive objective with reconstructive one. A similar contrastive objective is used in He et al. (2020), except with a memory bank used to sample negative examples, with this approach exlended to videos in Pan el al. (2021). The approach of Hyvarinen
and Morioka (0166) is also very similar in spirit, in which the idea is to learn temporal features of time series that differ across different time windows.
Although some of the works above deal with sequential data, they tend to be high-dimensional (videos (Pan et al. 202Ij, image patches (Aberdam et al.) 2020)) or data that is otherwise not directly interpretable by humans (andio (van den Oord et al. 2018), radio frequency signals (Li ct al. 2020 and do not consistently yield lower dimensional, readily interpretable, and easily composable functions of the form studied here. Fewer works have considered whether and how representation learning of such simple functions, such as the I-dimensional time series of the sort used in function learning experiments and studied here-this omission is notable since, despite their simplicity, such functions occur in a wide range of naturalistic settings (Dvenauu]
et al.
2013 Two particularly notable models of unsupervised time series learning are Triplet loss Jean-Yves
ct al. (Mikolov et al. 2013), and relies predicting the representation of "word" (here short window of the time series) from the representation of its "context" (here a longer window containing the "word"). In TNC, the timeseries is divided into disjoint segments. The encoder is jointly Iearned alongside a discriminator, in such a way that the discriminator is able to tell the difference between distant and proximal observations. In both TNC and Triplet loss, a recurrent encoder is used. An alternative approach to unsupervised learning of ID time series learning is through autoencoders. A popular choice here is a seq2seq architecture with reconstruction loss (Amiriparian et al. 2017: Lyu et al., 2018 Malhotra ct al.] 2017). Ma et al. (2019) augmented this setup with k-means objective to encourage clustering in the latent space. Compared to our approach, these involve considerably more complexity, through the use of an additional decoding step, as well as more intricate seq2seq architectures.
2019 and Temporal Neighborhood Coding (Tonckaboni et al. 2021 The first is inspired by word2vec
7.2 Function Learning and Gaussian Processes
The dominant framework for modcling of human function learning uses Gaussian processos, a statistical model that specifies a probability distribution over the infinite-dimennsioaal space of functions and allows for tractable inference procedures (Rasmussen and Williams 2006). Lucas et al. (2015) used Gaussian processes to capture a wide range of empirical function learning phenomena, while Wilson et al. (015 and Schulz et al. (217)) proposed specific families of kernels to model human extrapolation judgements. A limitation of the basic GP framework is its dependence on a choice of specific kernels or kernel families. Our approach sought to address this dependence through the use of unsupervised learning. Other approaches have taken a similar tack. The Spectral Mixture Kernel (Wilson and Adams 2013) and Variational GP (Tran et al. 2016 do so by introducing nonparametric families of kernels that can approximate arbitrary kernels to an arbitrary level of precision. Duvenaud et al. (2013) implement a similar idea, except by building up a family of kernels using operations of small number of atomic kernels, and performing a search over the resulting combinatorial space. Sun et al. (2018) perform a similar search except using a continuous relaxation and neural network. In Hinton and Salakhutdinov (2007), an appropriate kernel is found by fitting a Boltzmann machine. Neural Processes (aarncoo et al.) 2018b) go further and replace the Gaussian kernel with more ficxible paramctric family of distributions that can be learned using a neural network. This approach naturally exlends to et al.] 2018a Kim et al., 2019: Gondal et al.) 202I). This
modeling of conditional distributions/ approach also has the advantage that the encoder can accommodate both variable number of observations, as well as variable x-locations. Such approaches share our broad goal of trying to learn the structure of a space of curves without assuming any particular functional forms ahead of time. However, both differ from ours in
11

---

Published in Transactions on Machine Learning Research (07/2022)
that they build in more statistical machinery, by positing an explicit generative probabalistic model of the input distribution of curves.
8 Limitations
There are several limitations to our encoder. First, it differs from other models in that it was not designed to scale to very long time series. In particular, we use a feedforward convolutional encoder that processes the entire time series at once, while other techniques use some combination of recurrence and/or local windowing of the time series. Our time series have only 100 points, which is extremely short from the viewpoint of typical time series in learning models. However, the context of function learning, such short time series have face validity, because during function learning experiments people can only make use of limited information at a time (Villagra et al.] 2018). Thus, while it is not clear how well our encoder architecture would scale to very long time series, it is also not clear how well humans would do so either; and it remains to be determined how useful doing so would be for generalization in natural environments. These remain subjects for future rescarch. Second, the feedforward nature of our encoder also restricts it to processing time series of a fixed length and sampling frequency, as opposed to the recurrent encoders of the other models which can handle time series of variable lengths, or the CNN-stylee permutation-invariian encoder, which can handle both variable lengths and variable sampling froquencies. In principle this could be overcome by upsampling or downsampling as necessary (and this was the approach we took in the multiple choice completion task). While this kind of resampling may be benign helpful in certain circumstances (e.g., a form of context normalization (Webb ct al. 2020 ), there are also many applications in which it would instead be preferable to preserve the original resolution and accommodate variable lengths. It is also possible that an appropriate modification of the CNP-style encoder could be used to overcome this difficulty; but due to the relatively poor results of the contrastive-perm-iie model, further work is necessary to fully flesh out this idea.
9
Discussion
The contrastive encoder we presented exhibited superior performance to comparison models in tests of generalization involving categorization as well as free form extrapolation. This was the case, despite its greater simplicity than those models. Moreover, we found that the performance was significantly degraded if an MLP was used in place of the convolutional layers in the encoder, and was even further degraded when using architecture. On the other hand, the usage of a convolutional
Neural Process-style permutation-invarian encoder is not sufficient on its own to achieve this level of performance, as shown by the poor results of the VAE, which employed such an encoder. This suggests that the combination of 1d ievoouutions.ooettaei with the contrastive loss and specific family of augmentations, may be key to lcarning good representations of intuitive functions.
In the multiple choice task, all models found it easier correctly extrapolate prompt curves generated from the CG than curves generated from the SM, with this effect being most pronounced in the contrastive model. This is qualitatively similar to the corresponding empirical result from Schulz et al. (217)), regarding peoples' judgements in an analogous task. Thus our analysis suggests that such a bias may simply "fall out" as a consequence of a more general representation-llariin concept that the properties of representation learning algorithms can serve as an explanatory tool in the study of high-level human cognition such as function learning-
procedure. More generally, we regard this as a proof of
Indeed, several influential accounts of human intelligence posit the existence of elements of "core knowledge," " such abstract number sense and fundamental notions of Euclidean geometry (Seekee and Kinzler 2007 Chollet 2019 Sometimes referred to as "atoms" of knowledge, these primitives are assumed to be low dimensional forms of representation and/or simple constructs and functions (c.g., continuity of processing, simple forms of causality), on which more complex cognitive abilities responsible for human intelligence are built. It has been proposed that the availability and use of such primitives is a critical factor in distinguishing human gencralization capabilities from that of existing artificial systems (Lake et al., 2016). that rely on statistical estimation, and recent empirical evidence has been proposed in support of this claim (Kumar et al, 2020) Major efforts in cognitive science have assumed that such primitives are either genetically pre-specified,
12

---

Published in Transactions on Machine Learning Research (07/2022)
arise sufficiently early and predictably in development that they can be treated as predetermined. Based on this assumption, such efforts have focused research on the kinds of inference and learning mechanisms that. operating on such primitives, can compose them into more complex forms of processing (Lake et al. 2015 Ellis et al. 2020 Similarly, it has been proposed that such primitives should be considered as inductive biases when designing and comparing candidate computational architectures that seek to emulate human gencralization capabilities. In contrast to this approach, some have argued that is neither necessary nor accurate to assume that such primitives are pre-specific, but rather they arise from and are shaped by general purpose learning mechanisms interacting with and encoding statistical of present in the environment [Rumelhart and McClelland, 1986). While examples have been provided of how human-like concept formation and generalization can arise in this way (MeClelland and Rogers, 2003), these have generally relied on externally supervised forms of learning that are explicitly trained on tasks that elicit such structure. To date, it has been difficult to design artificial systems that can discover low dimensional, simple forms of structure that can be exploited for generalization, using unsupervisod or self-supervised forms of learning. Here, we have proposed one such mechanism, through combination of contrastive learning and topological augmentations, and have demonstrated its ability learn to basic classes of functions, and simple compositions thereof.
For testing free form extrapolation, we used a curriculum learning strategy that involved first learning categories and then learning category-specific forecasting rules. While this procedure was more complex than for the other heads, there is reason to believe that it in fact resembles the process by which people may learn to make inferences in sparse and underdetermined settings. The most direct evidence of this in the realm of function learning comes from Leon- Vill-ra and Lucas (2019), which showed that peoples' extrapolations of curves were dependent suggesting that people use calegory-dependdeu forecasting rules. More generally, our approach may be viewed as implementing a form of Hierarchical Bayesian model, which have been show to capture the structure of peoples' intuitive theories about abstract structures in the world Gershman and Niv 2010 Tenenbaum et al.
whether they judged the curves to lic in previously encountered category,
2011
Kemp and Tenenbaum 2008).
Our approach also fits with the idea that learning in natural agents involves adjudicating a tension between maintaining as much fiexibility possible (by optimizing Maximum Entropy objective) while at the same time maximizing efficiency of computation (e.g., by optimizing Minimum Energy objective). From this perspective, the contrastive encoder can be viewed as maximizing entropy, as implemented by the InfoNCE objective that we used, as it may be shown [Wang and Isola, 2020) that the second term in that objective is an estimator of the entropy of the distribution of codes in the latent space. Complementing this, our curriculum learning can be viewed as minimizing the energy of representations generated by a given calegory of function when presented with an instance of that function. This may strike a balance between flexibility (of generalization) and efficiency (of inference) that begins to approximate the balance observed in natural agents, and humans in particular FFanklland et al. 2021
10 Broader Impact
As this work is concerned with foundational properties of learning algorithms in an abstract setting, we do not foresee any negative societal consequences arising directly from this work.
References
A. Aberdam, R. Litman, S. Tsiper, O. Anschel, R. Slossberg, S. Mazor, R. Manmatha, and P. Perona. Sequencc-to-soquence contrastive learning for text recognition. arXie:2012.10873.1, 2020.
S. Amiriparian, M. Freitag, N. Cummins, and B. Schuller. Sequence sequence autoencoders for unsupervised representation learning from audio. In DCASE Workshop, 2017.
S. Bocker and G. Hinton. Self-organizing neural network that discovers surfaces in random dot stereograms. Nature, 1992.
13

---

Published in Transactions on Machine Learning Research (07 (07/2022)
A. J. Bell and T. I. Sejnowski. An information maximization mn approach to blind separation and blind deconvolution. Newral Compulation, 1995.
E. Bienenstock, L. Cooper, and P. Munro. Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. 1982.
L. Chen. The topological approach to perceptual organization. Visual Cognition, 2005.
Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
F. Chollet. On the measure of intelligence. arXiu:1911.01547, 2019.
E. L. DeLosh, J. R. Busemeyer, and M. A. McDaniel. Extrapolation: The sine qua non for abstraction in function learning. Journal of Experimental Psychology: Learving, Memory and Cognition, 1997.
D. Duvenaud, J. R. Lloyd, R. Grosse, J. B. Tenenbaum, and Z. Ghahramani. Structure discovery in nonparametric regression through compositional kernel search. In International Conference on Machine Learning, 2013.
K. Ellis, C. Wong, M. Nye, M. Sable-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama, and J. B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. (XXi:2006608388 2020.
S. Frankland, T. Webb, and J. Cohen. No coincidence, george: Capacity-limits the curse of compositionality. PsyAraiichttpps:/ddod htwpshttttss//oiioooooo org/ 10..1234///////// osf.io/cjuab, 2021.
M. Garnelo, D. Rosenbaum, C. J. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. J. Rezende, and S. M. A. Eslami. Conditional neural processes. arXiv: 1rX7: 1807.011701 2018a.
M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. A. Eslami, and Y. W. Teh. Neural processes. arvvv preprint arXiv:1807.0162. 2018b.
S. J. Gershman and Y. Niv. Learning latent structure: carving nature at its joints. Current Opinion in Neurobiology,, 2010.
M. W. Gondal, S. Joshi, N. Rahaman, S. Bauer, M. Wuthrich, and B. Scholkopf. Function contrastive learning of transferable meta-reppesentations. In International Conference on Machine Learning, 2021.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Computer Visiston and Pattern Recognition, 2020.
G. E. Hinton. Deterministic boltzmann learning performs steepest descent in weight-space. Neural Computa- tion, 1989.
G. E. Hinton and R. R. Salakhutdinov. Using deep belief nets to learn covariance kernels for gaussian processes. In Conference on Neural Information Processing Systems, 2007.
A. Hyvarinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlincar ica. In Advances in Neural Information Processing Systems, 2016.
N. Intrator and L. Cooper. Objective function formulation of the bem theory of visual cortical plasticity: Statistical connections, stability conditions. Neural Networks, 1992.
Jean-Yves, Yves, Franceschi, A. Diculeveut, and M. Jaggi. Unsupervised scalable representation learning for multivariate time series. In Conference on Neural Information Processing Systems, 2019.
Jean-
C. Kemp and J. B. Tenenbaum. The discovery of structural form. Proceedings of the National Academy of Sciences, 2008.
H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Tch. Attentive neural processes. 1.0x.01900.050 2019.
14

---

Published in Transactions on Machine Learning Research (07/2022)
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXin:1312.6114, 2014.
S. Kumar, I. Dasgupta, J. Cohen, N. Daw, and T. Griffiths. Meta-learning of structured task distributions in humans and machines. 2020.
P. Kwantes and A. Neal. Why people underestimate y when extrapolating in linear functions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 2006.
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and Bruin Sciences, 2016.
P. Leon-Villagra and C. G. Lucas. Generalizing functions in sparse domains. In Proceedings of the Cognilive. Science Society, 2019.
T. Li, L. Fan, Y. Yuan, H. He, Y. Tian, and D. Katabi. Information contrastive learning for sclf-supervised representations. arXiv:2112.0000 2020.
R. Linsker. Self-organization in a perceptual network. IEEE Computer, 1988.
C. G. Lucas, T. L. Griffiths, J. J. Williams, and M. L. Kalish. A rational model of function learning. Psychonomic Bulletin and Review, 2015.
X. Lyu, M. Hueser, L. Hyland, G. Zerveas, and G. Rätsch. Improving clinical predictions through unsupervised time series representation learning. arXi::1122.000 00490, 2018.
Q Ma, J. Zheng, S. Li, and G. W. Cottrell. Learning representations for time series clustering. In Advances in Neural Information Processing Systems, 2019.
P. Malhotra, L. Vishmu TV, Agarwal, and G. Shroff. Timonet.pre-trained deep recurrent neural network for time series classification. Xiv: 170.::1706.08838, 2017.
J. McClelland and T. Rogers. The parallel distributed processing approach to semantic cognition. Nature Reviews Neuroscience, 2003.
M. A. McDaniel and J. R. Busemeyer. The conceptual basis function learning and extrapolation: Comparison of rulc-basod and associativo-based models. Psychonomic Bulletin and Review, 2005.
T. Mikolov, 1. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 2013.
M. Neumann, S. Huang, D. E. Marthaler, and K. Kersting. pygps - a python library for gaussian process regression and classification. Journal of Machine Learning Research, 2015.
T. Pan, Y. Song, T. Yang, W. Jiang, and W. Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Computer Vision and Pattern Recognition, 2021.
C. E. Rasmusson and C. K. Williams. Gaussian processes for machine learning. MIT Press, 2006.
M. Roed and B. Simon. Functional analysis. Academic Press, 1980.
D. E. Rumelhart and J. L. McClelland. Parallel distributed processing: explorations in the microstructure of cognilion. Volume 1. Foundations. MIT Press, 1986.
E. Schulz, J. B. Tenenbaum, D. Duvenaud, M. Speckenbrink, and S. J. Gershman. Compositional inductive biases function learning. Cognitive Psychology, 2017.
F. S. Spelke and K. K. D. Kinzler. Core knowledge. Developmental Science, 2007.
15

---

Published in Transactions on Machine Learning Research (07 (07/2022)
S. Sun, G. Zhang, C. Wang, W. Zeng, J. Li, and R. Grosse. Differentiable compositional kernel learning for gaussian processes. International Conference on Machine Learning, 2018.
J. B. Tenenbaum, C. Kemp, T. L. Griffiths, and N. D. Goodman. How to grow a mind: Statistics,struccurr,, and abstraction. Science, 2011.
S. Tonekaboni, D. Eytan, and A. Goldengerg. Unsupervised representation learning for time series with temporal neighborhood coding, In International Conference on Learning Representations, 2021.
D. Tran, R. Ranganath, and D. M. Blei. The variational gaussian process. In International Conference Learning Representations, 2016.
A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv:1807.00744 2018.
P. L. Villagra, Preda, and C. G. Lucas. Data availability and function extrapolation. In Proceedings of the Cognitive Science Sociely, 2018.
T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, 2020.
T. W. Webb, Z. Dulberg, S. M. Frankland, A. A. Petrov, R. C. O'Reilly, and J. D. Cohen. Learning representations that support extrapolation. In International Conference Machine learning, 2020.
G. Wilson and R. Adams. Gaussian process kernels for pattern discovery and extrapolation. In International Conference on Machine Learning, 2013.
A. G. Wilson, C. Dann, C. G. Lucas, and E. P. Xing. The human kernel. In Conference on Neural Information Processing Systems, 2015.
E. Zeeman. Topology of the brain. In Mathematics and computer science in biology and medicine: Proceedings of the conference held by the Medical Research Council in Association with the Health Department, 1965.
16


Reviews:
[{"review_id": "398pLVMVbX", "reviewer": "Reviewer_83P8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- Focus on a \"simpler\" (compared to large scale, e.g. visual, tasks), yet sufficiently complex and well defined domain can generate interesting results.\n- The authors compare against a broad set of baselines\n- The authors performed a sizeable number of experiments for various baseline/hyperparameter/task variations.\n- Interesting results in section 6\n\n# Weaknesses\n\nI have grouped my feedback below into three categories, \"Clarity\", \"Lack of clearly formulated learnings\" and \"Methodological questions\".\nI've also added a \"Nitpick\" section containing more subjective opinions/minor issues that I would like to point out, but do not expect the authors to explicitly address in the rebuttal (but I think would strengthen the paper).\n\n## Lack of clearly formulated learnings/contributions\n\nThe most critical weakness of this paper for me is that the contribution is not clear. The authors state that they \"propose a framework for addressing these challenges\", meaning how to learn a good representation in a self-supervised fashion. However, this is a well-researched problem and the proposed framework is not novel. Neither the idea of unsupervised pre-training and learning small task-specific 'heads', nor the used loss function are new.\n\nThe authors further claim novelty on the investigated domain and the newly proposed augmentations. While the domain itself is not new, I am indeed not aware of deep-learning research on this domain for the specific question of self-supervised contrastive learning, although I am not an expert in this area. However, I believe a stronger justification for why this area is of interest would strengthen the paper. The authors make references to biological generalization, but these references are vague and possible insights gained are not clearly specified.\n\nThis connects to the experimental section, in which the results are interpreted as \"method X performs better than method Y\", but no deeper insights are formulated or discussed, for example about biological insights or reasons why some representation learning methods might perform better than others.\n\n## Clarity\n\nAt a few places, especially in the experimental section, I was not able to exactly understand the description:\n\n- Last paragraph before section 5.1: To me this paragraph reads as if each value in table 1 is the average/std over 30 values (3 copies * 10 hyperparameter choices). But does that mean each copy-hyperparameter pair was only evaluated on a *single* test-datapoint? Surely a larger test-set was used?\n- Section 5.1: How is the L2 penalty applied to the encoder? I thought the encoder remains fixed for this task?\n- Section 5.2: \"we constructed the candidate completion curves by computing the posterior mean with respect either to the SM kernel, or to the CG kernel family\". How was the kernel from the CG family chosen? For a CG prompt, was the \"correct\" kernel chosen or a random one? For a SM prompt, was a random or the best fitting CG kernel chosen?\n\n## Methodological questions\n\n- I am unsure about the choice to fix the hyperparameters for each kernel instead of fitting them to each given function. Because then the multiple choice task is not just between different kernel *types*, of which there are the described 14 (e.g. linear, rbf, SM), but actually between different *kernels* with different hyperparameters, which should make the choice much easier. (This also relates to my point in the 'clarity' section about how the corresponding CG kernels were chosen)\n- The results from section 6 is very interesting, but I believe it requires either some discussion/explanation or, even better, additional experiments explaining how this bias is introduced. It would also be good to confirm that the bias does not exist on the training dataset and that the observed bias is induced by the generalization of the learned representation.\n\n## Smaller points / Nitpicks\n\n- \"...that insights gained here are likely to shed light on the ability of biological and artificial agents to generalize more broad\". This is a strong claim that would benefit from support from either citations or a more detailed argument.\n- I am not sure about the intuitive explanation of the InfoNCE loss: If one approximates LSE as max(z_i), then this z_i won't be the most similar negative example, but it will be the positive example because the sum in the denominator is over *all* pairs, not just negative ones.\n- I think section 2.2 would greatly benefit from some visual examples for some of the kernels used.\n- I find it unsurprising that a CNN compares favourably to an MLP with the same number of parameters (CNNs have surprisingly few). A more interesting (IMO) comparison would have been to a larger MLP.\n- “Thus our analysis suggests that such a bias may simply “fall out” as a consequence of a more general representation-learning procedure. More generally, we regard this as a proof of concept that the properties of representation learning algorithms can serve as an explanatory tool in the study of high-level human cognition such as function learning, thereby opening up many exciting directions for future work.” I believe this claim requires some more concrete examples as support.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study representation learning in the space of 1D functions generated by various types of GP kernels.\nThey train 6 different types of encoders using either varying architectures or contrastive training losses.\nSubsequently, they compare the learned representations in three different supervised tasks by learning simple predictions head on top of the learned representation, using only small amounts of data.\n\nThe three tasks are:\n1. Classifying which type of kernel (out of 14 possible choices) a given function is most likely from \n2. Selecting one of two possible choices for continuation of a given function\n3. \"Freeform\" predicting the continuation of a given function by predicting auto-regression coefficients.\n\nThe 14 available kernels are selected from two different families of kernels: 13 generated from a compositional grammar (\"CG\", out of linear, rbf, periodic) and one spectral mixture kernel. The authors compare their results with biases found in humans, which exhibit a tendency to select CG kernels over SM kernels when choosing, and find a similar bias in their results.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please refer for details to the review sections above.\n\nAs most critical recommended change, I would suggest giving the paper a clearer \"take away\" message. The authors hint at potential relevance to social/biological sciences, but these are kept at a too high level. Furthermore, the discussion of experimental results should also better highlight potential conclusions. For example (but not necessarily these), things like: Why is InfoNCE better than the baselines? What mechanism leads to the bias get introduced in section 6? What does this tell us about human cognition? etc..\n\nOn the methodological side, two points should be addressed before I can recommend acceptance:\n* As the fixing hyperparameters for kernels instead of learning them is an unusual choice, which I believe could have quite strong influences on the qualitative results, it would be good to have some additional justification/discussion and, ideally, ablation studies (in the appendix is fine if space is scarce) showing that they don't change the qualitative results.\n* The results from section 6 are very interesting, but require more discussion, or even better additional experiments, further explaining how the observed bias is introduced. It should also be confirmed that the bias is not yet present on the training data (which, I believe, would point to a bug).\n\nLastly, please have a look at the \"Nitpicks\" in the review section above. However, these aren't critical for my recommendation.\n\n\nFurthermore, please improve the clarity of (experimental) descriptions. These should be small fixes.\n* Last paragraph of section 5.1 (the test setup)\n* How was L2 penalty used\n* Section 5.2: How was the kernel from the CG family chosen?\n\n\n\n", "broader_impact_concerns": "No broder impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7yW6XTbVwF", "reviewer": "Reviewer_ESBW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strength\n  1) The problem of learning a representation of 1D intuitive functions and extrapolating them is not only new but also very well-motivated by the cognitive science literature as discussed in the paper. While the majority of the prior work has focused on learning representations of images and texts, this paper introduces a relatively under-explored problem and makes a nice attempt to mimic how humans learn and extrapolate functions.\n  2) The proposed way of applying topological distortions sounds plausible and convincing.  \n  3) The empirical result looks convincing. Although there is little work on learning representations for 1D functions, the proposed method is still compared against reasonable and popular representation learning algorithms including CPC, Triplet loss, and VAE.\n  4) The paper is very well-written. The problem is very well-motivated, and the description of the method and experiment is very clear. \n\n* Weakness\n  1) There is no ablation study on the use and the specific choices of topological distortions. The proposed algorithm is a combination of 1) convolutional architecture, 2) contrastive loss, and 3) topological distortions. While there are ablation studies show the effectiveness of the first two, I could not find experiments showing the effectiveness of topological distortions unless I missed something. Showing this would make the result much more comprehensive. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a self-supervised learning method for learning a representation of 1-dimensional intuitive functions, which allows for better extrapolation. Specifically, the paper proposes 1) a 1D-convolutional encoder, 2) the use of contrastive loss, and 3) data augmentation via several topological distortions inspired by cognitive science literature. The experimental results show that the proposed method outperforms several baselines (e.g., CPC, triplet loss, VAE) on kernel classification, multiple choice extrapolation, and free-form extrapolation. In addition, the empirical result also shows that the proposed method has a bias towards Compositional Grammar (CG) like humans, which arguably makes it suitable for extrapolation in natural environments. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Show how much each topological distortion improves extrapolation.\n2. (Minor) Giving some examples of CG and SM kernels in the main text would be helpful for readers who are less familiar with them.  \n3. (Minor) Fix the wrong reference (Figure 3 -> Figure 1) in Page 4.\n4. (Minor) Fix squeezed tick labels (texts) in Figure 1, 2, 3. ", "broader_impact_concerns": "I do not have any ethical or broader impact concerns out of this paper. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ESe1mXpbqh", "reviewer": "Reviewer_84zM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Novelty, significance, relevance\n\nThe results and methodology in this paper appear novel. The work is certainly relevant to the representation learning community, though its connection to the psychology community is not immediately clear; however the psychology connection seems more of an aside in the paper anyways. The methodology represents a significant change to classical representation learning methodologies, where the experiment design is targeted towards emulating human behavior on a set of function learning tasks. The resulting experiments provide some interesting insights---though it's possible some of these insights may be misattributed (I will discuss more below).\n\n## Correctness\n\nBroadly, this paper did an exceptional job at matching empirical evidence to claims. Experiments were well designed with 30 samples used to estimate sample means of performance, uncertainty measures included in all tables, reasonable baselines provided, and special care given to training data to rule out potential sources of bias (i.e. data imbalance). \n\n### Confidence intervals\n95% confidence intervals are reported for every result in every table, however these confidence intervals are based on standard errors. It's not clear that proper assumptions are met for these 95% confidence intervals to be accurate, namely the normality assumptions on the underlying data as well as independence assumptions. There is underlying structure to the data (3 trials for 10 different hyperparameters = 30 total data points), which suggests that independence may be violated and also hints to me that normality plausibly would not hold. As a result, all of these uncertainty measures could be providing optimistically tight bounds on the sample mean estimate. \n\nIt should be noted, however, that in most cases the effect size appears to be quite large---there is a large difference in estimated mean performance between the proposed algorithm and baselines---so even if confidence intervals are optimistically tight, likely statistical significance still holds. These optimistic intervals may be adversely impacting conclusions in Table 3 where differences between the proposed method and the architecture ablation (contrastive-mlp) are often much smaller. In the other tables (Tables 1, 2, 4) these optimistic intervals likely do not impact conclusions severely. \n\n### Table 4 -- bias towards CG\n\nI believe the paper is actually underclaiming its results in this table. The bolding scheme and discussion in the paragraph at the bottom of page 9 suggest that the paper is seeking statistically significant differences between models. That is, the table is implicitly testing the claim:\n> Does our proposed model (contrastive) induce greater bias towards CG functions than other models?\n\nThe resulting conclusion is that there are no statistically significant differences among the results. However, I believe the goal of this section should _actually_ be measuring difference from 0 bias, rather than differences between algorithms. That is, the claim should be:\n> Do any of the models induce _any_ bias towards CG functions?\n\nAnswering this question would allow drawing conclusions that this empirical methodology yields similar biases as the human-centered methodology found in prior works. Interpreting the results in Table 4 in this way, it appears that the proposed method is statistically significant with a very meaningful difference (~20 points away from 0).\n\nRegardless of which claim is measured, however, I wonder if there are confounding effects that are harming the conclusions. Specifically, the paper suggests that these biases towards better accuracy on CG functions vs SM functions are due to the modeling decisions themselves. To some extent, this is clearly true: there are notable differences in bias between models. However some degree of this bias very likely is coming from the data. Specifically, look at the \"raw\" baseline. The bias is approximately ~5 points in favor of CG, despite the model having no predictive power (it is effectively the identity function). \n\nThe differences in the underlying data could be due to the effective function spaces induced by CG vs. SM. Each function construction methodology yields a space of possibly observed functions. Imagine the extreme case that CG yields only one possible function, while the space for SM is extremely large. Then despite the data having 50% of examples drawn from each set, the singular CG function has 250k samples that the estimator can learn from, while each SM function has only 1 relevant sample. In this setting, then, it would be unsurprising that the estimator has some bias towards CG. Clearly the function space of CG is much larger than 1 function as in my example, but how large is the effective function space compared to SM? Considering this alongside the result from the \"raw\" baseline, does this suggest some of the differences found in the study are due to the underlying data instead of the learning mechanism?\n\nAs an aside, taking into account the possible underlying data bias and my suggestion to change the statistical test in Table 4, the proposed method actually looks _even better_. The proposed method (and the single ablation constrastive-mlp) would plausibly be the only methods that have a bias toward CG functions, suggesting that the learning mechanisms resemble human biases more strongly on this dataset than other learning mechanisms. (In order to draw this conclusion, I subtracted 5.19 [the raw bias] from each model's bias to approximate offsetting by underlying data structure. This isn't exactly correct, but might give a ballpark estimate of possible conclusions).\n\n## Clarity\n\nI had a couple of minor clarity concerns, primarily with the objective function stated in Section 2.1. The first, Section 2.1 notes InfoNCE in its title, however this is not actually discussed or even mentioned in the section body. A citation to a relevant paper is provided without context. If InfoNCE is important enough to be in the section title, likely it should be discussed in the section body. \n\nThe second clarity concern in Sec 2.1 is with the definition of the objective. The objective assumes that we are provided with pairs: $(v_i, v'_i)$ with $N$ such pairs. Later the objective is defined with modular arithmetic and a summation that spans from $i \\in [1, 2N]$. It was not until reading the van den Oord et al. paper that I understood that the objective does not take advantage of the pairs structure (i.e. $v'$ is never used in the objective definition), but rather that these pairs are flattened into a single dataset of size $2N$. \n\nI don't believe the Spectral Mixture functions are defined in this work---and even Compositional Grammar functions are only intuitively defined. I recognize both come from prior works, however they are critical to this paper and clarity would be greatly enhanced if a little more time was spent detailing these classes of functions.\n\n## Pedantics\n\nA few pedantic minor points that shouldn't influence the decision:\n* There are several typos that shouldn't be possible with latex---notably several whitespace characters are dropped between a period and the start of the next sentence. \n* On page 6, there is a paragraph segment (paragraph starts on page 5) which is nestled awkwardly between two figures. It looks like a caption and made reading awkward.\n* Similar paragraph issue on page 8\n* I had difficulty writing a summary of contributions based on the abstract and/or introduction. That is, neither the abstract nor introduction gave a clear overview of what is contributed by the paper. Notably, I found the abstract rather misleading in thinking this paper would somehow be related to reinforcement learning (terms like \"agents\" and \"environments\" and my own biases mixed poorly). This paper is highly specific to CG and SM functions and a particular methodology for constructing function learning. I believe the abstract should be more upfront about this, instead of being high-level and, well..., abstract.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper builds a framework for studying general representations learned to estimate \"intuitive\" scalar functions. These intuitive functions are constructed from a set of functions that have previously been shown to be more intuitive or easy for humans to learn, providing a reasonable proxy for naturalistic function learning. The paper uses a self-supervised encoder (e.g. an autoencoder) to learn representations which are invariant to topological distortions of these intuitive functions. The paper studies a few-shot learning paradigm where representations are pretrained on a large corpus of data, then a specialized final layer is learned few-shot mapping from representation to the desired predictor.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* The changes marked \"pedantic\" above would be nice-to-have in order to improve readability.\n* A deeper definition of Spectral Mixture functions and Compositional Grammar functions would significantly strengthen the readability.\n* I think the work would benefit greatly from revisiting Table 4 and ensuring that it provides direct support for the claims the paper wishes to make. In order to better align with the abstract, I believe that a reinterpretation of Table 4 is necessary. However, the paper is strong even without this change.", "broader_impact_concerns": "The included broader impact concerns section seems sufficient for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies representation learning in the space of 1D functions. It generates functions from a set of kernels: 13 kernels are drawn from a family of compositional kernels (CG) and one spectral mixture kernel (SM). The paper optimizes a contrastive learning objective with data augmentation and explores several architectures for the encoder. It tests the learned representations on several downstream tasks, such as extrapolation and kernel classification.

One of they key results is that the contrastive learner with a 1D convolutional backbone exhibits a similar bias towards CG extrapolation compared to SM as humans do, albeit not quite as strong.

The reviewers all appreciated the focus on simpler functions, the quality of the writing, and the breadth of the experiments. The comparison with human data (section 6) is especially interesting. The main concerns were:

- A lack of clarity in terms of the specific contributions of the paper.
- Fixing kernel hyperparameters of each kernel rather than fitting them to data.
- More detailed discussion of the results in section 6.
- Ablations on the effects of topological distortions in relation to extrapolation.
- The validity of the reported confidence intervals.
- Various clarifications and definitions (e.g., mathematical definitions of CG and SM kernels).

The authors addressed all of these points during the rebuttal period and unanimously recommend acceptance. In my opinion, this paper outlines a very nice fundamental study, clearly and convincingly supports its claims, and would make a solid contribution to the TMLR community. 

Decision: Accept (poster)