SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tracey E. Mills  
MIT
temills@mit.edu
Joshua B. Tenenbaum  
MIT
jbt@mit.edu
Samuel J. Cheyette  
MIT
cheyette@mit.edu
People are adept at learning a wide variety of structured patterns from small amounts of data, presenting a conundrum from the standpoint of the bias-variance tradeoff: what kinds of representations and algorithms support the joint flexibility and data-paucity of human learning? One possibility is that people "learn by programming": inducing probabilistic models to fit observed data. Here, we experimentally test human learning in the domain of structured 2-dimensional patterns, using a task in which participants repeatedly predicted where a dot would move based on its previous trajectory. We evaluate human performance against standard parametric and non-parametric time-series models, as well as two Bayesian program synthesis models whose hypotheses vary in their degree of structure: a compositional Gaussian Process model and a structured "Language of Thought" (LoT) model. We find that signatures of human pattern learning are best explained by the LoT model, supporting the idea that the flexibility and data-efficiency of human structure learning can be understood as probabilistic inference over an expressive space of programs.
Humans, uniquely among animals, are exceptional learners of structure: we learn languages, develop and use systems of mathematics, make music, follow recipes, and play board games. It is very unlikely that we have evolved specialized modules for these specific abilities, save language, given their breadth and the fact that many were invented quite recently. Instead, people seem to have powerful inference mechanisms that enable learning and representing structured theories in many different domains. Yet the flexibility of human learning presents a challenge to models of intelligence based, for instance, on training large, data-intensive networks. While there are now machine learning systems that successfully mimic or even surpass certain human abilities, humans have general inductive abilities that exceed modern AI systems and, importantly, often require much less data.
Models of human structure learning historically fall under two broad approaches: symbolic approaches, based on grammars and logic, and statistical approaches, such as neural networks and Bayesian inference. Symbolic approaches have focused on capturing the productivity, systematicity, and compositionality of human thought [1]; however, they have often been implemented in ways that are insufficiently flexible to handle ambiguity and uncertainty which are also central features of cognition. In contrast, statistical approaches have rarely enforced this same degree of structure in learned representations, in order to achieve greater flexibility and robustness to ambiguity and uncertainty. Recently, the idea of modeling human learning as probabilistic program induction — in which probabilistic inference operates over structured, symbolic programs — has emerged as a compelling synthesis of these two approaches [2].
Program induction models have shown success in explaining patterns in human learning in a number of domains, including logical concepts, handwritten characters, counting, kinship relations, sequences,
list functions, and question asking, among others [e.g. 3-10]. For instance, Piantadosi et al. [6] used a program learning model containing primitives of objects, sets, set operations, and recursion to explain how children learn number words, capturing their progression from understanding the meaning of "one," "two," "three," and "four," to being "Cardinality Principle" knowers: understanding the relationship between the count list and cardinalities in general. Though this and other work supports the empirical validity of modeling human learning as probabilistic inference over programs, the evidence has largely been indirect, e.g. comparing relative learning rates of different concepts in people and models. Only in very constrained domains, such as learning concepts corresponding to Boolean or first-order logic programs, have there been systematic attempts to assess the specific inductive biases employed by human learners or to measure directly the extent to which people are making probabilistic inferences over candidate programs [3, 4].
There has been some work assessing the degree of programmatic structure present in people's inductive biases when learning in more complex or naturalistic domains. For example, past work has found that people have compositional inductive biases when learning 1D functions or time-series. Schulz et al. [11] built on Gaussian Process (GP) models of 1D function learning [12, 13] to show that human learners prefer compositional hypotheses over non-compositional alternatives matched for low-level visual properties. Specifically, Schulz et al. investigated models of human function learning as Bayesian inference over a grammar of GP kernels which specify the features of high probability functions (e.g. periodic, smoothly varying, linearly increasing). These models are capable of explaining complex patterns in data by combining kernel primitives to form compound kernels (e.g., a periodic deviation on top of a smoothly increasing trend) [14-16], while also maintaining a bias for simple functions by enforcing a prior for simple kernel structures. Though these compositional GP models offer one formalization of the joint flexibility and structure characteristic of human learning within this relatively unconstrained hypothesis space, they do not attempt to capture the full range of algorithmic structure that people can learn.
Other research has investigated human learning of more general program structures in the setting of 2D geometry and geometric sequence learning. Amalric et al. [8] found that both adults and children are strongly biased toward sequences described by programs that involve operations with symmetry, regular rotations, and repetitions in time. Relatedly, Sablé-Meyer et al. [17] argued that people represent geometric shapes in terms of high-level program control structures such as repetition, concatenation, and embedding, and found that participants' complexity ratings and memory of shapes was better predicted by a shape's minimum description length in a grammar that includes these concepts as primitives than in one which does not. However, these studies either did not directly model inference or did not test program induction against alternative accounts of learning.
Thus, although a strong basis has begun to form for the "learning as programming" account, there is still relatively little work that aims to thoroughly compare human inferences to those of both probabilistic program induction models and other plausible inductive models. That is our goal here. We experimentally test human predictions on a spatiotemporal pattern learning task that features a wide range of algorithmically interesting geometric structures, including variants of spirals, zig-zags, polygons, grids, staircases and other repeating and nested combinations of these motifs. In each trial of the task, participants repeatedly made predictions about where a dot would move next within a 2D region, based on previously observed points in a discrete sequence. While this task is quite simple, it requires participants to navigate a rich and theoretically unconstrained hypothesis space on the basis of very little data, with the potential to give insight into both the inductive biases and representations of uncertainty in people's hypotheses.
Our analyses compare the probability distributions of participants' predictions to those of multiple varieties of Bayesian program induction models as well as several competing accounts. Models tested include Bayesian polynomial regression, a Gaussian Process model that learns both a kernel and parameters, and two Bayesian program synthesis models with grammars that afford different degrees of flexibility and structure: a highly flexible Gaussian Process model that jointly learns kernel structure and parameters, and a structured "Language of Thought" (LoT) model with more traditional programming primitives such as loops, motor commands, and arithmetic operators. We also describe preliminary explorations with an LSTM-based neural network account.
We find that the two Bayesian program learning models provide the best accounts of both the mean performance of human learners and the distributions of their predictions at successive points of the observed sequences. The fine-grained signatures of human pattern learning, including both patterns
Figure 1: The first six timepoints of a sequence, shown to illustrate the task. The most recently revealed point in the sequence is shown as a black dot, previous points are shown in lighter gray, and human predictions are shown as red circles.
of successful and incorrect predictions, are best explained by the Bayesian LoT model, supporting the idea that the flexibility and data-efficiency of human structure learning can be understood as probabilistic inference over an expressive space of explicitly structured programs.
2 Experiment
2.1 Method
We recruited 50 adult participants from the online platform Prolific who each completed 20 trials in which they learned a unique sequence of 2D points. Participants were asked to predict successively the location of each point in the sequence conditioned on having seen the locations of all previous points. Each sequence contained 21 points including the initial point, so that participants made 20 predictions, with up to four attempts per prediction per trial. After completing each trial, participants responded to two questions with a sliding scale: "How confident are you that you learned the pattern?", from "not confident (0)" to "very confident (100)" and "How complicated was the pattern?", from "not complicated (0)" to "very complicated (100)." See Figure 1 for an illustration of the prediction task and the distribution of participants' predictions on each of the first six timepoints of one sequence. The set of all sequences is shown in full in Figure S3 in the Supplementary Materials.
At the beginning of each trial, a single point was shown on the screen within a display region with aspect ratio 4:3 and width in pixels  $w$  which was maximized based on a participant's screen dimensions. Participants were instructed to click where they predicted the next point would appear. Participants were allowed to click within a gray circular region which was centered at the previous point with radius  $0.21 * w$  pixels, and which excluded the area within  $0.020 * w$  pixels of the previous point (see Figure S2 in the Supplementary Materials). If participants guessed within an "acceptance distance" ( $0.026 * w$  pixels) of the next point in the sequence, their prediction would be considered correct and the next point would be revealed. Participants were given up to four incorrect attempts before the next point was revealed. Previous points in the sequence remained visible, but turned lighter according to how far back in the sequence they appeared. Participants were paid $7.25 for completion of the study, with a median completion time of 46 minutes.
2.2 Results
The full set of participants' predictions on all 50 sequences is visible in Figure 2, with early points in cooler colors (blue/purple) and later points in warmer colors (orange/yellow). Participants generally succeeded at the task, making correct predictions on their first attempt for 11.97 of the 20 predictions per sequence on average, though the number of correct first-attempt predictions varies considerably across sequences (see Figure 3). To more precisely quantify the extent to which participants successfully learned each sequence, we compute the relative absolute error of a participant's prediction at each timepoint as the absolute distance between their first prediction attempt  ${}^{1}$  and the true point, divided by the distance between the true point and the previous true point. We excluded the  ${10}\%$  of participants with the greatest mean relative absolute error across timepoints and sequences from our analyses.
Under this metric, participants successfully learned almost every pattern, with mean relative absolute error across participants and timepoints below 1 (the mean relative absolute error that would result if the location of each prediction attempt was simply the previous true point) for 49/50 sequences.
<sup>1</sup>Although participants were given up to four prediction attempts at each timepoint, we only include the first attempt in our analyses. The use of multiple attempts in the experiment primarily served as motivation for participants to remain engaged in the task.
Figure 2: Human predictions for each sequence across all timepoints, with predictions for earlier timepoints shown in cooler colors (blue/purple) and those for later timepoints in warmer colors (orange/yellow).
Learning varied noticeably by sequence, with mean relative error across participants and timepoints ranging from 0.11 to 1.04. Participants became more accurate as time went on within each sequence, with an average relative error across sequences of 0.94 at the first timepoint, 0.21 at the final timepoint, and 0.43 across timepoints. Figure 3 shows the distribution of the number of correct predictions (within  $0.026 * w$  pixels) participants made on their first attempt across all sequences. There are some sequences, such as a noiseless line, that participants got correct on their first attempt for all but the first or second prediction. On the other end of the spectrum, there are some sequences which participants struggled to learn even after 15 data points (e.g. sequences 49 and 50, as shown in Figure S3 in the Supplementary Materials). Overall, mean relative error had a strong positive relationship with participants' complexity ratings ( $r = 0.77$ ,  $p < .001$ ), and a strong negative relationship with confidence ratings ( $r = -0.87$ ,  $p < .001$ ).
3 Models
We evaluate several models on our task: first, we evaluate two models which have been used to explain human function learning in past work [12, 18-20], which fit parameters of either a fixed structure or a structure selected from a highly constrained set: a Bayesian polynomial regression model, and a non-compositional Gaussian Process (GP-NC) model where a single kernel and parameters are inferred. We also evaluate two structure-learning models: a Gaussian Process model where the compositional kernel structure and parameters are jointly inferred (GP-SL), and a Bayesian LoT program-learning model over motor and geometry primitives.
Each model was implemented in Gen [21] using Sequential Monte Carlo (SMC) with Markov Chain Monte Carlo (MCMC) rejuvenation steps. Specifically, all models run SMC with 20 particles which, after an additional observation becomes available, are resampled if their effective sample size falls below 10. There are then 100,000 rejuvenation steps on the inferred hypothesis (including noise parameters) for each particle. The rejuvenation steps use a generalization of Metropolis Hastings called Involutive MCMC that allows for custom-built kernels in the reversible jump MCMC framework. After rejuvenation, each resulting particle specifies a predicted location and noise estimate over this prediction for the next point in the sequence. Model predictions at each timepoint are marginalized over particle predictions, weighted by their posterior probability.
3.1 Fixed-structure models
Bayesian Ridge regression
Ridge regression is a linear modeling technique that uses  $\ell_2$  regularization to constrain fit model coefficients — a natural choice for a problem involving sparse data and highly correlated variables, which both apply to this task. Here, we use a Bayesian variant of Ridge regression, which finds
Figure 3: Boxplots for each sequence of the number of timepoints at which participants correctly predicted the location of the next point on their first attempt, with outliers removed. Sequence numbers correspond to those in Figure S3 in the Supplementary Materials.
estimates for regression weights  $\beta$  assuming a spherical Gaussian prior  $p(\beta \mid \lambda) = \mathcal{N}(\beta \mid 0, \lambda^{-1}\mathbb{I}_p)$ , where  $\lambda$  is a prior on the precision. The goal is to learn a mapping from previously observed  $x$ ,  $y$  coordinates and the time  $t$  that they were observed, to the next  $x$  value and next  $y$  value in the sequence — i.e., the  $x$  and  $y$  values at time  $t + 1$ . This is done via two regressions: one to predict subsequent  $x$  values and one to predict subsequent  $y$  values. The mean predicted values have the form:
$$
\mu_ {x} = \beta_ {x, 0} + \beta_ {x, 1} \cdot t + \beta_ {x, 2} \cdot x + \beta_ {x, 3} \cdot y
$$
$$
\mu_ {y} = \beta_ {y, 0} + \beta_ {y, 1} \cdot t + \beta_ {y, 2} \cdot x + \beta_ {y, 3} \cdot y
$$
The likelihood is then the product of  $\mathcal{N}(x|\mu_x,\sigma_x)$  and  $\mathcal{N}(y|\mu_y,\sigma_y)$ .
In the main text, we show results only for the linear model with weight coefficients on  $x$ ,  $y$ , and  $t$ , but in the Supplementary Materials we also show results for the Bayesian Ridge regression model using 2 and 3 degree features (i.e.,  $x$ ,  $y$ ,  $t$  and their two or three degree interactions, such as  $x^2$ ,  $x \cdot y$ ,  $x \cdot t$ , etc.).
Non-Compositional GP model
Inspired by the model developed in [12], we fit a Gaussian Process model that jointly infers both the kernel and its parameters, but does not allow compositional kernel structures (e.g.  $K_{1}(\cdot) \cdot K_{2}(\cdot)$ ). We use four base kernels each with at least one numeric parameter: Constant (with a value parameter), Linear (with an intercept parameter), Radial Basis Function (with a scale parameter), and Periodic (with scale and period parameters). Each kernel specifies a prior over functions and each is useful for modeling particular types of data; for instance, the Periodic kernel is often used for modeling periodically varying data, such as weather patterns. We assume that  $x$  and  $y$  observations are generated by independent kernels, which each take the previous  $x$  value, previous  $y$  value, or the current  $t$  value as an input; one kernel outputs a prediction for  $x$  and the other for  $y$  at time  $t + 1$ . Each base kernel additionally has a categorical parameter that specifies which dimension  $(x, y, \text{or } t)$  to take as input, allowing for inference about which input dimension is most predictive. We used uniform priors on the kernel type and on noise. The likelihood is determined by the probability of drawing the vector of true  $x$  and  $y$  values at each timepoint from a multivariate normal distribution with a mean vector of zeros and covariance matrix as determined by the noise parameter and kernel function for that dimension.
3.2 Program synthesis models
Both the GP-SL and LoT models have probabilistic context free grammars (PCFGs) which define a space of hypothetical program trees. During MCMC rejuvenation, Metropolis Hastings proposal moves manipulate the structure of inferred program trees as well as their parameters (see [22]).
GP Structure-Learning model
The GP-SL model, adapted from the one presented in [15] to handle multidimensional output, uses the same base kernels as the GP-NC model but additionally allows for kernel operators that compose the
base kernels. Specifically, the grammar additionally includes three composite kernels: Sum, Product, and Changepoint (with a location parameter which specifies an input point at which to smoothly transitions between kernels). During inference, Metropolis Hastings proposals over program tree structures include tree regeneration moves, which involve the replacement of random subtrees with trees generated from the prior distribution. As with the GP-NC model, the GP-SL model infers independent  $x$  and  $y$  output kernels and noise parameters, and the likelihood is determined by the probability of drawing the vector of true  $x$  and  $y$  values at each timepoint from a multivariate normal distribution with a mean vector of zeros and covariance matrix as determined by the noise parameter and kernel function for that dimension. In this case the prior is determined by the PCFG from which the  $x$  and  $y$  kernels are drawn in addition to the uniform prior on noise.
LoT Model
The LoT grammar draws inspiration from [17], which defined a simple Logo-like drawing language to model human representations of geometric shapes. This language includes a mixture of control operations (e.g. loops), motor commands (e.g. turn), and numeric expressions, which in combination efficiently generates myriad canonical and more complex geometric shapes. In this work, Sable-Meyer et al. found that the complexity of MDL programs in this grammar predicted people's ease of encoding and recalling the shapes generated by those programs. Our LoT grammar is organized similarly, but has some added complexity to allow for greater flexibility in the range of sequences it can generate. In particular, our grammar includes access to inner-state variables such as the current position or timepoint, as well as a wider range of motor commands and algebraic operators, making it highly expressive. Programs drawn from this grammar define algorithms for generating unboundedly-long sequences of  $x$  and  $y$  positions over time.
Operations (o)	Values (v)
(Controls)	(Actions)	(State variables)	(Numbers)	(Expressions)
Repeat(o, v)	Move()	θ (current angle)	N (naturals)	Plus(v, v)
Continue(o)	Stay()	s (current speed)	R (reals)	Minus(v, v)
Concat(o, o)	Turn(v)	x (current x-position)	Times(v, v)
Subprogram(o)	Accelerate(v)	y (current y-position)	Divide(v, v)
ChangeX(v)	t (current time)	Mod(v, v)
ChangeY(v)	n (function calls)	Sin(v)
SetX(v)
SetY(v)
Table 1: LoT model primitives.
Table 1 shows the full set of primitives in the grammar. Operations define the structure of the program, and values are numeric parameters to operations. Operations include control nodes which act over other operations: Repeat, which when evaluated repeats a given operation a certain number of times; Continue, which repeats a given operation indefinitely (when being evaluated, each program generated by our model is wrapped in an outer Continue call which repeatedly executes the program until it generates a sequence of the specified length); Concat, which sequentially executes two operations; and Subprogram, which executes an operation and then restores the previous internal program state. Operations also include action nodes: Move and Stay determine the next location in the sequence by setting the position at the next timepoint. Move changes the current position based on the current speed and angle, and increments the current time, while Stay increments the current time without changing the current position. Other action nodes simply update the internal state at the current timepoint: Turn, Accelerate, ChangeX, and ChangeY update the current angle, speed, and  $x$  and  $y$  positions respectively by a given value. SetX and SetY simply set the current  $x$  and  $y$  positions to a given value.
Value nodes include numbers (Naturals and Reals) as well as the current values of internal state variables: the angle  $(\theta)$ , where a value of 1 corresponds to 90 degrees, speed  $(s)$ ,  $x$  and  $y$  positions  $(x, y)$ , timepoint  $(t)$ , and number of times the program has been executed by the outer Continue call  $(n)$ . Values can also be composed in expressions with Plus, Minus, Times, Divide, Mod, and Sin nodes.
This model can compactly represent many simple structures, such as geometrical shapes. For instance, the simplest program generating a square is Repeat (Concat (Move(), Turn (1)), 4). However, by allowing sub-programs, recursive calls, and unbounded concatenation, the model