SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model. Our modeling approach is similar to The Number Game, except we now have a discriminative learning problem instead of a generative one, and an online learning setup where the learner observes a stream of examples. To model online learning, we draw fresh proposals from  $q$  for each training batch, and perform Bayesian inference over all proposals drawn so far. To model discriminative learning, each example is now a triple  $(B, T, Y)$ , where  $B$  is a batch of shapes,  $T$  is a test shape in that batch, and  $Y$  is zero or one depending on whether  $T$  in  $B$  is an example of the concept.
Our likelihood model assumes human subjects predict according to  $C$  with probability  $(1 - \epsilon)$  and otherwise predict randomly with a base rate  $\alpha$  of labeling an example as positive. Following [55, 56] we model a memory decay process where the relative importance of earlier observations falls off according to a power law with parameter  $\beta$ :
$$
p (Y = 1 \mid B, T, C) = (1 - \epsilon) \mathbb {1} [ (B, T) \in C ] + \epsilon \alpha \tag {8}
$$
$$
\log p \left(X _ {1: K} \mid C\right) = \sum_ {\left(B _ {k}, T _ {k}, Y _ {k}\right) \in X _ {1: K}} (1 + K - k) ^ {- \beta} \log p (Y \mid B, T, C) \tag {9}
$$
As before, we translate hypotheses from natural language into Python using code-davinci-002, and evaluate the likelihood term above by running Python code. We consider both pretrained and tuned priors. Our proposal distribution  $q$  comes from running GPT-4 on prompts that illustrate previous batches, either as truth tables (for propositional concepts) or as a raw list of previous observed batches (for higher-order concepts). We again place a learnable temperature on the posterior.
Bayesian Program Learning Baseline (BPL). We contrast with a strong BPL baseline. It uses a grammar over expressions in first-order logic, plus predicates for shape, color, and size, totaling 28 primitives that were selected by the creator of the logical concept learning dataset (A.3.3). The BPL baseline uses the same memory-decay likelihood model, and fits (tunes) its prior by estimating probabilities for each logical primitive. It is implemented in Fleet [57], the state-of-the-art in fast parallelized MCMC over grammatically structured hypothesis spaces.
Our model differs in two important ways. First, the baseline is given first-order primitives that were chosen specifically for this dataset. While our model can use natural language expressing first-order concepts (e.g., the only for  $\exists!$ ), it can also express concepts like objects with the least common color that are unrepresentable by the baseline, and which are not in the dataset.
The second difference is that our model supports efficient inference via bottom-up proposals, while this baseline performs a stochastic search over hypotheses (MCMC), requiring many more samples. It takes  $10^{6}$  Metropolis-Hastings proposals per batch, and per learning curve, totaling  $\approx 10^{9}$  proposals, which are deduplicated to yield  $\approx 45,000$  unique concepts, which provide the support of subsequent posterior inference. This means every BPL posterior is informed by the total  $\approx 10^{9}$  sampling moves.
Results. Our model's predictions generally align well with human judgments (Fig. 5). Using 100 proposals per batch, our model explains  $81\%$  of the variance in human responses  $(R^2 = .81)$ , which is much higher than GPT-4 on its own. The model is also more accurate at the actual task than GPT-4, and within  $3\%$  of human-level (Fig. 6C). The model also fits the human data somewhat better than the BPL baseline, even when that baseline is granted an exorbitant sampling budget. Tuning the prior proves critical, possibly because these logical concepts are more complex than the earlier number concepts, and so can be expressed in a wider variety of syntactic forms. The pretrained model is highly sensitive to syntax and cannot learn to attend to semantic features, unlike the tuned model.
Figure 5: Model fits on holdout data. Error bars: ±SEM over 3 runs. (Error bars often close to zero)
Although our model explains most of the variation in the human responses, nontrivial variation remains. One possible reason is that we are modeling the responses of many human subjects, and different subject groups per concept, so it is difficult to capture the full variability of the human responses: Building a model that accurately reflects the judgments of a population of humans may be much more difficult, and require a higher sampling budget, than building a model of a single person. The model also slightly underperforms humans, and so a higher fidelity model might come from simply performing the task better. (Although one can slightly surpass human accuracy by optimizing purely for task performance, doing so degrades model fit. See Fig. 6C, 'tune for accuracy').) More fundamentally, the data come from many successive trials, so they likely contain memory and order effects such as garden-pathing [58] and anchoring [39] which are not well accounted for by the pure probabilistic inference our model approximates, nor by our simple memory decay model.
At the same time, our model does account for many fine-grained details in the behavioral data, including predicting specific patterns of successes and failures (Fig. 6A). Because our approach is manifestly interpretable—it explicitly verbalizes its hypotheses in human language—we can inspect its maximum a posteriori concept at the end of each learning episode, and observe that its successes typically occur because its verbalization of the concept describes the correct first-order law. Conversely, when humans make highly selective failures, we can probe the model to suggest what alternative hypotheses humans may have incorrectly inferred (Fig. 6A, top right).
D
Figure 6: A. Left and middle: learning concepts isomorphic to bachelor and valedictorian. Dark ticks delimit batches. Above each plot is the ground-truth logical expression and the predicted natural language description. Right: the model can explain human mistakes by verbalizing what other concept humans might have been misled into learning. Both humans and the model seem to have learned largest blue instead of the largest and it also happens to be blue. B. Further learning curves, including somewhat worse fits, and getting the right answer despite having a slightly odd solution (rightmost). C. The model is close to human performance after extracting a prior from human judgments. D. New concepts run on new participants. All results on holdout learning curves.
Modeling new concepts. To test the flexibility of our model, we created two new concepts for evaluating both the model and a new group of human participants. These two concepts were shapes with the majority color, and shapes with the least common color. 16 human subjects participated in an IRB-approved study (A.2). Our study finds that humans rapidly learn these concepts (Fig. 6D.)
We also test our model on these new concepts, but using the prior estimated earlier on the data in [59]. Our model correctly predicts that humans will learn the concept of majority color after just 2 batches of examples. It predicts learning minority color after 4 batches, while humans need just 3, but the primary result—that this concept is very learnable—holds for both humans and the model.
Although these two new concepts are easy to explain in words, and could be expressed in first-order logic with the right primitives—set cardinality and quantification over colors—neither concepts are learnable by the BPL baseline. This is because both concepts are simply unrepresentable: despite being equipped with 28 primitives, those primitives were designed without anticipating these new concepts. This highlights the fact that it is difficult to preprogram a sufficiently broad set of primitives that can efficiently encode all the different concepts people might learn.
Putting humanlike inductive biases into machines. Humans excel at rapidly mastering new tasks and understanding new concepts in part because they have a good inductive bias or prior [60, 61]. imparting similar priors upon machines is therefore an important problem. Our work gives a recipe for training such a prior by fitting it directly to human judgments, marginalizing out the natural language, meaning we never need to elicit natural language from human subjects. Because our approach simply tunes a small network on top of a large open-source pretrained model, and because it can be fit to raw human judgments, we hope that it can serve as a broadly applicable engineering strategy for extracting and using human priors. Recent work has also explored complimentary strategies for instilling a humanlike prior upon a machine learning system. For example, Kumar et al. 2022 [9] show that training neural networks with auxiliary linguistic tasks, and with auxiliary programming tasks, causes their representations to better align with human priors.
Bayesian Program Learning (BPL). Our model has similar motivations to BPL. Because we translate the natural language into Python to compute the likelihood, it is possible to see our approach as BPL with an unusual prior:  $p(\text{program}) = \sum_{\text{NL}} p(\text{NL}) p(\text{program|NL})$ . In that sense, our work provides a new prior for BPL, together with the demonstration that it is possible and practical to do BPL over a Turing-complete language like Python. Relatedly, recent BPL modeling has found synergies between natural language and program representations for cognitive AI models [9, 62].
Beyond BPL, combining probabilistic reasoning with expressive symbolic representations has long been an appealing paradigm [63, 64, 65], although the expressivity of the symbolic language must be balanced against the tractability of inference [66, 67]. Guiding inference with a neural model is a natural choice, but this is hard because of the lack of natural training data (though synthetic data can help: [68, 69, 70]). Encoding knowledge in natural language allows pretrained neural models to guide inference, and it could be fruitful to examine statistical-relational AI [71] in light of that fact.
Large Language Models. Our work suggests that an out-of-the-box large language model is not an effective approach to inductive reasoning, at least on its own. Bayesian mechanics are needed to dampen the unpredictability of the language model. To the extent that being Bayesian is a normative account of rational behavior, our work offers a framework for enabling language models to draw more rational inferences, and ultimately generalize in more predictable and human-like ways by tuning their prior beliefs to match human data.
The Language of Thought. Our work connects to the Language of Thought Hypothesis [72], which says that human learning and thinking relies on an inner symbolic language. This has been a productive framework for computational modeling [15, 16, 18, 73]. In its most literal forms, language-of-thought models are afflicted by the curse of a compositional mind (Spelke 2022 [74]): the free-form recombination of concepts yields a combinatorial explosion, which here we address by using pretrained knowledge from language to guide the learner. Whatever the true Language of Thought looks like, however, it must have a wide array of composable basic concepts in order to explain the breadth, flexibility, and generality of human thinking. Natural language, even if it is not actually the same as our inner mental language, acts as a vast reservoir of human concepts,
and provides a flexible algebra for combining them. Therefore a reasonable near-term strategy for modeling human thinking may be to use natural language as a heuristic approximation to an inner Language of Thought, despite strong evidence that human thinking need not engage language-processing brain regions [75].
Rational Process Modeling. As a rational process model, our account of concept learning bridges the computational and algorithmic levels of the Marr hierarchy [28, 29]: We commit to a Bayesian computational-level theory, and a particular Monte Carlo algorithm as a rational approximation to that theory. One important sense in which our account is inadequate is that we do not actually explain how the bottom-up process works, or how it came to be learned. We merely require that it is stochastic and unreliable, but occasionally correct enough to not need many millions of proposals. Given those requirements, a modern large language model is a reasonable surrogate for this bottom-up process, even if its inner workings might differ greatly from human bottom-up proposal processes.
Generalizability of the theoretical framework. The basics of the model make few commitments, yet instantiating it requires selecting specific language models, engineering prompts and likelihoods, etc. More broadly, a high-resolution cognitive model, particularly a structured Bayesian one, requires domain-specific modeling choices. How much credit should we assign to the general theoretical framing, as opposed to particular engineering decisions? Although our paradigm introduces new degrees of freedom, such as which LLMs/prompts to use, it removes others, such as the grammatical structure of the symbolic hypothesis space. On balance, we are cautiously optimistic that the framework will generalize with less domain-specific tinkering, at least for abstract symbolic domains, because it replaces hand-designed symbolic hypothesis spaces with pretrained neural models, and because reasonable "default" neural networks worked well across our experiments.
Inference, Natural Language, and Recursive Reasoning. There is a rich literature on natural language and probabilistic inference from a social or communicative perspective, such as modeling pragmatic inference [76, 77, 78, 79]. These models often use a recursive reasoning formulation where each agent represents the beliefs of other agents, including beliefs about themselves, and so on, recursively. Our work does not consider the communicative function of natural language, and so does not engage with these issues. However, models of pedagogical inductive learning—where a helpful teacher selects the training examples—benefit from similar forms of recursive reasoning, because the teacher and learner must reason about each other's beliefs [80]. These insights could be important for building versions of our model that might simulate aspects of how humans learn from each other.
Limitations. Our model performs induction via discrete structure learning. Given the combinatorial difficulty of structure learning, it is unclear whether our approach can scale to inferring complex systems of symbolic rules. We believe recent work on iteratively refining language model outputs may be promising here [81, 82]. The present form of our model is also limited to processing discrete symbolic input-outputs. Actual human thinking connects with the messy perceptual world. It would be valuable to understand whether this limitation can be addressed using multimodal language models [83], or approaches that interface separate language and vision modules [84, 85]. It is worth noting that BPL models can straightforwardly interoperate with perceptional data [10, 16, 18], and that many outstanding challenge problems within AI have at their core a perceptual-reasoning process, such as Bongard problems [86] and the Abstraction and Reasoning Corpus [1].
Currently, the model relies on costly, energy-intensive models for its proposal distribution, a constraint that might be mitigated by open-source models and network compression [87].
Last, a strong justification for formal representations is that they allow specifying knowledge precisely and unambiguously [88]. Natural language is usually imprecise and ambiguous, which we deferred addressing by translating language into Python. It remains to be seen whether language models can be coaxed into producing sufficiently precise language to support representing knowledge solely in natural language, or if refinement into precise languages like Python offers the better scaling route.
Code and data available at: https://github.com/ellisk42/humanlike_fewshot_learning
Acknowledgements. We are grateful to Steven Piantadosi for providing the raw human data and Fleet results for the logical concepts, as well as Joshua Tenenbaum for providing his Number Game data, and Mathias Sablé-Meyer for comments on the manuscript and work.
[89] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
A.1 Supplemental Results
Fig. 7 illustrates model predictions across every Number Game concept in [43].
Figure 7: Model predictions across every Number Game concept in [43], using  $1000q$  samples
Recall that we deduplicated the proposals instead of performing actual importance sampling. Fig. 8 contrasts model fit for importance sampling and dedduplication. We originally did dedduplication simply because importance sampling is not possible with GPT-4, and GPT-4 proved necessary for the logical concepts. On number concepts we used code-davinci-002, from which we can construct an importance sampler because it exposes the log probability of its samples. On number concepts dedduplication provides a fit that is on-par (actually slightly better) compared to importance sampling (Fig. 8).
Figure 8: Monte Carlo inference using dedduplication instead of importance sampling does not harm model fit to human data. The above figures show Number Game models using a learned prior and 100 samples, and show predictions only on holdout data.
We also studied the effect of replacing the closed-source code-davinci-002 (Codex) with the open-source 70B LLaMA2 [89]. This helps in understanding if open LLMs can be used to build models like ours. Additionally, as the open models are presumably weaker, swapping out Codex for LLaMA2 serves as a way of softly ablating either the likelihood or proposal distribution, both of which use an LLM. We find that the (presumably weaker) LLaMA2 can substitute fully as a likelihood. Using LLaMA2 as a proposal distribution impairs model fit in the low-sample regime (Fig. 9), but with enough samples, the weaker model comes close to 'catching up'. This highlights the fact that an explicitly Bayesian reasoning process can compensate for deficiencies in the LLM, given enough Monte Carlo samples.
Figure 9: Comparing versions of our model that swap out the closed-source Codex model for the open-source LLaMA2 70B model, for both the likelihood and the proposal distributions.
A.2 Human Study
16 participants were recruited primarily through a Slack message sent to a channel populated by members of our academic department. Participants had an average age 28.1 (stddev 13.6, all over
18), and were 7 male/5 female/1 nonbinary/3 declined to answer. Participants were randomly split between the concepts of most common color / least common color. Each participant went through 15 trials, and took an average of 294s to complete those 15 trials. In exchange for participating in the study, participants received $10 in Amazon gift cards. Fig. 10 illustrates the web interface shown to our human participants, including the cover story.
Trial 1: Please read these instructions carefully
You are going to attempt to learn the meaning of a new word in an alien language, which the aliens call "Wudsy." On each trial, you are going to see a collection of shapes at the bottom of the webpage, and your job is to select which ones you think are "Wudsy." Afterward, the aliens tell you which shapes are "Wudsy."
The meaning of the word Wudsy is the same during the whole experiment. However, it is possible that whether something is Wudsy depends on what other shapes it is in the context of. Wudsy may or may not correspond to an English word.
To start with, no one has given you any examples of what counts as "Wudsy." So just do your best below and pick which ones you think might belong to the concept called "Wudsy." Right after you do so, the aliens are going to label the Wudsy objects by drawing a black box around them, and then you are going to get another round of guessing which objects are "Wudsy."
You will go through 15 trials of guessing what counts as Wudsy. Remember that the meaning of Wudsy does not change during the experiment, but it might depend on the other shapes in the collection.
(trial 1/15) Click yes on the objects that you think are Wudsy, and No on the other objects. Then click Next.
Figure 10: Cover story and web interface for our version of the logical concept learning study, which is based on [59]
A.3 Modeling
A.3.1 Temperature and Platt Transform
Adding a temperature parameter  $T$  to a model corresponds to computing the posterior via
$$
\begin{array}{l} p _ {\text {T e m p}} \left(X _ {\text {t e s t}} \in C \mid X _ {1: K}\right) \approx \sum_ {C \in \left\{C ^ {(1)}, \dots , C ^ {(S)} \right\}} w ^ {(C)} \mathbb {1} \left[ X _ {\text {t e s t}} \in C \right], \text {w h e r e} \\ w ^ {(C)} = \frac {\left(\tilde {w} ^ {(C)}\right) ^ {1 / T}}{\sum_ {C ^ {\prime}} \left(\tilde {w} ^ {(C ^ {\prime})}\right) ^ {1 / T}} \text {a n d} \tilde {w} ^ {(C)} = p (C) p \left(X _ {1: K} | C\right) \mathbb {1} \left[ C \in \{C ^ {(1)}, \dots , C ^ {(S)} \} \right] \tag {10} \\ \end{array}
$$
Adjusting the predictions of a model using a Platt transform corresponds to introducing parameters  $a$  and  $b$  which transform the predictions according to
$$
p _ {\text {P l a t t}} \left(X _ {\text {t e s t}} \in C | X _ {1: K}\right) = \text {L o g i s t i c} (b + a \times \text {L o g i s t i c} ^ {- 1} \left(p \left(X _ {\text {t e s t}} \in C | X _ {1: K}\right)\right)) \tag {11}
$$
For the number game, every model has its outputs transformed by a learned Platt transform. This is because we are modeling human ratings instead of human responses. We expect that the ratings correspond to some monotonic transformation of the human's subjective probability estimates, and so this transformation gives some extra flexibility by inferring the correspondence between probabilities and ratings. Logical concept models do not use Platt transforms.
A.3.2 Parameter fitting
Training consists of fitting the parameters  $T$ ,  $\theta$  (for the prior),  $\epsilon$  (for the likelihood),  $\alpha$  and  $\beta$  (for the logical concepts likelihood), and Platt transform parameters  $a$ ,  $b$  (for the Number Game). In practice, this amounts to around 400 parameters, almost all of which come from  $\theta$ .
We fit these parameters using Adam with a learning rate of 0.001. We perform 1000 epochs of training for the Number Game, and 100 epochs for logical concepts. There is a tenfold difference in the number of concepts, so this way they take about the same number of gradient steps.
For the number game we do 10-fold cross validation to calculate holdout predictions. For logical concepts we use the train-test split introduced in [59], which involves running different groups of human subjects on each concept twice, with different random examples. One sequence of random examples is arbitrarily designated as training data, and the other as holdout data.
All model were trained on a laptop using no GPUs. Training takes between a few minutes and an hour, depending on the domain and the model.