In-Context Learning Dynamics with Random Binary Sequences
Authors: Eric J Bigelow
Ekdeep Singh Lubana
Robert P. Dick
Hidenori Tanaka
Tomer Ullman
Venue: ICLR 2024 poster (2024)
Keywords: In-Context Learning
Large Language Models
Interpretability
Computational Cognitive Science
Primary area: visualization or interpretation of learned representations

Abstract: Large language models (LLMs) trained on huge text datasets demonstrate intriguing capabilities, achieving state-of-the-art performance on tasks they were not explicitly trained for. The precise nature of LLM capabilities is often mysterious, and different prompts can elicit different capabilities through in-context learning. We propose a framework that enables us to analyze in-context learning dynamics to understand latent concepts underlying LLMs’ behavioral patterns. This provides a more nuanced understanding than success-or-failure evaluation benchmarks, but does not require observing internal activations as a mechanistic interpretation of circuits would. Inspired by the cognitive science of human randomness perception, we use random binary sequences as context and study dynamics of in-context learning by manipulating properties of context data, such as sequence length. In the latest GPT-3.5+ models, we find emergent abilities to generate seemingly random numbers and learn basic formal languages, with striking in-context learning dynamics where model outputs transition sharply from seemingly random behaviors to deterministic repetition.

Published as a conference paper at ICLR 2024
IN-CONTEXT LEARNING DYNAMICS WITH RANDOM BINARY SEQUENCES
Eric J. Bigelow1.2 1.2 Ekdeep Singh Lubana 2.3
Robert P. Dick Hidenori Tanaka Tomer D. Ullman' 1.2
PPyyholoogy Department, Harvard University, Cambridge, MA, USA
"Center for Brain Science, Harvard University, Cambridge, MA, USA
&ECCS Department, University of Michigan, Ann Arbor, MI, USA *Physics & Informatics Laboratories, NTT Research, Inc., Sunnyvale, CA, USA
ABSTRACT
Large language models (LLMs) trained on huge text datasets demonstrate intriguing capabilities, achieving tfate-of-the-ar performance tasks they were not explicitly trained for. The precise nature LLM capabilities is often mysterious, and different prompts elicit different capabilities through in-context learning. We propose framework that enables us to analyze in-context learning dynamics to understand latent concepts underlying LLMs' behavioral patterns. This provides more nuanced understanding than success-or-faill failure evaluation benchmarks, but does not require observing internal activations as a mechanistic interpretation of circuits would. Inspired by the cognitive science of human randomness perception, we use random binary sequences as context and study dynamics of in-context learning by manipulating properties of context data, such as sequence length. In the latest GPT- 3.5+ models, we find emergent abilities to generate seemingly random numbers and learn basic formal languages, with striking in-context learning dynamics where model outputs transition sharply from seemingly random behaviors to deterministic repetition.
INTRODUCTION
Large language models (LLMs) achieve high performance on wide variety tasks they were not explicitly trained for, demonstrating complex, emergent capabilities (Brown et al., 2020: Radford et al., 2019; Wei et al., 2022a; 2023; 2022b; Lu et al., 2023; Bommasani al., 2021; Wei al., 2021; Kojima et al., 2022; Pan et al., 2023; Pallagani et al., 2023; Li et al., 2022). In-context learning (ICL. describes how task-specific patterns of behavior are elicited in L.LMs by different prompts (or contexts) (Brown et al., 2020; Wei et al., 2022b; Dong et al., 2022; Zhou et al., 2022: Wang et al., 2023; Shin et al., 2022: Min et al., 2022: et al., 2023; Xie et al., 2022: Zhang et al., 2023b). Although weight updates occur in ICL, different input contexts
activate, or re-weight, different latent algorithms in an LLM, analogous to updating model parameters to learn representations in traditional gradient descent learning methods (Goldblum al., 2023; Dai et al., 2023; Ahn et al., 2023; Von Oswald et al., 2023; Akyürek et al., 2022: Li et al., 2023a). However, two seemingly equivalent prompts can evoke very different behaviors in LLMs (Si et al., 2023). ICL makes interpreting what representations in an LLM are being evoked by given input more challenging than with task-specific language models, where the space of capabilities is more restricted and well-defined.
Our central motivation is to interpret latent concepts underlying complex behaviors in LLMs by analyzing in-context learning behavioral dynamics, without directly observing hidden unit activations or re- re-training models on varied datasets. Inspired by computational approaches to human cogni- tion (Tenenbaum, 1998; Griffiths Tenenbaum, 2003; Piantadosi et al., 2012; Spivey et al., 2009), we model and interpret latent concepts evoked in LLMs by different contexts without observing or probing model internals. This approach, which we call Cognitive Interpretability, is a middle ground between shallow test-set evaluation benchmarks one hand (Srivastava et al., 2022; Saparov & He, 2022; Min et al., 2022; Lanham et al., 2023; Ganguli et al., 2023; Bowman et al., 2022; Kadavath
"Correspondence to: eb ebigelow8g hateelowwghhhaaarre..r hww.arrraa... edu 'Equal contribution

---

Published as a conference paper at ICLR 2024
Tosk: Generate a sequence of 1000 random number samples from fair coin:
Input Context
0010010010
001001001001001100
p(h (x)
LM Concept
H
Space
ply (4)
Output
01000101 10010111000011
OT 101000111 120101001
00100100000 10010 10100010
Subjective Randomness
Formal Language Learning
Figure 1: Overview of our modeling framework. Given a pre-trained Large Language Model, we systematically vary input context prompts, in this case r (001)" with n {2, 43 (001 001, 001001001001), LLM outputs (Bottom) significantly as a function of x (Top), based on some unknown latent concept space embedded in the LLM. We model a subset of the algorithms embedded in the LLM's latent concept space H that are invoked by ICL. (Middle), to predict the LLM outputs y. With very little context (Left), GPT-3.5+ generates subjectively random sequences, whereas with enough context matching a simple formal language (a = 0010010010C1 = (001) it begins deterministically repeating patterns in x.
al., 2022; Perez et al., 2022; Prystawski & Goodman, 2023; Turpin al., 2023) and mechanistic neuron- and circuit-level understanding of pre-trained model capabilities on the other (Nanda et al., 2023; Goh et al., 2021; Geva al., 2020; Belinkov, 2022; Li et al., 2022; Wang al., 2022: Chughtai et al., 2023; Gurnee et al., 2023; Foote et al., 2023; Lubana et al., 2023; Lieberum et al., 2023; Barak al., 2022; Liu et al., 2022b; Zhong et al., 2023) (also see App. C). Unlike related work of feature interpretability in vision neural networks, we analyze ICL and text generation dynamics over output sequences, and unlike mechanistic work studying ICL, we observe learning dynamics in black box input-output behavior of state-of-the-art LLMs.
Specifically, we focus on interpreting latent concepts in LLMs the domain of random sequences of binary values (Fig. 1). Random binary sequences extensively in statistics, formal language theory, and algorithmic information theory (Sipser, 1996; Chaitin, 1966; Li et al., 1997). We use this domain to systematically test few-shot learning as a function of context length |x| across different binary token sequences x. Moreover, language generation trajectories over binary sequences can also be analyzed and visualized more easily than typical user-chatbot interaction trajectories (Zhang et al., 2023a; Berglund et al., 2023) since the token-by-token branching factor is only two. Random binary sequences have also been target domain in cognitive science (specifically, subjective randomness), where researchers have studied the mechanisms and concepts that underlie how people generate random binary sequences or evaluate the randomness of a sequence (Falk & Konold, 1997; Tversky& & Gilovich, 1989; Griffiths et al., 2018).
minimal domain that has been studied
Similar to computational cognitive theories of human concept learning (Chater Vitányi, 2003; Dehaene et al.. 2022; Piantadosi et al., 2012; Goodman et al., 2008; Ullman et al., 2012: Ullman & Tenenbaum, 2020), we argue that ICL. can interpreted as under-specified program induction, where there is no single "correct" answer; instead, an L.MM should appropriately re- re-weight latent algorithms. The domain of random sequences reflects this framing, in contrast to other behavioral evaluation methodologics, in that there is correct answer to a random number generation or judgment task (Fig. 3). If the correct behavior is to match a target random process, then the best way to respond to the prompt Generate N flips from a fair coin is uniform distribution over the tokens Heads and Tails, ss. instead of a specific ground truth token seeeenseuueeee sequence. If instead the correct behavior to match how humans generate random sequences, the LLM must represent more complex algorithm that matches the algorithm used by people. Overall, we make the following contributions in this work.
Evaluating ICL as Bayesian model selection in a practical setting (Sec. 2). We find evidence of in-context learning operating as Bayesian model selection in LLMs in the wild, compared to prior work that trains smaller toy neural networks from scratch.
Subjective Randomness
domain for studying ICL dynamics (Sec. 3), We introduce new domain, subjective randomness, which is rich enough to be theoretically interesting, but simple enough to enable analysis of complex behavioral patterns.
2

---

Published as a conference paper at ICLR 2024
Unnormalized hypothesis posterior pl (01)'
Next-Flip Prediction pl ym Tails [x=(01)")
0.30
Non- Nundom (01)"
10
Semouli
0.25
7
0.10
Averaging
Modal Selecion (Mad
0.85
Model Selection (Soft)
0.5
160
[+]
Figure 2: Sharp changes in the predictive distribution p(y]ær) suggest model selection (Left) Here we show an illustrative example of simple Bayesian model with two hypotheses in H: a random Bernoulli hypothesis, and a deterministic Non-Random hypothesis for the concept (01)". When hypothesis posteriors p(h(z)) cross as function of additional data, i.e. larger |a] the predictive distribution p(y|=:) for model selection suddenly transitions from one pattern of behavior to another (Right). With model averaging, instead there is steady, gradual change in p(y]ær (see App. D for further explanation). We find analogous S-shaped curves in ICL with L.M predictive distributions p(y(a)) in GPT-3.5+, suggesting model selection rather than model averaging (Fig. 4, 5). These phase changes are not predicted by theories of ICL as linear regression (Akyürek et al., 2022) or few-shot learning (Brown et al., 2020; Wei et al., 2022b), where loss steadily decreases with more context examples in 2.
Sharp transitions in abilities to in-context learning dynamics (Fig. 4, 5). We systematically analyze in-context learning dynamics in LLMs without observing updating model weights, demonstrating sharp phase-changes in model behavior. This is a minimal, interpretable example of how the largest and most heavily fine-tuned LLMs can suddenly shift from one pattern of behavior to another during text generation, and supports theories of ICL as model selection.
IN-CONTEXT LEARNING AS MODEL SELECTION
Bayesian inference is key methodological tool of cognitive modeling, and recent work has framed in-context learning as Bayesian inference over models (Xie al., 2022: Li al., 2023b; Hahn & Goyal, 2023). The output tokens y an LLM produces given input tokens x can be modeled as a posterior predictive distribution p(y| that marginalizes over latent concepts e in the LLM: p(vl Scec P(ylc) p(c|zr In other words, a context will activate latent concepts c in an LLM according their posterior probability p(c|z). which the model marginalizes over to produce the next token y by sampling from the posterior predictive distribution. This inference process takes place in network activation dynamics, without changing model weights.
In our experiments, we assume a hypothesis space H. that approximates the latent space of LLM concepts used when predicting the next token, i.e.,
x(yl=) Σ p(ul))) p(hlx)) NCH
We assume that an LLM operates over next-token prediction only, and so each generated token yr- will be appended to the context when the following token is sampled, i.e. 2; (2o,....t-1, (/-1)) 1) With this framework, we can understand ICL dynamics through the lens of hypothesis search. An LLM might not compute the sum in the above equation by taking a weighted average of latent capabilities. An LLM might perform Bayesian model selection, instead of model averaging, where for any given x, only a single latent concept h approximates the posterior predictive distribution: h" argmax, p(a(z), p(y[z) p(ylh"). We provide a visual demonstration of how Bayesian model selection leads sharp changes in the predictive distribution p(y|2²) with a toy model in Fig. 2.
Cognitive scientists have used Bayesian predictive and posterior distributions to model learning dynamics in human adults and children, as well as non-human animals (Tenenbaum, 1998; Goodman et al., 2008; Ullman& & Tenenbaum, 2020). A discrete hypothesis space in probabilistic model can give clear and meaningful explanations to learning patterns analogous to mode collapse and phase transitions in deep learning. When behavior suddenly shifts from one pattern, mode, of behavior to another, this can be understood as one hypothesis coming to dominate the posterior p(h[z) as grows in scale. This theory can explain conceptual shifts observed when children learn intuitive theories of domains such as intuitive physics (Ullman et al., 2012) and learning to count (Piantadosi et al., 2012). Such cognitive analysis of behavioral shifts generated from shift in
3

---

Published as a conference paper at ICLR 2024
posterior probability between hypothesis to another parallels recent work on learning dynamics in training and prompting LLMs. Sharp phase changes have been observed when training neural networks, corresponding to the formation of identifiable mechanisms such as modular addition circuits or induction heads (Nanda et al., 2023; Olsson et al., 2022; Zhong et al., 2023; Bai et al., 2023), and can be interpreted as Bayesian model selection (Hoogland et al.). Also see App. A, C.
To our knowledge, we provide the first demonstration of such sharp phase changes in ICL, particularly with state-of-the art blackbox LLMs. The minimal model used by Xie al. (2022), as well as other few-shot learning work (Brown et al., 2020), show linearly or logarithmically decreasing loss curves in ICL, rather than S-shaped curves with stable plateaus as we find. Michaud et al. (2023) propose that neural network knowledge and skills may be quantized into discrete chunks, and so training might consist of many such phase changes which disappear into a smooth curve when test loss is averaged over many samples. We propose that In-Context Learning may also consist of sudden phase changes, as well as other complex learning dynamics, which may be explained as search over a discrete latent concept space. Such phase changes may occur in typical few-shot learning domains (Wei et al., 2022b; Brown et al., 2020), where smooth, continuously decreasing loss curves might hide complex ICL dynamics corresponding to shifts in the latent algorithms driving an LLMs behavior.
3 ALGORITHMIC AND SUBJECTIVE RANDOMNESS
Next, we describe the domain of randomness explored in this work, and provide relevant background from computational cognitive sci- ence. To build intuition, consider: which se- quence of coin flips is more random, TTHTHHTH or HITTTTTT? While the odds of a fair coin producing either sequence are the same (1 in 256), most people say that the first sequence "looks more random". In bias termed the Gam- bler's Fallacy, people reliably perceive binary sequences with long streaks of one value as "less random', and judge binary sequences with taatter-taa-cchamme alternation rates as 'more random' than truly random sequences (Falk& & Konold, 1997). In addition to not using sim- ple objective measures when judging the ran- domness of sequences, people also have a hard time generating sequences that are truly ran- dom (Oskarsson et al., 2009; Griffiths et al., 2018; Gronchi & Sloman, 2021; Meyniel et al., 2016 6; Planton et al., 2021).
Problems
Pypathedk Space
Pastwrio Inference
G. Which sequence
move rindoo??
pradge prwaec))
mangelty
simple
TTHTHHTH
complex
pris())
medom
p(h
petsige)
fort mnninn
TTHTIBITH
HITIEIIT
(eyadonaaaaeei)) prea))
Figure 3: Determining whether sequence is random can be viewed as search for sim- ple program that could generate that sequence. HTTTTTTT can be described with a short determin- istic program s simpleSequence, and has a higher p(h) according to simplicity prior, compared to TTHTHHTH and complexsequence. Both se- quences can be generated by randomSequence, but with lower likelihood p(
Cognitive scientists studying Subjective Randomness model how people perceive randomness, or generate data that is subjectively random but algorithmically non-random, and have related this problem algorithmic information theoretic definitions of randomness. One way study this is to ask people whether a given data sequence was more likely to be generated by a Random process or a Ron-Random process (Fig. 3). HTTTTTTT might be described with a shorter program-one followed by seven Ts- - than HTTHTHHT, which cannot be compressed as well followwed by two Ts, then one H, then one T, then two Hs, then one T. Viewing this task as program induction, natural prior p(h) over program hypotheses assigns higher weights to programs with shorter description lengths, or lower complexity. Similar to our deseription of model selection, or hypothesis search, compared to model averaging, the space of non-random programs can be simplified to search for the single shortest program consistent with the data x. This search is equivalent to computing the Kolmogorov complexity of a sequence K(x) (Li et al., 1997) and has motivated the of "simplicity priors" in a number of domains in computational cognitive science Chater & Vitányi (2003); Goodman et al. (2008); Piantadosi et al. (2012).
As in Griffiths & Tenenbaum (2003); Griffiths et al. (2018), we define subjective randomness of sequence as the ratio of likelihood of that sequence under a random versus non-random model,
4

---

Published as a conference paper at ICLR 2024
1.0
CONOOUE
Generiee 1000 randorr samples megrited car,, with (t oit probatillt of Heads and gath prrbsiiiity of Taik
ply 0ly-1om Bandos [x]
0112
(seeuennc))
the tollowing secuence coin fips gsnersted by namdom procesa with pullem they genemtted by a nom-andom aigorii?? (jequennnt)
0.2
A sequarce mas gereented by
0.0
10
15
20
35
40
45
50
60
65
70
75 90
95
100
Sequence length
Figure 4: Prompt templates and GPT-3.5 Randomness Judgments (Left) Prompt templates used for the Randomness Generation (top) and Judgment (bottom) tasks. (sequence) is substituted with a list of context flips z, e.g. Heads, Tails, Tails, Heads. {p} is substituted with the probability of Tails, and {1 pt with the probability of Heads. (Right) In Randomness Judgment tasks, we observe sharp transitions in the predictive distribution pyy random] (z) for GPT-3.5, from high-confidence in x being generated by a random process, to high confidence in a non-random algorithm, as additional data is provided from the same concept. See App. G for additional concepts.
i.e, randomness( (z) = log P(z|random) - log ((x non-random). The likelihood given truly random Bernoulli process is p(ar) (rrddd))))) 2-1x1, since each of 2 binary sequences of length |ar has equal probability. The non-random likelihood p(z/nooo-a)ddmm
2-K(a) is the probability of the minimal description length program that generates æ, equivalent to Bayesian model selection: (on-rromr)))mm
maX p(=[h) p(h). See App. K for more background. In theory, the space H of non-random models encompasses every possible computation; in this work, we study a small subset of this space that includes formal languages and probabilistic models inspired by psychological models of human concept learning and subjective randomness (Griffiths et al., 2018; Nickerson, 2002; Hahn & Warren, 2009). Specifically, consider Bernoulli processes, regular languages, Markov chains, and simple memory-constraine probabilistic model as candidates for H.
4 EXPERIMENTS
Our experiments aim to evaluate how LLMs operate in the domain of Subjective Randomness. Specifically, our goal is to understand whether ICL and text generation dynamics suggest model selection, or hypothesis search, over a discrete concept space. We first evaluate whether there an emergent algorithm for generating subjectively random sequences, and if so, whether it is similar to cognitive models of human subjective randomness (Fig. 6. 7). We use open-ended text generation experiments (Fig. 4, Left) and analyze binary sequences produced by LLMs, and compare L.MM sequences to a Bernoulli distribution and our cognitively inspired (Falk & Konold, 1997: Hahn Warren, 2009) Window Average model. Second, we evaluate whether ICL. dynamics in searching over random and non-random hypotheses demonstrate ICL operating as model averaging, where learning is smooth and continuous, or as model selection, or hypothesis search, where sudden, dramatic changes can occur during learning.
Randomness Generation and Judgment Tasks In order to assess text generation dynamics and in-context concept learning, we evaluate LLMs on random sequence Generation tasks, analyzing responses according to simple interpretable models of Subjective Randomness and Formal Language Learning. In these tasks, the model generates sequence y of binary values, or flips, comma-separated sequences of Heads or Tails tokens. We also analyze a smaller set of randomness Judgment tasks, with the prompt: Is the following sequence of coin flips generated by a random process with no pattern, or are they generated by a non-random algorithm?. In both cases, y distribution over tokens with two possible values. Outputs y consist of a single Random or Non token, indicating whether the sequence was generated by a random process with no correlation, or some non-random algorithm. We analyze dynamics in L.M--generate sequences simulating weighted coin with specified p(Taiis), with [a| 0. An initial flip 'Heads' is used so the model's flips will be formatted consistently. Prompt context in these experiments includes specific probability, shown in Fig. 4, where the last marks where the model begins generating tokens y. We collect 200 output sequences y for each LLM at each P(Tails) E [.05, 1, 2, 33, 4, .49. 5, 551,60,, .60, 770, .80, -90, .95], cropping output tokens to yy = 50 to limit cost and for simplicity. Temperature parameters are set to 1, and other parameters follow OpenAl API defaults; see App. B for more details.
In the Judgment task, x includes the prompt question and the full sequence of flips. We systematically vary prompt context by varying the number of flips, denoted |z]] We test few-shot learning in L.LMs
5

---

Published as a conference paper at ICLR 2024
1.0
0900
0033
2.5
0150
71
320
010
0.2
1B
0131
1020
1.0
pyEC
13
1
180
101
0.6-
Cancept
15
800
1100
230
211
02
10
15
20
25
30
35
40
Contact Length
Figure 5: With enough context, GPT-3.55 learns simple formal language concepts, transitioning from generating subjectively random sequences to only generating values matching the concept (Left) GPT-3.5 next-token predictive distribution p(y (z)) visualized as a binary tree, where red arrows correspond to Heads (0), blue arrows Tails (1), and nodes matching the target concept C = (011)" are dark green. p(y|zr) changes with varying (z]. here |ac| = 39 and next-token predictions strongly follow the concept C. Also Fig 15, 16 in Appendix. (Right) In-context learning dynamies for simple formal languages = (HTH)" " (010) and x = (HTT)" (011) as function of context length = |r] Prediction accuracy computed as the total probability mass of p(y(z)) assigned valid continuations of the formal language x, as a function of prediction depth d = ly/ and context length |z|- Curves shown are for depth d = 6, where only 3 out of 64 possible length-6 binary sequences y match concept C. Note: bottom model is 3tt-3.5-tuu 5--5-tturbbr wnteiiiintttttttieeeoeee 0914. Also see App. F.
by evaluating output behavior on specific bit sequences, for example (01)" with varying n (e.g. "Heads, Tails, Heads, Tails, ..."). as well as zero-shot language generation dynamics when is empty and |x|| = 0 (in practice, we initialize x with a single flip to help the L.L match the correct format). Unlike typical cases of few-shot learning, where high-variance samples ('shots') are drawn from a broad category such as "chain-of-thoug""
solutions" (Wei et al., 2022b; Brown et al., 2020), all data in a sequence x corresponds to a single unobserved program.
Subjective Randomness Models We compare LLM-generater sequences to a ground truth "ran- dom" Bernoulli distribution with the same mean (μ TLLM), to simple memory-consstriiee probabilistic model, and to Markov chains fit to model-generated data y. The Gambler's 's Fallacy is the human bias to avoid long runs of a single value ('Heads' or "Tails") consecutively in row (Falk & Konold, 1997; Nickerson, 2002). Hahn & Warren (2009) theorize that the Gambler's Fallacy human memory limitations, where 'seeming biases reflect the subjective
emerges as a consequence experience of finite data stream for agent with a limited short-term memory capacity'. We formalize this as a simple Window Average model, which tends towards a specific probability p as a Tz-w....)))
function of the last flips: p(y|z)) max(0,
Sub-Sequence Memorization and Complexity Metrics Bender et al. (2021) raise the question of whether L.MS are 'stochastic parrots' that simply copy data from the training set. We test the extent to which L.Mss generate subjectively random sequences by arranging memorized patterns of sub-sequences, rather than according to more complex algorithm as our Window Average model represents. To measure memorization, we look at the distribution of unique sub-sequences in If an LLM is repeating common patterns across outputs, potentially memorized from the training data, this should be apparent in the distribution over length K sub-sequences. Since there are deep theoretical connections between complexity and randomness (Chaitin, 1977; 1990). we also consider the complexity of CPTT-poodcedd sequences. Compression is metric of information content, and thus of redundancy over irreducible complexity (MacKay, 2003; Delétang et al., 2023), and neural language models have been shown to prefer generating low complexity sequences (Goldblum et al., 2023). As approximations of sequence complexity, we evaluate the distribution of Gzip-compressed file sizes (Jiang al., 2022) and inter-sequence Levenshtein distances (Levenshtein et al., 1966).
Formal Language Learning Metrics In our Formal Language Learning analysis, x is a subset of regular expression repetitions of short token sequences such (011)" where longer sequences x correspond to larger n. This enables us to systematically investigate in-context learning of formal languages, |ar| corresponds to the amount of data for inducing the correct program (e.g. (011))

---

Published as a conference paper at ICLR 2024
Bemoulll BOXE
Rurning Avg. Generelie
2.0
0.8- 0.6
0.4
02-
c.a
10
30 40
Output Ind 1or%
Figure 6: GPT-335 generates subjectively random binary sequences that deviate from Bernoulli process. (Left) Running averages for flip sequences generated by each model, where denotes 'Heads' and denotes 'Tails'. Compared to Bernoulli process (top), sequences generating using GPT (middle) and those of our Window Average model (bottom) stay closer to the mean, repeating the same patterns more often. (Right) Empirical conditional probabilities for third-order Markov Chain fit to sequences y generated by GPT-3.5 text-daving cetavinnni00.... a Bernoulli process centered the mean of GPT sequence and our Window Average model (w 5). In the simulated Bernoulli process, edges are fairly uniform; edges for GPT-3.5 and the Window Average model are non-uniform, but similar one another.
from Bernoulli processes, as well as the space of all possible algorithms. A Markov chain of sufficiently high order & can implicitly represent regular language its weights (see Fig. 6, Right). However, the number of parameters required is exponential in Markov order k; and weights are difficult to interpret. For example, 3-order Markov chain fit to data = (011) 1)) will represent (011)* corresponds to high weights in the conditional probability distribution for p( ((0[11), p(2101), p(1[10), and low values for other conditional probabilities. To match the goal of the Generation task, we use variation of the prompt in Fig. 4: "Generate a sequence of 1000 samples that may be from fair coin with no correlation, or from some non-random algorithm'
In Randomness Judgment tasks, we assess formal concept learning dynamics by changes in the predictive distribution p(y random| = Cl=l), i.e. the probability of the final token being 'Random' vs. 'Non', as function of context length |z/]. The method for measuring concept learning dynamics in Randomness Generation tasks requires computing p(y|z) over possible binary flip sequences instead of y being single token as with Judgment tasks. In binary sequence Generation tasks, we estimate the predictive probability p(u E C/2,, Vo....t11) assigned to a given regular language concept C' by computing the total probability mass of p(y) x)) for all unique token sequences yo...a up to some depth d. such that the exactly matches a permutation of the concept. If tokens in x and y have two possible values, or 1, the probability table for p(ye (9......11) is a weighted binary tree (see Fig. 5, Left) with depth d, where edges represent the next-token probability p(y)(Bt-1), and paths represent the probability a sequence of tokens. For example, with = (011)". there will be paths yo...d corresponding
the permutations (011)" (101)². (110)², x)) ply = '1, 1, O. 1, 1, ..... .)) + p(y = "1, O, 1. 1, 1, ... x) We use the notation p(y C C Fig. 5 (Right) refer this predictive probability of y matching a concept C. In both Judgment and Generation tasks for formal concept learning, we use next-token prediction probabilities p(y (o.....-1..)) .)) directly from the OpenAl API, for cost efficiency and since this is equivalent to repeated token samples.
of 2d possible sequences y, and
so: p(y E (011)" [ x) ply = '0, 1, 0, 1,
RESULTS
5.1 SUBJECTIVELY RANDOM SEQUENCE GENERATION
In the GPT3.5 and higher models-IIstrueiGGP (text sont-ddvvncii-000 003),, ChatGPT and GPT-4-we find an emergent behavior of generating seemingly random binary sequences (Fig. 6). This behavior is controllable, where different p(Tails) values lead to different means of generated sequences J. However, the distribution of sequence means, as well as the distribution the length of the longest runs for each sequence, deviate significantly from Bernoulli distribution centered y, analogous the Gambler's Fallacy bias in humans (see Fig. 7).
(.pt-3.5-turl
7

---

Published as a conference paper at ICLR 2024
P[Tails) 0.1
P[Talis) - 0.51
P(Talis) 0.9
20
20
Jon: conti
Denoity
10
Serncul
Window Average
10-
0
0.4
1.0
0.0
0.6
0.8
Sequence Mean
0.15
0.75
Densty
0.05
0.05
0.00
0.00
0.00
Length Longest Run
Figure 7: GPT-3.5 shows a Gambler's fallacy bias of avoiding long runs. (Top) Distribution of mean values flip sequences (μ E, 3/) generated by GPT-3.5 the specified p(Tails), compared with a Bernoulli process and our Window Average model with the same mean as the GPT-3.5 flips. Flips generated by GPT approximately follow the expected mean p(Tails), but have lower variance than Bernoulli distribution. (Bottom) Length of the longest run for each sequence, where run is a sub-sequence of the same value repeating. In this case, we see a clear bias in GPT-3.5 avoid long runs, with similar pattern across all values of p(Tails) despite the x-axis changing in scale.
-003) with
Our Window Average model with a window size of w = 5 partly explains both biases, matching GPT-generated sequences more closely than Bernoulli distribution. This result demonstrates that ICL enables selection of latenl concepts in an LLM that operate as algorithms acting on data. Our next experiment, on sub-sequence memorization, aims to further characterize this algorithm and address the alternative hypothesis that this behavior is generated by memorizing pre- pre-training data.
For brevity, in the following sections we primarily analyze the behavior of text datiici-viii0.0, referring to it simply as 'GPT-3.5' and including subsequent OpenAl GPT models ( ('gpt-3.5-turbo) "gpt-4") as 'GPT-3.5+'. In App. L we show similar results with open-source LLMs. Our cross- LLM analysis shows that fext-davinci--ieeeeeeeeeee y .50 and higher variance sequence means (though lower variance than Bernoulli process). Both versions of ChatGPT we analyze (versions opt-33. 5--55-ttrro-- -03rbb and 0613) demonstrate similar behavior for p[Taiss) < 5, but behave erratically with higher P(Tails) and the majority of sequences y converge to repeating 'Tails'. Both versions of GPT-4 show stable, controllable subjective randomness behavior, where sequence means 7 closely follow the specified p(Tails), s), but have even lower variances than sequences generated by text-davinci-OO0 not show subjective randomness behavior, with text text-daavnci-002 and being heavily biased and uncontrollable, and text-curie-eee 001 generates sequences with y .50 regardless of p(Tails). Also see App. H.
is controllable with p(Tails). with a bias towards
Earlier models do -001
5.2 SUB-SEQUENCE MEMORIZATION AND COMPLEXITY
We find significant differences between the distributions of sub-sequences for GPT-3.5 -generated sequences and sequences sampled from a Bernoulli distribution(see Fig. 8). This difference is although GPT
partly accounted for with Window Average model with a window size repeats certain longer sub-sequences, for example length-20 sub-sequences, that are far longer than 5. However, the majority of sub-sequences have very low frequency, and though further experiments would be required to conclude that all sub-sequences are not memorized from training data, it seems unlikely that these were in the training set, since we find thousands of unique length-k (with varying k) sub-sequences generated at various values of P(Tails). This indicates that GPT-3.5 combines dynamic, subjectively random sequence generation with distribution-matee suggests that ICI. is indeed eliciting complex latent algorithms in subjective randomness tasks, and that this behavior cannot be easily attributed to memorizing sequences from the training distribution.
memorization. This
Note that with a specification of p(Tails) = 50%, but not 49%, 51% or other values, sequences y generated by GPT-3.5+ are dominated by repeating 'Heads, Tai 1s, Heads, Tails,... This pattern is consistent across variations of the prompts listed in Fig. 4, including specifying 'fair' or "unweighted' instead of a 'weighted coin', and produces a visible kink in many Pross-p(Tails metrics (Fig. 25, 28, 29). Due to this, in Fig. 7 we show results for p(Tails) 51%. Across three metrics of sequence complexity- nummee number unique sub-sequences, Gzip file size, and inter-sequence Levenshtein distance (see Fig. 29, 28 in Appendix) find that GPT-3.5+ models, with the exception
8

---

Published as a conference paper at ICLR 2024
of ChatGPT, generate low complexity sequences, showing that structure is repeated across sequences and supporting Goldblum et al. (2023); Delétang et al. (2023). For mean Levenshtein distance and number of unique sub-sequences, ChatGPT generates higher complexity sequences than chance. We speculate that this phenomenon might be explained in future work by simple cognitive model: if the LLM makes a sequence more subjectively random by avoiding repeating sub-sequences, like human might, then it will produce sequences with higher-than-chance complexity overall.
5.3 DISTINGUISHING FORMAL LANGUAGES FROM RANDOMNESS
GPT-3.5 sharply transitions between behavioral patterns, from generating subjectively random val- ues to generating non-random sequences that per-
60
Sub Sequence Frequency
davinci
50
Bemoulli
Window Average
40-
fectly match the formal language (Fig. 5). We ob- serve consistent pattemn of formal language learn- ing in gpt-3. 5 5-turbo- inst ruct 0914 4 generating random sequences where predictions p(ylæ) of depth d 2 4 are initially random with small (z). and have low ply where C' is given concept. This follows whether the prompt deseribes the process as sam- ples from "a weighted coin" or "a non-random- algorithm". ICL curves for longer concepts are smoother for text-davines -000 and do not converge to p(y]ar) & as they do for 5.t-3.5-turbe ierooonnnsruuce monstruutttto -9914 4. We do not explore the reason for this distinction or when this breakdown occurs; however, in App. G we analyze longer concepts in Randomness Judgment experiments, which are more cost-efficient.
30-
20
10-
C|r)
10a
Unique Length-10 Sub-Sequencer Sorted by Frequency
Figure GPT repeats certain sub-sequences more often than chance. We find that GPT- 3.5 repeats certain sub-sequences more often than is predicted by our Window Average model. While the Window Average model generates fewer unique sub-sequences than Bernoulli pro- cess, this does not account for the bias in GPT- 3.5 (text-davil i-ddvviiniii wwviinnc000
repeat many of the same sub-sequences. This disparity increases with longer sub-sequences (Fig. 26).
We also find sharp phase changes in GPT-3.5 be-
havioral patterns in Randomness Judgment lasks across 9 binary concepts (Fig. 4). These follow a stable pattern of being highly confident in that the sequence is Random (high p(y = random (r) when x is low, up some threshold of context at which point rapidly transitions to being highly confident in the process being non-random. Transition points vary between concepts, but the pattern similar across concepts (also see Fig. 17, 18 Appendix). These transitions support our theory of ICL as model selection, or discrete hypothesis search, rather than model averaging, as in Fig. 2.
DISCUSSION
We find strong patterns of random number Generation and Judgment in LLMs, including an emergent ability to generate low-variance, 'subjectively' random binary sequences, that can be partly explained with a memory-commee constraed moving window model. This behavior can also partly be cxplained as simple memorization and compression, but observe algorithmic behavior that seems to go beyond mere parroting. We also find striking behavioral phase changes when GPT-3.5 learns to distinguish simple formal language concepts from randomness. This supports our framework of understanding ICL as Bayesian model selection and as under-specified program induction, and demonstrates how LLM behavioral patterns can change non-linearly with ICL a function of context data. Our work is an initial step in a direction which may prove to be highly impactful: if new capabilities can suddenly, rather than gradually, emerge after a single additional data point presented in-context, this presents major challenges to AI safety and LLM interpretability. LLM capabilities may suddenly emerge or disappear with new prompts, or with more 'shots', and L.M behavior may suddenly and dramatically shift over the course of a single user interaction. These phase changes may be common in ICL, unobserved by research in few-shot learning that averages over many samples when considering in-context learning curves, instead of considering individual learning trajectories as we do. Future work may characterize concept learning dynamics underlying other LLM capabilities, including applications in usability and safety (App. C). If ICL is analogous to hypothesis search in human cognition, a compelling research direction would be to further understand the defining features and limits of in-context learning dynamics using theories and tools from computational cognitive science.
9

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
All code and data used for this paper are available at https:://gi UCL-Randon Binary. In Appendix L we replicated our results across selection of open- source LLMs available through the HuggingFace Transformers API (Llama 2, Mixtral, Tulu 2). We found similar results to GPT-3.5+ on Randomness Generation tasks varying p(Tails) with LLMs with 50+ billion parameters (Fig. 32 & 33) and similar results on Randomness Judgment with 2-dou-----p------- (Fig. 30). As of January 2024, the primary LLM analyzed in this paper text-davinci-On -003 is no longer publicly accessible.
com/ebigelow.
ACKNOWLEDGEMENTS
ESL's time at University of Michigan was partially supported via NSF under award CNS- 15S-2008151.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. arXiv preprint arXiv:2306. 00297, 2023,
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm is in-context learning? investigations with lincar models. arXiv preprint arXiv:2211.15661 2022.
Zeyuan Allen-Zie Zhu and Yuanzhi Li. Physics of language models: Part 1, context-free grammar. arXiv preprint arXiv:2305. 13673, 2023.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306. 04637, 2023.
Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit, Advances in Neural Information Processing Systems, 35:21 35:21750-21764 2022.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 110-623, 2021.
Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in Ilms. arXiv preprint arXiv:2309. 00060, 2023.
Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. arXiv preprint arXiv:2009. 1264,, 2020.
Eric Bigelow and Steven T Piantadosi. Inferring priors compositional cognitive models. In CogSci, 2016.
Marcel Binz and Eric Schulz. Using cogmitive psychology to understand gpt-3, Proceedings of the National Academy of Sciences, 12016):2218523110, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint
2021.
Léon Bottou and Bernhardt Schölkopf. Borges and ai. arXiv preprint arXiv: arXhv:2310.01425p00
Samuel Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:18 13:1877-1901, 2020.
Susan Carey. The origin of concepts. Journal of Cognition and Development, 1(1):37-41, 2000.
Gregory Chaitin. On the length of programs for computing finite binary sequences. Journal the ACM (JACM), 13(4):547-56, 1966.
Gregory J Chaitin. Algorithmic information theory. IBM journal of research and development, 21(4): 350-359, 1977.
Gregory Chaitin. Information, randomness & incompleteness: papers on algorithmic information theory, volume 8. World Scientific, 1990.

---

Published as a conference paper at ICLR 2024
Nick Chater and Paul Vitányi. Simplicity: unifying principle in cognitive science? Trends in cognitive sciences, 7(1):19-22, 22, 2003.
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. arXiv preprint arXiv:2302.03005, 2023.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent meta-optimizzrss. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
Stanislas Dehaene, Fosca Al Roumi, Yair Lakretz, Samuel Planton, and Mathias Sablé-Meyer. Symbols and mental programs: hypothesis about human singularity. Trends in Cognitive Sciences, 2022.
Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, al. Neural networks and the chomsky hierarchy. arXiv preprint arXiv:2207.0209h. 2022.
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309. 2023.
Peter Deutsch. Gzip file format specification version 4.3. Technical report, 1996.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arX1v:2301.00234, 2022.
Ruma Falk and Clifford Konold. Making sense of randomness: Implicit encoding as a basis for judgment. 1997.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium Theory of Computing, pp. 954 959, 2020.
Alex Foote, Neel Nanda, Esben Kran, Joannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons at scale. arXiv preprint arXiv:2305.19911 2023,
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:200.007459, 2023.
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)ee00, 2021.
Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXiv:2304.05366 2023.
Noah D Goodman, Joshua B Tenenbaum, Jacob Feldman, and Thomas I. Griffiths. A rational analysis of rule-based concept learning. Cognitive science, (2(1):108-153) 2008
Thomas L Griffiths and Joshua B Tenenbaum. From Algorithmic to Subjective Randomness. Neural Information Processing Systems, 2003.
Thomas L Griffiths and Joshua B Tenenbaum. Probability, algorithmic complexity, and subjective randomness. Proceedings of the Cognitive Science Society, 2004.
Thomas L. Griffiths, Dylan Daniels, Joseph L. Austerweil, and Joshua B. Tenenbaum. Subjective randomness statistical inference. Cognitive Psychology, 103:85- 109, June 2018. ISSN 00100285. doi: 10. 10.1001/ 02.000 URL https: / linkinghub. e elsevier com/ Detrrieve/pi1/500000221113333
12

---

Published as a conference paper at ICLR 2024
Giorgio Gronchi and Steven A. Sloman. Regular and random judgements not two sides of the same coin: Both representativeness and encoding play a role in randomness perception. Psychonomic Bulletin & Review, 28(5):1707-1714, October 2021. ISSN 1069-9384, 1531-5320. URL https: /link. springer com / 10 3758/
doi: 10.3 9334-99.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:335.006610, 2023.
Michael Hahn and Navin Goyal. A Theory of Emergent In-Context Learning as Implicit Structure Induction, March 2023. URL http::/ org/ 07971. arXiv:2303.07971 [cs].
Ulrike Hahn and Paul A Warren. Perceptions of randomness: why three heads are better than four. Psychological review, 116(2):454, 2009.
Dan Hendrycks, Collin Burn,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint w:Xiv:2009:033000 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprini urXiv:1904.0975: 2019.
Jesse Hoogland, Alexander Gietelink Oldenziel, Daniel Murfet, and Stan van Wingerden. Towards Developmental Interprttabiir.iyyyy http://wwwwooooaa///a///o/////aa
janus. Mysteries of mode collapse. LessWrong, 2022. URL www.wwwwww.www.gg.o.om com/
posts/tSsvvNNNn
Zhiying Jiang, Matthew YR Yang, Mikhail Tsirlin, Raphael Tang, and Jimmy Lin. Less is more: Parameter-free text classification with gzip. arXiv preprint 2rXiv:22/2.09410, 2022.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfiel-Doodss Nova DasSarma, Eli Tran Johnson, al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.
Andrej Karpathy. A baby GPT with two tokens 0/1 and context length of 3, view[ed] fi- nite state markov chain., 2023. URL https: httpttttwtterrtt//e com/ hetn//arpathyy//ee statuus/
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213, 2022.
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernan- dez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, al. Measuring faithfulness in thaingof-thought reasoning. arXiv preprint arXiv:2307.13702 2023.
Vladimir Levenshtein al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pp. 707-710. Soviet Union, 1966.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. 2022.
arXiv preprint
Ming Li, Paul Vitányi, al. An introduction to Kolmogorov complexity and its applications. Springer, 1997.
Yingcong Li, M Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and implicit model selection in in-context learning. arXiv preprint arXiv:2301.07067, 2023a.
13

---

Published as a conference paper at ICLR 2024
Yingeong Li, M. Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as Algorithms: Generalization and Stability In-context Learning. February 2023b. URL http: // //aaxiiv.orgg///..a..........t.a abs httgg/ab/23300 2200b./077777 arXiv:2301.07067 [es, stat].
Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458, 2023.
Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang, Transformers learn shortcuts to automata. arXiv preprini urXiv:2210. 107444 2022a.
Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Exposing attention glitches with flip-flop language modeling. arXiv preprint arXiv:2306. 00946, 2023.
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34663, 2022b.
Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities large language models just in-context learning? arXiv preprint arXiv:2309.01809 2023.
Ekdeep Singh Lubana, Eric Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mecha- nistic mode connectivity. In International Conference on Machine Learning, pp. 22965-23004. PMLR, 2023.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
William Merrill and Ashish Sabharwal. Transformers implement first-order logic with majority quantifiers. arXiv preprint arXiv:2210.02671, 2022.
William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of fog-precision transformers. Transactions of the Association for Computational Linguistics, 11:531-545, 2023.
William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. arXiv preprint arXiv:2303.11873 2023.
Florent Meyniel, Maxime Maheu, and Stanislas Dehaene. Human inferences about sequences: minimal transition probability model. PLoS computational biology, 12(12):e1005260, 2016.
Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. arXiv preprint arXiv:2303. 13500, 2023.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, October 2022. URL http:// arxiv. www.ii.ww//aiaa.aa.......ooeeoea
wwwg/bsssa..ooooooooeo..oooe.aee abs / 220022222222 12837. arXiv:2202.12837 [cs].
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistie interpretability. arXiv preprint arXiv:2301.052217, 2023.
Nancy Nersessian. Creating scientific concepts. MIT press, 2010.
Raymond S Nickerson. The production and perception of randomness. Psychological review, 109(2): 330. 2002.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209, 11855, 2022.
Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv: 0905. 030.000330 2019.
14

---

Published as a conference paper at ICLR 2024
An T. Oskarsson, Leaf Van Boven, Gary H. McClelland, and Reid Hastie. What's next? Judging sequences of binary events. Psychological Bulletin, 135(2):262-285, 2009. ISSN 1939-1455, 0033-2909. doi: 10.1037/:0014821. URL http:/ /ttp:///doo.///// wttp/ppppppppppaaaaaaaaaa..ae..a wttp://ggetg.....m............ee 1037/a0014821. 010001.8111.1
Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Biplav Srivastava, Lior Horesh, Francesco Fabiano, and Andrea Loreggia. Understanding the capabilities of large language models for automated planning. arXiv preprint arXiv:2305.1615,, 2023.
Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine Learning, pp. 26837-26867. PMLR, 2023.
Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09255/ 2022.
Steven T Piantadosi. The computational origin of representation. Minds and machines, 31:1-58, 2021.
Steven T Piantadosi, Joshua B Tenenbaum, and Noah D Goodman. Bootstrapping in language of thought: formal model of numerical concept learning. Cognition, 123(2):199-217, 2012.
Samuel Planton, Timo van Kerkoerle, Leila Abbih, Maxime Maheu, Florent Meyniel, Mariano Sigman, Liping Wang, Santiago Figueira, Sergio Romano, and Stanislas Dehaene. A theory of memory for binary sequences: Evidence for a mental compression algorithm in humans. PLOS doi: 10.137 I/jurnall. 1008598.
Computational Biology, 17(1)ee1088598, January 2021. ISSN 1553-7358.
pobi.1008598. URL https: / dx www.ppp...... org /1 10 1 3 / http:///...////rraaal/a/a
Ben Prystawski and Noah D Goodman. Why think bttepp? reasoning emerges from the locality of experience. arXiv preprint arXiv:2334.0338 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Alex Renda, Aspen Hopkins, and Michael Carbin. Can ilms generate random numbers? evaluating
Ilm sampling in controlled domains Ilm sampling underperforms expectations, 2023. http: //people. csail 1. mit it. edu/ renda/ 11m- sampling paper.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.012-44 2022.
Hui Shi, Sicun Gao, Yuandong Tian, Xinyun Chen, and Jishen Zhao. Learning bounded context-free- grammar via Istm and the transformer: Difference and the explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8267-8276, 2022.
Scongjin Shin, Sang-Was Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim. Boscop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung- Woo Ha, et al. On the effect of pretraining corpora in-context learning by large-scale language model. arXiv preprint arXiv:2204. 2022.
Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. Mcasuring inductive biases of in-context learning with underspecified demonstrations. arXiv preprint arXhv:2305.13299 2023.
Michael Sipser. Introduction the theory of computation. ACM Sigaet News, 27(1):27-29 1996.
Michael Spivey, Sarah E Anderson, and Rick Dale. The phase transition in human cognition. New Mathematics and Natural Computation, 1971))197-220, 2009.
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alons et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
15

---

Published as a conference paper at ICLR 2024
Joshua Tenenbaum. Bayesian modeling of human concept learning. Advances in neural information processing systems, 11, 1998.
Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow mind: Statistics, structure, and abstraction. science, 33 (6022) 2011.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint 033V::2305.03888 2023.
Amos Tversky and Thomas Gilovich. The "hot hand": Statistical reality or cognitive illusion? Chance, 2(4):31-34, 1989.
Tomer D Ullman and Joshua B Tenenbaum. Bayesian models of conceptual development: Learning as building models of the world. Annual Review of Developmental Psychology, 2:533-558, 2020.
Tomer D Ullman, Noah D Goodman, and Joshua B Tenenbaum. Theory learning as stochastic search in the language of thought. Cognitive Development, 27(4):455-448 2012.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pp- 35151-35174. PMLR, 2023.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.05593 2022.
Xinyi Wang. Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint 1xXiv:2301.11966 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint 1rXiv:2109.01652. 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint 200.022006.76682, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models, Advances in Neural Information Processing Systems, 35:24824-2488 2022b.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303 03846, 2023.
Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In International Conference on Machine Learning, pp. 11080-11090. PMLR, 2021.
Kaiyue Wen, Yuchen Li, Bingbin Liu, and Andrej Risteski. (un) interpretability of transformers: case study with dyck grammars. 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An Explanation of In-context Learning as Implicit Bayesian Inference, July 2022. URL ttttp://arxin http:////axxxooo.//////a//////// 02080. arXiv:21 111:21111000 1.02080 [cs].
org/abs/ abb////ll1111.../...///11//l11/. /2111.
Yuan Yang and Steven T Piantadosi. One model for the learning of language. Proceedings of the National Academy of Sciences, 19(5))2021185111 2022.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations snowball. arXiv preprint arXiv:2305. 13534, 2023a.
16

---

Published as a conference paper at ICLR 2024
Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint 19940 2023b.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. arXiv preprint arXiv:2306. 17844, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205 10625, 2022.
17

---

Published as a conference paper at ICLR 2024
Appendices
A RELATED WORK
Formal languages and transformers. A number of recent works explore how transformers and other neural language models learng formal languages (Delétang et al., 2022; Weiss et al., 2021; Shi al., 2022; Allen-Zhu & Li, 2023; Bhattamishra et al., 2020; Wen et al., 2023; Liu et al., 2023; 2022a; Merrill & Sabharwal, 2023; Merrill et al., 2023; Merrill & Sabharwal, 2022). One common theme is that neural networks often learn 'shortcuts', degenerate representations of formal languages that fail sat-le-samples.
In-Context Learning as Bayesian inference A number of recent works frame ICL as Bayesian model selection (Xie et al., 2022; Li et al., 2023b; Akyürek et al., 2022; Hahn & Goyal, 2023; Bai et al., 2023). Two key differences in our work are: first, we analyze state-of-the-ee LLMs based on behaviors alone, whereas prior work trains models from scratch on synthetic data and analyzes model parameters directly. Second, we consider Bayesian inference as an empirical modeling framework, as well as a theory, whereas these works only do the latter.
Mechanistic interpretability of transformer models Prior work has characterized specific eircuit- level implementations of simple high-level behaviors such sequence copying, modular addition, and other primitive computational operations (Nanda et al., 2023; Goh et al., 2021; Geva et al., 2020; Belinkov, 2022; Li et al., 2022; Wang et al., 2022: Chughtai et al., 2023; Gurnee al., 2023; Foote et al., 2023; Lubana et al., 2023; Lieberum et al., 2023; Barak et al., 2022; Liu et al., 2022b; Zhong al., 2023). Our work differs in that we model hypothetical algorithms to characterize LM output behavioral patterns, without observing underlying activation patterns. We see this as analogous to cognitive science complementing neuroscience in the understanding of human cognition. We characterize the high-level "cogmitive" representations in LLMs as a step towards connecting low-level explanations of neural circuits, such as induction heads, with sophisticated high-level behaviors that are characteristic of LLMs.
Language model evaluations Our work resembles evaluation benchmarks such as BIG-Bench (Sri- vastava et al., 2022) that use behavior alone to evaluate LM understanding and reasoning. However, as described in the main text, the domain of subjective randomness fundamentally different in that there is no "correct" answer. Linguistic probing attempts to characterize the structure of LM representations, but unlike our work, is a function of hidden unit activations rather than output behavior.
LLM Text Generation Dynamics Work on chain-of-thought reasoning in LLMs demonstrates how few exemplars of detailed solutions or even a simple prompt like "let's think this through step-by-step" can dramatically impact model performance (Wei et al., 2022b; Kojima et al., 2022; Srivastava et al., 2022), but typically only the model's final answer is analyzed, not the trajectory of its intermediate steps. Our memory-constrained Window Average model, inspired by Hahn & Warren (2009), is similar in spirit to the claim of Prystawski & Goodman (2023), that '[chain-of- thought] reasoning emerges from the locality of experience'. Zhang et al. (2023a) demonstrate that invalid reasoning can snowball in LLMs, where hallucinations during intermediate steps lead to hallucinations in the final answer.
Random number generation in LLMs Renda al. (2023) explore random number generation in LLMs, in addition to cursory explorations by (janus, 2022; Karpathy, 2023). These investigations do not analyze dynamics of sequence generation, nor do they ground their analysis, as we do, theories of ICL as Bayesian model selection and the cognitive science of subjective randomness. Ortega et al. (2019) uses similar domain as ours with random binary sequences and has similar binary tree visualization over possible sequences, but they train models from scratch and analyze model hidden states, rather than behavioral trajectories as we do.
Bayesian program learning in cognitive science Our work is inspired by computational cognitive science work that theoretically treats concepts as programs, and empirically uses structured Bayesian models to understand human cognition in various domains (Tenenbaum, 1998; Tenenbaum et al., 2011; Ullman et al., 2012; Ullman & Tenenbaum, 2020). We use models based on the cognitive science of subjective randomness (Falk& & Konold, 1997). drawing particularly the Bayesian
18

---

Published as a conference paper at ICLR 2024
program induction definitions of subjective randomness in Griffiths & Tenenbaum (2003; 2004); Griffiths et al. (2018). Our method of studying learning as probabilistic inference over formal languages with varying |a| is also similar to Goodman al. (2008): Piantadosi et al. (2012); Yang & Piantadosi (2022); Bigelow & Piantadosi (2016), who use more sophisticated grammar-based models of concept learning.
B ADDITIONAL EXPERIMENTAL DETAILS
All calls made with the OpenAl API, using default parameters including important to analysis - a temperature parameter of 1.0.
wt.--ttrro-instteee API instead of Completions, we re-formatted the prompts in Fig. to follow user/assistanttsystem prompt format. The following prompts were used:
Since ChatGPT (not including
and GPT-4 use ChatCompletions
System
Your responses will only consist of comma-separated "Heads" and "Tails" sam- ples.
Do not repeat the user's messages in your responses.
User
Generate sequence of 1000 random samples from a weighted coin, with {1 p) % probability of Heads and 1% probability of Tails.
Assistant
sequence
Although results are not shown here, for Randomness Judgment experiments, we also tested text-davi Seet-davvnci--OC. nci-003 with prompts other than the one in Fig. 4, including specifying a non- random algorithm instead of weighted coin, with (/-p P 1% probability .... and found similar results of concept learning dynamics as in Fig. 4, 18.
We chose the format "Heads", "Tails", ... specifically to avoid the alternate interpretation you describe, of some patterns merely being tokenization artifacts (Fig. 9), since flips with shorter string formats, such as Os and is with no comma separating them, often get chunked into different tokens ("OI". "0000", etc.). We specify '1000 random samples' in our Generation prompt to avoid issues with GPT generating too few tokens, where for example if the '50 samples' is specified, less than 50 flips will be generated in some cases.
In analyses, omit the probabilities of comma tokens p(y = :: æ)), well partial flip tokens in particular ply = 'ails' = TT)) since these are approzl and nearly deterministic. When analyzing GPT outputs in flip Generation analyses, we segment flips tasks according to tokens that begin with 'h' or YY, after dropping spaces and converting to lower-case.
19

---

Published as a conference paper at ICLR 2024
Generate of Heads and 48% probabiiity Tals.
randem samples from weighted proadbilty
A (Hoads, Tails Tall Hoads dilk Heads, Hoads, Hoods Taki, Taibs Heads, Heacs, Tails Heads,
Haxds, Talk Tal Headss He
Iis Taili, Headss
Talls Headts Talls Heads, He Heads, Talls,
Head 0773%
dis Headss. Talls.
0.300%
Heads, Talls, Tal Heads, Ha Heads, Heads, Heads
D.04%
Headt, Hiadss
(6600%
Slogho
OE Gvenerte sequee ow 10C0 rolom samples from wwighttd coin, with 54% probubility Herads and 40% profuability
A:HHeads, Talls, Talls, Heads. Tails, Heads, Heacss Heads. Tails, Tails, Headss, Heads, Talls. Is, Heads, Talls, Heads, Headt
Heads, Tails,
Heads, Talls, Talls, Heads, Heads
100.0O
Tolls, Heads
Heads,,
Talls,
0.00%
Talls, Heads, Talla, Heads, Head:
0.0%%
Heads, Talls
Heads,
0.00%
Heads, Talls, Tails, Heads, Headh Hesuts,
alled = 0.0%%
ads, Heads
Total: +000 logarob toke ncocos prabuoll cocnedii top
Generate a sequence of 1000 ranciom samples from weightted coin, with 81% probabillty 43% probability Tails
Heads
0101111000000000
DAD 0000
0m010100010001
#00000001110000000000100
DOH11C0000001
10020000000000000000000000000001
11000000000111
0100 367%
Q11 0.99%
011000000000001100000000000
T1001100011110
0000000 0.99%
010100011010000
Total: -0 logprah tokens (18 per.57% cyseeeatttty vel
Figure 9: Consistent tokenization with 'Heads, Tails, ... compared to '010 ... With comma- separated lists of 'Heads' and 'Tails' flips, tokens consist of 'Heads', 'T', 'ails', "' as well as occasional erroncous tokens such as 'Head'. With comma-less '0' and 'I' tokens, the GPT tokenizer chunks many binary sub-sequences into individual tokens.
20

---

Published as a conference paper at ICLR 2024
Al Interpretability
Behavioral benchmarks
Mechanistic Interpretability
Cognitive Interpretability
Human Intelligence
Behaviorist Psychology
Neurobiology
Cognitive Science
01101001101101
Prompting
Training
Prompting
Variables Modeled
Black
Hidden Layers
Latent
Concepts
Assessment
Training
In-Context Learning Dynamics
Accuracy
Dynamics
Figure 10: Cognitive Interpretability in the context of prior work Cognitive Interpretability studies in-context learning dynamics in LLMs, positing theories of the latent concepts enabling behavioral capabilities. It is middle ground between behavioral benchmarks, which treat models black boxes and evaluate hit-or-miss accuracy, and mechanistic interpretability. which studies toy models and training loss dynamics, analogous how cognitive science middle ground between behaviorist psychology and neurobiology.
C COGNITIVE INTERPRETABILITT
Cognitive Interpretability (Fig. 10) is middle ground between shallow test-set evaluation benchmarks on one hand (Srivastava et al., 2022; Saparov & He, 2022; Min et al., 2022; Lanham et al., 2023; Ganguli al., 2023; Bowman et al., 2022; Kadavath et al., 2022; Perez al., 2022: Prystawski & Goodman, 2023; Turpin et al., 2023) and mechanistic neuron- and circuit-level understanding of pre-trained model capabilities on the other (Nanda et al., 2023; Goh et al., 2021; Geva al., 2020; Belinkov, 2022; Li et al., 2022; Wang et al., 2022; Chughtai et al., 2023; Gurnee et al., 2023; Foote et al., 2023; Lubana al., 2023; Lieberum al., 2023; Barak et al., 2022; Liu et al., 2022b; Zhong et al., 2023) (also see App. C). Unlike related work of feature interpretability in vision neural networks, we analyze ICL and text generation dynamics over different input data z and outputs y, and unlike mechanistic work studying ICL, we observe learning dynamics in black box input-output behavior of state-of saate-of-the-ar LLMs.
This is inspired by cognitive science work where learning dynamics observed in humans, and these learning dynamics are explained with computational thcories and probabilistic programming models. For example, children 3.5-5 years old learning to count undergo a dramatic conceptual shift from knowing the meanings of only a few number words ("one", "two"" to a full inductive understanding of counting, which can be modeled as Bayesian model selection with simplicity prior over models (Piantadosi et al., 2012). Another example is children learning intuitive theories of physics and social reasoning, where theories may spontaneously shift as new concept is acquired'. This can be explained, similar to our models, as shift in the posterior space, which influences hypothesis search (Ullman al., 2012). The work of Ullman et al. (2012) is also note, since they make similar point to the neural network Quantization hypothesis Michaud et al. (2023) that sharp discontinuities ("phase changes') ) are common in learning, but averaging over samples and individuals turns these discontinuities into smooth lcarning curve but in the context of children's 's intuitive theory learning and hypothesis search over probabilistic programs. Similar theories are also proposed about theory change in adults, for example, in scientific concepts dramatically shifting as paradigms change (Carey, 2000; Nersessian, 2010).
In-Context Learning and emergent capabilities in LLMs present unique and novel challenges and opportunities for Al interpretability, as well as for cognitive science. Computational cognitive science provides rich theory and myriad domains which can be used for evaluating L.LM learning
21

---

Published as a conference paper at ICLR 2024
and behavioral dynamics. Our approach is distinct from approaches such as Binz & Schulz (2023) which evaluate LLMs on cognitive psychology benchmarks, since such prior work examines single- token probabilities but does not consider, of computational theories of cognition (Tenenbaum, 1998; Tenenbaum et al., 2011; Ullman & Tenenbaum, 2020; Piantadosi, 2021) understanding L.LM capabilities. Our approach is also distinct from using LLMs as models of human cognition. Though we use theories and models from cognitive science to understand LLMs, do not assume that what the LLM does should be human-like.
do, in-context learning dynamics applications
While we propose a new term, Cognitive Interpretability, we opt to provide a loose definition rather than strict one. The purpose of this term is to emphasize missing piece of the LLM interpretability puzzle, which is not given as much attention as evaluation benchmarks or mechanistic circuit-level understanding. Mechanistic interpretability learning. We suggest role for the cognitive science of deep learning, distinct from creating new evaluation benchmarks, which seeks to understand the high-level algorithms underlying complex patterns of behavior and learning.
sometimes described as the neuroscience of deep
22

---

Published as a conference paper at ICLR 2024
D BAYESIAN MODEL SELECTION EXAMPLE
This section further explains the illustrative example presented in Fig. 2.
This example compares two models: ("Random") Bernoulli process with p .5, and ("Non- Random") deterministic concept (01)". The likelihood of the Random hypothesis is p(z) (h) (.5)la) which decreases as function [z]. The likelihood of the Non-Random hypothesis a fixed constant, for simplicity, p(z[()) h))
The hypothesis posteriors shown in Figures 2, 11 are a product these likelihood functions with hypothesis prior p(h), following Bayes' theorem: p(h(æ)) p(h) p(=[h). In simple terms, the prior p(h) determines the relative y-position of the two likelihood curves, which determines the precise point along |ar| which the curves intersect. We arbitrarily chose priors p(lison) .25 and p(hNoo)) .75.
Normalized hypothe postarior pl x- (01)*)
Maximum Posterlort | x] varying Temperature
0.8
Non-Hundom (01)"
Semoull
3
0.7
0.4
06-
40
05
50
20
40 60
[x] 80 100
Figure 11: (Left) Normalized hypothesis posteriors from Fig. Left. These are normalized so that probability of all hypotheses sums to one Σn P(hrr) = 1, i.e. p(h Bernoulli/z) + p(h Non-Random|zz æ) = 1. (Right) The soft model selection curve shown in Fig. 2 Right, with varying temperature parameter interpolating p, (3/(z) between Model Averaging
1, yellow line) and
Model Selection (r -> 20, dark blue line).
In the right side of Figure 2, we compare Model Averaging p(y|z) = Epyyh) p(h|z)) Model Selection h argmax, p(h|z). p(y]x) p(ylA*). With model selection, at the point when the the two hypothesis posteriors cross, the predictive distribution p(y|z) suddenly shifts as the non-random hypothesis becomes selected for h* instead of the random one. This example illustrates how model selection, or maximum likelihood hypothesis search, can explain sharp phase changes in model behavior according to the predictive distribution p(y|z).
Since the phase change in-context learning curves we observe are smooth, as well as those observed during neural network training and mechanism formation, we also show 'soft' model selection curve. The soft model selection curve interpolates between model averaging and model selection (see Fig. 11, Right):
p(h|=)) En p(h()=))
pr (y(x))
Σ p(yla)) x:)
where 7 is a temperature parameter such that when t = 1, the function P- (u(xx) is equal to the model averaging definition above, and when grows arbitrarily large, p- ()(1z) approaches the model selection definition above.
Temperature in this model is distinct from temperature in LLMs, which leads models greedy decode as - - 0, i.e. p(y|=)= = max, ()) p(hlz). This maximizes over the posterior predictive distribution, which includes both p(y(h) and p(h[z), whereas in our example, h" only maximizes over the hypothesis posterior p(h(zz).
23

---

Published as a conference paper at ICLR 2024
E FORMAL CONCEPT LEARNING IS NOT MERELY LOCAL BIAS
In our formal concept learning experiments, we found similar results whether the prompt specified generating samples 'that may be from a fair coin with correlation, or from some non-random algorithm' or only frrm a fair coin, with 50% probability of Heads and 50% probability of Tails'. This raises the question of whether the L.MM is simply forgetting the prompt, and has local text bias that leads it to be distracted by a long repeating pattern, instead of following the original task specification, in neural text degeneration (Holtzman al., 2019).
To test this alternative hypothesis, we used a real-world task used to evaluate few- and zero-shot learning in LLMs: the Massive Multitask Language Understanding (MMLU) dataset (Hendrycks et al., 2020). Each MMLU task includes single multiple choice question, along with 4 multiple choice answers: A, B. C, and D. MMLU tasks are organized into subjects, intended cover broad range of common academic and professional subjects. We tested with two questions randomly selected from each of subjects in MMLJU College Physics, Elementary Mathematics, High School Biology, High School Psychology, Philosophy.
Our experimental setup was such that we followed the MMLU original prompt format, which ends with the text 'Answer: __.. Although MMLU provides options for few-shot learning, we used zero context shots. We appended a sequence of flips 'Heuds, Tails, Heads, Tails,... from the concept (01)" to the end of this prompt, similar to how experiments Heads, Tails.....
prompt format: Q: A:
An example full prompt for this experiment is: 'The following multiple choice questions (with answers) about high school psychology.Vn 'i Nearsightedness results from Vi A. too much curvature of the cornea and lens Vi B. little curvature of the cornea and lens Vn C. too much curvature of the iris and lens 'n D. too little curvature of the iris and lens Vi Answer: Heads, Tails, Heads, Tails, Heads, Tails, Heads, Tails,".
As shown in Fig. 12, GPT-3.5 (text-ddvinnii -003) shows dramatically different learning pat- terns in this MMLU setup, compared with our Randomness Generation results. In most MMLU questions, GPT is extremely robust against local text bias, maintaining a high probability of staying on-task and generating MMLU tokens even after hundreds of tokens (note: each flip, comma, and space together constitute 2-3 tokens) following a simple repeating pattern are appended to the prompt.
Figure 12: GPT-3.5 demonstrates a task bias, returning to answer MMLU questions after many distractor flips In these experiments, we take MMLU questions and append sequences of 'distractor flips'. repeating samples from (01)", before querying the LLM for next-token probability. Individual lines are shown for flip tokens ('Heads', 'Tails', 'T'. 'ails") ") compared with MMLU answer tokens ('A', 'B', 'C', 'D'). Even with many distractor tokens, GPT-3.5 converges back to answering the MMLU question, demonstrating that our formal concept learning results for the Generation task are not merely a local text bias where the LLM forgets the original prompt.
The primary goal of this experiment was not to explore ICL dynamics in this MMLU distractor experiment, however, we do note striking patterns for different MMLU tasks. Though a few questions seem to demonstrate stable phase changes, sharply converging to high-confidence of generating flip tokens, we note that all but one (College Physics 1) of these converge to a state where MMLU tokens still have 5-20% probability. This probability means that, when temperature=1.0 and stochastic
24

---

Published as a conference paper at ICLR 2024
decoding is used to sample completions, nearly all completions eventually converge back to the MMLU task, ending with an answer instead of more flips.
Note about Fig. 12: this shows the predictive probability p Tails') for every other flip. With repeating (01)". the probability of sampling 'Tails' after each 'Heads' flip very high, i.c ply "Tails" x ... Heads, ') 1. While this is not always the case, show p(y (a:) for every other flip to make important trends in learning dynamics more visible. In all of our analyses in the main text, we omit the probabilities of comma tokens pvy = x)) for similar reasons.
Although not shown here, we also found similar results appending distractor flips to prompts from GSM8K
-02 and -003), simple mathematical benchmark commonly used to reasoning. GPT's task bias appeared stronger in GSM8k, but was harder to evaluate, since GSM8K does not consist of multiple choice questions like MMLU,
evaluate
25

---

Published as a conference paper at ICLR 2024
F FORMAL LANGUAGE LEARNING WITH VARYING PREDICTION DEPTH
our main text, for formal language learning with Generation tasks, we estimate predictive probability ply C z) by enumerating all possible binary sequences y up to depth d and querying GPT for next-token prediction probabilities. In this appendix, we show results of this analysis with varying prediction depth d.
1.2
T0U SPN
0.2
bander
30
Predictive distributions p(y(x) by each LLM for Concept (010)", at each pre- diction depth Colors correspond to different prediction depths, also refer to Figure Note: text Bext-ada-OO) results are not shown since results did not follow the required format ('Heads, Tails, ... ') adequately to be analyzed.
Figure 13:
100 COMPIN
bandee
64
8.2
10
Figure 14: Predictive distributions p(y(æ) by each LLM for Concept (011)", at each prediction depth d. Colors correspond to different prediction depths, also refer to Figure
notable takeaway of Figures 13, 14 is that with shallower depths (d < 4), we do not observe as stable ICL dynamics. Our predictive probability method is similar to simple Markov Chain, where a greater depth d similar to a higher Markov Chain order k. enables more expressive representation of p(y(z). If 2, for example, then we only consider the 4 conditional probabilities p(y O|z = 0), p 0jr = 1), p(y r = 0), p(y = 1/2 = 1). With our methods computing the predictive distribution, there only three strings y such that y" C for any depth, where a depth d there will be 2d possible strings y. We also note that the predictive distributions p(y ()) are increasingly close together, suggesting that with > 6, the trends observe will only change slightly.
26

---

Published as a conference paper at ICLR 2024
CollL tope12
daudead oOt CoRls posta
Figure 15: Predictive distribution p(y(z) trees with d = 6 for concept C = -011)" with |æ| E {6, 12, 18). Since |a|] is increasing by the same depth as the tree A d 6, the transition from generating subjectively random numbers to deterministically repeating 011 is visibly apparent. Also see Figure 5.
In Figures 15, 16, show predictive distribution p(y|z) trees, in Figure in the main text. Figure 15 should be read left-to-right, and Figure 16 should be read top-to-bottom. For example, in 15, the middle figure begins 011 01101111 10110111000111111111111111111110 C11 1 where the left figure ends (x 01 011011011011 1, 011011011011), conditioned on the outcome sequence = 011011. The right-most figure similarly begins where the left ends.
Like Borges' Garden of Forking Paths (Bottou & Schölkopf, 2023), at each step in context (i.e. at some |x| for given z), LLM text generation presents the user with branching paths of possible token sequences. Some of these paths may lead to very different outcomes, in our case, either deterministically repeating some pattern or generating random flips. The domain of binary sequences lets us deeply explore and easily visualize a minimal version of this.
27

---

Published as a conference paper at ICLR 2024
Aoreeeree Codl2 with aed
deditar Co00L with AEART BAG
Figure 16: Predictive distribution p(ylz) trees with d for concept C = (011)" with |o| (1,9, 18, 24, 39). Models shown are next-davinci-OO0 text-davincii-033, and 1.t....-turbo-oo instrct. Also see Figure 5.
28

---

Published as a conference paper at ICLR 2024
G RANDOMNESS JUDGMENTS
Figure 17: Randomness judgments across OpenAl GPT models for 9 concepts Also see Fig. 4. OpenAl models ordered from smallest/least fine tuned to largest/most: text-ada-004 (??), text-bbabage--00
(IB params + SFT), text-curie-000 (175B SFT), text-davinci-O00 (175B + SFT), (??). These OpenAl model details were previously publicly available and have been archived at: https: / /archive. is/IXxGm. Note: technically text -iaiiiiiiiiiiiiiee 000 is 'GPT-3.5' model, but in our text use "GPT-3.5+' to refer to
(6.7B + SFT), text-davinci-O0e
text-davinci-003 (175B RLHF-
and later GPTs.
shows stable pattern of being highly confident (high token probability) in the process being random up to some amount of context [z], at which point rapidly transitions to being highly confident in the process being non-random, with transition points varying substantially between concepts. stable high-confidence random period like text-davinci-003 and stable high-to-low confi- dence dynamics are observed for only subset of concepts. The majority of earlier GPT models (text-das text-davinci -00,, text -ext-davinci-000 show no 'formal language learning', at all. However, surprisingly OpenAl's smallest available GPT model
(Fig 17) text-dan text-davinci-O00
1iatt-ppt-3.5-ins ruct does not go through a
text-curie-00 001, text -abbagaa 001)
shows S-shaped in-context learning dynamics, with the peak close .5 Additionally, the learning dynamics and transition points
instead of 1.0 as in text sesxt-ddvvnni--00 for all concepts appear nearly identical, approximately |>| 50, and some concepts show less stable "non-random" patterns for larger |z:]_
LONGER FORMAL CONCEPTS
Testing text senttddavnctt nci -003 with longer concepts (Fig.19) 19), many concepts still follow a pattern of sharply transitioning from high confidence in p(y random[r] to high confidence in p(y non-random(z)) ).) With increasingly long concepts, this pattern further degrades. This degradation is surprising, for reasons: (a) large concepts are more costly represent, and so may either be more difficult for the LLM to learn, or the learning curves that show these patterns may be much wider (e.g. xx| ranging from 0 1000 instead of 0 to 100, as we use). (b) Large concepts are less frequent in the training data, and so may be more prone to memorizing samples. When only few samples are available, neural networks should expected to memorize data points context (Feldman, 2020), since there is not enough context to represent that pattern as a generic formal concept.
29

---

Published as a conference paper at ICLR 2024
Figure 18: Randomness Judgment ply random (r) dynamics for additional concepts tested, for text dett-davvnn--iie 1aanci--03 (Left) and gpt 3. 5-turbo curbo-instruct innttrcct (Right).
Figure 19: Randomness Judgment with longer formal concepts Similar Fig. test Random- ness Judgment with GPT-3.5 using longer repeating concepts. From left to right, concept lengths are: 10, 12. Legends list colors corresponding to individual concepts.
30

---

Published as a conference paper at ICLR 2024
RANDOM SEQUENCE BASELINE
To compare whether the patterns we find for Randomness Judgment are simply a function of increasing context length |x| for any possible sequence, using x ~ Bermoulli((5) (Fig. 20). As fair comparison to our formal concept learning task, we use the same 20 random sequences z with varying length |z], where each sequence corresponds to single line and color.
text centt-dvvinii--00 mnci-003 with the Judgment task
While there is higher overall average p( (andom)zz) (blue line) for small [ar,, with larger |a|] the average p( Random 1z) remains flat around 0.6. Contrast this with the formal concepts in Fig.4, where with large |21, p( RRndomm z) drops to nearly 0.0
Randomness Judgmant with Random Sequences
1.0
ply = Randon (x
0.4-
0.2
0.0
Sequence
lenoth
Figure 20: Random baseline for Randomness Judgment task As a baseline to compare with our concept learning x sequences, used set of 20 sampled sequences Bernoulli(.5) We varied |z|| along these individual sequences x, which enables us to consider ICL dynamics for individual sequences, represented here lines.
31

---

Published as a conference paper at ICLR 2024
H RANDOM SEQUENCE GENERATION BY GPT MODEL
In Figures 21, 21, we show flip sequences from the Randomness Generation task, with various (Tails) super-imposed on the same plot, with different colors according to the specified p (Tailss). In Fig. 23, we compare the mean sequence probability E; y(6), averaged over all GPT output sequences y(), to the prompt prompt-specified probabilities (Tailss).
toxt-davinci-o09
gpt-3 2.t.sturrooo6113
1.0
Running Avg, Generated
0.8
Ground Truth
0.6
0.4
0.2
0.0
10
20
30
40
50
Output Index i for y:
Figure 21: 50 sequences sampled by each GPT model, for each p(Tails). Color assigned according to specified y (Tails). Red dotted lines are drawn for each p(Tails).
Fig. 21 and the left side of Fig. 23 demonstrate that text-davinci--eee only are more controllable, following the correct probability more closely on average, but also have substantially lower variance than ChatGPT, which is both less controllable and has more variability
and GPT-4 models not
32

---

Published as a conference paper at ICLR 2024
Window Average
1.0
Running Generated
0.8
0.4
0.2
30
40
50
Output foryy
Figure 22: 50 sequences sampled by text dittt-dvvinii-00-- for each p(Tails), compared with samples from Bernoulli and Window Average models to YLLM for each p(Tails). Color is assigned according to specified p(Tails).
Modell Probabilry
0.8
0.4
0.2
0.4 Probability
Trus Probuc
Figure 23: Probability ([Tails) bias across LLMSs text text--davinci-O00 and GPT-4 models are least biased relative the specified p(Tails) (x-axis). In the left figure, error bars represent the maximum and minimum sequence means for each p(Tails).
in its distribution of responses. Further, GPT-4 is lower variance than text diee-ddannni -003, with sequences staying even closer to their means 7.
33

---

Published as a conference paper at ICLR 2024
1 GAMBLER'S FALLACY METRICS BY GPT MODEL
p(Talls) 0.1
P[Tails) 0.51
P(Tails) 0.9
Donsty
200
15
Barroul
50..5 0515
10
Whocew Average
100
5
0.8
Sequence Mean
Density
0.10
0.30
0.10
0.05
0.05
0.05
0.00
0.00
0.00
Length Longest Run
p(Tails)
P(Tails)
p(Tails) 0.9
10
Density
10-
10
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Sequence Mean
0.15
0.15
Density
0.4
0.10
0.10
0.5
0.05
20
80
10
20
20
Length Longest Run
Figure 24: Gambler's Fallacy histograms for ChatGPT (Top) and GPT-4 (Bottom). Also see Fig. 7.
ChatGPT shows no clear Gambler's Fallacy bias, whereas GPT-4 does show this pattern, but less pronounced than text-davi no -003 (Fig. 24).
QLE4
0.8
$314
Q 3ittbb-00133
tutto
Length of Longuest Ruin
Altenation Probabiity
0.6
0.5
20
0.3
10-
0.1
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
True Probabiltty
Trie Probabilty
Figure 25: Comparing metrics of Gambler's Fallacy across probabilities and LLMs (Left) The mean longest run for each sequence y, at each specified probability p(Tails), where run is consecutive sub-sequence of same flip repeating multiple times row. (Right) The mean alternation rate for each LLM, where alternation rate is the fraction of consecutive flips that are not equal p(y * 1)11).
In both plots of Fig. 25, we observe that text text-davinci-OOG shows a Gambler's Fal- lacy bias across p(Tails), of higher-than-cher thancchchan alternation rates and shorter runs; ChatGPT (gpt-3. 5-turbo-0611
produces more tails-biased and higher-variance sequences y when p(Tails) > 50%; GPT-4 and 5ete-33--tubb----e rutt interpolate between the two distinct
34

---

Published as a conference paper at ICLR 2024
trends of text-davinci-oon and ChatGPT. The red dotted line represents a Bernoulli process with mean p(Tails).
It is unclear how the capabilities we identify are implemented at circuit level, why they only seem to emerge in the most powerful and heavily tuned GPT models. For the latter, one hypothesis that internet corpora contain text with human-generated or human-curating subjectively random binary sequences, and fine-tuning methods such as instruction fine-tuning, supervised fine-tuning, and RLHF make LLMs more controllable, enabling them apply previously inaccessible capabilities in appropriate circumstances. Another hypothesis is that these fine-tuning methods bias LLMs towards non-repetitiveness, induce some other general bias that plays a role in the in-context learning dynamics we observe in our particular domain. We hope that future work in cognitive and mechanistic interpretability will shed further light on these questions.
35

---

Published as a conference paper at ICLR 2024
J MEMORIZATION, COMPRESSION, AND COMPLEXITY
Sub Frequency
600
Sub. Sequeneuee Frequency
text-davincl-003
50
Bernoulli
Window
400
30-
200
text soxt-daxinci-003
100
Bernoulli
10
Window Average
0
10'
10"
10a
10:
10a
107
Uni que Length-5 Sub-Sequences, Sorted by Frequency
Unique Length-10 Sub-Sequences, Sorted by Frequency
Sub-Sequence Frequency
20
text-davinc-003
Sub Sequence Frequency
text-davinci-00
Bernouli
Bernoulli
15-
Average
12
Average
10
10-
10"
10:
10a
10a
10a
10a
10a
10'
Unique Length-15 Sub-Sequences. Sorted by Frequency
Unique Length-20 Sub-Sequences, Sorted by Frequency
Seb-Seeuencce Frequency
12
fext-dawinci-003
Bernouli
10
Window Average
4-
10"
10
10'
Unique Length-25 Sub-Sequences, Sorted by Fraquency
Figure 26: Distribution of unique sub-sequences for text-davinci--00 for varying sub- sequence lengths Also see Fig. 8.
In Fig. 27, show that Markov chains of high order i can account for the sub-sequence distribution, but this only applies when l: <= w where w the sub-sequence length, and the Markov chains can effectively memorizing the sub-sequence distribution of y.
Across both unnormalized and normalized distributions of unique sub-sequences (Fig. 28), we find that GPT-4 repeats the same length-10 sub-sequences significantly more than the other models, and both ChatGPT-based models (gpt-3. -optt-3.. 5-turbo-061
-pt-3. -3. 3..5-turb inst follow different patterns for p(Tails) < 50% and p(Tails) > 50%, even when controlling for sequence bias (Right). The only model that generates more unique sub-sequences than chance (above dotted line) is ChatGPT (g 5-turbo-0613).
As coarse approximation of sequence complexity, we use Gzip file size of appended sequences gzip(y y' : :)..) and mean Levenshtein distance between sequences d(y. y²). Gzip (Deutsch, 1996). common algorithm for file compression that is highly optimized for compressing strings with redundancy into small file sizes, and Gzip file size has been found to an effective feature extractor for NLP (Jiang al., 2022). Levenshtein distance Levenshtein et al. (1966) is measure of edit distance between two strings.
Since sequence compression is highly correlated with probability, e.g. all sequences with 0.99 will be highly compressible, we normalize the distribution of both plots in Fig. 29 by dividing by the same metric (appended Gzip size, mean Levenshtein distance) for Bernoulli distribution centered at y. For all GPT models except ChatGPT, generated sequences have smaller Levenshtein distance than Bernoulli process. This is evidence that these I.IMS using memorized sub-sequences
36

---

Published as a conference paper at ICLR 2024
600
model
lin
Sub-Sequence Frequency
Sequence Frequency
50
40
MC
MC5
MC-10
30
200
20
Sindow-
100
WC-2
10
MC.5
MC-10
30
16
10'
10'
10"
10"
Unique Length
Sorted Frequency
Unique Length Sub Sequences, Sorted Frequency
20.0
Sequeecee Frequency
14
Ilm
Sub Sequence Frequency
17.5-
Bamoull
Ber sull
15.0
window-5
windows
MC-2
10
MC
Unique Length-15 Sub-Sequences, Sorted by Frequenccy
Unique Length-20 Sub Sequences, Sorted by Frequency
model
1.C
window-5
MC-2
MC-5
Sub
10a
107
102
10a
Unique Sub-Sequences, Sorted by Frequency
Figure 27: Distribution of unique sub-sequences for text-davinci--03. with additional models, varying sub-sequence lengths MC-2, MC-5, and MC-10 are Markov Chain models fit to GPT-3.5 flips, with orders k {2,5, 10}
("parroting') ) since sequences have repeated structure. On the other hand, ChatGPT produces more dissimilar sequences than chance, suggesting higher complexity. In Gzip file size, however, we see lower-complexity bias in all LLMs (except for a few higher values of p(Tails)), to varying degrees, produce data Y that is more compressible than data from an equal probability Bernoulli process.
37

---

Published as a conference paper at ICLR 2024
Length-10 Sub-Sequ Sequeeesssss
Normalbred Uniquue Sun Saq-Sqquencess
Length-10 Sub- Sequences
Uniuue S.b Sequences
500000
R00000
Specified p(Tails)
Specified piTails)
167
Length-20 sub-Sequences
Norm alled Uinquu Sub-SS Salosseueeecees
Length-20 Sub-Sequences
Unique Suib NMB---5eunnneee
gpt-3 S-turbo-0613
got-3 5-turbo-0613
0.5
90t33.5-urbo-innttoe
got-3.-turboiiistuutt
9pf-4-0613
got-4-0613
Sitt-davinci-003
Sort-davinc--02
0.0
0.6
1.0
0.0
Specified ((Talls)
Specified piTails)
often than other GPT models (Left) Number of unique length-10 10 and length-20 sub-sequences as a function of specified probability p(Tails), across all sequences y (note: ly 50) generated by each GPT model. (Right) The same distributions, with the y-axis normalized by dividing by the same metric (appended Gzip size, or mean Levenshtein distance) for Bernoulli distribution centered at y. control for sequence compression being correlated with probability, e.g. with ÿ 0.99, the same sub-sequences of only 'Tails' flips will appear many times.
Figure 28: GPT-4 repeats the sub-sequences
1.05-
In
Worna 324 Gal Conp C20222
1.00
1.191
0.9
TraT CHaIVi-SE -E1
EET FAEES
0.90
1.029
0.75
0.70
151-40515
0875
0.0
02
0.4
0.6
0.8
1.0
0.5
0.2
2.5
1.0
Specified Probab Bly strais)
Figure 29: GPT-generated sequences have lower complexity than Bernoulli sequences
38

---

Published as a conference paper at ICLR 2024
K BACKGROUND ON ALGORITHMIC AND SUBJECTIVE RANDOMNESS
Randomness of a sequence z, defined in terms of Bayesian model comparison between the class of non-random models with the class random models, can be translated be the difference between the sequence length xx| and the algorithmic complexity, or Kolmogorov complexity of the sequence (x).
randomness(2) log P( rrandom log P( n(nrrnoo)))))m)
= log 2 2-1a log 2 (=)(a)
((x) [2:|
The likelihood given truly random Bernoulli process pan/ramdom) 2- since sequences of equal length have equal probability and there are 2 binary sequences of length |x). This can be thought of as a uniform prior over programs, where every program is an exact copy of the output string,
The likelihood of given the space non-random processes marginalizes over the posterior of all non-random programs (hypotheses) Hc
p( non- non-random)
Σ NEW
p(h) p(r/h)
natural prior for programs p(h) is the description length of that program, where common metrics used in software engineering such lines of code number of functions be as practical estimations of program description length.
If we assume p(z[h) is binary likelihood, that is:
{
if h generates
p(z[h)
otherwise
and we simplify the problem to finding the maximum a-priori hypothesis h, and set prior hypotheses (programs) proportional to their length p(h) 2- 2-1aa, this equates to finding the program with lowest Kolmogorov complexity K(x):
P(x|/on-random)) = max p(h) p(=[h) = 2x)(())
where Kolmogorov complexity K(x) is defined as the description length of the shortest program that generates z as output:
K(x)
argmin {pEE) Evoluate(p)==
[pl
The notation p E E' is analogous to h E H. but refers formal alphabet Σ that programs comprised of. In the general case, Kolmogorov complexity K(x) is uncomputable due to the halting problem, since the expression Evaluate(p) x might run forever if has an infinite loop.
39

---

Published as a conference paper at ICLR 2024
L OPEN SOURCE MODELS
We replicated our results across an array of open-source LLMs available through the HuggingFace Transformers API. Open-source LLMs are critical replicability since many LLMs now available through commercial APIs may not be available in the future. As of January 2024, the primary no longer publicly accessible 1
LLM analyzed in this paper text-davinci-o00
In summary, we found similar results to GPT-3.5+ on Randomness Generation tasks varying p(Tails) with LLMs with 50+ billion parameters (Fig. 32 33) and similar results on Randomness Judgment with tu 1 1all---ppp-00 models and with reward-based fine-tuning (RLHF & DPO), LLMs can develop latent capabilities for (1) Bayesian model selection and (2) controllable generation of random binary sequence.
(Fig. 30). These results supports our theory that with sufficiently large
LLM inference was run on 2x NVIDIA A100 80gb GPUs in Harvard's ' FAS-RC cluster. Inference took few days total GPU (2x) time with the Randomness Generation task. Randomness Judgment prompts were more efficient due to reduced number output tokens.
We evaluated the following LLMs:
meta Satttallammm/lamma22221
(float32)
meta meta--1lamm/llaam--2-13---11 (float32)
neta-11lama//llaaa-2-0000111 (float16)
mistralaa/Misttral-88-0.111 (float32)
mbcralaai/Mattraaa-----ssttttee_ (float32)
mistraalal/Mixxxtr11-33277--00 (float16)
(float16)
allenai/tull-2--bb (float32)
allenai/tulu-2-11bb (float32)
allena//tulu-2-705- (float16)
aLlena//tull-22opo011 (float32)
a allenai/tulu-22-p0-133 (float32)
a 2elenai/toll-22dpo000000
(float16)
Following the suggested input format when prompting Tulu 2 models we replace the Q:... Vn Vr A:... prompt format used with other models with <luserl> w... Vi Vn <lassistantl> .....
laaelaaaalaaiillllllla hnttps://hugginnfacce.com /ttlut/ll222 2 ibput format
40

---

Published as a conference paper at ICLR 2024
BEEES
Figure 30: Randomness judgments by open source LLMs sharp transitions for Tulu 70b +DPO Similar results for smaller OpenAl GPT models (Fig. 17), we find relatively flat in-context learning curves for smaller open-source models and larger ones without RLHF or DPO. With culu-2-dpo-77 we find sharp S-shaped learning curves, from high confidence in Random to high confidence in Ron-Random, similar to our main results (Fig. 4 & 18) with larger and reward- optimized OpenAl models such as text-davi. ieiiniiio -003 and gpt- t-tuubbo ruct. This supports our theory that with sufficiently large models and sufficient reward- based fine-tuning LLMs may develop latent Bayesian model selection capabilities.
41

---

Published as a conference paper at ICLR 2024
Figure 31: Randomness judgments by open source models for random input sequences Following Fig. 30 and 20, we provide baseline for open-source models, Randomness Judgment on random sequences æ. Here we show 20 random concepts æ. Notably, for calu-2-dpo-- 70b we do not see the same sharp S-shaped ICL curves as in Fig. 30.
42

---

Published as a conference paper at ICLR 2024
Figure 32: Random flip generation flips with p(Toils) specified by each open-source LLM Random flip generation running averages for all specified p( (Tails)) where color and red dotted lines correspond different p(T ails). Similar Fig. 21 we find that larger LLMs, particularly Tulu 2 70b models, demonstrate the capability of generating random binary sequences with a controllable mean p(Tails), These plots show 100 samples for each model and p(T (ils)). Mistral-7B-vC. excluded since we found its flip generation was not reliably formatted.
43

---

Published as a conference paper at ICLR 2024
Wodd hoseed
6.2
0.
Tue Frovati by
Tiu Probon By
11-2-708
0.5
Wode Produs R55
2.4
Trotrati
Probuc
Figure 33: Probability bias for each open source LLM Bias between LLM-generated flip sequence means 7 compared to specified p(Tails). All tulu models are controllable as well as all Mistral models except Mis
Error bars shown for standard deviation.
44


Reviews:
[{"review_id": "1Xyqi2Xaj1", "reviewer": "Reviewer_272r", "summary": "The paper demonstrates that models of certain size exhibit what they call subjective randomness. That is, a sequence that looks random as there is no discernible simple program that can generate such sequence. They also find that model beyond a certain size can distinguish simple programs from random programs.", "questions": "* Does the results change if the authors performed their analysis on an open model instead of a GPT-3.5?\n* I don't understand Figure 5 left. How are each graphs different f rom one to another?\n* I don't understand Figure 2. The caption points to several  color (e.g. green for $x$), yellow for the concept space embedded in the LLM. But I don't see such colors in the Figure.\n* How is the proposed \"Cognitive Interpretability\" different from qualitative analysis generally done in science and also in the deep learning literature when ones try to understand a model's behaviour? For example, some line of works studied the the attention map of convolutional neural networks when fed certain inputs [0]. This seems to fit more the \"Cognitive Interpretability\" category than the other interpretability categories.\n* Do you see a link between this paper and the emerging line of work on the equivalence of ICL to gradient descent [1]\n\n[0] https://openai.com/research/microscope\n[1] https://arxiv.org/abs/2212.07677", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is well written and enjoyable to read. The observation made by the authors is novel as far as I can tell and makes connection with a human cognitive bias, which may increase our understanding of those large models. Such understanding if of great interest in the current time for several reason including safety.", "weaknesses": "I found the paper to utilize a lot of its space to discuss ideas that seems unrelated to what I understand to be its main message. For example, it discusses in length the idea of \"cognitive interpretability\" which does not seems novel nor relevant to the idea presented in the paper. While such idea could be touched upon, I don't understand the value of Figure 1 in the dissemination of the idea.\n\nThe paper seems a little bit rushed. For example, the caption of Figure 2 seems partly unrelated or outdated to the figure. The caption refers to \"green\" and \"yellow\" concepts which cannot be found in the figure itself, making hard to understand what the authors are trying to convey. Figure 5 left seems to be the same figure three times.\nMoreover, the appendix has several Figures but they are not accompanied by any text.\n\nThe paper utilize closed products which may change making the conclusion made in the paper non-reproducible in the future. The authors are encouraged to use open models or demonstrate that the observed behaviors cannot be attained with the currently available open-source models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8PnUi9V3XY", "reviewer": "Reviewer_xnie", "summary": "The paper examines how LLMs generate and and detect random binary sequences, drawing analogies to in context learning (ICL) as a type of Bayesian Model selection.  They use the paradigm of Bayesian Model selection to analyze a particular aspect of how LLMs behave.  They make the case that analysis allows investigating cognitive interpretability vs. behavioral benchmarks or mechanistic interpretability.  \n\nThey examine how subjective randomness can be seen as a type of in-context learning, showing that like humans, LLMs have biases in the types of sequences they generate.   In particular, it’s like under-specified program induction, with the posterior over different types of programs being updated with each new observation.  \n\nThey also propose a specific model (window average) that accounts for some of the behavior of the generated random sequences better than the true generative process which limits access to more recent history. They go on to show that earlier versions of text-davinci do poorly at sequence generation, while the most recent version does have “subjective randomness behavior”, similar to their proposed model.  They also show that specific sub-sequences occur more frequently than would be expected from a true Bernoilli process. \n\nThe authors then go on to show how many repeats of a given seed pattern are required to allow chatGPT to identify the sequence as non-random. \n\nAnd finally, they also show many repeats of a given “seed” sequence are required to make chatGPT only repeat that final pattern, vs. continuing with a psuedo-random sequence.   This sudden phase shift in how an LLM generates a pattern is argued to be similar to switching hypothesis in Bayesian model averaging", "questions": "> ChatGPT generates higher complexity sequences than chance. We speculate that this phenomenon might explained by a cognitive model that avoids sampling with replacement.\n\nI’m unsure what this means - can you elaborate what you mean by higher complexity in this context?\n\nWhat is the tokenizer doing?  Given that the whole paper is based around heads/tails sequences, how are they being represented?  Are there equal number of tokens being used for Heads and Tails?  Are tokens straddling between Heads/Tails at all, or on the punctuation, which may induce some of the observed patterns?  \n\nCorrespondingly, does it matter how the random sequence is represented?  Heads/tails might have their own strong biases not represent in random token choices as there are many examples in the training data of these tokens in various non-random contexts.\n\nWhy do the prompts specify generating 1000 tokens, but only ever the first 50 are used?  Is there a particular reason for that mismatch?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "## Originality:\nDetermining whether a binary number sequence is assessed to be random by a human has been pretty extensively studied in the literature, and it is an interesting extension to see how it compares to the behavior of an LLM judging a sequence to be random.  Some parts of their framing are novel, in particular their experiments w.r.t to the effect of seeding with a repeating pattern.  Their Window average model appears to match well the real performance of the LLMs, and they have a clear example of the Gamblers fallacy in this model during the generation.  Overall, their biggest novel contribution is that LLMs can be biased by strong local patterns when trying to generate new random sequences, and that this could be interpreted as Bayesian model selection. \n\n## Quality:\nSee weaknesses for more discussion about the potential mismatch in what are claiming and what their experiments are actually doing.  With that proviso, the experiments generally appear to be consistent with their claims.  They have an extensive literature review that is well written and easy to follow.  Their setting is well chosen, with potential to give some insights into how LLMs are finding local patterns.  \n\n## Clarity:\nThe paper is sufficiently well written, and easy enough to follow.  \n\n### Other minor comments:\n> For this reason, in Fig. 6 we show results for P (Tails) = 51%.\nIt would be clearer to also update Figure 6 to show that it is 51%, as opposed to only mentioning it in the text.\n\n> In randomness Generation tasks, we asses concept learning according\nShould be “assess”", "weaknesses": "The analogy of generating random sequences to in context learning is limited in that the model depends on LLMs getting confused when they are told they are provided with a random sequence, but in fact the “starting” sequence has strong local patterns. Admittedly, this is a short-coming of the LLM as it shouldn’t be “distracted” by this local pattern.  However, this artifact may go away when the base model is improved, and not “distracted” by the local misleading start, which could limit the usefulness of that part of the analysis. \n \nIn other words, it can be argued that the analysis is simply examining a failure model of an LLM, namely in a regime where local patterns start to dominate the predictive behavior of the system and it “forgets” the prompt. That doesn’t mean the transformer is doing any sort ICL - it’s just dominated by the local signal.   \n\nFor example, if you put in any prompt, then a long enough sequence of 001001, it may ignore the prompt and start predicting 001001… as it’s “forgotten” about the prompt due to the strong local predictable signal.     This makes the connection to in context learning more tenuous due to the potential disconnect between the prompt and the task, which may not happen in larger/newer/more refined models or in a more complicated ICL problem.  \n\nOne way to address mismatch could be the prompt being explicit about it being an ICL problem.  For example:\n\n“”“Continue the following potentially random sequence: <...>”””\n\nThen measuring the success rate of the continuation as you know the underlying generate model.   This would remove the mismatch between the prompt and the model. \n\nAdditionally, experiments for determining whether a sequence is random should include some truly random baselines.  For example, Figure 4 has no truly random baseline.  Hence, it is not clear from the current results if the model incorrectly assumes that all sequences are not random given enough observations.  This could be due to limitations in the attention or number of layers, for example, or biases in the training data.  What is the pattern where the LLM starts to fail?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KnN3SW0CZ", "reviewer": "Reviewer_F7kD", "summary": "This paper studies in-context learning (ICL) of LLMs with experiments inspired by subjective randomness.\n\nSubjective randomness is concdididerned with the human perception of randomness. A main takeaway of this framework is that strings are more likely to be perceived as random if they can be generated by a short (deterministic) program. More formally, the subjective randomness of a string can be defined as the difference between the log-likelihood of it being generated by uniformly at random, and it being generated by a determinstic program. As the former quantity is fixed (for a given length), the subjective randomness is proportional to the Kolmogorov complexity of the string.\n\nThis paper presents experiments evaluating the ability of several GPT-3.5+ models (hereafter, GPT) to generate seemingly random strings, and distinguish uniformly random strings from deterministicly generated strings. These experiments are carried out by explicitly prompting the model to accomplish each task.\n\nIn the generation experiments, the authors find that the statistics of strings generated by GPT differ from uniformly random. For example, the distribution of the running average bit is tighter around the mean as compared to uniformly random strings (Figure 5); more specifically, GPT avoids long single-value runs, i.e., exhibits the gamber's fallacy (Figure 6). Furthermore, strings generated by GPT have simple sub-sequences (as measured by Gzip file size or Levenshtein (aka edit) distance).\n\nFor the distinguishing experiments, GPT is prompted with strings of increasing length, generated either uniformly at random or from a simple regular language (e.g. (01)^n, or (010)^n). The authors find that each language has a sharp phase shift in GPT's distinguishing ability: for example, when feeding contexts from (01)^n, the likelihood that GPT predicts the context to be determinstically-generated increases from ~0.5 (a random guess) at n=3, to ~0.8 at n=4.", "questions": "- Thank you for introducing me to the area of subjective randomness. Can you confirm that my understanding of it is correct, as far as needed to understand this paper? Am I missing a key insight from this area?\n- When talking about distinguishing ability, the correct quantity to look at is the \"advantage\" of the distinguisher: Denoting the uniform distribution, the pseudorandom generator, and the distinguisher by $U$, $G$, and $D$ (respectively), this is the quantity $|Pr_{x \\gets U}[D(x)=1] - Pr_{x \\gets G}[D(x) = 1]|$. Note that the minuend is hard to compute (requires enumerating over all strings of the given length), but can be estimated with uniformly random samples. Can you confirm that this is the quantity portrayed in Figure 8? If not (for example, if it is $Pr_{x \\gets G}[D(x) = 1]$ that is portrayed), the graph might falsely claim that the distinguishing ability increases with context length, without any real gain in advantage.\n- Am I missing any key experimental result presented in the body of the paper? How do the experiments support the claim that ICL is Bayesian model selection? (How would a different outcome of the same experiments falsify this claim?)", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important and formidable question: what is really happening in ICL? Considering the complexity and opacity of LLMs, this creative approach could make a welcome addition to the literature.\n- Specifically, this paper is, to my knowledge, the first to propose subjective randomness as a method of analyzing LLMs. This is an original idea.\n- This work can serve as an interdisciplinary bridge for computer scientists into cognitive science. As more works are concerned with the \"cognitive abilities\" of LLMs (and in general, adopt a humanizing/personifying lens on LLMs), it makes sense to draw from the existing rich literature developed in the field of cognitive science itself.\n- As a reader with a computer science background (esp. pseudorandomness), I found the high-level introduction to subjective randomness to be well-written and accessible.\n- The methods presented in this paper are black-box: they only require prompting access to the language model, but do not use the weights, architecture, or training data. This is increasingly important as not all researchers reveal these attributes upon release of an LLM.", "weaknesses": "In decreasing order of importance.\n### Usage of the term \"pseudorandomness\"\nThe use of the term \"pseudorandomness\" in this paper differs significantly from how the term is commonly used in computer science literature. In particular, a distribution can only be pseudorandom with respect to a (computationally-bounded) class of distinguishers. However, throughout this paper, the ability of GPT to generate \"pseudorandom\" numbers is discussed without an explicit specification of a distinguisher. Perhaps the use of the term \"pseudorandomness\" is more appropriate in the experiments in which GPT is the distinguisher (rather than the generator), but there the generator itself is extremely trivial---it is a determinstic function, which therefiore can be distinguished by an almost-computatioanlly-trivial class of functions (that output 1 iff their input matches the (fixed!) output of the generator).\n\nTo avoid confusion with a long-standing and clearly-demarcated field of research, I suggest the authors to avoid using the term \"pseudorandom\" in this paper. Instead, they could use a term such as \"subjectively-random\" or even \"seemingly-random\", which avoids alluding to a concept that is not appropriate in this experimental setting.\n\n### What do the experimental results actually mean?\nA significant contribution of this paper is in demonstrating how subjective randomness can be used to experiment on LLMs. However, I am not sure what insight can be garnered from the particular experiments presented in this paper.\nMy understanding of the results from the experiments presented in the body of the paper is:\n1. GPT does not generate uniformly random numbers; in particular, it exhibits the gambler's fallacy.\n2. As context length increases, GPT can more confidently distinguish uniformy random strings from extremely simple, deterministically-generated strings (such as (01)^n).\n\nI find neither of these conlcusions particularly insightful towards a deeper understanding of ICL dynamics. As a suggestion for improvement, I would expand the setup of the distinguishing experiments (item 2 above) along two axes:\n- Stretch (length of generated string): Rather than testing on deterministically-generated strings (which are NOT pseudorandom in any non-trivial sense of the word), have GPT distinguish between uniformly random strings and strings generated by an actual pseudorandom generator; that is, one that takes as input a short random seed, and stretches it into a longer string.\n- Generator complexity: The automaton for detecting the language used in the current experiments is only a handful of cells large. Yet, GPT has billions of parameters. If you insist on using deterministically-generated strings in the distinguishing experiments, at least have them be generated by significantly more complex programs.\n\nOne of the main contributions claimed in page 3 was that the subjective randomness framework will provide convincing evidence that ICL carries out a form of Bayesian model selection. I could not see how the experimental results I listed above support this claim. Please let me know if I am missing a connection here.\n\n### Figures\n- The colors in the caption of Figure 2 do not align with the figure itself. This is one of the main figures in the paper! On that note, I'd suggest using \\color{} in the caption when referring to the different colors.\n\n- Please review all figures in the body of the paper for colorblind-accessability. I am colorblind and had a hard time deciphering Figure 8. There are plenty of guides online on how to make accessible figures, and I am happy to point you to specific ones if that is needed.\n\n- What does \"Probability of Concept\" mean on the y axis of Figure 8? My understanding is that it is the probability that, when prompted by a context of length $|x|$ from the regular language, the model responds that the input is deterministically-generated. Is that correct? (See also the last question in Questions section.)\n\n### Writing on page 7\nThe mathematical notation in the sum on page 7 was confusing (the righthand side that following \"and computing\"): I didn't understand what is $(y_d == 1)^{(i)}$--though I may have some guesses, please give a full definition of notation you are using in your paper.\n\nThere is a typo in the same paragraph: \"asses\" should be \"assess\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MZJoxtJytr", "reviewer": "Reviewer_rqBo", "summary": "This paper proposes a cognitive interpretability framework (IMO, a Bayesian model selection framework similar to those used in iterated learning or rational speech model) to analyze the dynamics of LLM conducting in-context learning (ICL). The analysis can help us understand what the latent concept is (it seems hard to understand this term without reading [1]) in ICL. Specifically, the paper considers a random binary sequence-generating problem and GPT series models for experiments. They find the model exhibits very interesting behaviors, e.g., it matches the Bayesian hypothesis selection process well; there is a sharp phase change when manipulating the prompting input length, indicating specific hypothesis becomes dominant, etc. Although the paper studies a very important problem by combining results in different fields, I cannot give a clear-cut conclusion of what is the core contribution of this paper. It would be helpful if the authors could highlight some core concepts in a relatively simple way and explain the results more. Also, I have several concerns about the technique details of the analysis (see the weakness part). After reading the paper several times, I believe the paper has big potential. I would give the current version a 3, but I would be happy to increase my score if my concerns are tackled during the rebuttal.\n\n[1] Xie, Sang Michael, et al. \"An explanation of in-context learning as implicit Bayesian inference.\" ICLR 2022", "questions": "1. For Figure 2, what do x (green), LLM (yellow), and ICL (purple) mean? There is no such color in the figure.\n2. I feel confused about the notation $p(y=Random | x)$, I guess $y\\in[ H, T ]^n$ or $y\\in [ 0, 1 ]^n$ is the output sequence. Should it be $p(h=Random | x)$?\n3. In section 4, the notation of $\\bar{x}_ {t-w…t}$ in $p(y|x)$ is not quite clear. I guess it should be $\\bar{x}_ {t-w,…,t}$.\n4. Figures like Figure-5-right are quite hard to understand. What is the take-away message? Is this saying GPT performs more similar to the window average model and less similar to the Bernoulli model? (in terms of how wide the curves spread around the converged mean)\n5. In the last paragraph of section 4, what is the $C$ in $x=C^{|x|}$ and $y_t\\in C$ means. Where is $C$ defined? (I can find $c$ and $\\mathcal{C}$ in section 2, but cannot find $C$).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This is an interesting interdisciplinary study of deep learning and cognitive science, which is quite novel to me. The results of the simple binary sequence are quite persuasive.", "weaknesses": "1. The paper is very hard to follow. Section 1, 2, and 3 provides the background, preliminaries, and the system setting of the paper, but I find it hard to extract the system setting of the paper. Most of the text in this part discusses some insights and hypotheses from related works in these two fields. It would be helpful to squeeze this part and discuss the related works in another section.\n2. The results on binary sequence are good, but it is not sure whether the findings can be extended to other realistic tasks. It would be helpful if the author could verify these findings in more real datasets, e.g., math problems, question answering, etc.\n3. There is not so much discussion about the Bayesian model. Going deeper in this direction will help the paper a lot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)