Title: Schema-learning and rebinding as mechanisms of in-context learning and emergence
Venue: NeurIPS 2023
Accepted: yes
Keywords: mechanistic interpretability

in-context learning

emergence

large language models
Abstract: In-context learning (ICL) is one of the most powerful and most unexpected capabilities to emerge in recent transformer-based large language models (LLMs). Yet the mechanisms that underlie it are poorly understood. In this paper, we demonstrate that comparable ICL capabilities can be acquired by an alternative sequence prediction learning method using clone-structured causal graphs (CSCGs). Moreover, a key property of CSCGs is that, unlike transformer-based LLMs, they are {\em interpretable}, which considerably simplifies the task of explaining how ICL works. Specifically, we show that it uses a combination of (a) learning template (schema) circuits for pattern completion, (b) retrieving relevant templates in a context-sensitive manner, and (c) rebinding of novel tokens to appropriate slots in the templates. We go on to marshall evidence for the hypothesis that similar mechanisms underlie ICL in LLMs. For example, we find that, with CSCGs as with LLMs, different capabilities emerge at different levels of overparameterization, suggesting that overparameterization helps in learning more complex template (schema) circuits. By showing how ICL can be achieved with small models and datasets, we open up a path to novel architectures, and take a vital step towards a more general understanding of the mechanics behind this important capability.

Schema-learning and rebinding as mechanisms of in-context learning and emergence

Abstract

In-context learning (ICL) is one of the most powerful and most unexpected capabilities to emerge in recent transformer-based large language models (LLMs). Yet the mechanisms that underlie it are poorly understood. In this paper, we demonstrate that comparable ICL capabilities can be acquired by an alternative sequence prediction learning method, namely clone-structured causal graphs (CSCGs). A key property of CSCGs is that, unlike transformer-based LLMs, they are interpretable , which considerably simplifies the task of explaining how ICL works. We show that ICL in CSCG uses a combination of (a) learning template (schema) circuits for pattern completion, (b) retrieving relevant templates in a context-sensitive manner, and (c) rebinding novel tokens to appropriate slots in the templates. We go on to marshall evidence for the hypothesis that similar mechanisms underlie ICL in LLMs. For example, we find that, with CSCGs as with LLMs, different capabilities emerge at different levels of overparameterization, suggesting that overparameterization helps in learning more complex template (schema) circuits. By showing how ICL can be achieved with small models and datasets, we open up a path to novel architectures, and take a vital step towards a more general understanding of the mechanics behind this important capability.

1 Introduction

In a pre-trained sequence model, in-context learning (ICL), or few-shot prompting , is the ability to learn a new task from a small set of examples presented within the context (the prompt) at inference time. Surprisingly, large language models (LLMs) trained on sufficient data exhibit ICL, even though they are trained only with the objective of next token prediction [1, 2]. A good deal of the ongoing excitement surrounding LLMs arises from this unexpected capacity, since it dramatically enlarges their set of potential applications. Ongoing attempts to understand this capability take a variety of forms, including higher-level normative accounts using Bayesian inference [3], and mechanistic explanations involving implicit gradient descent [4] or induction heads [5]. Despite this, the mechanisms that underlie ICL in LLMs remain somewhat mysterious.

We take an alternative approach, studying a sequence-learning model called a clone-structured causal graph (CSCG) [6, 7] to reveal the conditions that drive ICL. We show that ICL can be explained as a combination of (a) learning template circuits for pattern completion, (b) retrieving relevant templates in a context-sensitive manner, and (c) rebinding of novel tokens to appropriate slots in templates [8]. Unlike n-gram models, CSCGs allow transitive generalization in the latent space: they assign semantically sensible non-zero probabilities to sequences never seen during training to ensure that the contexts (prompts) used for retrieval are not pure memorizations. In addition, the binding of novel tokens to slots in learned templates allows the same structural knowledge to be applied to entirely novel inputs. We hypothesize how similar mechanisms could exist in transformer-based LLMs. By

elucidating the principles that underpin the mechanics of ICL, we hope to pave the way for the design of novel architectures for abstraction and generalization, while the building blocks we identify guide the search for mechanistically interpretable [9] and editable [10] circuits in transformers [11].

2 Rebinding algorithm for clone-structured causal graphs

2.1 Background on clone-structured causal graphs (CSCGs)

Consider an agent executing a series of discrete actions a 1 , . . . , a N -1 with a n ∈ { 1 , . . . , N actions } , e.g. walking in a room. As a result of each action, the agent receives a perceptually aliased observation [12], resulting in the stream of random variables X 1 , . . . , X N with observed values x 1 , . . . , x N , where each x n ∈ { 1 , . . . , N obs } . CSCG [6] is a probabilistic sequence learning model that introduces a latent explanatory variable Z n at each timestep n , with values z n ∈ { 1 , . . . , N latent } , to model the action-conditional stream of observations as

Atransition tensor T : T ijk = P ( Z n = k | Z n -1 = j, a n -1 = i ) ∀ n represents the action-conditional dynamics. T defines a directed multigraph, whose nodes correspond to the values of z . Conditioned on an action, each entry of T is the weight of a directed edge between two nodes (from the row index to the column index of that entry). A CSCG can thus recover a graph that represents the latent causal structure [13] of the environment (see Fig. 1D for an example) which can then be used for planning.

An emission matrix E : E ij = P ( X n = j | Z n = i ) ∀ n represents the observation probabilities. CSCGs have a deterministic observation model: for any latent value z , the same observation x is always emitted. Multiple values of z can result in the same observed x , making the model overcomplete [14]. The restriction to deterministic E makes CSCGs less general than a hidden Markov model (HMM), but easier to learn [6]. A CSCG can disambiguate multiple aliased percepts (same observation x ) into distinct causes (different latent values z ) given a sufficiently long context. If the observations correspond to word tokens, CSCGs can also be used as a language model, with a single action that accesses the next token 1 .

2.2 Rebinding in CSCGs

On encountering a new environment with a similar structure but different observations, an agent can learn that environment faster by reusing the latent graph T from prior experience and relearning just the emission matrix, through a rebinding process [15]. Rebinding can be interpreted as a soft intervention on the agent's model [16, 17]. See Fig. 1D & F for examples of two rooms that share the same latent structure but different observations. When a new emission matrix binds to an existing schema, it has to respect the clone structure of the original emission matrix (Fig. 1E). The clone structure function C ( · ) ∈ 1 , . . . , N obs partitions the latent state in N obs slots : two latent states z = i and z = i ′ belong to the same slot iff C ( i ) = C ( i ′ ) . An emission matrix respects the clone structure C if C ( i ) = C ( i ′ ) = ⇒ E ij = E i ′ j ∀ i, i ′ , j . The 3-tuple { T, C , E } defines a grounded schema , the tuple { T, C} defines an ungrounded schema with clone structure and T alone is a schema [15].

2.2.1 Fast rebinding by attending to surprise

Often, environment changes are localized such that most of the latent structure and observation mapping is preserved while just a few observations need to be rebound: for example, just replacing the carpet in a room while the wall colors remain the same, or getting exposed to a new word in a familiar context. This insight can be utilized to derive an algorithm that focuses the update of the emission matrix only to those observations that were found surprising by the existing model.

Suppose that at test time, a grounded schema { T, C , E 0 } is exposed to a sequence with novel observations. Algorithm 1 proposes a fast procedure to update the emission matrix to the new observations by only performing local updates, and to bind the updated emission matrix to the existing schema T , defining a new grounded schema { T, C , E rb } . We call this process fast rebinding .

Room 1

slot 1

slot 2

slot 3

slot 4

slot 5

slot 6

slot 7

slot 8

slot 9

slot 1

slot 2

slot 3

slot 4

slot 5

slot 6

slot 7

slot 8

slot 9

emission matrix learns mapping

from slots to observations

observations

Room 2

clones (latent space)

slot 1

slot 4

slot 2

slot 7

slot 3

slot 6

Room 1

t=0

observations

actions

…

A

D

E

F

B

C

clones

observations

z

n

z

n

+1

x

n

x

n

+1

observations

latent states

z

n

z

n

+1

x

n

x

n

+1

a

n

emission

matrix 2

emission

matrix 1

different

floor & wall colors

Given a prompt ( x 1 , . . . , x N ) and a surprise threshold, Algorithm 1 proceeds by (a) identifying emission matrix entries that need updating then (b) updating these entries using the ExpectationMaximization (EM) algorithm [18]. The conditional probability P ( X n = j | x \ n ) of tokens at timestep n given all other tokens is used to identify timesteps and latent states that are surprising. Step 3 identifies anchors , i.e., latent states corresponding to observations that are correctly predicted with high confidence: anchors are not rebound. Step 4 identifies candidates for rebinding as latent states (a) not among the anchor states and (b) corresponding to timesteps at which observations are incorrectly predicted with high confidence. Finally, instead of re-learning the whole emission matrix [15, Appendix A.2], Step 5 (detailed in Appendix A) locally updates the emission matrix by only applying EM on the latent states and timesteps identified in Step 4. As a result, only a small subset of rows differ between E 0 and E rb . Protected rows correspond to either (a) anchors in the current prompt or (b) slots not relevant to the current prompt but possibly relevant to future observations. Section 6 discusses how a similar mechanism could be implemented in transformers.

Algorithm 1 - Fast rebinding algorithm

Input: Grounded schema { T, C , E 0 } , pseudocount ϵ , prompt ( x 1 , . . . , x N ) , surprise probability threshold p surprise .

Output: Rebound emission matrix E rb

1: Define ˜ E 0 ∝ E 0 + ϵ , with normalized rows.

2: For timestep n , use the emission matrix ˜ E 0 to compute P ( X n = j | x \ n ) = P ( X n = j | x 1 , . . . , x n -1 , x n +1 , . . . , x N ) , ∀ j ≤ N obs

3: Identify latent states and timesteps that can act as anchors: A = { ( i, n ) ∣ ∣ P ( X n = x n | x \ n ) > p surprise , and C ( i ) = x n }

̸

4: Identify latent states to be rebound (and their timesteps): R = { ( i, n ) ∣ ∣ P ( X n = j | x \ n ) > p surprise , j = x n , ( · , n ) / ∈ A , ( i, · ) / ∈ A and C ( i ) = j }

5: Fix T , and use EM to update the emission matrix (initialized with E 0 , and without using any pseudocount) by only using the beliefs for latent states i and timesteps n such that ( i, n ) ∈ R .

[

B

D

A

C

E

]

[

B

D

A

C

E

]

M

P

K

N

R

[

B

D

A

C

E

]

one

eating

robber

river

bank

resort

milk

honey

bread

butter

\n

and

at

one

eating

robber

river

bank

resort

milk

honey

bread

butter

\n

and

at

clones

clones

separation of contexts

transitive generalization

flexibly branching/merging sequences with sharing

A

B

E

F

G

reverse sequence

reverse lengths 3 to 5

reverse with rapid rebinding

[

B

D

A

C

E

F

]

copy sequence

C

clones

[

1

0

]

parity

D

clones

After rebinding, we complete the prompt by performing MAP inference conditioned on the provided prompt in the rebound CSCG. We run the max-product algorithm [19] forward (the backward messages are all uniform) thus generating a series of MAP observations for the tokens following the prompt. We stop once we generate a delimiter token. See Algorithm 2 in Appendix B for details.

3 Outline of the overall argument using CSCG

3.1 Context-dependent latent representations and transitive generalization

The clone structure of CSCGs allows context-based separation and appropriate blending for language modeling. For example, the sense of the word 'bank' in 'bank robber' is different from the one in 'river bank'. CSCG learning disambiguates these contexts in the latent space by wiring them to different clones to improve predictive accuracy. In Fig. 2A, the sentences 'river bank resort', and 'one bank robber' use different clones of 'bank'. Sequences can have probabilistic branching: 'one bank robber' can terminate at ' \ n', or continue to 'eating at river bank resort' or 'eating bread and honey', or 'eating bread and butter at river bank resort' (Fig. 2B). CSCGs also allow the merging of contexts that result in transitive generalization: even if training data has only the sequences 'bread and butter', and 'milk and honey', if they go through the same clone state 'and', the model will generalize to 'bread and honey' and 'milk and butter', assigning non-zero probability to those sequences. Due to the combination of context-sensitive separation and transitivity, related topics, concepts, and algorithms get clustered into sub-networks that pass through the same clones. A prompt's context would activate its sub-network, and transitive generalization allows for prompts that are not exact memorizations. As we show in Section 4, the Bayesian inference perspective on ICL [3] corresponds to this context-sensitive and transitively generalizing storage and retrieval alone, and is insufficient to explain the ICL properties we consider in the next sections.

3.2 Learning flexible schemas (template circuits) and rebinding

Just like learning room layouts, CSCG can learn automata circuits [20] for sequence-to-sequence (seq2seq) algorithms. See Fig. 2 for CSCG circuits for computing parity, copying a sequence, and reversing sequences of multiple lengths. The list reversal circuit in Fig. 2E is bound to the specific symbols A,B,C,D,E used in training. For use as a template, slots in this graph must be able to appropriately bind to contents (arbitrary symbols) that occur in context at test time [8, 21]. The rebinding mechanism (formalized in Algorithm 1) can intuitively be understood as operating based on prediction errors - when the latent context strongly predicts the latent state corresponding to a time instant, but the actual observation is mismatched, rebinding adjusts the emission matrix to wire all the clones of that latent state to the surprising observation. Such a mechanism to mix and gate previous knowledge with new content allows circuits learned during training to become flexible templates

with slots that can dynamically bind to new inputs as required. For example, in the list reversal schema in Fig. 2F, tokens '[', and ']' are prior contents that detect the beginning and end of the list - these act as anchors for grounding the schema in the observations. Probabilistic branching based on the end of list token ']' allows for length generalization, whereas absorbing arbitrary symbols into the slots corresponding to A,B,C,D,E allows the algorithm to generalize to new symbols. Fig. 2G illustrates the outcome of this rebinding mechanism where the slots emitting A,B,C,D,E are respectively rebound to symbols K,M,N,P,R from the input prompt. Similarly, in the sentence 'I wrote in a notebook using a dax', rebinding can absorb the new token 'dax' into the context by binding it to a clone corresponding to 'pencil' or 'pen', and use the new word in those contexts.

3.3 Instruction-based or content-based retrieval and completion of tasks

Zero-shot task recognition as content-based retrieval using rebinding: Many striking examples of zero-shot learning involve recognizing tasks from prompts, and repeating them on new inputs. For example, given a prompt 'Input: [p, q, r, s] Output: [p, p, q, q, r, r, s, s]; Input: [l, m, n, o] Output: [l, l, m, m, n, n, o, o]' LLMs can infer the task as repeating the elements of the sequence, and apply that to complete the output for a new input prompt even when the tokens 'p, q, r, s, l, m, n, o' were not seen during training, in association with this task. The rebinding mechanism offers a natural explanation for this. Given the prompt, expectation maximization (EM) [18] simultaneously evaluates the different rebindings to multiple latent algorithm schemas to infer the best binding, which is then applied to complete the query prompt.

Instruction-based retrieval: When algorithms are trained with prefixed language instructions, CSCGs learn instruction sub-networks that directly point to the circuits that represent the algorithms (see Section 4.2). The algorithm can be retrieved by direct prompting with language instructions that can be significantly different from training instructions due to transitive generalization and rebinding.

3.4 Emergence

We hypothesize and empirically demonstrate in Section 4, that emergence is explainable as the combined effects of the above properties (context-separation, transitive generalization, schemaformation, and rebinding), model capacity, and patterns in the data. Training on a bigger dataset results in the induction of more templates that might not have occurred in the smaller dataset. Learning the schematic circuits for more complex algorithms or more patterns in the data requires greater model capacity because overparameterization helps in the optimization process.

4 Results

Wesubstantiate the above argument using empirical results on three datasets: (a) the GINC benchmark introduced in [3], (b) a suite of algorithm learning tasks that we introduce in our LIALT datasets, and (c) a zero-shot word usage induction task on a CSCG language model.

4.1 Context-sensitive retrieval on GINC dataset matches Bayesian inference explanation

Dataset: The GINC dataset [3] introduced for studying ICL, is generated from a uniform mixture of five factorial HMMs [22]. Each factorial HMM is referred to as a concept . A document is created by concatenating independent sentence samples from a concept. The in-context test prompts have examples of lengths k ∈ { 3 , 5 , 8 , 10 } , varying in number from n = 0 to n = 64 , with 2500 prompts for each setting ( k, n ) . Each prompt uniformly selects a concept, samples n -1 examples x (1) : k , . . . , x ( n -1) : k of length k , and one example x ( n ) : k -1 of length k -1 . The in-context task is to infer the most likely last token of the last example, i.e., argmax x ( n ) k -1 p ( x ( n ) k -1 | x (1) : k , . . . , x ( n -1) : k , x ( n ) : k -1 ) . Since the vocabulary is shared among different latent concepts, observations in GINC are aliased like in natural language, and solving the task requires the model to disambiguate the aliased observations to correctly infer the latent concepts.

Training: We train a single CSCG with 50 clones on the GINC dataset for 100 full-batch EM iterations using a pseudocount [6] of ϵ = 10 -2 . Given a test prompt, CSCG infers the most likely hidden sequence for that prompt, then predicts the next most likely observation.

B

In-context accuracyon theGINCdataset

trainedonthe.GINCdataset

CSCGwith50clones

CSCGwith10clones

In-context accuracy

0.8

0.6

0.4

0.2

+

+

0

k=5

k=10

k=5

+

k=10

0

No.of examples

60

0

No. of examples

60

CLatent state inference for different prompt lengths

oy

oywr

mroya

oywrmro

oywrmroyaj

Results: CSCG learns different latent sub-networks corresponding to the five latent concepts in the GINC dataset ( Fig. 3A), and inference on a supplied prompt retrieves the correct latent subnetwork (Fig. 3C). Increasing the prompt length improves the localization of the sub-network and the particular states within the sub-network. Figure 3C visualizes the decoded latent state distribution for an example prompt in the zero-shot setting ( n = 0 ). The decoding starts out uncertain, and improves as the prompt gets longer. This localization (on the graph) results in effective schema retrieval, and hence accurate prompt completion. Figure 3B[left] reports the in-context accuracy-defined as the average ratio of correct predictions-for each ( k, n ) pair of the GINC test set. CSCG in-context accuracy matches the patterns exhibited by LSTMs and transformers in [3], while slightly improving their performance. Fig. 3B also shows that a CSCG with larger capacity, i.e. with 50 clones per token, better separates the latent concepts and significantly outperforms a CSCG with only 10 clones per token. Fig. 9[left] in Appendix C displays the CSCG in-context confidence: for larger contexts, CSCG is better at disambiguating aliasing and the averaged predictions probabilities are higher. Finally, Fig. 9[right] shows that similarly to the transformer and LSTM in [3], CSCG fails at ICL when test prompts are sampled from concepts unseen during training. The GINC results match the context-based retrieval argument in Section 3.1: ICL in this setting is the retrieval of a shared latent concept between the prompt and the model. By using the long-range coherence of concepts in the training documents, the model learns to separate concepts into different latent representations. Despite the train and prompt distribution mismatch [3], CSCG succeeds at prompt completion because the representation allows transitive mixing.

4.2 Learning schemas for seq2seq algorithms and generalization using rebinding

Training dataset: To test the ability of CSCG to learn algorithms that generalize to novel inputs not seen during training, we construct the Language Instructed Algorithm Learning Tasks (LIALT) dataset. The LIALT training set contains demonstrations of 13 list and matrix algorithms displayed in Fig. 4A[top-left]. A demonstration consists of a multi-word language instruction-each algorithm has five different instructions-followed by 10 input-output examples of that algorithm. See Tables 2 & 3 in Appendix D.1 for the complete list of instructions used. For each instruction, the dataset contains 20 demonstrations. Within a demonstration, the language instruction and the examples are separated by a '/' delimiter. Demonstrations are separated by a ' \ n' delimiter. The input lists and matrices values are created by uniformly sampling from a vocabulary of 676 tokens, created by random pairings of uppercase letters. List operations examples vary in length from 3 to 6 , and the matrix operations are of sizes 2 × 2 or 3 × 3 . Fig. 4A [bottom-left] shows the training data format.

A

Language Instructed Algorithm Learning Tasks

B

Example learned circuit

Clone-stacked view

Unrolled view

Algorithms

Training set format and examples

repeat twice

reverse

print alternate even/odd

circ shift forward/backward

return nth element

return element at index

roll columns 1 step

transpose

diagonal

list

operations

matrix

operations

algo

k

language description /

in1 algo

k

(in1

)

/.../

in

M

algo

k

(in

M

)

/

five variations

reverse the list / [ PZ LM RT ] [ RT LM PZ ] / [ QR FC JJ ] [ JJ FC QR ] /

[ 2 r G J 7 ] [ 7 J G r 2 ] / [ a b c d ] [ d c b a ]

[ a b 1 d m ]  a / [ X a 2 3 ]  X

flip the list / [ QM AY JQ HH ] [ HH JQ AY QM ] /

Test set 1: instruction based retrieval

Test set 2: example based retrieval

language instruction / novel input completion

in1 algo

k

(in1

)

/

completion

reverse the list / [ XY KL MR] [ MR KL XY]

prompt

prompt

reverse the list

reverse the list

In-context accuracy by task

Overallocation ratio

Instruction

based

prompts

Example

based

prompts

In-context

accuracy

In-context

accuracy

0

0

1

1

0

0

1

1

0.1

0.3

1.0

3.0

0.1

0.3

1.0

3.0

Overallocation ratio

Test dataset: LIALT has two test datasets, respectively containing: (a) instruction-based retrieval prompts, and (b) example-based retrieval prompts. An instruction-based retrieval test prompt consists of a natural language instruction followed by a single input. An example-based retrieval test prompt consists of a first input-output example of an algorithm, without any natural instruction, followed by a second input. All the lists and matrices in the two test datasets contain novel tokens. For both types of prompts, the in-context task is to predict the algorithm's output when applied to the (last) input. Note that for an example-based prompt, CSCG has to infer the algorithm used from the first example. Each test set contains 100 prompts, constructed by uniformly sampling instructions, and list or matrix tokens. Fig. 4A [right] shows the formats of these two test sets.

Training: For each token, a CSCG allocates its number of clones proportionally to the number of distinct contexts in the training data in which it occurs 2 . We parameterize CSCG capacity via this proportionality factor - the 'overallocation ratio'. We train CSCGs for an increasing sequence of overallocation ratios on the training data with 500 EM iterations and a pseudocount of ϵ = 10 -6 . After running EM, we run 10 iterations of Viterbi training [23].

A

88

matrix operations

alt. odd

list operations

repeat twice

elem. at idx

8888

euoe

alt.

even

8

8

transpose

88

1st elem.

8

circ shift bw

8

roll columns

2nd elem.

circ shift fw

B

0

p(Xn

二

Tn  n)

1

/[

rtgosk

][krtgos

/[

[xjd]

1/

ｒtｇｏsk］[ｋsoｇtr]

] / [xjd]

C

before rebinding

...

./[xjd］[NPFIZ］

O

before rebinding

.../[x jd］[NP FI LZ］

after 1 EM iter.

··.

./[xjd］[dxj]

after 1 EM iter.

.../[xjd][djx]

8

X

Results: CSCGs with sufficient model capacity successfully learn the algorithms in the training set, and rebinding generalizes those algorithms to novel tokens. Fig. 4B shows the extracted circuit for the list reversal algorithm. Fig. 5[left] presents the in-context accuracy of CSCGs (using ϵ = 10 -6 and p surprise = 0 . 1 ) on the two LIALT test sets: the best performing CSCG (a) successfully rebinds the learned schemas to the test prompts' novel tokens and (b) correctly infers the algorithm from a single input-output pair for example-based prompts. Fig. 5 also shows that model size drives ICL performance [left] even when breaking down the performance by tasks [right].

The learned CSCG (initialized with an overallocation ratio of 3 ) is visualized in Fig. 10 in the Appendix, using stacked clones. Fig. 6A shows the transition graph using the Kamada-Kawai algorithm [24]. It reveals thirteen loosely connected clusters corresponding to the thirteen algorithms present in the LIALT dataset. Fig. 6B illustrates the rebinding process, with the decoded distributions over latent states of the learned CSCG model, for two different example-based prompts. Even before any rebinding, the identification of anchors and slots already restricts the decoding to schemas compatible with the prompt structure -in this case based on brackets & delimiters. However, the structure is insufficient to disambiguate completely between the compatible schemas (list operations corresponding to reversal, circular forward shift, and circular backward shift), and both the chosen prompts result in the same latent state distribution. Hence, the decoded distribution after the first E-step localizes to the three compatible schemas. In the M-step that follows, the slots in all three schemas will be rebound for this prompt. At the end of the first EM iteration, the new bindings for slots in the correct schema will be highly certain given the consistent evidence, while inconsistent evidence will lead to uncertain bindings for the other slots. In the E-step of the second iteration, the respective levels of certainty in the bindings then help promote the correct algorithm schema to become the most likely decoding-and complete the prompt appropriately. Note that a single EM step is sufficient to derive the correct rebinding in these examples. Compare Figs. 5 & 11, and the

tables in Appendix Sec. D.3 for how the in-context completion performance after the first EM step in the rebinding process is very similar to that at the end of EM convergence.

The LIALT results substantiate the arguments we made in Sections 3.2 and 3.3. Bayesian inference of the latent context based on long-term coherence (sufficient for the GINC results in Section 4.1) does not explain the remapping of a latent representation to completely new tokens as required for generalizing on the LIALT algorithms. Without rebinding, even a prompt containing a full-length example of an algorithm but with novel tokens does not retrieve the correct algorithm schema or produce the correct completion based on inference over the latent states alone (Fig. 6B, first row). By contrast, simultaneously inferring the rebindings and the latent states results in accurate retrieval of the algorithm schema and the correct prompt completion (Fig. 6B, second row). CSCGs are thus able to learn seq2seq algorithms and generalize those algorithms to novel tokens using rebinding.

Emergence: ICL performance of CSCG on the LIALT dataset shows characteristics attributed to emergence. In-context accuracy has a clear dependency on the level of overparameterization of CSCG, offering evidence in support of our hypothesis in Section 3.4.

4.3 Dax test

In language, the 'dax' test [25] is used to demonstrate the capability of a model to absorb the usage of an entirely new word from a single presentation. To test for this capability, we train a CSCG on the PreCo dataset [26], which is a large-scale English dataset for coreference resolution. We then test the model on five word-replaced query prompts, where certain words in the prompts do not appear in the training set. We use Algorithm 1 with ϵ = 10 -6 and p surprise = 1 16 to rebind the emission matrix on each of these prompts, each time probing the model for completing a sentence by filling in the blanks (uncertain inputs) using MAP inference. Fig. 7 shows these results.

Replacement query prompts

Fill in the blanks probing

after rebinding

... make global heating worse !

global heating may be a big problem, ...

->

heating

warming

... victors receive a new computer ...

contest victors will be announced ...

->

victors

winners

... have n't found life on other terras yet .

seven other terras also go around the sun .

->

planets

... the field of AG intelligence .

.... dangerous in this AG intelligence progress then ?

->

AG

artificial

... most people ride cycles to school .

... more people ride their cycles around the world .

->

cycles

bikes

unseen word

5 Related work

In-context learning: Similar to how humans learn by analogy [27] and how synaptic plasticity allows the brain to rapidly adapt to a new task [28], ICL capabilities [1] allows a pre-trained model to learn a new task given only a few examples. [29, 30] showed how demonstrations that explicitly guide the reasoning process improve the ICL performance of transformers on new complex tasks. We clarify below some concepts that should not be confused with ICL, and then discuss some works that aim at understanding ICL and the factors that influence it.

Supervised learning (SL) and few-shot learning (FSL): SL approaches learn a mapping that minimizes a loss on the training data: gradient methods are a popular paradigm [31, 32, 33]. In FSL, a model learns to rapidly adapt to a new task from a limited number of supervised examples [34, 35, 36], and performs this same task at inference. In contrast, ICL tasks are only revealed at inference. [37, 38] showed that finetuning transformers on ICL instructions improves their ICL performance.

Meta-learning: The meta-learning paradigm aims at learning to adapt to a new task with only a few examples [39, 40, 41] by using multiple learning experiences. In contrast, ICL directly emerges from the pre-trained model. [42, 43] proposed a meta-learning framework for ICL where the model is fine-tuned: it learns to leverage few-shot examples and to adapt to new tasks at inference time.

How ICL works: [3] explained ICL as implicit Bayesian inference and constructed the GINC dataset (see Section 4.1) for demonstrating ICL. [44] abstracted ICL as an algorithm learning problem and found that a transformer can implicitly infer a hypothesis function. Similarly, [45] showed that a transformer can be trained to perform ICL of unseen linear functions, with performance

comparable to the optimal least squares estimator. [46] showed that, in the linear case, transformers implicitly implement gradient descent and train an implicit linear model on the ICL examples. [4] proposed a dual between transformer attention and gradient methods and suggested pre-trained models as meta-optimizers. They presented ICL as implicit finetuning, where the forward pass on the demonstrative examples produces meta-gradients. Finally, [5] showed the existence of 'induction heads' in transformers, that emerge during training, copy previous patterns, and drive ICL capacities.

What influences ICL: [1, 47] indicated that LLMs' ICL performance 'emerges' then keeps improving when the model size increases. [48] proposed a substitute for the positional encoding, and demonstrated how transformers can learn schemas for algorithmic tasks and generalize to test sequences longer than any seen during training. Some works have highlighted the role of the training data in ICL. [49] showed that ICL emerge when the training data has a large number of rare classes and when examples appear in clusters. while [50] demonstrated that ICL emerge when a model is trained on a combination of multiple corpora, and that low perplexity and ICL performance do not always correlate. [51, 52] found that ICL is highly unstable and is influenced by the prompting template, the selection of in-context examples, and the order of the examples. [53] showed that the ICL performance is driven by the exposure to the label space, the input distribution, and the overall format of the sequence. Similarly, [54] found that selecting ICL examples with closer embeddings to ICL test sample improves ICL performance, and [55] showed that adding explanations in-context improves performance. Finally, [56] recently claimed that the sharp emergence of ICL in larger models might be an artifact of the metrics, not a fundamental property of the model.

6 Discussion

With ICL deconstructed into schema learning, schema retrieval, and slot rebinding, an interesting avenue for future work would be to probe various sequence models for how robustly each of these components are manifested - or even construct models around these principles. Here we consider how this framework might map to transformers, where the phenomenon of ICL was originally observed. Unlike CSCGs, transformers buffer the inputs and represent location as positional encoding, allowing attention to gate by the structure of the prompt, along with the contents. Prior explanations [3, 4] do not distinguish the role of sequence positions vis-a-vis contents; we argue that theories might need to emphasize this distinction (see Fig. 8A) to fully understand the inductive biases behind ICL. We conjecture (see Fig. 8B) that layers of the transformer implement multiple mixed templates of positions and content, evaluated at different offsets of a prompt. The template assembly that can auto-regressively match the prompt wins out the competition to gate the content. The rebinding mechanism requires only a few iterations of sparse updates to the emission matrix, and can be temporally 'unrolled' into a forward pass, allowing ICL behavior with fixed weights since the slotting process lives in the space of activations.

content-based

predictor

position-based

predictor

content & position

based predictor

0

1

2

3

4

5

[

A

B

C

D

]

template

0

1

2

3

4

5

[

A

B

C

D

]

template

0

1

2

3

4

5

[

A

B

C

D

]

template

gating

gating

0

1

2

3

4

5

[

A

B

C

D

]

[

A

A

B

B

C

C

D

D

]

[

P

Q

R

S

T

]

[

P

6

7

8

9

10

11

12

13 14

15

16

17

18

19

20

21

22

23

24

Template 1 evaluated at di

erent o

sets

Template 1 evaluated at di

erent o

sets

Template 1 evaluated at di

erent o

sets

Template N evaluated at di

erent o

sets

selection among templates

P

gating

0

1

2

3

4

5

[

A

B

C

D

]

0

1

2

3

4

5

[

A

B

C

D

]

0

1

2

3

4

5

[

A

B

C

D

]

0

1

2

3

4

5

[

A

B

C

D

]

[

A

A

B

B

C

C

D

D

]

[

P

Q

R

S

T

]

[

P

6

7

8

9

10

11

12

13 14

15

16

17

18

19

20

21

22

23

24

Template 1 evaluated at di

erent o

sets

Template 1 evaluated at di

erent o

sets

Template 1 evaluated at di

erent o

sets

P

position

content

A

B

template

template

template

gating

gating

gating

selection among templates

Template N evaluated at different offsets

Coming back to CSCGs, implementations can scale to larger models and datasets by exploiting sparsity and parallelizing computations in the EM steps. Allowing a factorized latent space and adding skip connections would also allow compositionality while enabling scalability. Further, while we have illustrated here the concept of rebinding to attach new symbols to existing slots, rebinding 'through time' can also target connections between clones, enabling compositional behavior incontext. We leave these explorations for future research. Our goal here has been to elucidate a general framework for ICL behavior, leveraging the interpretability of CSCGs. We hope this demystifies the ICL behavior observed in LLMs by analogy, showcases avenues for further research on ICL capabilities, and provides broad impetus for interpretable methods.

References

Tom Brown et al. 'Language models are few-shot learners'. In: Advances in neural information processing systems 33 (2020), pp. 1877-1901.

Taylor Webb, Keith J Holyoak, and Hongjing Lu. 'Emergent Analogical Reasoning in Large Language Models'. In: arXiv preprint arXiv:2212.09196 (2022).

Sang Michael Xie et al. 'An explanation of in-context learning as implicit bayesian inference'. In: arXiv preprint arXiv:2111.02080 (2021).

Damai Dai et al. 'Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta Optimizers'. In: arXiv preprint arXiv:2212.10559 (2022).

Catherine Olsson et al. 'In-context learning and induction heads'. In: arXiv preprint arXiv:2209.11895 (2022).

Dileep George et al. 'Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps'. In: Nature communications 12.1 (2021), p. 2392.

Antoine Dedieu et al. 'Learning higher-order sequential structure with cloned HMMs'. In: arXiv preprint arXiv:1905.00507 (2019).

Murray Shanahan and Melanie Mitchell. 'Abstraction for Deep Reinforcement Learning'. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 . 2022, pp. 5588-5596.

Neel Nanda et al. 'Progress measures for grokking via mechanistic interpretability'. In: arXiv preprint arXiv:2301.05217 (2023).

Kevin Meng et al. 'Locating and Editing Factual Associations in GPT'. In: Advances in Neural Information Processing Systems 36 (2022).

Ashish Vaswani et al. 'Attention is all you need'. In: Advances in neural information processing systems 30 (2017).

Lonnie Chrisman. 'Reinforcement learning with perceptual aliasing: The perceptual distinctions approach'. In: AAAI . Vol. 1992. Citeseer. 1992, pp. 183-188.

Judea Pearl. Causality . Cambridge university press, 2009.

Vatsal Sharan et al. 'Learning overcomplete hmms'. In: Advances in Neural Information Processing Systems 30 (2017).

J Swaroop Guntupalli et al. 'Graph schemas as abstractions for transfer learning, inference, and planning'. In: arXiv preprint arXiv:2302.07350 (2023).

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

Daniel Eaton and Kevin Murphy. 'Exact Bayesian structure learning from uncertain interventions'. In: Artificial intelligence and statistics . PMLR. 2007, pp. 107-114.

Arthur P Dempster, Nan M Laird, and Donald B Rubin. 'Maximum likelihood from incomplete data via the EM algorithm'. In: Journal of the royal statistical society: series B (methodological) 39.1 (1977), pp. 1-22.

Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference . Morgan kaufmann, 1988.

Bingbin Liu et al. 'Transformers learn shortcuts to automata'. In: arXiv preprint arXiv:2210.10749 (2022).

Oren Kolodny, Arnon Lotem, and Shimon Edelman. 'Learning a generative probabilistic grammar of experience: A process-level model of language acquisition'. In: Cognitive Science 39.2 (2015), pp. 227-267.

Zoubin Ghahramani and Michael Jordan. 'Factorial hidden Markov models'. In: Advances in Neural Information Processing Systems 8 (1995).

Frederick Jelinek. 'Continuous speech recognition by statistical methods'. In: Proceedings of the IEEE 64.4 (1976), pp. 532-556.

Tomihisa Kamada, Satoru Kawai, et al. 'An algorithm for drawing general undirected graphs'. In: Information processing letters 31.1 (1989), pp. 7-15.

Haley A Vlach and Catherine A DeBrock. 'Remember dax? Relations between children's cross-situational word learning, memory, and language abilities'. In: Journal of memory and language 93 (2017), pp. 217-230.

Hong Chen et al. 'PreCo: A large-scale dataset in preschool vocabulary for coreference resolution'. In: arXiv preprint arXiv:1810.09807 (2018).

Patrick H Winston. 'Learning and reasoning by analogy'. In: Communications of the ACM 23.12 (1980), pp. 689-703.

Katie C Bittner et al. 'Behavioral time scale synaptic plasticity underlies CA1 place fields'. In: Science 357.6355 (2017), pp. 1033-1036.

Jason Wei et al. 'Chain of thought prompting elicits reasoning in large language models'. In: arXiv preprint arXiv:2201.11903 (2022).

Denny Zhou et al. 'Least-to-most prompting enables complex reasoning in large language models'. In: arXiv preprint arXiv:2205.10625 (2022).

Léon Bottou, Frank E Curtis, and Jorge Nocedal. 'Optimization methods for large-scale machine learning'. In: SIAM review 60.2 (2018), pp. 223-311.

Sebastian Ruder. 'An overview of gradient descent optimization algorithms'. In: arXiv preprint arXiv:1609.04747 (2016).

Diederik P Kingma and Jimmy Ba. 'Adam: A method for stochastic optimization'. In: arXiv preprint arXiv:1412.6980 (2014).

Yaqing Wang et al. 'Generalizing from a few examples: A survey on few-shot learning'. In: ACM computing surveys (csur) 53.3 (2020), pp. 1-34.

Li Fei-Fei, Robert Fergus, and Pietro Perona. 'One-shot learning of object categories'. In: IEEE transactions on pattern analysis and machine intelligence 28.4 (2006), pp. 594-611.

Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 'Human-level concept learning through probabilistic program induction'. In: Science 350.6266 (2015), pp. 13321338.

Jason Wei et al. 'Finetuned language models are zero-shot learners'. In: arXiv preprint arXiv:2109.01652 (2021).

Victor Sanh et al. 'Multitask prompted training enables zero-shot task generalization'. In: arXiv preprint arXiv:2110.08207 (2021).

Devang K Naik and Richard J Mammone. 'Meta-neural networks that learn by learning'. In: [Proceedings 1992] IJCNN International Joint Conference on Neural Networks . Vol. 1. IEEE. 1992, pp. 437-442.

Sachin Ravi and Hugo Larochelle. 'Optimization as a model for few-shot learning'. In: International conference on learning representations . 2017.

Sepp Hochreiter, A Steven Younger, and Peter R Conwell. 'Learning to learn using gradient descent'. In: Artificial Neural Networks-ICANN 2001: International Conference Vienna, Austria, August 21-25, 2001 Proceedings 11 . Springer. 2001, pp. 87-94.

Sewon Min et al. 'Metaicl: Learning to learn in context'. In: arXiv preprint arXiv:2110.15943 (2021).

Yanda Chen et al. 'Meta-learning via language model in-context tuning'. In: arXiv preprint arXiv:2110.07814 (2021).

Yingcong Li et al. 'Transformers as Algorithms: Generalization and Stability in In-context Learning'. In: ().

Shivam Garg et al. 'What can transformers learn in-context? a case study of simple function classes'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 30583-30598.

Ekin Akyürek et al. 'What learning algorithm is in-context learning? investigations with linear models'. In: arXiv preprint arXiv:2211.15661 (2022).

Jason Wei et al. 'Emergent abilities of large language models'. In: arXiv preprint arXiv:2206.07682 (2022).

Yuxuan Li and James L McClelland. 'Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks'. In: arXiv preprint arXiv:2210.00400 (2022).

Stephanie Chan et al. 'Data distributional properties drive emergent in-context learning in transformers'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 1887818891.

Seongjin Shin et al. 'On the effect of pretraining corpora on in-context learning by a large-scale language model'. In: arXiv preprint arXiv:2204.13509 (2022).

Zihao Zhao et al. 'Calibrate before use: Improving few-shot performance of language models'. In: International Conference on Machine Learning . PMLR. 2021, pp. 12697-12706.

Yao Lu et al. 'Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity'. In: arXiv preprint arXiv:2104.08786 (2021).

Sewon Min et al. 'Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?' In: arXiv preprint arXiv:2202.12837 (2022).

Jiachang Liu et al. 'What Makes Good In-Context Examples for GPT3 ?' In: arXiv preprint arXiv:2101.06804 (2021).

Andrew K Lampinen et al. 'Can language models learn from explanations in context?' In: arXiv preprint arXiv:2204.02329 (2022).

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 'Are Emergent Abilities of Large Language Models a Mirage?' In: arXiv preprint arXiv:2304.15004 (2023).

A Locally updating the emission matrix with the transition matrix fixed

We reuse the same notations as in [15], Appendix A.2. The authors describe the EM algorithm for learning the emission matrix of a CSCG with a fixed transition matrix. In particular their M step defines the new emission matrix as:

where E ( j ) is a column of the emission matrix corresponding to the emission j , 1 X n = j is an indicator function, ⊘ is the element-wise division and γ ( n ) is derived by the authors from the forward and backward probabilities. The i entry of the vector E ( j ) is then defined as:

In contrast, in Section 2.2, Step 5 of Algorithm 1 only updates the row i for which we can find a pair ( i, n ) ∈ R , by only using the beliefs at timestep n . For this row, the j th entry becomes:

The pseudocount used in Step 1 of Algorithm 1 is an uncertainty parameter that lets the model smooth over incorrect observations. More details of this parameter are available in [6].

B Prompt completion algorithm

Algorithm 2 describes the prompt completion algorithm introduced in Section 2.2. It implicitly considers a single action, which takes the next sequence element.

Algorithm 2 - Prompt completion

Input: Grounded schema { T, C , E rb } with rebound CSCG emission matrix E rb , delimiter token x ∅ , prompt x ( prompt ) = ( x 1 , . . . , x N )

Output: A completed prompt x ( completed ) = ( x 1 , . . . , x N , x N +1 , . . . , x N + P = x ∅ )

1: Run max-product for MAP inference and return z MAP = ( z 1 , . . . , z N ) = argmax z P ( z | x ( prompt ) ) .

̸

2: Set ℓ = 0 . While x N + ℓ = x ∅ , increment ℓ ← ℓ +1 and sample the next most likely observation: z N + ℓ ∈ argmax j T z N + ℓ -1 , j and x N + ℓ ∈ argmax j E rb z N + ℓ , j .

C Additional materials for the GINC dataset

First, we present two additional plots for the GINC experiment.

In-context performance on the GINC dataset

0

10

20

30

40

50

60

Num examples

0.30

0.35

0.40

0.45

0.50

In-context confidence

In-context confidence on seen concepts

3

5

8

10

0

10

20

30

40

50

60

Num examples

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

In-context accuracy

In-context accuracy on unseen concepts

k=5

k=10

Second, we present the table of results associated with Fig. 3 for the CSCGs with 10 and 50 clones.

CSCG performs better on zero-shot prompts than on few-shot prompts: We observe that, for short contexts, CSCG in-context accuracy is higher on zero-shot prompts n = 0 than on few-shot prompts n = 1 , 2 , . . . . We hypothesize that the difference between the training and the prompt distributions creates a gap that lowers few-shot in-context accuracy. The performance gap disappears

for larger contexts k ∈ { 8 , 10 } as they 'overpower' the train-test distribution divergence. [3] made a similar observation for transformers. However, their performance gap was also observable for larger contexts.

D Additional materials for the LIALT dataset

D.1 Natural language instructions

Tables 2 and 3 present the natural language instructions respectively used for the nine list algorithms and four matrix algorithms of the LIALT dataset. Language instructions are grouped in clusters of five: all five instructions within one cluster describe to the same algorithm. As described in the main text, each demonstration of the LIALT training and first test set uniformly selects one instruction.

D.2 Learned CSCG model

Our next Figure 10 displays the transition graph of the CSCG model trained on the LIALT dataset with an overallocation ratio of 3, with stacked clones for each symbol.

altern

Begk

D.3 Results on the LIALT dataset

D.3.1 After a single EM iteration

Presented below are the tables of results associated with Fig. 5. Table 4 contains the in-context accuracies averaged on the entire test set, Table 5 contains the in-context accuracies per task on instructions-based prompts, and Table 6 contains the in-context accuracies per task on example-based prompts.

D.3.2 After EM convergence

Figs. 11 & 12 present analogues of Figs. 5 & 6 but after the EM algorithm in the rebinding process has converged. We note that the results are mostly identical. Table 7 contains the in-context accuracies averaged on the entire test set, Table 8 contains the in-context accuracies per task on instructions-based prompts, and Table 9 contains the in-context accuracies per task on example-based prompts.

In-context accuracy by task

Overallocation ratio

Instruction

based

prompts

Example

based

prompts

In-context

accuracy

In-context

accuracy

0

0

1

1

0

0

1

1

0.1

0.3

1.0

3.0

0.1

0.3

1.0

3.0

Overallocation ratio

p(Xn = n I n)

A

0

1

rtgosk

]

krtgos

]

]

]/

rtgosk][ksogtr]/[xjd]

.

B

before rebinding

／[xjd][NPFILZ］

before rebinding

／[xjd］[NPFILZ］

after 1 EM iter.

.../[xjd][dxj]

after 1 EM iter.

... /[xjd］[djx］

0

after EM convg.

/[xjd］［dxj］

after EM convg.

/[xjd］[djx］

0

D.4 Example failures

Finally, we present a few examples which illustrate the failure modes of our approach. These are primarily a consequence of imperfections in the learned CSCG model. Each example is presented in the format (prompt, ground truth correct output, actual model response).

For these failures, the instruction circuit has been wired to the wrong algorithm circuit (possibly driven by the ambiguity of the forward slash delimiter separating the instruction from the example), resulting in the retrieval of the wrong schema.

For these failures, the schema has been learned incorrectly.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors attempt to analyze the reason for the success of In-context learning (ICL) for few-shot learning regimes. They apply ICL to clone-structured causal graphs (CSCGs) that can be used to interpret how ICL works in LLM. The CSCG is constructed as causal graph model where for the task of next token sequence prediction. The authors use CSCGs as interpertable language models to which they apply ICL to analyze key-properties of the learning mechanism. The authors go on to show how CSCG can perform similarly to ICL tasks designed for Transformer models and then show similar properties of CSCG to Transformer models (such as over-parameterization improves performance). The insights can help design new models.\n\nSTRENGTHS:\n1. The authors use CSCG an interpretable model to study a learning method (ICL) that is of high importance to the research community. \n2. The experiments are thorough, and the authors show interesting capabilities of CSCG that they compare to a Transformer model.\n\nWEAKNESSES:\n1. It is a difficult paper to evaluate as the authors evaluate CSCG ability on standard ICL benchmarks such as 4.3 Dax test and the use of CSCG as a proxy to study ICL, e.g. 4.2 \"Emergence\". Expanding on my point, the authors choose to evaluate CSCG as a direct replacement to LLM (which does not appear to be either their premise or motivation of the work). \n2. Related to 1.; the results and figures, lack conclusions. For example, I would expect to know how / what the experiment explains for how ICL works. An example where this is done successfully is Figure 5. `In-context accuracy (with standard errors) per task on the two LIALT test sets: for each task, overparametrization improves performance.`, but is not present in other figures or sections. \n3. The jump on conclusions from a Transformer model to a CSCG is quite large. As such it appears the authors are under-delivering on their original premise, while the paper is of great interest without making the reference.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper implements Cloned Structured Causal Graphs (CSCG), a model that was previously used in Neuroscience to explain Hippocampal cognitive maps, on language tasks that require in-context learning (ICL). They show the success of CSCG's across a variety of language benchmarks and come up with a theory on the mechanisms behind ICL and validate this theory using the results from ICL in CSCG's. The properties they claim are necessary for ICL include context-based separation, context-based merging for transitive generalization, and the presence of general abstract schema that can implement certain abstract content-independent operations that can also be rebound as necessary. Targeted experiments are conducted to illustrate each of these properties with a handful of relevant benchmarks, both old and new - GINC (old), LIALT (new), and PreCo (old).\n\nSTRENGTHS:\n* I have not seen CSCG's being used for the language domain, so I think that is a novel contribution inofitself. \n\n* The properties that explain the mechanisms of ICL are very interesting and, although some I think can only be uniquely revealed through the architecture of CSCG's, seem like architecture-agnostic generalizable principles that add to the interpretability literature on ICL. \n\n* Experiments are targeted, well-organized, and thorough. The contribution of the LIALT dataset also can be a valuable test-bed for ICL abilities.\n\nWEAKNESSES:\nThe paper makes some (though I think reasonable) conjectures on how these properties are implemented in transformers. However, there isn't any empirical evidence to verify these claims. I don't think this is a strong weakness, though, because the empirical evidence from CSCG is a valid contribution by itself and this can be the topic of future work.\n\nQUESTIONS:\nWhat was the process of finding and identifying the schemas shown in Figure 2? Is there a principled way to detect abstract generalizable schemas learned during training?\n\nLIMITATIONS:\nI don't think the authors have included explicit sections discussing the limitations of using CSCG's in language modeling or on the broader societal impacts of this work. Would appreciate including this in a rebuttal, possibly with the extra page that is given to authors during the rebuttal phase.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to replicate the in-context learning (ICL) phenomenon in large language models (LLM) with clone-structured casual graphs (CSCG), which is roughly trying to learn hidden states in PODMP such that the transition matrix is invariant new environments, and only the emission matrix needs re-learning.\n\nCSCG has the properties of context-separation, transitive generalization, schema-formation, and refining, which can be used to explain ICL behaviors. Results on three benchmarks show CSCG can match some LLM ICL behaviors.\n\nSTRENGTHS:\n- ICL is important and emerging, and its understanding is timely. The paper contributes to a novel view into ICL with some interesting constructions and experiments.\n\n- The context-dependent latent representation and transitive generalization make sense.\n\n- Experiments seem interesting and help replicate some ICL behaviors on toy datasets, e.g. accuracy of ICL depends on overparametrization.\n\nWEAKNESSES:\n- CSCG is unfamiliar to most readers and there could be some confusions that hinder understanding the paper (see questions). Importantly, **I do not understand how is the transition learned in the first space**, even before any \"rebinding\". And what is even the training data for each experiment??\n\n- The setup of CSCG is still far from LLM ICL, and there should be acknowledgements of such limitations if the goal is to explain LLM ICL via CSCG. For example, learning and architecture, training task, test task...\n\nI confess I do not understand CSCG fully. It might be a super smart and cool idea with a lot of potential, and I'm willing to raise my score these are addressed.\n\nQUESTIONS:\n- How is learning (of transition matrix) done? The paper seems to only talk about re-binding for learning the emission matrix...\n\n- Why transition T is assumed to be fixed, and what kinds of train-test split do we assume this? (LLM might be hard to assume this?)\n\n- Is the latent symbol space for z huge? Is \"clone\" just defined as #latent_symbols/#observed_symbols? Also, why not several latent tokens between observation tokens? Might make symbol space exponentially more efficient and more like language compositionality?\n\n- Figure 3, is accuracy kind of low? does it really match Transformer models?\n\n- Is this whole CSCG thing kinda like a latent automation? Do you think LLM is possible to recover such symbolic structures?\n\nLIMITATIONS:\nDid not see any limitation section in the paper, but there should be some (see weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the clone-structured causal graph (CSCG) with rebinding as a model of in-context learning in language models. The authors conduct several experiments showing that CSCGs can learn latent graphs corresponding to meaningful concepts seen in the training data, while also generalizing to instantiations of those concepts containing novel tokens seen at test time.\n\nSTRENGTHS:\nThe presentation of the CSCG as a model of in-context learning is new to me and very interesting, addressing a hot topic in LLMs that is relevant to a large portion of the community. The experiments show that CSCGs can be learned in a way that leads to meaningful concepts, as well as the fact that overparameterization is useful for CSCGs as with typical LLMs. Finally, the experiments show that CSCGs can perform fast binding of novel tokens to previously-seen structures in a \"dax test\", which is an important capability the authors argue is not adequately explained by existing frameworks.\n\nWEAKNESSES:\nKey claims about interpretability are not substantiated. In particular, the abstract and intro clearly emphasize the importance of interpretability in models of language/ICL, but the experiments do not meaningfully explore how CSCGs are more interpretable than existing LLMs.\n\nSome experiments are somewhat superficial or small scale, particularly 4.3, where only a few qualitative examples are shared, without any quantitative analysis showing that the results are representative. The largest datasets are orders of magnitude smaller than the datasets used by real LLMs. This shortcoming isn't absolutely critical, but of course the results would be much more convincing if ICL phenomena could be shown at something closer to real-world scale.\n\nQUESTIONS:\nHow does the size of the latent graph for e.g. list reversal scale with list size?\n\nWhat is the computational cost of fitting a CSCG with EM? PreCo is many orders of magnitude smaller than typical pre-training corpora. Is CSCG plausible even for very large corpora?\n\nAre the results in Figure 6 cherry picked? How often is rebinding + MAP inference successful?\n\nIs there a typo in the emission matrix/clone structure definition? The number of columns of the emission matrix is the size of the observation space when it is defined, but it is the size of the latent space when the clone structure condition is defined later in 2.1.\n\nLIMITATIONS:\nThe authors should add some discussion about the limitations of CSCGs for modeling the behaviors of real-world LLMs, particularly whatever computational challenges may exist in scaling CSCG-like models to billions or trillions of tokens.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}