Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting
Authors: Suraj Anand
Michael A. Lepori
Jack Merullo
Ellie Pavlick
Venue: ICLR 2025 Poster (2025)
Keywords: Natural Language Processing
In-Context Learning
In-Weights Learning
Active Forgetting
Primary area: interpretability and explainable AI
TL;DR: We explore why in-context learning abilities on rare and unseen tokens disappears over training, and introduce a training intervention that fixes these failures.

Abstract: Language models have the ability to perform in-context learning (ICL), allowing them to flexibly adapt their behavior based on context. This contrasts with in-weights learning (IWL), where memorized information is  encoded in model parameters after iterated observations of data. An ideal model should be able to flexibly deploy both of these abilities. Despite their apparent ability to learn in-context, language models are known to struggle when faced with unseen or rarely seen tokens  (Land & Bartolo, 2024). Hence, we study $\textbf{structural in-context learning}$, which we define as the ability of a model to execute in-context learning on arbitrary novel tokens -- so called because the model must generalize on the basis of e.g. sentence structure or task structure, rather than content encoded in token embeddings. We study structural in-context algorithms on both synthetic and naturalistic tasks using toy models, masked language models, and autoregressive language models. We find that structural ICL appears before quickly disappearing early in LM pretraining. While it has been shown that ICL can diminish during training (Singh et al., 2023), we find that prior work does not account for structural ICL. Building on Chen et al. (2024) 's active forgetting method, we introduce pretraining and finetuning methods that can modulate the preference for structural ICL and IWL. Importantly, this allows us to induce a $\textit{dual process strategy}$ where in-context and in-weights solutions coexist within a single model.

Published as a conference paper at ICLR 2025
DUAL PROCESS LEARNING: CONTROLLING THE USE OF IN-CONTEXT VS. IN-WEIGHTS STRATEGIES WITH WEIGHT FORGETTING
Suraj Anand
Michael A. Lepori
Jack Merullo
Ellie Pavlick
Department of Computer Science Brown University
Correspondence to 6urajk61 0 gma coom. com.
ABSTRACT
Language models have the ability to perform in-context learning (ICL), allow- ing them to flexibly adapt their behavior based on context. This contrasts with in-weights learning (IWI.), where memorized information is encoded in model parameters after iterated observations of data. An ideal model should be able flexibly deploy both of these abilities. Despite their apparent ability to learn in-context, language models are known to struggle when faced with unseen or rarely seen tokens (Land & Bartolo 2024 Hence, we study structural in-context learning, which we define as the ability of a model to execute in-context learning arbitrary novel tokens so called because the model must generalize on the basis of e.g. sentence structure or task structure, rather than content encoded token embeddings. We study structural in-context algorithms on both synthetic and naturalistic tasks using toy models, masked language models, and autoregressive language models. We find that structural ICL appears before quickly disappearing early in LM pretraining. While it has been shown that ICL can diminish during training (Singh et al.) 2023), we find that prior work does not account for strue- tural ICL. Building on Chen et al. (2024)² 's active forgetting method, we introduce pretraining and finetuning methods that can modulate the preference for structural ICL and IWL. Importantly, this allows to induce dual process strategy where in-context and in-weights solutions cocxist within a single model
INTRODUCTION
fundamental trait of transformer language models (LMs) is their ability integrate context to adjust model representations and behavior without weight updates. This ability enables emergent phenomenon such as 'in-context' learning (ICL) (Brown al. [ong al. [023]] Garg et al.) 2023), and more generally allows models to flexibly accommodate variations in language. For instance, a model likely memorize that the token green is typically used as an adjective, yet still recognize that it is used as a noun in the sentence The child sat on the main green based on contextual information.
However, this flexibility breaks down on truly novel/unseen tokens. Recent research has found that models cannot successfully perform ICL when given undertrained (Land & Bartolo 2024 Rumbelow & Watkins
2023) or newly- -nwly-intooduced tokens (e.g. when adding languages to existing model) (Chen et al. 2024). For example, otherwise- pernsseeeeffommatt language models produce bizarre responses when queried to simply repeat undertrained token (Rumbelow & Watkins (023). Notably, this task does not require any semantic content to be encoded within the embedding of the token of interest, and so one might expect a model to implement a solution that robust undertraining.
In light of this, we distinguish two types of strategies that a language model might implement when faced with a task presented in context: conditional ICL refers strategies that sensitive to
We release code at ducibility
process- learning for repro-

---

Published as a conference paper at ICLR 2025
Natural Language Setting
Synthetic Setting
BERT
Probe (POSP)
Sequance
Query
Language
Shuchwe
cop sadj
Struuue
cadip camart
Exanpei
POSP
Prnsumi
Powt
Exampte
Hapoy happy happy
the dop happy
dog-
happy dog deg
Phroury Miod
POSP
Uneeen (Stortonn [CL
Utsison (Stuctival
cunseenti In suntemn7
carseendo cumeenns
POBP
Pircert
Temporary Forgetting
Structural In-Context Learning
Preference of fead for In-Seiktrs
Strucburil In-Contaat Peformance
Pure
1980
2006
Training
ActineTTergooaa Forpetling
Figure I: (Top Left) our naturalistic setting, we train a part-of-speech probe on BERT representa- tions of sentences from Penn Treebank 3 and evaluate it templatic examples (Section 3 (Top Right) In our synthetic setting, we train a small masked language model (MLM) on sequences where the expected response is determined based on the part-of-spece
of the query token (Section 4
(Bottom Left) An idealization of two main findings: (1) structural ICL is transient (i.e. decays over training) in both naturalistic and synthetic settings, and (2) Active/temporary forgetting maintains structural ICL in the synthetic setting. (Bottom Right) Temporary forgetting induces structural ICL when applied for N > 0 steps, enabling generalization to unseen random tokens. In-weights preference is coarsely controllable by varying temporary forgetting parameter N.
the semantic content of all tokens, whereas structural ICL refers to strategies that are invariant to the information (or lack thereof) encoded the embedding weights of least one token. Refer to Section for more precise definition. While not all tasks permit structural ICL strategies, several fundamental syntactic (Section 3 and logical (Section 7 tasks do.
A recent influential line of work has studied the development of transformers throughout training through the lens ICL vs. in-weights learning (IWL). This has resulted several notable findings, including (1) that models often adopt either ICL IWL strategies, unless the data distribution has specific. language- like properties (Chan et al.) 2022b), (2) that ICL is transient, disappearing as the models become overtrained (Singh al. (023), and (3) that Tzz-regularization mitigates ICL. transience, but instead leads to IWL transience (Singh et al.) 2023). We expand upon this framework study the relationship between conditional ICL, structural ICL, and IWL. Moreover, we aim to translate insights from these studies into actionable techniques to encourage models to flexibly deploy both IWL and structural ICL strategies. We refer this capability as dual process learning in loose analogy to Dual Process Theory (Kahneman] 2011), as IWL implements automatic, memorized operations for IID settings (à System 1) and structural ICL enables flexible, context-sensitive operations for out-of-distributie
settings (à la System 2).
In the present study, we find that structural ICL is also transient. However, while regularization provides path to persistence for conditional ICL (Singh et al.) 2023), it does not for structural ICL. Therefore, we propose an extension to active forgetting - recent weight resetting technique introduced by Chen et al. 22024 to help augment models with new tokens render structural ICL persistent. Our modification allows us to coarsely control the strategies that the model adopts during pretraining, enabling to induce dual process learning: (structural) ICL for and unseen tokens and IWL for common tokens. Finally, we demonstrate a proof-of-concepp fine-tuning strategy to induce dual process learning in pretrained causal language models.
2

---

Published as a conference paper at ICLR 2025
In summary, main contributions are:
We define and study the concept of structural ICL in both large models and toy models. We discover that both masked and autoregressive LMs exhibit (limited) form of structural in-context learning that emerges early in training, but this ability quickly vanishes.
We show that active forgetting (Chen et al. 2024) maintains structural ICL in models.
We introduce temporary forgetting, which enables one to control how much model relies on in-weights vs. in-context strategies. We find that temporary forgetting enables us to induce dual process learning under variety of data distributions, where our model uses in- -n-weights strategy for frequently tokens and a (structural) in-context solution for rarely seen tokens.
We introduce probabilistic temporary forgetting, which enables one to induce structural ICL pretrained causal language model. We demonstrate a proof-of-concey by fine- tuning GPT-2 and demonstrating structural ICL in a simple logical reasoning task.
DEFINITIONS
In-Context vs. In-Weights Learning We follow Reddy (202)), which defines in-weights learning (IWL) to be "query-respone" relationships encoded in the weights of the network" while in-context learning (ICL) emerges due to "common structural element[s]" and "can be exploited perform zero-shot learning on novel tasks that share this structure." Notably, word embeddings are purely in-weight representations, which are enriched with contextual information by attention layers.
We formulate our ICL prediction tasks as I'M (y Pi: Zi.)) where are the label(s), Pi.m is the set of positional embeddings, Z1:1 is the set of word embeddings for a sequence of length n, and M refers to the parameters of a language model.
Structural vs. Conditional ICL. We define structural ICL. precisely via an empirical test: a model exhibits structural ICL if it can complete task that presented in context in a way that is invariant to the content of one or more embeddings. For one or more word embeddings at specified position(s)
Frandom- This removes the in-weight information contained within the word embedding and forces the model to rely on on contextual information and structural analogy. We state that model can perform conditional ICL when it succeeds prediction task when Z in remains unmodified. This the standard ICL setting studied in the literature (Chan et al., 2022b Singh et al. 2023 Garg et al., 2023 Akyürek et al. 2024). Note that a model that exhibits conditional ICL will not necessarily exhibit structural ICL.
i E T, we replace replace
Head vs. Tail In skewed token distributions, we refer to the most frequently occurring tokens (typically 10%) as the head of the distribution and the least frequently occurring tokens (typically x 10%) as the tail. As token distributions increase in skew, tail tokens are seen less frequently. This dichotomy relates our analysis of structural ICL because tail tokens an interpolation between fully-trained tokens and random tokens. By accommodating random tokens through structural ICL, can also recover performance on infrequent tail tokens.
3 (STRUCTURAL) IN-CONTEXT LEARNING IS TRANSIENT
Recent work has discovered that conditional ICL capabilities slowly degrade over the course of long training in synthetic setting (Singh et al. 2023). In this section, we study the transience of conditional and structural ICL over the course of training in a naturalistic setting using BERT-style models (Devlin et al.) 2019). Using syntax probing task, we find that structural ICL rapidly degrades to completely random performance after fairly few training steps, while conditional ICL abilities degrade during significantly longer timescale. To perform this analysis,
intermediate checkpoints released from the MultiBERTs (Sellam et al. 202T), averaging all of our results across seeds 0, 1, and 2. We calculate error bars in Figure as +1 standard error of the mean (SEM).

---

Published as a conference paper at ICLR 2025
3.1 TASK
Determining the part of speech (POS) of each word in a sentence is a fundamental step toward understanding that sentence. We identify two strategies that model might employ to determine the POS of a token: (1) an in-weights strategy, where the model explicitly memorizes the POS of a token in its weights, and (2) an in-context strategy. where the model infers the POS of given token from context information. We created several templatic evaluation datasets in order to tease these two strategies apart. Each dataset contains sentences that obey the template: The <noun> is <adj> (e.g. The dog is happy).
Our evaluation datasets are defined as follows:
1. Head (Tail): Contains sentences where filler tokens are sampled from the most (least) frequent 1500 nouns and most (least) frequent 15000 adjectives in the training set of Penn Treebank 3 (PTB-3) (Marcus et al. [993). We sample 1500 unique words because this comprises R 10% of all unique nouns in PTB-3.
2. Head (Tail) Switch: Contains sentences where tokens are sampled as in the "Head" ("Tail") dataset, but where noun tokens fill the <adj> slot and adjective tokens fill the <noun> slot (e.g., The happy is dog). These datasets put the IWI. strategy and ICL strategy into conflict.
3. Random Token: Contains sentences where the <noun> and <adj> slots are filled by randomly- -inttiiiiit embeddings. This dataset evaluates structural ICL performance. This dataset appeals to the intuition that it should be possible to infer the POS of nonce tokens in sentences like "the gluck is wug."
For each layer and MultBERT checkpoint, we train separate binary POS probe on representations of nouns and adjectives from sentences the training set of PTB-3 (Marcus et al. 1993). We then evaluate these trained probes on our evaluation datasets in order to understand the strategies that models employ to determine POS at various points through training. For multi-token words, average representations across tokens (See Appendix A for additional details). Note that the MultiBERTss are trained following Devlin et al. (2019) on a combination of BookCorpus (Zhu et al. 2013) and English Wikipedia collected by Ture et al. (2019). As such, the distribution of the training data is fixed, and our experiments are constrained to the natural distribution of language. As BookCorpus does not have POS tags readily accessible, we employ PTB-3 estimate the noun and adjective distribution of the training data. We classify a word as either nouns (adjectives) if that word appears as noun (adjective) over 80% of the time. See Figure (Top Left) for more details.
3.2 TRAINING DYNAMICS
We examine (1) structural in-context learning and (2) the tradeoff between in-context and in-weight strategies over the course of training.
Structural ICL We find that the MultBERTss are able to perform structural ICL early in training, but that this capability is transient. We measure structual ICL by evaluating pretrained probe performance on the Random Token evaluation dataset. model determines POS using an in-context strategy that is invariant to the content of the probed token, then it should succeed at inferring the POS of the random tokens inserted in both slots. In Figure 2 (Left), we present accuracy on the Random Token dataset using probe trained on representations from Layer 7, as this layer achieves the highest probing validation performance on PTB-3 (See Appendix [B] for results across all layers). We find a clear signature of structural ICL transience: probe performance MultiBERT training before dropping to chance by the end of training. These results suggest that there is an inductive bias toward structural ICL that diminishes information is encoded in the model weights As structural ICL confers the ability to generalize to rare and new tokens, this finding raises questions about how we can train models that maintain this ability throughout training.
random tokens spikes early in
In-Context vs. In- In-Weights Strategies Much like Singh et al. (2023). we observe that conditional ICL strategies decay over training as information encoded model weights (e.g., token embed- dings). To approximate how much model relies on contextual information to infer the POS of tokens, we consider the difference in performance between probes trained on Layer 0 (the embedding layer) and probes trained on Layer 7 on the Head and Tail evaluation datasets. Layer 0 must rely only
We also observe structural ICI. transience in Pythia- (See Appendix
4

---

Published as a conference paper at ICLR 2025
Structural In-Context Performance
from In-Context
for In-Weights Strategy
Rendom Token Accuracy
Layer?
Val. Accuracy
0.30
Tail Heed
Random Baseline
0:35
Gain
Switch Accuracy
0.75
0.20
0.70
0.7
0.15
0.65
0.
0.00
0.00
0.00
0.05
0.33
0.50
Head
Thi
250
ADD 250 1003
1250 1960 1750 1000
132
10"
MultIBRTT
Taining Step (in 1000s)
MultiBERT Training Stop
MultiBERT Training Step
Figure 2: (Left) Structural ICL is transient, as Random Token accuracy first peaks and then decays. (Middle) We investigate the benefit of contextualization over memorization in Head and Tail datasets by examining the difference in Layer 7 Accuracy (where both in-context and in-weights strategies possible) and Layer O Accuracy (where only an in-weights strategy is possible). These differences become negligible after sufficient training. (Right) Using the Head Switch and Tail Switch datasets, we find that models begin to encode POS using an IWL strategy over time. Note that the x-axis begins at training step 20,000 for (Middle) and (Right).
on information encoded in the embedding matrix, as there is no in-context information available; in contrast, Layer 7 can contextualization to achieve higher performance (Tenney al. 2019 Hewitt jet al. 2021). We find that the benefit of contextualization fades for both Head and Tail datasets, but dissipates more quickly for the head of the distribution than the tail (See Figure 2 Middle). We hypothesize that this occurs because there are far more gradient updates to head token embeddings P
Concurrently, we measure Layer probe performance on the Head Switch and Tail Switch datasets. We observe that the model shifts from an in-context to an in-weights strategy, preferring to infer POS from token identity, rather than token position (See Figure 2 Right). other words, models become more reliant on in- weights strategies and less reliant on in-context strategies over the course of training. This finding aligns with Singh et al. (2023).
DATA DISTRIBUTION IMPACTS IN-CONTEXT LEARNING
We develop a synthetic masked language modeling task to characterize how data distributional parameters affect structural ICL, conditional ICL, and IWL. Our synthetic task requires the model to determine which of two classes word belongs to. A token's class may be inferred from contextual information or memorized in the embedding layer. This task was crafted as a simplified version of the naturalistic task investigated in Section 3
Our vocabulary contains tokens that represent nouns, adjectives, and copula (i.e., is). Each input sample is created by selecting (1) sequence S, (3) query Q, (3) two filler tokens Tnown Ladj- The query uniquely determines a response pattern P. Our MLM is trained to predict P(P.SS Q) for all (O...., Pl 1} (i.e. the probability of each pattern token). The sequence and pattern are designed so that no exceedingly simple heuristic can solve this task. Specifically, sentence templates are defined using the following elements:
sequence S: Either <noun> <copula>
<adj>
<copula>
<adj> <noun>.
query Q: Either <noun> or <adj>.
response pattern P: Either <adj> <noun> <noun> if the query is <noun> or <adj> <adj>> <adj> if the query is <adj>.
These templates populated by Znoun filling the <noun> slots and Todj filling the <adj> slots. This task is designed such that the model must make a POS classification on the query token, and then perform additional operations conditioned that classification (i.e., copying specifie token identities in a specific order). See Figure [ (Top Right) and Appendix [ for examples.
We parameterize the task with vocabulary size v, the sampling distribution skew for noun/adjective fillers (where we select Tnoun, Ladj Zi)f(a)), and the ambiguity of token POS e. The ambiguity
'e observe that performance gain due to the model's use of in io-tontext information decreases across wide range of syntactic phenomena as embeddings enriched during training. We term this the "Pushdown Phenomenon" " and explore thoroughly in Appendix H
5

---

Published as a conference paper at ICLR 2025
Structural In-Context Performance
Pref. of Head for In-Context
Pref. of Tail for In-Context
Random Token Accuracy
Uniform = 1.0001
Head Switch Accuracy
AliCL
Tail Switch Accuraccy
All K3.
2.5
08
9.8
a=
2.6
1.5
0.6
0.6-
Nit
0.--
AII
1C000 20000 a00co 40000 scoco 6coc0
10000
20000
20000 40000 50000 50000
100co 200co
aciic 400ca scoco acoco
Step
Step
Step
Unseen Token Accuracy
Head Switch Accuracy
Tail Switch Accuracy
3.8
Mp
0.6-
0.4-
0.6-
0.4-
Uniform
1.0001
0.2-
a= 1.2
a=
0.0
All INL
10000
20000
30000 40000 sc0c0 60000
10000 200co acoco 40000 scóco acoco
Step
Step
Step
Figure 3: Comparative amalysis of in-context learning performance across training methodologies and data distributions. (Top) In-context performance by distribution with vanilla training: (Bottom) In- context performance by distribution with active forgetting. The parameters used are v 10000, E 0.10. Note that the Uniform distribution does not have a head or tail, and we present results the head graphs. (Top Left) Vanilla training results in structural ICL transience across all distributions. (Top Middle, Top Right) Conditional ICL is asymptotically nonzero for most distributions, unless they are highly skewed (i.e., 1.5). (Top Middle) However, IWL is often preferred for head tokens and (Top Right) conditional ICL is preferred for tail tokens. (Bottom Row) In contrast, active forgetting preserves structural ICL and removes all preference for IWL. across distributions and datasets. Note: The y-axis in the bottom left is relabelled "Unseen Token Accuracy" to emphasize that the random token evaluation dataset does not contain any random embeddings seen during active forgetting.
parameter determines the percentage of filler tokens can fill both <noun> and <adj> slots, and is inspired by the ambiguity of POS found in natural language. For primary experiments, fix f We investigate how training dynamics change as the skew changes, and additionally compare uniform sampling distribution.
In this task, an ICL strategy to infer the POS of the query token may achieve perfect accuracy by utilizing in-context information (c.g. copula always followed first by an adjective, then a noun). In contrast, an IWL strategy may achieve an accuracy of (1 e/2) at most due to ambiguous tokens. Thus ambiguity provides mild pressure develop an ICL strategy. In order make fair comparisons, we only evaluate our models on the subset of tokens that are not ambiguous; thus, both an ICL and IWL solution could achieve perfect accuracy.
Our task is formatted in a cloze-style where each token in the pattern is masked. We employ a BERT- style MLM (Devlin et al. (219) to predict the identities of these masked tokens, with hyperparameters described in Appendix !] Our models achieve near-perfect validation accuracy after < 60, 000 steps in all experimental settings.
In addition to performance model's preferred strategy throughout training, similar to Section 3 All examples these datasets contain novel [nnow Radj } combinations. Much like naturalistic setting in Section 3.1 create Head, Tail, Head Switch, Tail Switch, and Random Token datasets. In this setting, our head and tail datasets use the top and bottom 10% of the token distribution by count, respectively.
randomly selected validation set, create datasets to evaluate the
Interestingly, find that must be greater than zero for in-context solution emerge all.

---

Published as a conference paper at ICLR 2025
Accuracy Token Location Distribution
Pref. of Head for In-Weights
Preff Tail for In-Context
S-Wich Accuretyy
Sentch BccuPery
04
Wrrlh T-Ivio
propory Fororting
Distribution 2w35
Head
Heed
Tal
Urceen
Dictiuution
Distiibtion (0) Token Location
Figure 4: (Left) Temporary forgetting achieves near perfect unseen random token performance across distributions, indicating structural ICL. (Left, Green) Vanilla training skewed distributions renders tail token performance poor: (Left, Blue) In contrast, tail token performance is almost perfect after temporary forgetting. (Right) Temporary forgetting can maintain preference for IWI. for the head of the distribution while maintaining preference for ICL for the tail of the distribution i.e., temporary forgetting induces dual processes learning. Parameters used are 10000, & 0.10 and optimal hyperparameters k, N are found using a grid search.
4.1 TRAINING DYNAMICS
Transience of Structural ICI. We reproduce our results from the naturalistic setting presented in Section structural in-context strategies emerge quickly, but are transient. This is shown by the model's performance on the Random Token dataset over the course of training, which peaks early and then quickly degrades (See Figure 3 Top Left). This trend holds across all tested distributions. Thus, both synthetic and naturalistic training settings result in structual ICL transience, as hypothesized in Figure I (Bottom Left). Critically, we find that models retain conditional ICL strategies, even after structural ICL performance degrades. Across most data distributions, performance both the Head Switch and Tail Switch datasets reveal a nonzero reliance on conditional ICL strategies, even while Random Token accuracy remains zero (See Figure (Top Middle, Top Right)).
Structural ICL has Practical Importance In highly skewed distributions (e.g. Zipf a 2 1.5) where tail tokens are very rare and head tokens are very common, the disappearance of structural ICL eventually precipitates total loss of ICL abilities (Figure 3 Top, Red Line). We find that common tokens are memorized, resulting in high overall performance. However, the model completely fails when presented with tail tokens (See Figure 4 Left, {1.2, 1.5)). Even when conditional ICL strategies remain after training less-skewed distributions, the least-frequent subset of tail tokens result in poor performance. Inducing structural ICL would recover performance on these undertrained tokens.
In- In-Context Learning confficts with In- Reights Learning We analyze how the skew of the training distribution applies pressure toward adopting an IWI. or ICL. strategy. We find that increasing the skew of a distribution increases the pressure toward an IWL strategy for the head of the distribution, and increases the pressure toward an ICL strategy for the tail the distribution. Furthermore, training distributions with Uniform sampling distribution show a comparatively higher conditional ICL preference (and thus lower IWL preference) than any Zipfian sampling distribution (See Figure Top Middle). We explore how to mitigate this competition in Section
5 MAINTAINING STRUCTURAL ICL WITH ACTIVE FORGETTING
In Sections and have demonstrated structural ICL is transient across models and tasks. In an effort to promote the persistence of structural ICL, we utilize active forgetting (Chen et al. 2024). Henceforth, refer the standard training procedure vanilla training.
Active Forgetting When training a model using active forgetting, we re- re-initialize the embedding matrix every & steps during training. The intuition behind this is that the model must employ in- context strategies to achieve high accuracy, as each embedding is no longer guaranteed to encode
Additional experiments exploring the effect of ambiguity are located in Appendix L and the effect of vocabulary size located in Appendix
7

---

Published as a conference paper at ICLR 2025
1.00
1.00
Vanilla Training
Singh Et Ali, 2023 Solution
0.75
0.75
Temporary Forgetting
0.50
0.50
0.25
0.25
0.00
0.00
Rnd
1
Training Frequency Decile
Training Frequency Decile
Figure 5: Performance by token decile and on randomly initialized embeddings (Rnd). (Left) With vanilla training on a skewed distribution (Zipfian 1.5). low decile tokens show poor performance. However, overall performance remains good because these tokens are rare. (Right) Temporary forgetting induces structural ICL to recover performance on tail, undertrained, and unseen tokens compared with fingh et al. (2023)s Lo-regularization procedure, which was proposed to preserve conditional ICL.
in-weight information. This renders unseen/undertrained embeddings in-distribution, whereas they were previously outoo-diistrbbutin. We further explore this effect in Appendix N
Results We test k 100, 1000, 5000 and settle on k 1000 after preliminary exploration. Training our models with active forgetting asymptotically promotes structural ICL across all tested skews. We evaluate this using the Random Token dataset, and find near-perfect performance after sufficient training on all distributions (See Figure 3 Bottom Left). Given random embeddings to fill the <noun> and <adj> slots, the model can now (I) derive the POS of these tokens by ICL and (2) output novel labels corresponding to the identity of these embeddings in the desired pattern
As the skew of the distribution of nouns and adjectives increases, there is greater pressure to memorize the head of the distribution (as these tokens are observed more frequently). Thus, takes longer for the model exhibit a preference towards in-context solutions for head tokens (e.g., almost 60,000 steps for the a = 1.5 setting) and there is a much larger dip in performance after every instance of forgetting the embedding matrix. However, we find that our active forgetting results generally match our idealized result from Figure (Bottom Left).
DUAL PROCESS LEARNING WITH TEMPORARY FORGETTING
While active forgetting successfully induces structural ICL strategy, our model loses the ability to memorize information in its embeddings. This is detrimental in a variety of cases, such as when in-context information insufficient to generate an appropriate response. An optimal model would encode a dual process strategy: maintaining structural ICI. solution while also memorizing useful linguistic properties.
Temporary Forgetting We modify the paradigm of active forgetting in order to induce a bias for structural in-context strategies for the tail of the distribution while preserving in-weights strategies for frequently-obserr observed tokens. We introduce temporary forgetting, where we perform active forgetting every l steps for the first N steps (N >> k) of training. After this point, allow the embedding matrix to train normally. As a baseline, we compare to Singh et al. (202))'s solution to conditional ICL transience, L2 regularization. Crucially, we wish to understand whether L2 regularization helps maintain structural ICL, which was not tested in the original work.
Results We study highly skewed distribution, with parameters 10000, 0.10, = 1.5. We find that by varying N, we can vary the model's dependence on in-weights information for frequently seen tokens while maintaining structural ICL performance (See Figure I Bottom Right). At the extremes, setting N to very large mimics the behavior of active forgetting and setting to be small only sometimes maintains structural ICL performance. We can control the preference for IWL versus ICL on observed tokens by modifying N.
Thus, temporary forgetting enables model to successfully encode two distinct strategies for the same task. We can now induce this behavior for any distribution a 2 1.0 (See Figure Right),
TThi is possible because the emhedding and unembedding matrices heing tied.
8

---

Published as a conference paper at ICLR 2025
Vanilla Training
Active Forgetting
Temporary Forgetting
3.0
Nut Tusrg
Benson Berais
Tamponey Rapeting
6.9
Accuracy
Accuracy
semurt
Values
Conditinnal IDL
80000
85000
2000
Structuraal Category
feed
Step
Step
(a) Results from replication study the Chan et al. 2022b) task with autoregressive transformer. (Left) With vanilla training, structural ICL is transient while conditional ICL is persistent. (Middle) Training with active forgetting preserves structural ICL. (Right) When training our model to a skewed token distribution (Zipfian = 3), vanilla training results in memorization of head tokens and chance performance on unseen tokens; in contrast, temporary forgetting evokes dual process, which significantly improves structural ICL performance while preserving IWL common tokens. Further task and experiment details in Appendix B
Syllogism Task
Vanilla Fine-Tune GPT-2
Prob. Temporary Forgetting GPT-2
Strutive
1.0
Genastinn
Syllogism Task Accuracy
are Thersfore, all abs are
0.0
Example
0.6-
ALL are V. Tserefors, all
Unseen (Stustual ICL
All sunseens are W.
Therefors all cLtseste
0
500
1000
1500
2000
500
1000
1500
2000
line Tunnnuungggggg Step
Fine-luniig Step
(b) Probabilistic temporary forgetting enables GPT-2 to perform structural ICL. (Left) A diagram representing the syllogism task. (Middle) Vanilla fine-tuning GPT-2 large Wikitext fails to confer structural ICL, resulting poor Random (Unseen) Token Performance. (Right) In contrast, we find that probabilistic temporary forgetting successfully confers structural ICI., drastically improving Random Token Performance.
Figure 6: Autoregressive Transformer Structural ICL Experiments
while also inducing structural ICL behavior on all distributions we test (See Figure Left) In contrast, we find that the strategy employed by (Singh et al., 2023) does not eliminate structural ICL transience: undertrained and random tokens result in poor performance, as seen in Figure [ In summary, temporary forgetting significantly enhances our ability to balance between in-context and in- in-weights strategies, overcoming inherent biases in naturally occurring data. After critical number of training steps, we can stop the forgetting mechanism and retain structural ICL.
STRUCTURAL ICL IN AUTOREGRESSIVE TRANSFORMERS
Replication using Chan et al. (002bb Task In this section, we replicate our main findings using an autoregressive transformer task similar to Chan et al. (202b)) We modify the task presented
in Chan et al. (2022b) to enable us to examine structural ICL (See Appendix E) We find that the phenomena described in Sections
and all extend this task. In particular, Figure ba
(Left) demonstrates the transience of structural ICI, even when conditional ICL persists. Figure 6a (Middle) demonstrates that active forgetting preserves structural ICI.. Finally, Figure [6a (Right) demonstrates that temporary forgetting induces dual process strategy, where structural ICL is maintained and IWL is deployed on the head of skewed distribution.
7 Distributions where a S 1.0 would likely only rely on an in-context strategy.
TThis modification ensures the tail distribution of these tokens undertrained/uutriincd and thereby resemble the "glitch tokens" of Rumbelow & Watkins 2023

---

Published as a conference paper at ICLR 2025
Probabilistic Temporary Forgetting Induces Structural ICL in GPT-2 Thus far, our interven- tions have focused exclusively on pretraining to induce structural in-context learning (ICL). In this section, expand this approach to explore whether fine-tuning can also facilitate dual process learning. As a proof-of-concept, we introduce probabilistic temporary forgetting. Our methodol- ogy involves fine-tuning GPT-2 model using causal language modeling objective, with one key modification: during each fine-tuning step, we replace approximately 10% of tokens in the batch with randomly-initializze embeddings. Importantly, after each gradient update, we restore the embedding matrix to its original pretrained values. For additional details, please refer to Appendix C
We designed a simple syllogism task inspired by Lampinen et al. (202)) and Kim et al. (2024) (Figure 6b] Left). Notably, we include condition where the subject term in the first premise and conclusion of the syllogism replaced by a random embedding. As we use an unconditionally valid syllogism, this should not interfere with the model's ability to perform logical inference. We fine-tune GPT-2 large (Radford et al. 2019) on Wikitext (Merity et al. (016)) sentences for 2000 steps using both (I) vanilla training and (2) probabilistic temporary forgetting.
The results reveal a striking contrast between methods. With standard fine-tuning, the model demonstrates persistently low accuracy on our syllogism task when tested with unseen tokens (Figure 6b| Middle). In contrast, when fine-tuned using probabilistic temporary forgetting, the model shows substantial improvement in handling random (unseen) tokens while maintaining comparable performance on baseline conditions (Figure 6b, Right).
8 DISCUSSION
As discussed throughout the work, the present study is intimately related to the burgeoning literature examining the trade-offs between in- contett and in- in-weights learning (Chan et al.) 2022b Reddy] 2023) Raparthy et al.] 2023 Fu et al.] 2024). Additionally, this work connects to vast literature on forgeting in neural networks. Most prior work on forgetting characterizes this phenomenon as undesirable (Kemker et al., 2017: Kirkpatrick et al. 2017 McCloskey & Cohen 1989; Rateliff 1990). However, some work has shown that intentional forgetting (via resetting a subset of parameters) may be beneficial in certain contexts. On computer vision tasks, forgetting has been shown to help with generalization and sample efficiency (Alabdulmohsin et al. 202
Related Work
Taha et al. 2 Ramkumar et al.) 2023 Additionally, Zhouu et al. (2022) show that a forget-and- relearn paradigm helps shape the learning trajectory of neural networks. Our method of forgetting
embeddings is directly inspired by [Chen al. (2024), which shows that forgetting during pretraining boosts linguistic plasticity for multilingual learning.
Conclusion The ability to flexibly deploy in-context and in- weights algorithms has been described as an "important and useful [behavior] for model," as it enables models to both memorize information about commonly-sen difficult to ensure that models reliably acquire both forms of processing. This has led prior work to celebrate the ability to maintain dual strategies even for a limited set of distributions and suggest interventions such as "engineer(ingl data distributions to evoke this behavior in models" (Chan et al., 2022b). In contrast, the present work demonstrates method for engendering dual process learning across a range of distributions. Additionally, we extend our analysis to structural in-context learning.
inputs and generalize to new inputs Chan et al. (20226). However, it has proven
In summary, we find that structural ICL is transient in LMs, as they initially learn to generalize to unseen tokens, before losing this ability. We find that active forgetting recovers structural ICL, the expense of IWL. We introduce temporary forgetting and probabilistic temporary forgetting to induce dual process learning, enabling models to leverage IWL for common tokens and structural ICL for rare or unseen tokens-thin approach holds across various distributions. These strategies may prove particularly valuable for training models in domains characterized by highly skewed distributions.
This study opens several promising directions for future work. A critical next step involves deter- mining whether temporary forgetting can be effectively incorporated into large-scale pretraining curricula, which would establish the method's broader impact. Additionally, we provide only proof-of- for probabilistic temporary forgetting, more comprehensive analysis of this tech- nique is essential. From an implementation perspective, investigating whether probabilistic temporary forgetting can be integrated with parameter-efficien fine-tuning methods (Hu et al. (2022) represents an important advancement toward making this approach practical for large language models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank the members of the LUNAR, Serre, and LNCC laboratories at Brown University for their valuable feedback on this research. In addition, would like to thank Vignesh Pandiarajan, Anish Anand, and Akash Anand for proofreading the manuscript.
REFERENCES
Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architec- tures and algorithms, 2024. URL htt.t:tt///// arxiv. wwt./os/////////o///o..ooss.ssse abs /abbs/4411... 12973
Ibrahim Alabdulmohsin, Hartmut Maennel, and Daniel Keysers. The impact of reinitialization on generalization in convolutional neural networks, 2021.
Nora Belrose, Quintin Pope, Lucia Quirke, Alex Mallen, and Xiaoli Fern. Neural networks learn statistics of increasing complexity, 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel MosserttVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.1 14165, 2020. URL httpt:ttp//oomm...... / org/abs 200...1166..
Stephanie C. Y. Chan, Ishita Dasgupta, Junkyung Kim, Dharshan Kumaran, Andrew K. Lampinen, and Felix Hill. Transformers generalize differently from information stored in context vs in weights, 2022a.
Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in- context learning in transformers, 2022b.
Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp, Sebas- tian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. 2024.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. When bert forgets how to POS: amnesic probing of linguistic properties and MLM predictions. CoRR, 00052000600995, 2020. URL pestps://arxiae org abs 2006. 00995
Jingwen Fu, Tao Yang, Yuwang Wang. Yan Lu, and Nanning Zheng. How does representation impact An exploration on a synthetic task, 2024. URL URL/sttpa///opeeeeiiie net/ wwwp:m//ooddoooooooooeoee
in-context lcarning: Icarming.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes, 2023.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word repre- sentations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:
11

---

Published as a conference paper at ICLR 2025
Human Language Technologies, Volume (Long and Short Papers), pp. 4129 4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 11/8866//3111// 1419.
John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher Manning. Conditional probing: measuring usable information beyond baseline. In Mlarie-Francing Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 1626-1639 1639, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 16653/1221221 emnlp-main. 122. URL ht..tttpps... //ac h org/ 2021 emn 1 p ma i n. 122
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Ronald Kemker, Angelina Abitino, Mare McClure, and Christopher Kanan. Measuring catas- trophic forgetting neural networks. ArXiv, fos/1708.00072 2017. URL https://api
semanticschol wtw.:olaaw..rr/ooossiiiiaa. 229100766
Geonhee Kim, Marco Valentino, and André Freitas. A mechanistic interpretation of syllogistic
reasoning in auto-regressive language models. arXiv preprint arXiv:2408.08599, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Demis Hassabis,
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521 3526, March 2017. ISSN 1091-6490. doi: 10. 10.1100/ss.s.00 06611835114 14. URL http://dx> doi org.
1073/pnas 1611835177
Andrew K Lampinen, Ishita Dasgupta, Stephanic C Y Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models, like humans, show content effects on reasoning tasks. PNAS Nexus, 3(7))pgae 07 2024. ISSN 2752-6542. doi: (0.1003/pnassexuu/gaae33. URL (tthps::: /ttp::://ooiooo/ 1ttp:/1..19911eeeeeeee http://000/////paeeespepp/eeeee
Sander Land and Max Bartolo. Fishing for magikarp: Automatically detecting under-trained tokens in large language models, 2024.
Linguistic Data Consortium. Ontomotes release 5.0. httpss:/caaas og. 1dc. upenn. edu
LDC2013T13 2013. Accessed on December 10, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Compur. Linguist., 19(2):313 330, jun 1993. ISSN 0891-2017.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower (ed.), Psychology of Learning and Motivation,
volume 24 of Psychology of Learning and Motivation, pp. 109-165 165.. Academie Press. 1989. doi: http:://dbooog////oooooooooss11d
8, URL https: //tw:///wwww.. www. www.scaa scwencedirecttttt..cceeeoeoeee..
com/
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundann Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. Universal Dependency annotation for multilingual parsing. In Hinrich Schuetze, Pascale Fung, and Massimo Poesio (eds.), Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 92- 97, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https //aclanthology. wtt/p13-201000
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
12

---

Published as a conference paper at ICLR 2025
Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers of classification neural networks, 2023.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. CoNLL- 2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Sameer Pradhan, Alessandro Moschitti, and Nianwen Xue (eds.), Joint Conference on EMNLP and CoNLL Shared Task, pp. 1-40, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https: / ac aattooothhoooo www.oggg.ooooooooooo.oooo....ooo org woopo//2-//000 4501
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Vijaya Raghavan T. Ramkumar, Elahe Arani, and Bahram Zonooz. Learn, unlearn and relearn: An online learning paradigm for deep neural networks, 2023.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 28729-28745. PMLR, 23-29 Jul 2023, httttttts://pooooott mlr Sttesss/2222//aagaaaaa
html
Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu. Gener- alization to new sequential decision making tasks with in-context learning, 2023.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97 2:285-308 308, 1990. URL htttht://aaat /api
semant cestiiisschhhaaaa wwtp/oooooooiiiiie00.CC
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task, 2023.
Jessica Rumbelow and Matthew Watkins. Solidgoldmagikarp (plus, prompt generation). Less- http::/gww....oooo//////aaaaaaaa gol.:cg....cckkarr-p--ppppppppp-
Wrong, 2023. URL //www
Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D' D'mmuur, Tal Linzen, Jasmijn Bastings, Julia Ture, Jacob Fisenstein, Dipanjan Das, Ian Tenney, and Ellie Pavlick. The multiberts: BERT reproductions for robustness analysis. CORR, abs/2106. 16163, 2021. URL
https: /ttps:://axivve wwwp:///......//...../. 16163
Aaditya K Singh, Stephanie C.Y. Chan, Ted Moskovitz, Erin Grant, Andrew M Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. In Thirty-seventh Conference on Neural Information Processing Systems. 2023, URL
net cutt/forum?icee OFOGBZowBF
Ahmed Taha, Abhinav Shrivastava, and Larry Davis. Knowledge evolution in neural networks, 2021.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 11111166611111///
URL httpp:://claatthllla
oxq//19-11422
Iulia Ture, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better:
On the importance of pre-training compact models, 2019.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks, 2022.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19-27, 2015. doi: 10.1 1.110
13

---

Published as a conference paper at ICLR 2025
A PROBING SETUP
We provide probing background in this section, borrowing some notation from Elazar et al. 2020
Given set of labeled data of points X 21,... En and task labels Y Ui..... . analyze model / that predicts the labels Y from X : Di f(zi). We assume that this model is composed of parts: (1) an encoder h that transforms input и; into learned representation vector ha; and (2) classifier that is used for predicting d based on hg, such that øi c(h(z;)). We refer to as the probe and the model containing h as the model.
Given this setup, we evaluate particular model's performance across various layers and training steps for our POS task. Each encoder h is associated with a specific training step and layer h We probe the residual stream after layer
In this research, we are interested the model's ' choice of strategy at a particular time step. That is, we seek to describe the change in prediction of due to varying 1, of encoder ht Accordingly, we fix as a single linear fully-connected layer.
B STRUCTURAL ICL ACROSS LAYERS
Structural In-Context Performance
Layer
Layer
Layer
Layer
Layer
Layer 5
Unseen Tuken Accurec
Layer
Layer
Layer
Layer
Layer 10
Layer 11
Layer 12
Random Baseline
0.4
0.31
1.5
MulbiBRT Training Step
Figure 7: We find that structural ICL is transient across all layers of MultBERTss (seeds 0, 1, 2 averaged). The middle layers show the most structural ICL during early in training, whereas very early and very late layers remain about random throughout training.
We find that structural ICL consistently approachs random levels as training progresses across layers in the MultiBERTs. This signifies that the model fully loses the ability to process unseen tokens as training continues. This one explanation for the "glitch tokens" described Land Bartolo (2024), for which LMs fail to output sensible content.
14

---

Published as a conference paper at ICLR 2025
C STRUCTURAL ICL IN GENERATIVE DECODER-ONLL LANGUAGE MODELS
C.I SYLLOGISM TASK
We designed a syllogism task that requires symbolic reasoning based on the context to show that (1) structural ICL is transient decoder-only transformer based generation and (2) a variant of temporary forgetting can remedy structural ICL on real-world natural langauge model.
Our task is formulated follows: Our task requires abstract reasoning untrained tokens in decoder-only transformer. The model must complete the following syllogism.
All <X> are <Y>
All <Y> are <Z>, Therefore, all <X> are
The correct answer is <2>, We examine accuracy, which we define as the probability of choosing <②> compared the probability choosing <Y>, We test baseline performance over training steps where <x>, <Y>, <Z> are chosen from the set of tokens representing A-7,, and we test unseen token performance by replacing <x> with an unseen token this formulation (<Y>, <2> are still chosen from A-Z). This task was inspired by[ Lampinen et al. (2024).
C.2 STRUCTURAL ICL IS TRANSIENT PYTHIA 1.4B
Structural ICL on Pythia 1.4B Syllogisms Task
0.9
0.8
0.7
Accuracy
0.6
0.5
0.4
0.3
0.2
Baseline Random Tokens
0
20000
40000
60000
80000
100000
120000
140000
Step
Figure 8: find that structural ICL is transient for the decoder-only Pythia- on a syllogisms task. Performance common tokens continues improving near-perfect accuracy. Results averaged over three trials.
In the above syllogism task, we observe structural ICL transience in Pythia 1.4B checkpoints. We find that structural ICL. consistently spikes and then approaches below random levels as training progresses across layers the Pythia 1.4B model (Biderman et al.) (023), as shown by the random (unseen) token accuracy in Figure The model loses the ability to perform syllogisms tokens as training continues. We chose the Pythia-1.4B model show the generalizability of our finding natural language decoder-only models. We employ publicly released training checkpoints to run our experiments (starting at step 0, then every 5000 steps starting from step 1000 until 141000).
unseen
D PROBABILISTIC TEMPORARY FORGETTING FIXES STRUCTURAL ICL IN GPT-2
We finetune GPT-2 large (Radford et al. 2019) on Wikitext (Merity et al. 2016) sentences taken from Wikepedia articles for 2000 steps. We the AdamW optimizer with learning rate of 3e 5 and
15

---

Published as a conference paper at ICLR 2025
linear optimization schedule with 500 warmup steps. Note that unseen token syllogism performance on the pretrained GPT-2 large even worse than on the pretrained Pythia 1.4B. To accommodate the fine-tuning setting. tokens in the batch with p = 0.10 with randomly initialized embeddings. After the step, we set the embedding matrix back its original values, hence maintaining the spirit of temporary forgetting. In this method, our pretrained embeddings remain unchanged.
probabilistic variant of temporary forgetting: every step, replace
After fine-tuning with probabilistic temporary forgetting Wikipedia sentences, we find that syllogism accuracy with unseen tokens jumps from 0.02 to 0.927 while the baseline syllogism accuracy goes from 0.933 to 0.923, as seen in Figure 6b In addition, when we fine-tune without probabilistic temporary forgetting (i.e. vanilla fine-tuning), we see that unseen token syllogism accuracy remains substantially below-random. Our probabilistic temporary forgetting rectifies structural ICL on a downstream task in a real natural language model.
16

---

Published as a conference paper at ICLR 2025
E AUTOREGRESSIVE TRANSFORMER SYNTHETIC SETTING
To show the broadness of our structural ICL results, we also replicate our findings using modified version of the synthetic task presented in Chan et al. (2022b).
E.1 MODIFIED CIIAN ET AL. (2022B) TASK
(a) Model, inputs, and outputs.
(b) Sequences for training.
bursty
transformer (causal)
218
e22
h
21
Sio
62i
C
non-bursty
resnet embed
ho
4?2
184
h
45
1000
821
C
210
800
C
image
labal
context
query
query
(c) Sequences to evaluate in-context learning.
(d) Sequences to evaluate in in-weights learning.
X
X
Z
X
Z
Z
X
Z
X
Fro
h
1008
S79
contaxt
cuary
contexx
query
Figure 9: This is the task formulation of Chan et al. (202bb) (Replicated from Figure of Chan et al. (2022b)) We use similar task, but with token embeddings that are learned during training rather than ResNet encodings of Omniglot.
Similar to Chan et al. (2022b)'s 's task formulation, we have training data comprised of sequences of tokens and labels where the context is made up of the first 16 elements (8 token-label pairs), and the final element is the 'query' token. The aim of the model is to predict the correct label for the query. There are 1600 tokens, each mapped to label. With some ambiguity probability (e.g., 0.05), a token is mapped to different label randomly chosen from the set of labels (in order to confer ambiguity, as in Section 4.. Sequences are bursty, with the query-label pair as well different token-label pair each occurring 3 times in the context. We evaluate the trained models three types of sequences to measure (1) structural ICL, (2) conditional ICL, and (3) IWL.
Again borrowing from Chan et al. (2022b), our context for the conditinns is random ordering of two token-label pairs with 4 examples each, and the query is selected randomly from one of the two tokens. While label-pairs the two tokens are randomly re-assigned to either 0 or for each sequence. We calculate few-shot accuracy by considering only probabilities assigned
fixed in training (up to the ambiguity parameter), the labels for
and (resulting chance performance of 0.5). In evaluating structural ICL, we generate sequences consisting of random tokens and labels, while conditional ICL sequences consisted of tokens previously seen by the model during training. We test on tokens drawn from uniform and zipfian distributions, where experiments are with Zipf = 1.0001 token sampling distribution unless otherwise specified.
To measure IWL, we considered non-bursty sequences where the query-label is not located in the context. The only way for model to correctly predict the label is to rely on information in weights ensured unique, non-query token-label pairs in the context.
Note that the difference from Chan et al. 2022b) setup is that we use randomly initialized tokens embeddings rather than Omniglot Resnet-encoded images and our autoregressive transformer is also smaller. This enables us to test for structural ICL by replacing token identities with random vectors. Another method for to test structural ICL could have been to random images, but this would not maintained the analogy to undertrained/umseen "glitch tokens" in language models, unlike our current setup
17

---

Published as a conference paper at ICLR 2025
E.2 MODEL DESCRIPTION
We use 4-layer GPT-2 architecture as our autoregressive transformer with 4 attention heads per decoder layer and embedding size of 64 (Radford et al., 2019). To optimize, AdamW with learning rate of 5 x 10 and a linear warmup schedule with 1/T0 of the total number of steps as warmup steps (Loshchilov Hutter 2019).
We ensure that on validation similar to the training set, there is near-perfect performance by the completion of training.
E.3 VANILLA TRAINING
We find across setting that settings where ICL arises, there is structural ICL and it disappears abruptly with vanilla training. This is true for different levels of burstiness (0.8, 0.95, 1.0), different levels of ambiguity (0.05, 0010, 0.20), and different distributions (Uniform, Zipf with 1.0001, 1.5, 2, 3). In-weights learning varies based on the distribution.
Ambo=0.00 05, a=0
Ambb=0.05 d=1.0001
Amb=0.00 05, d=2
Amb 05, a=3
Conditional ICL Scructural ICL Basaline
calos sole vile elno men
Amb-0.10.cmm a-0
Amb-0.10, 8=1.0001
Amb-0.10, d-2
Amb-0.10.0m a-3
Amb-0020 d=0
Amb=0.20 0,20, 1=000001
Amb=0.20, a=2
Amb=0.20, am3
1906
Step
Step
Stap
Figure 10: Structural ICL. disappears while conditional ICL remains across different combinations of ambiguity and skew in our autoregressive few-shot task described in Appendix Interestingly, skewed distributions with high ambiguities show some variance in structural ICL accuracy after the initial disappearance.
18

---

Published as a conference paper at ICLR 2025
E.4 ACTIVE FORGETTING
Active forgetting preserves structural ICL, but completely removes any use of IWL. We see this across tested distributions (Uniform, Zipf with a 1.0001, 2). We use k 500 because this worked well with initial experiments (although the other tested parameters of 1000, 2000 also worked almost equivalently).
a=1.0001
a=2
Rocueety
Accuratty
Canditional
Structural ICL
Randeen Roseline
20,000
2008
Stop
Stop
Stop
Figure 11: Active forgetting preserves structural ICL across different skews our autoregressive few- shot task described in Appendix B converge quicker.
Interesting, increasing the skew seems make active forgetting
E.5 TEMPORARY FORGETTING
In our temporary forgetting setting, we use burstiness parameter of 0.95 for experiments. We use 1000, N 8000 because these parameters worked well in initial experiments. We did exhaustively search over parameters. We tested whether we could evoke a dual process ICL and IWI. across distributions (Zipf with
1. 1.0001, 3), active forgetting, where we cannot learn information in-weights (Figure 13 and vanilla training,
in Figure 12 This is in contrast
where we cannot asymptotically perform above a random baseline for structural ICL (Figure I0
Structural ICL
IWL Head/Common Tokens
Random Baseine
Vanilla Taaning
Temporry Forgeting
n7-
Distibbttonn
Distr Eustiibbtion
Figure 12: Temporary forgetting preserves structural ICL across different skews in our autoregressive few-shot task described in Appendix E, as opposed to vanilla training (i.e. standard training). In addition, it enables IWL for common tokens instead of completely removing it like active forgetting. achieves about 90% the IWL use for these. Note we consider the smaller set between top 100 tokens and top 10% of the probability when choosing common tokens evaluate IWL on.
Active Forgelting Training
Temporary Forgelling Training
Structural IWL on All Tokens
Jocurncy
20000
40000
69/00
2000
Figure 13: Temporary forgetting enables us to learn IWL while preserving structural ICI., whereas active forgetting forces only structural ICI. This is seen by the developmental accuracies in this figure (note & = 500 for active forgetting whereas 1000, N = 8000 for temporary forgetting).
19

---

Published as a conference paper at ICLR 2025
F
DUAL PROCESSES FOR SKEWED DISTRIBUTIONS
Chan et. al Dual Process
Temporary Forgetting Dual Process
ICL on holdout classes
IWL
ICL random tokens
1.00
1.00
1.00
Switch Accurco
Accurtyy
0./3
0.73
0.50
0.50
0.25
0.25
6.00
1.001 1.
17401 ppl Exponent
1.0001
0.00
1.5
2657 Exponent
Figure 14: Temporary forgetting's ability to invoke dual processes (in yellow) various distributions of our synthetic POS task compared with Chan et al. (2022b) observational baseline. Structural ICL and IWI. are able to be co-occur in networks now trained data distributions of any skew with 2 1, as opposed to being limited to specific "sweet spot" distribution.
G PUSHDOWN DATASETS
We use the train/dev splits from the English UD Treebank for the f-pos. and dep tasks McDonald
itt al. (003)); the train/dev splits from Ontonotes-v5 in the CoNLL-2012 Shared Task format for the ner, phrase start, and phrase end tasks Linguistic Data Consortium (2013); Pradhan al. 2012); the train/dev splits from Penn Treebank-3 for the depth and dist tasks Marcus et al. (1993): and generated token sequences for the prev, dup, and ind tasks.
We reproduce baselines from Elazar et al. (2020) to verify the correctness of our probing setups for c-pos, f-pos, ner, dep, phrase start and phrase end and from Hewitt & Manning (2019) for depth and dist.
H PUSHDOWN SIGNATURE OBSERVATION IN SYNTAX
31222 9300032
cpos
tpos
KCM
nar
6.75
6.10
phrase start
phrose end
dep
depth
dist
Layar 03011
140
00 10
2000
100
1000
Figure 15: The "Pushdown Phenomenon" is observed across syntactic features, suggesting that a transition from IC to IW strategies happens across these features. early steps of training, representing syntactic information occurs in later layers, which are more contextualized. However, as training progress, the same properties are better encoded in earlier layers due to memorization of token-level and n-gram level information. The n-gram level information requires attention to build, which explains why performance in dep, depth, and dist does not propagate all the way to embeddings.
20

---

Published as a conference paper at ICLR 2025
We observe "Pushdown Phenomenon", which describes phenomenon where in early steps of training. computing roken-wise syntactic properties occurs in later layers, which have more in-context information. However, training progresses, the same properties are better encoded in earlier layers until only the first couple layers are required for representing syntactic properties.
We examine whether the "Pushdown Phenomenon" exists in various syntactic properties in BERT. To do so, we employ our probing setup (Appendix AA for the tasks of named entity recognition (ner), coarse part of speech (c-pos), fine-grained part of speech (f-pos), dependency parsing (dep), syntactic constituency boundaries which indicate the start and end of phrase (phrase start, phrase end), depth in the parse tree (depth), and distance in the parse tree (dist). We probe each property across the axes of (1) training time steps and (2) layers. We repeat this process for three seeds of the MultiBERT's (Sellam et al. 2.2T). For all tasks, we probed all layers MultiBERT seeds O, 1, and for timesteps from O to 200,000 increasing by 20,000; 200,000 to 1,000,000 increasing by 100,000; and 1,000,000 to 2,000,000 increasing by 200,000. If specific word is composed of multiple subword tokens, we follow Hewitt & Manning 2019) and average the encoding across tokens.
We observe the "Pushdown Phenomenon" all our examined tasks. However, we find that across tasks, syntactic information is "pushed down" at different rates. Early layer accuracy increases approximately follow a pattern of ner + phrase start + cpos/fpos - phrase end + dep - depth - dist. We leave it to future work to explore whether this timing a function of (1) complexity of high-achieving rules/heuristics consistent with Belrose et al. (2024) or (2) a naturally occurring dependency hierarchy of syntactic relationships suggestive of implicit curriculum learning. One possible intuition for why the "Pushdown Signature" of memorization often coincides with poor maintenance of in-context strategies might be neural collapse (Parker al., 2023 Rangamani et al.) 2023). although this should be further investigated by future studies.
I SYNTHETIC POS TASK EXAMPLES
Here, we provide further details regarding the design of our synthetic POS task. Our task is designed to 1) minimally emulate a subtask performed in language models Part-oof-Speed tagging) while 2) controlling for various confounds. In particular (1) it does not allows heuristics based on token position and (2) is not deterministic based on the query. Here are a couple clarifying examples (<sequence>
<query> + <pattern>):
1. (a) is happy dog (b) dog is happy Note that in this example, we show that using two templates rules out simple position-based heuristic. If model assumes that the noun occupies the 3rd position of the sequence, then the model will believe happy is a noun in the second example and falsely predict a response pattern of dog dog dog.
dog happy dog dog
dog happy dog dog
2. (a) dog is happy dog + happy dog dog
(b) dog is sad dog sad dog dog
Note that in this example, both queries are dog, yet the predicted pattern is different. Context is necessary for correct prediction.
J TOY MODEL
We employ a 6-layer BERT model across the synthetic setting experiments. Experiments were performed with an MLM syntactic structure is much more difficult infer in autoregressive models as they are only exposed to an ordered subset of the tokens in a sentence. This model has attention head per layer, 64-dimensional hidden dimensions, 128-dimensional intermediate representations, and tied weights for the embedding and unembedding layers. We optimize model parameters with AdamW with learning rate of 5 x 10-5 {Loshchilov & Hutter 2019). The hidden dimension sizes were decided per minimax strategy, i.e. this representation dimensionality was the smallest such that we achieved near perfect accuracy on a validation set for the downstream task. Future work should better examine the effect of representation size in in-context vs. in-weights learning.
21

---

Published as a conference paper at ICLR 2025
K PERFORMANCE BY TOKEN DECILE
Vanilla Increased Weight Decay
Active Forgetting
Temporory Forgetting
0-10% 10-20% 20-10%
90-40%
40-50%
50-80%
60-70% 70-90016
800 80-9905
90-100%%
Training Percenblle
Figure 16: Increased weight decay has little/no effect on the failure of the structural ICL strategy (we increase weight decay from 0.01 0.1). In contrast, active and temporary forgetting boosts rare token validation accuracy significantly, seen in the tail of the distribution. Parameters are v 10000, 0.10, & = 1.5
We find that on highly skewed distributions, the tail of the distribution suffers immensely due to undertraining. This phenomenon cannot be rectified by /Singh et al. (2023)'s 's method of promoting asymptotic ICL. However, we find that both active forgetting and temporary forgetting correct this behavior to boost performance on tail tokens in skewed distributions from near-zero to near-por sear-perfect levels.
22

---

Published as a conference paper at ICLR 2025
L
AMBIGUITY (5) EXPERIMENTS
Structural In-Context Performance
Head In-Context Preference
In-Context Preference
Uniform
= 1.0001
Unsoon Tokoon Accuracy
10-
2 - 1.5
Thll Switch Accuracy
20008
2000
80000
3008
Unscen Taken Accurac)
Tall Switch Accuracy
Head Switch
20012
2002
0000
20005
100.0
12008
20000
2000
1000
2000
3000
2008
8000
50035
Unsoon token Accuracy
16
Tall Switch Accuracy
Stop
Stop
Stop
Figure 17: (Top) & = 0.01, (Middle) 6 = 0.10, (Bottom) & = 0.50. Overall in-context strategy dependent by amount of ambiguity in the labels. With 50% of the tokens ambiguous, all unambiguous tokens use an in-context strategy; with 10%, there is a mixed strategy dependent on where the distribution the example is; with 1,, almost unambiguous tokens use a memorized strategy. The vocab size is v 10000.
In all of our ambiguity experiments, structural ICL is transient (even when 50% of tokens are ambiguous). The ambiguity parameter significantly alters the model's overall strategy. With low ambiguity parameter, the model prefers memorization (IWL strategy) of unambiguous tokens and with high ambiguity parameter, the model prefers ICL strategy, Across all ambiguity parameters, there is difference in tail and head behavior.
23

---

Published as a conference paper at ICLR 2025
M
VOCABULARY SIZE (v) EXPERIMENTS
Structural In-Context Performance
Head In-Context Preference
In-Context Preference
Uniform
2 1.0001
Unsoon Tokeo Accuracy
2 - 1.5
Tall Switch Accuracy
Mamnnry Man laol
20008
2003
Unscen Taken Accurac)
Head Switch Accuracy
Tall Switch Accuracy
20012
2002
0000
20005
100.0
12008
2000
20000
2000
18000
20005
3000
20008
6000
20035
Unsoon token Accuracy
12-
Tall 19990i0 Accurasy
Stop
Stop
Stop
Figure 18: (Top) v = 1000, (Middle) r = 10000, (Bottom) v = 20000. The strength of an in-context solution depends the interaction between vocabulary size and skewedness of the distribution Too small of a vocabulary size (i.e. v = 1000) encourages more memorization in general but fixes performance in a 1.5 setting. The ambiguity is 0.10.
In all of our vocabulary experiments, structural ICL is transient. As expected, find that vocabulary size has a similar effect to the skewedness of the distribution. That is, increasing the vocabulary without bound would lead to poor tail ICL performance. Too small of vocabulary size seems to increase ICL among very skewed distributions but decrease ICL among all other distributions.
24

---

Published as a conference paper at ICLR 2025
N EMBEDDING ANALYSIS
We perform qualitative analyses on the embeddings produced by vanilla training (i.e. standard training without modification), active forgetting, and temporary forgetting in order to better understand how these training regimens impact model representations. These analyses, consisting of principal component analysis (PCA) and probing for POS, are located in Appendix N
After vanilla training. the learned embeddings cluster according to their POS, far from the distribution of randomly-ine inittilliie tokens. We train linear probe on these learned embeddings, and find that can almost perfectly partition nouns and adjectives. Note that the disappearance of structural ICL occurs at the same time as the probe achieves above-random POS probing (i.e. memorization).
As expected, we do not see any structure in the embeddings produced after active forgetting. As such, a linear POS probe trained on these embeddings never achieves above random chance throughout training. The embedding distribution looks quite similar to the random initialization distribution, indicating that no information has been encoded these embeddings. See Figure 19
Finally, the temporary forgetting setting reflects aspects of both vanilla training and active forgetting; that is, the head of the token distribution learns to partition nouns and adjectives whereas the tail of the distribution does not learn any structure. The tail embeddings much more closely resemble the initialization distribution with temporary forgetting than with vanilla training. This results in a unseen token generalization in addition to memorized information. See Figure 20
Vanilla Training
Active Forgetting
Random (Init)
Component 2
Token Type
Acijs
Nouns
Componeni
Component 1
Figure 19: Vanilla training imposes structure the adjectives and nouns such that randomly initialized (unseen) tokens are out-of-distributioo whereas active forgetting embeddings resemble the initial distribution. Parameters used are 10000, 1.0001, 0.10.
25

---

Published as a conference paper at ICLR 2025
Head Tokens
Tail Tokens
Legend
Noun
Acjective
Vanilla Training
Probe Accuracy: 0.95
Probe Accuracy:
Active Forgetting
Probe Accuracy: 0.52
Probe Accuracy: 0.54
Temporary Forgetting
Probe Accuracy: 0.88
Probe Accuracy:
Figure 20: Vanilla training learns to partition noun and adjective embeddings in the head of the distribution, and some structure in the tail. Active forgetting learns separation between noun and adjective embeddings. Temporary forgetting learns structure the head of the distribution and no structure in the tail of the distribution. Parameters used are 10000, 1.2, 0.10.
26

---

Published as a conference paper at ICLR 2025
O OTHER RANDOM DISTRIBUTION GENERALIZATION
Note that while we define structural in-context learning as free from reliance on any encoded semantic information, it is important note that this does not mean that structural in-context learning assumes no geometry of the space. In fact, this would be practically impossible to achieve because connectionist networks function in geometric space and take advantage of orthogonality, translation, scaling, etc. If we cannot make assumptions about the distribution from which the data is sampled, then we deprive our networks of their toolbox. Still, we test on random sampling distributions for the embeddings other than our initialization distribution. Namely, we test on uniform distribution from to I and large normal distribution with mean of and standard deviation of
Training
Porgetting
Temporary Forgetting
Randon Larg
Handon (int)
Composeri1
Componett
Cont porppont
Fandor (Eaget
Skp
20g
Figure 21: Vanilla training fails all random tokens, whereas active/temporary forgetting succeed on the random distribution of initialization. Active and stop forgetting do not generalize to arbitrary random distributions, although show some generalization to normal distributions with large means and variances.
P REQUIRED COMPUTE FOR EXPERIMENTS
We employed compute resources at a large academic institution. We scheduled jobs with SLURM. For our naturalistic experiments, each MultiBERT seed required 24 separate runs (one per tested checkpoint at a particular timestep), which totaled = 100 hours on an RTX A5000 with 24 GB of GPU memory. Over seeds, this was 300 hours GPU usage. For our synthetic setting, the vanilla training required 64 separate runs (one per hyperparameter combination of vocab size, ambiguity, and sampling distribution), which totaled == 250 hours of RTX A5000 usage. Likewise, active forgetting and temporary forgetting interventions took similar amount of GPU usage. Therefore, in total, our GPU usage for all synthetic experiments summed up to about 750 hours. We experiments mostly in parallel with SLURM to iterate quickly. Compute was significant limitation for the development time and informed our development of training interventions in a synthetic setting. In total, our GPU usage was significantly higher than the reported number due to various failed/modified experiments. The total compute likely was around 20,000 GPU-hours on RTX A5000s, although this is rough estimate.
27


Reviews:
[{"review_id": "WDhZTOYLFT", "reviewer": "Reviewer_Szdj", "summary": "This paper studies the emergence and disappearance of in-context vs in-weights learning in masked language models; in particular, a kind of ICL termed “structural ICL”, which refers to the ability to infer (and then use) information about the structural role (e.g., POS) of words from the context. In experiments, this ability appears and then disappears over the course of MLM training. A recently proposed “active forgetting” approach helps keep the ability, leading to models that use both in-context and in-weights learning.", "questions": "See weaknesses.\n\nMinor note: Line 317-8: it seems the sentence is missing a verb", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper studies a question of substantial interest, nicely continuing prior work on different kinds of learning\n- considers both real-world LM and toy models\n- Provides novel insights by distinguishing \"structural\" from \"conditional\" ICL\n- Provides method for mitigating the loss of structural ICL in a toy setup", "weaknesses": "- Structural ICL is operationalized using performance on random token embeddings. However, as the paper also demonstrates (Figure 18), such embeddings are drastically OOD for a trained model. Wouldn't a fairer test use novel embeddings that match the distributional properties of the embeddings of the trained model?\n- Results on Structural ICL in MultiBERT (Section 3) are based on POS = noun vs adjective as the only property. It remains unclear how robust findings are to other properties, such as other pairs of POS, or other properties, such as grammatical number.\n- Results on Structural ICL in MultiBERT (Section 3) are based only on probing accuracy, not behavior of the model. This seems suboptimal given that in-context learning as an emergent property of LLMs is generally thought of as an ability appearing in next-token prediction (e.g., complete a prompt with an appropriate label), One could test for ICL in properties such as POS behaviorally by creating contrastive examples where the key word has different POS and comparing model probabilities.\n- The term “Dual Process” here is used to refer to models using both in-weights and in-context learning, and the paper refers to Kahneman 2011; Miller 2000. The link to dual process theory appears tenuous: dual process theory refers to thse use of implicit, unconscious vs explicit, conscious processes; the link to in-weights vs in-context inference is at best hand-waved in line 90-91, but no explicit justification for the link is given. The choice of the term and the link to dual process theory thus appears a bit of a stretch to apply the in this context.\n- It remains unclear if the proposed strategy for keeping structural ICL abilities would transfer to real-world LMs. While training an LLM is understandably out of scope, even a modest LM could provide substantial insight here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fijbWvuAhy", "reviewer": "Reviewer_pXza", "summary": "This paper examines \"structural\" in-context learning (ICL), a specific type of ICL where some tokens in the model’s prompt are randomly initialized. The motivation for studying structural ICL is based on prior observations that ICL behaves unexpectedly when prompts include tokens that are rarely encountered during training, i.e., undertrained tokens. This paper uses synthetic tasks to demonstrate that structural ICL develops early in training but diminishes as training progresses. The authors show that periodically reinitializing token embeddings during training (active forgetting) preserves structural ICL abilities, albeit at the expense of memorizing information in the model’s embeddings. The paper further demonstrates that both structural ICL and memorization are maintained if active forgetting is employed, but only for the first N steps of training. This procedure is referred to as “temporary forgetting.”", "questions": "1. What are the limitations of temporary forgetting? Is it strictly beneficial, or did the authors find that some model abilities deteriorated? Given that evaluation wasn’t performed on downstream tasks, what synthetic experiments can be conducted to evaluate the trade-offs of temporary forgetting?\n2. Suggestion: Since no evaluation was performed on downstream tasks, this paper could be improved by expanding on why studying undertrained tokens is important (beyond what is already discussed in the introduction). I am familiar with this area, so the introduction was adequate for me, but adding such a section might make this paper more broadly accessible.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I appreciated that this work distinguishes structural ICL from ICL in general. This seems like a natural and interesting extension to prior work that studies in-weights versus in-context learning.\n2. Despite the lack of experiments on downstream tasks, I found the breadth of the experiments satisfactory. I also recognize the creativity employed by the authors given their compute constraints.", "weaknesses": "1. No experiments on downstream tasks. While the motivation to address the issue of undertrained tokens is interesting, it is difficult to assess how training with temporary forgetting will affect language models in real-world applications. Given the author’s limited computational resources, they might consider a variant of temporary forgetting that can be employed as a fine-tuning step, e.g., fine-tune with active forgetting for $k$ steps before resetting the token embeddings to their original states, leaving only the model parameters changed.\n2. Limited technical novelty. Unless I misunderstood the paper, temporary forgetting is simply active forgetting applied for $K$ steps. This is only a minor weakness in my view and is excusable if the authors could find a way to perform experiments on downstream tasks.\n3. The presentation can be improved. For example, there are instances where the figure legends contain terms that are not defined. In Figure 6 (right), “Vanilla Forgetting” is mentioned. Figure 11 uses the term “Stop Forgetting,” which I believe is intended to mean “Temporary Forgetting.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9I5GX3pAUt", "reviewer": "Reviewer_E33F", "summary": "The paper introduces a distinction between *structural* and *conditional* in-context learning, where structural ICL can generalise to novel tokens because it does not depend on token embeddings. They show that by default, structural ICL emerges and then sharply diminishes during pretraining, and they introduce a novel pretraining method based on periodically reinitialising the embedding matrix, which prevents the loss of structural ICL.", "questions": "**Q1**\n\nTo what extent does active forgetting impair other model capabilities, if at all? What about temporary forgetting? I'd be interested to hear hypotheses, and particularly any experimental results you have. \n\n**Q2**\n\nDo you believe the problem of glitch tokens will recur in future, and in such a case, do you think preservation of structural ICL would be a useful strategy?\n\n**Q3**\n\nCan you give an example (or ideally a few examples) of cases where models might have a hard time performing a natural task because they are incapable of performing structural ICL on a rare token? \n\n---\n\nI have provisionally put down a marginal acceptance, but would be happy to revise my score upwards if these questions are appropriately addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The central thesis is clear\n- The contributions are well-grounded in prior literature\n- The distinction between structural and conditional ICL is compelling\n- The experimental results support the thesis well, and are quite extensive, especially on the dynamics of structural versus conditional ICL", "weaknesses": "I feel there are two main weaknesses.\n\nFirstly, it is not clear to me how important structural ICL actually is as an ability for models\n  - The connection to glitch tokens is fair, but I don't expect these to be very common, and even when they do occur I'm not sure that preserving structural ICL is the best response - for instance, it wouldn't have fixed all the canonical solidgoldmagikarp problems, because structural ICL only preserves syntactic knowledge of glitch tokens, but fundamentally cannot give semantic knowledge of unknown tokens\n  - The application to very rare tokens is likewise fair but by definition not very common, and again it is unclear to me whether in these cases there are any important capabilities which are purely syntactic\n\nSecondly, it is not clear to me how far active/temporary forgetting impairs other abilities in models\n- My intuition is that repeatedly reinitialising the embedding matrix is encouraging structural ICL by damaging other kinds of learning\n- While the temporary forgetting approach might ameliorate this, it is not clear to me by how much or in what way it does this\n\n\nOverall, I find much of the paper compelling in its presentation and exploration of structural ICL as a phenomenon, but I would be even more enthusiastic about it if it were clearer to me:\n- that structural ICL is an important property (which could be demonstrated by natural examples of its relevance)\n- what the side-effects are of the proposed methods for preserving structural ICL", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cf9ov6tPsm", "reviewer": "Reviewer_zYZG", "summary": "This paper studies the limitations and corresponding solutions for in-context learning with unseen tokens (i.e., structural ICL). Under this scenario, language models (LMs) learn the sentence structure (i.e., simple syntax structure) instead of token semantic meanings. Thus, with carefully designed forgetting strategies, the authors show that they can let LMs perform well under the structural ICL setting.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.  This paper investigates an interesting research question: comparing ICL with parameter-tuning. Some explorations are helpful and provide useful insights. \n2.  Besides, the idea of combining ICL with structures is inspiring. Most previous ICL works run experiments on the sentence classification datasets, which are highly semantic-related. The discussion about sentence structures instead of prompt structures are limited.", "weaknesses": "1. Some statements about ICL explanation are not up-to-date. The authors mentioned that typical ICL algorithms do not have stable learning manner and named it as conditional ICL. However, some recent works [1,2] have shown that ICL learns the task composition: the model only learns how to compose the task based on the learned tasks from pretraining, instead of learning from scratch based on the ICL examples. Thus, some claims in paragraph 2 of the introduction should be updated.\n\n2. Experiment settings are a bit problematic. This work is based on previous findings. However, these findings are not general across all settings. For example, Singh et al., 2023 find that ICL slowly dissipates as models are overtrained ONLY for decoder-based LMs without pretraining. In their limitation section, they also mentioned that the conclusion may not hold for pretrained LMs (e.g., Multi-BERT here) under other scenarios. I can’t list all of them, but in general, what I mean is that we should be careful about the assumptions of previous work, and cannot simply generalize and rely on previous conclusions.\n\n    Second, since this paper uses encoder-based LMs, it may not be able to perform ICL as there is little discussion about it. Besides, ICL ability is often used for large models. Implementing experiments on Multi-BERT with probing accuracy instead of generation is not convincing to show the conclusions of this paper is useful in practice.\n\n3. The sentence structure is learned from pretraining instead of predefined. Thus, the model may not treat sentences as human do (i.e., with linguistic structures). Even if for the simple POS triple structure it doesn’t matter, other more general or complex structures may not work.\n\n4. The potential impact of this work may be a bit limited. The motivation of injecting novel tokens under the ICL setting is not clear to me. As aforementioned, ICL is often used for large models, where their tokenizer vocabulary is large enough and it’s not necessary to add extra new tokens (e.g., LLaMA 3-8B). While for small models that may require to extend the tokenizer (frequently), it may not perform ICL well and they are not expected to be used under this scenario. \n\n[1] Hahn M,; Goyal N. A theory of emergent in-context learning as implicit structure induction[J]. arXiv preprint arXiv:2303.07971, 2023.\n\n[2] Li, J.; Hou, Y.; Sachan, M.; and Cotterell, R. What Do Language Models Learn in Context? The Structured Task Hypothesis. ACL 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)