Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting

## Abstract

Abstract Language models have the ability to perform in-context learning (ICL), allowing them to flexibly adapt their behavior based on context. This contrasts with in-weights learning, where information is statically encoded in model parameters from iterated observations of the data. Despite this apparent ability to learn in-context, language models are known to struggle when faced with unseen or rarely seen tokens. Hence, we study structural in-context learning , which we define as the ability of a model to execute in-context learning on arbitrary tokens – so called because the model must generalize on the basis of e.g. sentence structure or task structure, rather than semantic content encoded in token embeddings. An ideal model would be able to do both: flexibly deploy in-weights operations (in order to robustly accommodate ambiguous or unknown contexts using encoded semantic information) and structural in-context operations (in order to accommodate novel tokens). We study structural in-context algorithms in a simple part-of-speech setting using both practical and toy models. We find that active forgetting, a technique that was recently introduced to help models generalize to new languages, forces models to adopt structural in-context learning solutions. Finally, we introduce temporary forgetting , a straightforward extension of active forgetting that enables one to control how much a model relies on in-weights vs. in-context solutions. Importantly, temporary forgetting allows us to induce a dual process strategy where in-context and in-weights solutions coexist within a single model. 1 1 1 We release code here for reproducibility

## 1 Introduction

A distinguishing trait of transformers is their ability to perform ‘in-context’ learning (ICL) (Brown et al., ; Dong et al., ; Garg et al., ) – the ability to use context at inference time to adjust model behavior, without weight updates, to generalize to unseen input-output combinations. This ability enables the models to flexibly accommodate variations in language. For instance, a model is likely to memorize that the token green is typically an adjective, but recognize that it is used as a noun in the sentence The child sat on the main green. If queried to predict the part of speech (POS) of green, a model using the in-weights strategy would likely incorrectly predict adjective, while the in-context strategy would allow it to infer noun.

Recent research has studied the tradeoff between ICL and in-weights learning (IWL) in transformers (Chan et al., ; Singh et al., ; Reddy, ; Chan et al., ). Chan et al. () found that language-like data distributional properties play a critical role in the emergence of ICL. Importantly, they found that ICL and IWL strategies often appear to be in opposition; only with a particular skew of the label distribution were they able to promote both strategies to co-occur in a model. With a similar setup, Singh et al. () found that ICL strategies smoothly decrease in strength across distributions; furthermore, while they found regularization mitigates ICL transience, they never arrive at a method that allows ICL and IWL to permanently co-exist in the same model. An ideal model would encode dual processes: flexible, context-sensitive operations for out-of-distribution settings and memorized, static operations for ambiguous contexts or IID settings (Kahneman, ; Miller, ).

Figure: Figure 1: (Top Left) In our natural setting, we use a part-of-speech probe trained on BERT representations of sentences from Penn Treebank 3 and evaluate on templated examples (Section 2). (Top Right) In our synthetic setting, we train a small masked language model (MLM) on a grammar where the expected response is conditioned on the part-of-speech of the query (Section 3). (Bottom Left) An idealization of our main finding: structural ICL is transient (i.e. decays over training) in both natural and synthetic settings. Active/temporary forgetting maintains structural ICL in the synthetic setting. (Bottom Right) T-SNE visualization of token embeddings after standard vanilla MLM training on synthetic setting (van der Maaten and Hinton, ). We see that embeddings in the head of the distribution clusters together, as do the unseen token embeddings. The embeddings in the tail of the distribution bridge between the two clusters. Models using conditional ICL would only generalize to the heldout examples that exist within the head token distribution. Models using structural ICL would freely generalize to all token embeddings.
Refer to caption: extracted/5703567/figures/task_figures_pure.png

Moreover, prior work (Singh et al., ; Chan et al., ) has focused on what we refer to as conditional in-context learning. That is, they focus on ICL which generalizes to heldout inputs which are imbued with semantic information, and thus can be seen as interpolations of seen inputs. Such conditional ICL algorithms would likely fail to predict that in the sentence The child sat on the main bluk., the new word bluk is a noun. Conditional ICL algorithms fail when they include tokens that are undertrained (Land and Bartolo, ; Rumbelow and Watkins, ) or newly-introduced (e.g. when adding languages to an existing model) (Chen et al., ). This breakdown in ICL performance occurs because the model is not encoding a truly content-independent in-context strategy, and rare and unseen embeddings are often out-of-distribution after vanilla training, as shown in Figure 1 (Bottom Right). In contrast, we define structural in-context learning to be the ability of a model to perform in-context learning on arbitrary tokens, or extrapolations from seen inputs. We test this by assessing performance on unseen tokens in a naturalistic and synthetic setting described in Figure 1 (Top Left, Top Right). While conditional ICL fails on the tail of highly-skewed distributions Chan et al. (), structural ICL would maintain performance.

We find that structural ICL is also transient. However, while regularization provides a path to persistence in conditional ICL (Singh et al., ), it does not for structural ICL. Therefore, we propose an extension to active forgetting – a recent weight resetting technique introduced by Chen et al. () to help augment models with new tokens – to make structural ICL persistent. Our modification allows us to coarsely control the strategies that the model adopts, enabling us to induce a dual process strategy: (structural) ICL for rare and unseen tokens and IWL for common tokens.

Our main contributions are:

- •
We define and study the concept of structural ICL in both large models and toy models using a simple part-of-speech probing task. This allows for true generalization of in-context strategies for completely unseen tokens. We discover that MLMs exhibit a (limited) form of structural in-context learning that emerges early in training, but that this ability quickly vanishes.
- •
We show active forgetting (Chen et al., ) maintains structural ICL in models. We introduce temporary forgetting, a straightforward extension of active forgetting that enables one to control how much a model relies on in-weights vs. in-context solutions.
- •
We demonstrate that when training with skewed token distributions, temporary forgetting enables us to induce a dual process strategy where our model uses an in-weights solution for frequently-seen tokens in the head of the distribution and a (structural) in-context solution for rare tokens in the tail.

## 2 (Structural) In-Context Learning is Transient

Recent work has discovered that conditional ICL capabilities slowly degrade in synthetic settings over the course of training (Singh et al., ). Building on this work, we track the tradeoff of conditional IC vs. IW algorithms in a naturalistic syntax probing task over the course of training for encoder-only language models (LMs). More importantly, we also track structural ICL over the course of training.
We study the MultiBERTs, averaging all of our results across seeds 0, 1, and 2. We calculate error bars in Figure 2 as $\pm 1$ standard error of the mean (SEM).

### 2.1 Task

We design a task that employs templated stimuli to determine the tradeoffs between different strategies for assigning part of speech to tokens – this task permits both structural IC and IW solutions. For instance, in the sentence the dog is happy, there are at least two ways of determining that dog is a noun: (1) memorize that the token identity “dog” is a noun or (2) extract that dog is the subject of the sentence from the context. For each layer and MultiBERT step, we train a binary POS probe on representations of nouns and adjectives from sentences in the training set of Penn Treebank 3 (PTB-3) (Marcus et al., ). For multi-token words, we average representations across tokens. See Appendix A.1 for additional details about our probing setup. We then evaluate the pretrained MultiBERT and probe on a suite of test sets designed to assess the adoption of in-context or in-weights strategies. Each dataset contains sentences that obey the template:
The <noun> is <adj> (e.g. The dog is happy).
Our evaluation datasets are defined as follows:

- 1.
Head: Templated examples where tokens are sampled from the most frequent 1500 nouns and most frequent 1500 adjectives in the training set of PTB-3.
- 2.
Tail: Templated examples where tokens are sampled from the least frequent 1500 nouns and most frequent 1500 adjectives in the training set of PTB-3.
- 3.
Head Switch: Templated examples where tokens are sampled as in the “Head” dataset, but where nouns appear in the adjective position and adjectives appear in the noun position (e.g., The happy is dog).
- 4.
Tail Switch: Defined similarly to “Head Switch”, except where the tokens are sampled from the tail of the token distribution.
- 5.
Unseen Token: Templated examples where “nouns” and “adjectives” are sampled from a set of 1,500 randomly initialized tokens. This metric evaluates structural ICL performance(^2^22We are able to generate novel labels not seen during train time because the embedding and unembedding matrices are tied in the MultiBERT models.).

Note that the MultiBERTs are trained following Devlin et al. () on a combination of BookCorpus (Zhu et al., ) and English Wikipedia collected by Turc et al. (). As such, the distribution of the training data is fixed, and our experiments are constrained to the natural distribution of language.
As BookCorpus does not have POS tags readily accessible, we employ PTB-3 to estimate the noun and adjective distribution of the training data. We defined nouns and adjectives as words that appeared as each POS, respectively, over 80% of the time.
We chose 1500 examples as this is $\approx 10\%$ of the number of unique nouns.

Figure: Figure 2: (Left) We exhibit the transience of structural ICL by examining the Unseen Token Accuracy over time. (Middle) We show the trend of memorization of tail versus head of distribution over training steps by examining the difference in Layer 7 Accuracy, where both in-context and in-weights strategies are possible, and Layer 0 Accuracy, where only an in-weights strategy is possible; (Right) We display the preference for in-weights strategy when conflicting with in-context strategy over time.
Refer to caption: extracted/5703567/figures/bert_figure_new.png

### 2.2 Training Dynamics

We examine (1) structural in-context learning and (2) the tradeoff between in-context and in-weight strategies over the course of training.

#### Structural ICL

We find that the MultiBERTs are initially able to perform structural ICL, but that this capability is transient. In Figure 2 (Left), we present results from a probe trained on representations from Layer 7 as this layer achieves the highest probing validation performance on PTB-3. This is consistent with prior research which demonstrates that syntactic structures are encoded in the middle layers of MLMs Tenney et al. (); Limisiewicz and Mareček (). Furthermore, results across all layers are presented in Appendix A.2.
Structural ICL transience is evident as probe performance on Unseen Tokens tend to spike early in MultiBERT training before dropping to chance by the end of training. These results suggest that there is an inductive bias toward structural ICL that diminishes as information is encoded in the embeddings. As structural ICL confers the ability to generalize to rare and new tokens, this raises questions about how we can train models that maintain this ability throughout training.

#### In-Context vs. In-Weights Strategies

Next, we compare conditional in-context vs. in-weights strategies for observed tokens. First, we observe that ICL strategies dissipate over training, as more information is encoded in token embeddings. We approximate the use of in-context information for determining POS as the difference in performance between Layer 0 (the embedding layer) and Layer 7. Layer 0 must rely only on in-weights information as there is no in-context information available; in contrast, Layer 7 uses contextualization to achieve higher performance (Tenney et al., ; Hewitt et al., ). Early in training, this additional in-context information leads to higher probe accuracy; however, this benefit disappears over time. Figure 2 (Middle) demonstrates this trend across tokens at the head and tail of the distribution. Notably, the benefit of in-context information disappears more quickly for the head of the distribution than the tail, likely because there are far more gradient updates to head token embeddings.(^3^33We observe that performance gain due to the model’s use of in-context information decreases across a wide range of syntactic phenomena as embeddings are enriched during training. We term this the ”Pushdown Phenomenon” and explore it more thoroughly in Appendix A.4.)

As the benefit of the model’s use of in-context information dissipates, we observe that the model shifts from an in-context to an in-weights strategy in Figure 2 (Right). Specifically, we find that a model’s preference toward assigning POS on the basis of token identity (i.e. an in-weights solution) increases slightly over time when in-context and in-weights information are in conflict. In other words, models becomes more reliant on in-weights strategies and less reliant on in-context strategies over the course of training. This finding aligns with Singh et al. (), which analyzed a similar phenomenon using toy models and a synthetic task. Additionally, we observe that the degree to which the model adopts an in-weights strategy varies significantly for tokens selected from the head versus the tail of the distribution. When assigning POS to tokens in the the head of the distribution, the model relies almost exclusively on an in-weights solution, while the model relies on both in-weights and in-context solutions when assigning POS to tokens in the tail.

In summary, we find that (1) the benefit of the model’s use of context information disappears over time and (2) reliance on in-weights information increases over time, varying depending on the distributional properties of the token that we are probing.

## 3 Synthetic Task: Distributional Parameters Impact In-Context Learning

We develop a synthetic masked language modeling task to reproduce the above trends, in order to characterize how distributional parameters affect the learning strategy that the model adopts. Our synthetic task requires the model to determine which of two classes a word belongs to. This may be derived either from in-context information or by memorizing token identity-class associations in the embedding layer. We draw analogies between these classes and POS in natural language.

Our vocabulary contains tokens that represent nouns, adjectives, and a copula (i.e. is). Each sentence is created by selecting (1) a sequence $S$, (2) a query $Q$, and (3) a response pattern $P$. Our MLM is trained to predict $\mathbb{P}(P_{i}|S,Q)$ for all $i\in\{0,\ldots,|P|-1\}$ (i.e. the probability of each pattern token). The sequence and pattern are arbitrary and designed so that no exceedingly simple heuristic may solve this task.

- •
sequence $S$: Either <noun> <copula> <adj> or <copula> <adj> <noun>.
- •
query $Q$: Either the <noun> or <adj> from the sequence.
- •
pattern $P$: Either <adj> <noun> <noun> if the query is a <noun> or <adj> <adj> <adj> if the query is an <adj>.

This task is designed such that the model must make a POS classification on the query token, and then perform some additional operation conditioned on that classification (copying specific token identities in a specific order). See Appendix A.5 for more details. See Figure 1 for an example.

We parameterize the task with vocabulary size $v$, the sampling distribution skew for nouns/adjectives $\alpha$ (where we select $\texttt{<noun>, <ad>}\sim\text{Zipf}(\alpha)$), and the ambiguity of token POS $\varepsilon$. The ambiguity parameter determines the percentage of tokens can act as both as noun and an adjective, and is inspired by the inherent ambiguity of POS in natural language. For our primary experiments, we fix $\varepsilon=0.10$. Note, we find that $\varepsilon$ must be greater than zero for an in-context solution to emerge at all. We compare our skewed distribution results to sampling tokens from a Uniform distribution.

In this task, an ICL solution to derive the POS of the query may achieve perfect accuracy by utilizing in-context information (e.g. a copula is always followed first by an adjective, then a noun). In contrast, an IWL solution to derive the POS of the query may achieve at most an accuracy of $(1-\varepsilon/2)$ due to ambiguous tokens. To account for this, we evaluate our models only on tokens that are not ambiguous; thus, both an ICL and IWL solution could achieve perfect accuracy. (Ambiguous tokens always use an ICL solution.)

Our task is formatted in a cloze-style where each token in the pattern is masked. We employ a MLM (Devlin et al., ) to predict the identities of these masked tokens, with hyperparameters described in Appendix A.6. Near-perfect validation accuracy is achieved after <60,000 steps on all experimental settings.

In addition to performance on a randomly selected validation set, we create datasets to evaluate the model’s preferred strategy throughout training, similar to Section 2. All examples in these datasets contain novel <adj>, <noun> pairs. Much like our naturalistic setting metrics in Section 2.1, we create Tail, Head, Head Switch, Tail Switch, and Unseen Token Accuracy metrics. In this setting, our head and tail metrics use the top and bottom 10% of the token distribution by count, respectively.

Figure: Figure 3: (Top) In-context performance by distribution with vanilla training; (Bottom) In-context performance by distribution with active forgetting. The parameters used are $v=10000,\varepsilon=0.10$. Note that the Uniform distribution does not have a head or a tail, so its results are in the head graphs.
Refer to caption: extracted/5703567/figures/icl_by_dist_new.png

### 3.1 Training Dynamics

#### Structural ICL

We largely reproduce the results from the natural language setting presented in Section 2: structural in-context solutions emerge quickly, but are transient. This is shown by the early peak of Unseen Token Accuracy, followed by its steep drop. This trend holds across all tested distributions in Figure 3 (Top Left). As such, both the syntactic and naturalistic settings align with our idealized graph of structural ICL transience exhibited in Figure 1 (Bottom Left). However, the disappearance of a structural in-context algorithm occurs extremely quickly compared to our MultiBERT experiments, likely due to the simplicity of our synthetic task.

#### In-Context vs. In-Weights Strategies

In this section, we analyze whether models adopt conditional ICL or IWL strategies over the course of training. Our results are presented in Figure 3. Importantly, we find that increasing the skew of a distribution increases the pressure toward an IWL strategy. Conversely, examples with tokens drawn from a Uniform sampling distribution show a comparatively higher ICL preference (and thus lower IWL preference) than any Zipfian sampling distribution in Figure 3 (Top Middle). Among Zipfian skewed distributions, the model’s strategy varies based on whether the adjective and noun are in the head or the tail of the token distribution, much like in our naturalistic task. As in Section 2, we find that all skewed distributions prefer a IWL strategy for head tokens. However, for tail tokens, distributions of moderate skew ($\alpha=1.0001,\alpha=1.2$) prefer an ICL strategy as shown in Figure 3, while highly skewed distributions ($\alpha=1.5$) fail altogether as shown in Appendix A.7. This is likely due to the fact that these tokens are rarely observed in the training data. This illustrates an important distinction between structural ICL and conditional ICL – a structural ICL solution would maintain performance on the tail of highly skewed distributions.
Additional experiments exploring the effect of ambiguity are located in Appendix A.8 and the effect of vocabulary size are located in Appendix A.9.

## 4 Maintaining Structural ICL with Active Forgetting

In Sections 2 and 3, we have demonstrated that as information gets memorized in the embeddings, the benefits of in-context information dissipate and models shift to an IWL strategy.
In an effort to promote structural ICL, we utilize a recently-introduced training procedure: active forgetting (Chen et al., ). When training a model using active forgetting, we re-initialize the embedding matrix every $k$ steps during training. The intuition behind this is that the model must employ in-context strategies to achieve high accuracy, as no information can be preserved in each token’s embedding. In other words, the model can no longer assume that the input embeddings encode any particular information and thus must develop a structural ICL strategy. While after vanilla training, these unseen embeddings are out-of-distribution as illustrated in Figure 1 (Bottom Right), we hypothesize that these unseen embeddings would align with seen embeddings after training with active forgetting. We explore this hypothesis in Section 6.

Training our models with active forgetting successfully engenders structural ICL, enabling the model to approach perfect performance on the Unseen Token Set (See Figure 3, Bottom Left). Given two random embeddings representing a noun and an adjective, the model can now (1) derive the POS of these tokens and (2) output the identity of these out-of-distribution embeddings in the desired pattern. Note that we see a slightly more stochastic version of our idealized trend from Figure 1 (Bottom Left) due to the resetting mechanism.

We test $k=100,1000,5000$ and settle on $k=1000$, as this worked well in our preliminary exploration. With active forgetting, both the head and the tail of the training distribution prefer an asymptotic in-context strategy across all tested skews (See
Figure 3, Bottom).
Still, as the skew of the distribution of nouns and adjectives increases, there is greater pressure to memorize the head of the distribution (as these tokens are observed more frequently). Thus, it takes longer for the model to exhibit a preference towards in-context solutions for head tokens (e.g. almost 60,000 steps for the $\alpha=1.5$ setting) and there is a much larger drop-off in performance after every instance of forgetting the embedding matrix.

## 5 Dual Process Learning with Temporary Forgetting

Figure: Figure 4: In-weights preference is coarsely controllable by varying temporary forgetting parameter $N$. All $N>0$ settings in figure induce success on completely abstracted generalization for all $N$. Note $N=0$ is vanilla training and $N=\infty$ is active forgetting. Parameters used are $v=10000,\varepsilon=0.10,\alpha=1.5$.
Refer to caption: extracted/5703567/figures/sf_final_control.png

While active learning successfully induces a structural ICL strategy, our model loses the ability to memorize information in its embeddings. This is detrimental in a variety of cases, such as when in-context information is insufficient to generate an appropriate response. An optimal model would encode a dual process strategy: maintaining a structural ICL solution while also memorizing useful linguistic properties (Chan et al., ). We modify the paradigm of active forgetting to attempt to induce a bias for structural in-context strategies in the tail of the distribution while preserving the in-weights solutions for frequently-observed tokens. We introduce temporary forgetting, where we perform active forgetting every $k$ steps for the first $N$ steps ($N>>k$) of training. After this point, we allow the embedding matrix to train as normal.

We find that by varying $N$, we can vary the model’s dependence on in-weights information on frequently seen tokens while maintaining structural ICL performance as displayed in Figure 4. If $N$ is too large, this training procedure mimics the behavior of active forgetting, eliminating in-weights solutions in favor of structural in-context solutions. Additionally, if $N$ is too small, the training only sometimes maintains structural ICL performance; note, however, that this seems to be an all-or-nothing effect. The sweet spot for $N$ depends on the skew of the distribution. We show that in the $\alpha=1.5$ case, we can specifically control the preference for an in-weights strategy over an in-context strategy on observed tokens by modifying $N$ (See Figure 4). In general, by manipulating the $k$ we reset the embeddings and $N$, we can calibrate the relative strength of in-context vs. in-weights strategies.

Thus, temporary forgetting enables a model to successfully encode two distinct strategies for the same task. While this dual process strategy was previously demonstrated in Zipfian distributions with $\alpha\approx 1.0$, we can now induce this behavior for any distribution $\alpha\geq 1.0$, while also inducing structural ICL behavior on all distributions (See Figure 5).(^4^44Distributions where $\alpha\leq 1.0$ would likely only rely on an in-context strategy) Note that the control granted by temporary forgetting over head IWL preference has limits – we can push up to almost 90% the original IWL preference while maintaining a high tail ICL preference.

Temporary forgetting imparts an incentive that significantly enhances our ability to balance between in-context and in-weights strategies, overcoming inherent biases in naturally occurring data. By tuning the hyperparameters ($k,N$), one can bias the model toward either type of solution.

Figure: Figure 5: (Left) Temporary forgetting achieves near perfect unseen token performance (structural in-context) asymptotically among distributions. (Right) In addition, temporary forgetting can asymptotically hold preference for an in-weights strategy in the head of the distribution while holding preference for an in-context strategy in the tail of the distribution (i.e. learn dual processes). Parameters used are $v=10000,\varepsilon=0.10$ and optimal hyperpameters $k,N$ over gridsearch.
Refer to caption: extracted/5703567/figures/sf_comparison_new1.png

## 6 Embedding Analysis

We perform qualitative analyses on the embeddings produced by vanilla training, active forgetting, and temporary forgetting in order to better understand how these training regimens impact model representations. These analyses, consisting of principal component analysis (PCA) and probing for POS, are located in Appendix A.10.

After vanilla training, the learned embeddings cluster according to their POS, far from the distribution of randomly-initialized tokens. We train a linear probe on these learned embeddings, and find that it can almost perfectly partition nouns and adjectives. Note that the disappearance of structural ICL occurs at the same time as the probe achieves above-random POS probing (i.e. memorization).

As expected, we do not see any structure in the embeddings produced after active forgetting. As such, a linear POS probe trained on these embeddings never achieves above random chance throughout training. The embedding distribution looks quite similar to the random initialization distribution, indicating that no information has been encoded in these embeddings.

Finally, the temporary forgetting setting reflects aspects of both vanilla training and active forgetting; that is, the head of the token distribution learns to partition nouns and adjectives whereas the tail of the distribution does not learn any structure. The tail embeddings much more closely resemble the initialization distribution with temporary forgetting than with vanilla training. This results in a unseen token generalization in addition to memorized information.

## 7 Related Work

#### In Context v. In Weights

A body of recent literature closely examines in-weights versus in-context learning (Chan et al., , ; Reddy, ; Raparthy et al., ; Fu et al., ).
The emergence of in-context learning abilities in transformers has been shown to depend on the distributional properties of the training data such as burstiness, training class rarity, and dynamic meaning (Chan et al., ; Reddy, ). While we employ a similar analytical framework to this work, we (1) consider truly random heldout inputs and novel outputs/labels, (2) evaluate on large, natural language models, and (3) consider structural ICL. Additionally, while transience of in-context solutions has been noted in Singh et al. (), we find transience of structural ICL, and find that the adoption of conditional ICL actually increases over training for our synthetic setting. Additionally, unlike Singh et al. (), we find that increasing L2-regularization does not affect the transience of structural ICL in our synthetic setting (See Appendix A.7). Finally, we introduce temporary forgetting to solve what both Singh et al. () and Chan et al. () suggest to be an extremely useful behavior: the co-existence of in-context learning and in-weights learning.

More broadly, the conflict between context-dependent and context-independent (or reflexive) solutions has been well-studied in the cognitive and computational neuroscience literature (Russin et al., ; Rougier et al., ; Russin et al., ). A key feature of human intelligence, termed cognitive control, is the ability to maintain dual strategies and flexibly deploy either one in response to particular stimulus. Any artificial system that aspires to producing human-like behavior must therefore be capable of maintaining both of these solutions.

#### Weight Forgetting To Help Learn.

While most literature on forgetting characterizes this phenomenon as undesirable (Kemker et al., ; Kirkpatrick et al., ; McCloskey and Cohen, ; Ratcliff, ), recent neuroscience literature has shown that intentional forgetting may have positive roles in certain contexts (Srivastava et al., ; Pastötter et al., ; Levy et al., ; Anderson and Hulbert, ). Intentional forgetting in neural networks is accomplished by resetting a subset of parameters during training. On computer vision tasks, this resetting procedure has been shown to help low compute and data resource generalization (Alabdulmohsin et al., ; Taha et al., ; Ramkumar et al., ). Additionally, Zhou et al. () show that a forget-and-relearn paradigm helps language emergence. Our method of forgetting embeddings is directly inspired by Chen et al. (), which shows forgetting during pretraining boosts linguistic plasticity for multilingual learning. As far as we know, we are the first to propose using forgetting to induce ICL.

## 8 Discussion

This research provides insights into the interplay between structural ICL, conditional ICL and IWL within transformers. We shed light on several critical factors determining how models manage and utilize the encoded and contextual information when faced with novel tokens and tasks.

#### Structural In-Context Learning

One of our key findings is the transience of structural ICL in LMs. Initially, models exhibit a strong ability to leverage structural ICL, generalizing algorithms to unseen tokens. However, this capability dissapears as training progresses, suggesting an initial inductive bias towards structural ICL that wanes as the model learns. This transience limits generalization on rare tokens and new tokens. We find that active forgetting maintains structural ICL by repeatedly reinitializating the embeddings. Our temporary forgetting training procedure enables a dual process strategy through strategic re-initialization of weights. This enables adaptability while still leveraging accumulated knowledge.

#### Implications for Model Training and Application

Our findings are useful to design training protocols that result in flexible models. A significant reason for the success of LMs is their capacity for ICL and IWL strategies to co-exist, a behavior that organically occurs with a moderately skewed Zipfian distribution. However, most natural domains such as protein discovery, network traffic, and video recording face even more skew, breaking down this ideal behavior. Our temporary forgetting technique facilitates a dual process strategy regardless of skew, which could potentially bring some of the profound success of LMs to other domains.

#### Future Directions and Limitations

The research opens up several avenues for future investigation. Future research should examine Structural ICL across different model architectures and configurations. One significant limitation is that our temporary forgetting experiments were not performed on LMs. Our compute resources limited such experiments, but we believe this is a critical future step to refining this training intervention. Another limitation of our work is that the optimal hyperparameters to temporary forgetting are not known a priori, and might require several runs to tune. Finally, another avenue of fruitful future research may be the translation of structural ICL algorithms into symbolic systems. As structural ICL does not rely on the content of the input, it should be possible to use techniques like circuit analysis (Räuker et al., ) to reverse-engineer an explicit symbolic representation of the algorithm that the neural network uses to solve a task.

#### Conclusion

This study deepens our understanding of a model’s adoption of structural ICL, conditional ICL, and IWL strategy during training. The techniques introduced here not only enhance our theoretical understanding but also offer practical tools for improving model training and functionality in real-world applications.

## Appendix A Appendix / supplemental material

### A.1 Probing Setup

We provide probing background in this section, borrowing some notation from Elazar et al. [].

Given a set of labeled data of points $X=x_{1},\ldots x_{n}$ and task labels $Y=y_{1},\ldots,y_{n}$, we analyze a model $f$ that predicts the labels $Y$ from $X:\hat{y_{i}}=f(x_{i})$. We assume that this model is composed of two parts: (1) an encoder $h$ that transforms input $x_{i}$ into a learned representation vector $\textbf{h}_{x_{i}}$ and (2) a classifier $c$ that is used for predicting $\hat{y_{i}}$ based on $\textbf{h}_{x_{i}}$, such that $\hat{y_{i}}=c(h(x_{i}))$. We refer by probe to the classifier $c$ and refer by model to the model from which the encoder $h$ is a subset of.

Given this setup, we evaluate a particular model’s performance across various layers and training steps for our POS task. Each encoder $h$ is associated with a specific training step and layer $h^{t,l}$. We probe the residual stream after layer $l$.

In this research, we are interested in the model’s choice of strategy at a particular time step. That is, we seek to describe the change in prediction of $\hat{y_{i}}$ due to varying $t,l$ of encoder $h^{t,l}$. Accordingly, we fix $c$ as a single linear fully-connected layer.

### A.2 Structural ICL across Layers

Figure: Figure 6: We find that structural ICL is transient across all layers of MultiBERTs (seeds 0, 1, 2 averaged). The middle layers show the most structural ICL during early in training, whereas very early and very late layers remain about random throughout training.
Refer to caption: extracted/5703567/figures/layerwise_sicl.png

We find that structural ICL consistently approachs random levels as training progresses across layers in the MultiBERTs. This signifies that the model fully loses the ability to process unseen tokens as training continues. This is likely the reason for the "glitch tokens" described in Land and Bartolo [], for which LMs fail to output sensible content.

### A.3 Pushdown Datasets

We use the train/dev splits from the English UD Treebank for the c-pos, f-pos, and dep tasks McDonald et al. []; the train/dev splits from Ontonotes-v5 in the CoNLL-2012 Shared Task format for the ner, phrase start, and phrase end tasks Linguistic Data Consortium [], Pradhan et al. []; the train/dev splits from Penn Treebank-3 for the depth and dist tasks Marcus et al. []; and generated token sequences for the prev, dup, and ind tasks.

We reproduce baselines from Elazar et al. [] to verify the correctness of our probing setups for c-pos, f-pos, ner, dep, phrase start and phrase end and from Hewitt and Manning [] for depth and dist.

### A.4 Pushdown Signature Observation in Syntax

Figure: Figure 7: The "Pushdown Phenomenon" is observed across syntactic features, suggesting that a transition from IC to IW strategies happens across these features. In early steps of training, representing syntactic information occurs in later layers, which are more contextualized. However, as training progress, the same properties are better encoded in earlier layers due to memorization of token-level and n-gram level information. The n-gram level information requires attention to build, which explains why performance in dep, depth, and dist does not propagate all the way to embeddings.
Refer to caption: extracted/5703567/figures/syntax_easy.png

The "Pushdown Phenomenon" suggests that in early steps of training, computing token-wise syntactic properties occurs in later layers, which have more in-context information. However, as training progress, the same properties are better encoded in earlier layers until only the first couple layers are required for representing syntactic properties.

We examine whether the "Pushdown Phenomenon" exists in various syntactic properties in BERT. To do so, we employ our probing setup (Appendix A.1) for the tasks of named entity recognition (ner), coarse part of speech (c-pos), fine-grained part of speech (f-pos), dependency parsing (dep), syntactic constituency boundaries which indicate the start and end of a phrase (phrase start, phrase end), depth in the parse tree (depth), and distance in the parse tree (dist). We probe each property across the axes of (1) training time steps and (2) layers. We repeat this process for three seeds of the MultiBERTs [Sellam et al., ]. For all tasks, we probed all layers of MultiBERT seeds 0, 1, and 2 for timesteps from 0 to 200,000 increasing by 20,000; 200,000 to 1,000,000 increasing by 100,000; and 1,000,000 to 2,000,000 increasing by 200,000. If a specific word is composed of multiple subword tokens, we follow Hewitt and Manning [] and average the encoding across tokens.

We observe the "Pushdown Phenomenon" in all our examined tasks. However, we find that across tasks, syntactic information is "pushed down" at different rates. Early layer accuracy increases approximately follow a pattern of $\textit{ner}\to\textit{phrase start}\to\textit{cpos/fpos}\to\textit{phrase end
}\to\textit{dep}\to\textit{depth}\to\textit{dist}$. We leave it to future work to explore whether this timing is a function of (1) complexity of high-achieving rules/heuristics consistent with Belrose et al. [] or (2) a naturally occurring dependency hierarchy of syntactic relationships suggestive of implicit curriculum learning. One possible intuition for why the "Pushdown Signature" of memorization often coincides with poor maintenance of in-context strategies might be neural collapse [Parker et al., , Rangamani et al., ], although this should be further investigated by future experimentation.

### A.5 Synthetic Data Generation Formulation

Our synthetic data generation can be formally representated as a probabilistic context-sensitive grammar (PCSG). Mathematically, we parameterize our vanilla PCSG (without POS ambiguity) as follows:

$$ $\textbf{G}=(N,\Sigma,P,S,\alpha,v)$ $$

where $N=\{S,Q,Q_{N},Q_{A},P_{N},P_{A}\}$ is the set of nonterminal symbols, $\Sigma=\{N_{init},A_{init},N_{r},A_{r},C\}$ is the set of terminal symbols, $S$ is the starting point (and notationally also represents sequence), and $\alpha,v$ characterize the sampling probability distribution of our terminal symbols.
Our production rules $P$ are

$$ $F\to\begin{cases}S\ Q_{N}\ P_{N}\\ S\ Q_{A}\ P_{A}\end{cases}\text{with eq. prob.}$ $$

$$ $\displaystyle S\to\begin{cases}N_{init}\ C\ A_{init}\\ C\ A_{init}\ N_{init}\end{cases}\text{with eq. prob.}$ $\displaystyle Q\to\begin{cases}Q_{N}\\ Q_{A}\\ \end{cases}\text{with eq. prob.}$ $\displaystyle Q_{N}\to N_{r}$ $\displaystyle Q_{A}\to A_{r}$ $\displaystyle P_{N}\to A_{r}\ A_{r}\ A_{r}$ $\displaystyle P_{A}\to A_{r}\ N_{r}\ N_{r}$ $$

with terminal symbols sampled from

$$ $\displaystyle N_{init}\sim\text{Zipf}\left(\alpha,0,\frac{v}{2}-1\right)$ $\displaystyle A_{init}\sim\text{Zipf}\left(\alpha,\frac{v}{2},v-1\right)$ $\displaystyle C\to v$ $\displaystyle N_{r}\to N_{init}$ $\displaystyle A_{r}\to A_{init}$ $$

$N_{init}$ captures a specific token that corresponds to a token and all references to $N_{r}$ use this token exactly, enforcing strict consistency.

Note our sampling distribution Zipf is a truncated Zipfian parameterized by the tuple $(\alpha,s,e)$ with a probability mass function of

$$ $\mathbb{P}(X=k)=\frac{k^{-\alpha}}{H(\alpha,e-s)}\text{ for }k=s,s+1,\ldots,e, \text{ where }H(\alpha,n)=\sum_{k=1}^{n}k^{-\alpha}$ $$

We select tokens for $\texttt{<noun>}\in\left\{0,1,\ldots\frac{v}{2}-1\right\}$ and $\texttt{<adj>}\in\left\{\frac{v}{2},\frac{v}{2}+1,\ldots v-1\right\}$. Thus, given a particular vocabulary size $v$ and Zipf parameter $\alpha$, $\texttt{<noun>}\sim\text{Zipf}\left(\alpha,0,\frac{v}{2}-1\right)$ and $\texttt{<adj>}\sim\text{Zipf}\left(\alpha,\frac{v}{2},v-1\right)$. To add further control to this setting, we introduce the parameter $\varepsilon$ to describe ambiguity in the solution - that is, a proportion of $\varepsilon$ tokens in each of $n=10$ bins grouped by probability mass do not have a fixed POS but instead may be a noun or adjective with equal likelihood.

Note that when $\alpha=0$, this distribution degenerates into $\text{Unif}(s,e)$ and when $\varepsilon=0$, each token has a fixed identity.

### A.6 Toy Model

We employ a 6-layer BERT model across the synthetic setting experiments. Experiments were performed with an MLM as far less prior work has examined syntactic tasks with autoregressive models. Structure is much more difficult to intuit in autoregressive models as they are only exposed to an ordered subset of the tokens in a sentence. This model has 1 attention head per layer, 64-dimensional hidden dimensions, 128-dimensional intermediate representations, and tied weights for the embedding and unembedding layers. We optimize model parameters with AdamW with a learning rate of $5\times 10^{-5}$ [Loshchilov and Hutter, ]. We chose a thin and long representation to examine how representations evolve after each attention operation (for better granularity). The hidden dimension sizes were decided per a minimax strategy, i.e. this representation dimensionality was the smallest such that we achieved near perfect accuracy on a validation set for the downstream task. Future work should better examine the effect of representation size on in-context vs. in-weights learning.

### A.7 Performance by Token Decile

Figure: Figure 8: Increased weight decay has little/no effect on the failure of the structural ICL strategy (we increase weight decay from 0.01 to 0.1). In contrast, active temporary forgetting boosts rare token validation accuracy significantly, as seen in the tail of the distribution. Parameters are $v=10000,\varepsilon=0.10,\alpha=1.5$
Refer to caption: extracted/5703567/figures/failure_mode.png

We find that on highly skewed distributions, the tail of the distribution suffers immensely due to undertraining. This phenomenon cannot be rectified by Singh et al. []’s method of promoting asymptotic ICL. However, we find that both active forgetting and temporary forgetting correct this behavior to boost performance on tail tokens in skewed distributions from near-zero to near-perfect levels.

### A.8 Ambiguity ( ε 𝜀 \varepsilon italic_ε ) Experiments

Figure: Figure 9: (Top) $\varepsilon=0.01$, (Middle) $\varepsilon=0.10$, (Bottom) $\varepsilon=0.50$. Overall in-context strategy is dependent by amount of ambiguity in the labels. With 50% of the tokens as ambiguous, all unambiguous tokens use an in-context strategy; with 10%, there is a mixed strategy dependent on where in the distribution the example is; with 1%, almost unambiguous tokens use a memorized strategy. The vocab size is $v=10000$
Refer to caption: extracted/5703567/figures/icl_by_amb.png

In all of our ambiguity experiments, structural ICL is transient (even whe 50% of tokens are ambiguous). The ambiguity parameter significantly alters the models overall strategy. With a low ambiguity parameter, the model prefers memorization (IWL strategy) of umambiguous tokens and with a high ambiguity parameter, the model prefers an ICL strategy. Across all ambiguity parameters, there is a difference in tail and head behavior.

### A.9 Vocabulary Size ( v 𝑣 v italic_v ) Experiments

Figure: Figure 10: (Top) $v=1000$, (Middle) $v=10000$, (Bottom) $v=20000$. The strength of an in-context solution depends on the interaction between vocabulary size $v$ and skewedness of the distribution $\alpha$. Too small of a vocabulary size (i.e. $v=1000$) encourages more memorization in general but fixes performance in $\alpha=1.5$ setting. The ambiguity is $\varepsilon=0.10$.
Refer to caption: extracted/5703567/figures/icl_by_vocab.png

In all of our vocabulary experiments, structural ICL is transient. As expected, we find that vocabulary size has a similar effect to the skewedness of the distribution. That is, increasing the vocabulary without bound would lead to poor tail ICL performance. Too small of a vocabulary size seems to increase ICL among very skewed distributions but decrease ICL among all other distributions.

### A.10 Principle Component Analysis of Embeddings

Figure: Figure 11: Vanilla training imposes structure on the adjectives and nouns such that randomly initialized (unseen) tokens are out-of-distribution whereas active forgetting embeddings resemble the initial distribution. Parameters used are $v=10000,\alpha=1.0001,\varepsilon=0.10$.
Refer to caption: extracted/5703567/figures/pca_embeddings.png

We find that while vanilla training results in embeddings that lie on a manifold, active forgetting results in embeddings that look similar to the initial distribution. This helps motivate our use of temporary forgetting as we would like to preserve embedding structure. Moreover, note that in the above figure we use $\alpha=1.0001$ and PCA whereas in Figure 1 (Bottom Right), we use $\alpha=1.5$ and T-SNE. The tail tokens in the higher skew distribution see fewer gradient updates and thus resemble the randomly initialized (unseen) tokens more (in addition to T-SNE likely being a better visualization tool).

Figure: Figure 12: Vanilla training learns to partition noun and adjective embeddings in the head of the distribution, and some structure in the tail. Active forgetting learns no separation between noun and adjective embeddings. Temporary forgetting learns structure in the head of the distribution and no structure in the tail of the distribution. Parameters used are $v=10000,\alpha=1.2,\varepsilon=0.10$.
Refer to caption: extracted/5703567/figures/pca_head_tail.png

### A.11 Other Random Distribution Generalization

Note that while we define structural in-context learning as free from reliance on any encoded semantic information, it is important to note that this does not mean that structural in-context learning assumes no geometry of the space. In fact, this would be practically impossible to achieve because connectionist networks function in a geometric space and take advantage of orthogonality, translation, scaling, etc. If we cannot make assumptions about the distribution from which the data is sampled, then we deprive our networks of their toolbox. Still, we test on random sampling distributions for the embeddings other than our initialization distribution. Namely, we test on a uniform distribution from 0 to 1 and a large normal distribution with mean of 5 and standard deviation of 5.

Figure: Figure 13: Vanilla training fails on all random tokens, whereas active/temporary forgetting succeed on the random distribution of initialization. Active and stop forgetting do not generalize to arbitrary random distributions, although show some generalization to normal distributions with large means and variances.
Refer to caption: extracted/5703567/figures/other_random_dist.png

### A.12 Required Compute for Experiments

We employed compute resources at a large academic institution. We scheduled jobs with SLURM. For our naturalistic experiments, each MultiBERT seed required 24 separate runs (one per tested checkpoint at a particular timestep), which totaled $\approx$ 100 hours on an RTX A5000 with 24 GB of GPU memory. Over 3 seeds, this was $\approx$ 300 hours of GPU usage.
For our synthetic setting, the vanilla training required 64 separate runs (one per hyperparameter combination of vocab size, ambiguity, and sampling distribution), which totaled $\approx$ 250 hours of RTX A5000 usage. Likewise, our active forgetting and temporary forgetting interventions took a similar amount of GPU usage. Therefore, in total, our GPU usage for all synthetic experiments summed up to about 750 hours.
We ran experiments mostly in parallel with SLURM to iterate quickly. Compute was a significant limitation for the development time and informed our development of training interventions in a synthetic setting. In total, our GPU usage was significantly higher than the reported number due to various failed/modified experiments. The total compute likely was around 20,000 GPU-hours on RTX A5000s, although this is a rough estimate.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses.

Minor note: Line 317-8: it seems the sentence is missing a verb
soundness: 2
strengths: - The paper studies a question of substantial interest, nicely continuing prior work on different kinds of learning
- considers both real-world LM and toy models
- Provides novel insights by distinguishing "structural" from "conditional" ICL
- Provides method for mitigating the loss of structural ICL in a toy setup
summary: This paper studies the emergence and disappearance of in-context vs in-weights learning in masked language models; in particular, a kind of ICL termed “structural ICL”, which refers to the ability to infer (and then use) information about the structural role (e.g., POS) of words from the context. In experiments, this ability appears and then disappears over the course of MLM training. A recently proposed “active forgetting” approach helps keep the ability, leading to models that use both in-context and in-weights learning.
weaknesses: - Structural ICL is operationalized using performance on random token embeddings. However, as the paper also demonstrates (Figure 18), such embeddings are drastically OOD for a trained model. Wouldn't a fairer test use novel embeddings that match the distributional properties of the embeddings of the trained model?
- Results on Structural ICL in MultiBERT (Section 3) are based on POS = noun vs adjective as the only property. It remains unclear how robust findings are to other properties, such as other pairs of POS, or other properties, such as grammatical number.
- Results on Structural ICL in MultiBERT (Section 3) are based only on probing accuracy, not behavior of the model. This seems suboptimal given that in-context learning as an emergent property of LLMs is generally thought of as an ability appearing in next-token prediction (e.g., complete a prompt with an appropriate label), One could test for ICL in properties such as POS behaviorally by creating contrastive examples where the key word has different POS and comparing model probabilities.
- The term “Dual Process” here is used to refer to models using both in-weights and in-context learning, and the paper refers to Kahneman 2011; Miller 2000. The link to dual process theory appears tenuous: dual process theory refers to thse use of implicit, unconscious vs explicit, conscious processes; the link to in-weights vs in-context inference is at best hand-waved in line 90-91, but no explicit justification for the link is given. The choice of the term and the link to dual process theory thus appears a bit of a stretch to apply the in this context.
- It remains unclear if the proposed strategy for keeping structural ICL abilities would transfer to real-world LMs. While training an LLM is understandably out of scope, even a modest LM could provide substantial insight here.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. What are the limitations of temporary forgetting? Is it strictly beneficial, or did the authors find that some model abilities deteriorated? Given that evaluation wasn’t performed on downstream tasks, what synthetic experiments can be conducted to evaluate the trade-offs of temporary forgetting?
2. Suggestion: Since no evaluation was performed on downstream tasks, this paper could be improved by expanding on why studying undertrained tokens is important (beyond what is already discussed in the introduction). I am familiar with this area, so the introduction was adequate for me, but adding such a section might make this paper more broadly accessible.
soundness: 3
strengths: 1. I appreciated that this work distinguishes structural ICL from ICL in general. This seems like a natural and interesting extension to prior work that studies in-weights versus in-context learning.
2. Despite the lack of experiments on downstream tasks, I found the breadth of the experiments satisfactory. I also recognize the creativity employed by the authors given their compute constraints.
summary: This paper examines "structural" in-context learning (ICL), a specific type of ICL where some tokens in the model’s prompt are randomly initialized. The motivation for studying structural ICL is based on prior observations that ICL behaves unexpectedly when prompts include tokens that are rarely encountered during training, i.e., undertrained tokens. This paper uses synthetic tasks to demonstrate that structural ICL develops early in training but diminishes as training progresses. The authors show that periodically reinitializing token embeddings during training (active forgetting) preserves structural ICL abilities, albeit at the expense of memorizing information in the model’s embeddings. The paper further demonstrates that both structural ICL and memorization are maintained if active forgetting is employed, but only for the first N steps of training. This procedure is referred to as “temporary forgetting.”
weaknesses: 1. No experiments on downstream tasks. While the motivation to address the issue of undertrained tokens is interesting, it is difficult to assess how training with temporary forgetting will affect language models in real-world applications. Given the author’s limited computational resources, they might consider a variant of temporary forgetting that can be employed as a fine-tuning step, e.g., fine-tune with active forgetting for $k$ steps before resetting the token embeddings to their original states, leaving only the model parameters changed.
2. Limited technical novelty. Unless I misunderstood the paper, temporary forgetting is simply active forgetting applied for $K$ steps. This is only a minor weakness in my view and is excusable if the authors could find a way to perform experiments on downstream tasks.
3. The presentation can be improved. For example, there are instances where the figure legends contain terms that are not defined. In Figure 6 (right), “Vanilla Forgetting” is mentioned. Figure 11 uses the term “Stop Forgetting,” which I believe is intended to mean “Temporary Forgetting.”

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 4
questions: **Q1**

To what extent does active forgetting impair other model capabilities, if at all? What about temporary forgetting? I'd be interested to hear hypotheses, and particularly any experimental results you have. 

**Q2**

Do you believe the problem of glitch tokens will recur in future, and in such a case, do you think preservation of structural ICL would be a useful strategy?

**Q3**

Can you give an example (or ideally a few examples) of cases where models might have a hard time performing a natural task because they are incapable of performing structural ICL on a rare token? 

---

I have provisionally put down a marginal acceptance, but would be happy to revise my score upwards if these questions are appropriately addressed.
soundness: 4
strengths: - The central thesis is clear
- The contributions are well-grounded in prior literature
- The distinction between structural and conditional ICL is compelling
- The experimental results support the thesis well, and are quite extensive, especially on the dynamics of structural versus conditional ICL
summary: The paper introduces a distinction between *structural* and *conditional* in-context learning, where structural ICL can generalise to novel tokens because it does not depend on token embeddings. They show that by default, structural ICL emerges and then sharply diminishes during pretraining, and they introduce a novel pretraining method based on periodically reinitialising the embedding matrix, which prevents the loss of structural ICL.
weaknesses: I feel there are two main weaknesses.

Firstly, it is not clear to me how important structural ICL actually is as an ability for models
  - The connection to glitch tokens is fair, but I don't expect these to be very common, and even when they do occur I'm not sure that preserving structural ICL is the best response - for instance, it wouldn't have fixed all the canonical solidgoldmagikarp problems, because structural ICL only preserves syntactic knowledge of glitch tokens, but fundamentally cannot give semantic knowledge of unknown tokens
  - The application to very rare tokens is likewise fair but by definition not very common, and again it is unclear to me whether in these cases there are any important capabilities which are purely syntactic

Secondly, it is not clear to me how far active/temporary forgetting impairs other abilities in models
- My intuition is that repeatedly reinitialising the embedding matrix is encouraging structural ICL by damaging other kinds of learning
- While the temporary forgetting approach might ameliorate this, it is not clear to me by how much or in what way it does this


Overall, I find much of the paper compelling in its presentation and exploration of structural ICL as a phenomenon, but I would be even more enthusiastic about it if it were clearer to me:
- that structural ICL is an important property (which could be demonstrated by natural examples of its relevance)
- what the side-effects are of the proposed methods for preserving structural ICL

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: See weakness
soundness: 2
strengths: 1.  This paper investigates an interesting research question: comparing ICL with parameter-tuning. Some explorations are helpful and provide useful insights. 
2.  Besides, the idea of combining ICL with structures is inspiring. Most previous ICL works run experiments on the sentence classification datasets, which are highly semantic-related. The discussion about sentence structures instead of prompt structures are limited.
summary: This paper studies the limitations and corresponding solutions for in-context learning with unseen tokens (i.e., structural ICL). Under this scenario, language models (LMs) learn the sentence structure (i.e., simple syntax structure) instead of token semantic meanings. Thus, with carefully designed forgetting strategies, the authors show that they can let LMs perform well under the structural ICL setting.
weaknesses: 1. Some statements about ICL explanation are not up-to-date. The authors mentioned that typical ICL algorithms do not have stable learning manner and named it as conditional ICL. However, some recent works [1,2] have shown that ICL learns the task composition: the model only learns how to compose the task based on the learned tasks from pretraining, instead of learning from scratch based on the ICL examples. Thus, some claims in paragraph 2 of the introduction should be updated.

2. Experiment settings are a bit problematic. This work is based on previous findings. However, these findings are not general across all settings. For example, Singh et al., 2023 find that ICL slowly dissipates as models are overtrained ONLY for decoder-based LMs without pretraining. In their limitation section, they also mentioned that the conclusion may not hold for pretrained LMs (e.g., Multi-BERT here) under other scenarios. I can’t list all of them, but in general, what I mean is that we should be careful about the assumptions of previous work, and cannot simply generalize and rely on previous conclusions.

    Second, since this paper uses encoder-based LMs, it may not be able to perform ICL as there is little discussion about it. Besides, ICL ability is often used for large models. Implementing experiments on Multi-BERT with probing accuracy instead of generation is not convincing to show the conclusions of this paper is useful in practice.

3. The sentence structure is learned from pretraining instead of predefined. Thus, the model may not treat sentences as human do (i.e., with linguistic structures). Even if for the simple POS triple structure it doesn’t matter, other more general or complex structures may not work.

4. The potential impact of this work may be a bit limited. The motivation of injecting novel tokens under the ICL setting is not clear to me. As aforementioned, ICL is often used for large models, where their tokenizer vocabulary is large enough and it’s not necessary to add extra new tokens (e.g., LLaMA 3-8B). While for small models that may require to extend the tokenizer (frequently), it may not perform ICL well and they are not expected to be used under this scenario. 

[1] Hahn M,; Goyal N. A theory of emergent in-context learning as implicit structure induction[J]. arXiv preprint arXiv:2303.07971, 2023.

[2] Li, J.; Hou, Y.; Sachan, M.; and Cotterell, R. What Do Language Models Learn in Context? The Structured Task Hypothesis. ACL 2024

Meta-review: The authors continue the thread of existing work on ICL and token-sensitivity to define "structural in-context learning". They show that this has a different pattern of emergence than general ICL and suggest a pre-training method to improve it. Issues raised about applicability to different models and real-world relevance are discussed.

The paper is relevant and the experiments solid if not revolutionary, recommend accept as poster.

Decision: Accept (Poster)