Injecting structural hints: Using language models to study inductive biases in language learning
Authors: Isabel Papadimitriou
Dan Jurafsky
Venue: EMNLP 2023 Findings (2023)
Keywords: transfer learning
pretraining
recursion
context-sensitivity
TL;DR: We use transfer learning from formal languages to natural language in order to understand the effect of the inductive biases of recursion, crossing links, and Zipfian vocabulary on a language learner.

Abstract: Both humans and transformer language models are able to learn language without explicit structural supervision. What cognitive inductive biases make this learning possible? Here, we examine the effect of different inductive learning biases by actively controlling the inductive biases of artificial learners: we structurally bias models by pretraining on synthetic formally-structured data, and evaluate these structural biases by fine-tuning on three typologically-distant human languages: English, Japanese, and Basque. We investigate the effect on downstream language perplexity of three types of inductive bias: 1) recursive, hierarchical processing 2) unrestricted token-token dependencies that can't be modeled by context-free grammars, and 3) a Zipfian power-law vocabulary distribution. We show that complex, non-context-free interactions between tokens form the best inductive biases. Our study leverages the capabilities of transformer models to run controlled language learning experiments that are not possible to run on humans, and surfaces hypotheses about the structures that facilitate language learning in both humans and machines.

Injecting structural hints:
Using language models to study inductive biases in language learning
Isabel Papadimitriou and Dan Jurafsky
Computer Science Department Stanford University edu
Abstract
Pretraining: Inject structural inductive blas
Structurally- biased learner
Untrained GPT-2 model
Both humans and large language models are able learn language without explicit struc- tural supervision. What inductive biases make this learning possible? We address this funda- mental cognitive question by leveraging trans- former language models: we inject inductive bias into language models by pretraining on formally-structun data, and then evaluate the biased learners' ability to learn typologically- diverse natural languages. Our experimental setup creates a testbed for hypotheses about in- ductive bias in human language learning. investigate the effect of injecting models with three types of inductive bias: 1) recursive, hi- erarchical processing, 2) crossing token-token relationships that can't 't be modeled by context- free grammars, and 3) a Zipfian power-law vocabulary distribution. We show that context-free relationships form the best induc- tive biases. Our study leverages the capabil- ities of transformer models language learning experiments that are not pos- sible to run on humans, and surfaces hypothe- ses about the structures that facilitate language learning in both humans and machines.
Formal language
220
Wikitext
225 64 225
Structural Principle: Recursion, or Context sennxttitisiiiittty.. or
Metric: perplexity
after fine-tuning
Figure 1: Our method: we take GPT-2-sized model and pretrain it with formal language corpus (see Fig- ure for examples). We then take these pretrained models and fine-tune them Wikipedia data asses each formal structure as an inductive bias for learning English, Japanese, and Basque.
of context-sensitive dependencies, and bias for power-law Zipfian vocabulary distribution. We inject untrained transformer language models with each structural bias by pretraining on synthetic structured data and then evaluating language mod- cling fine-tuning on human languages (English, Japanese, and Basque). Our inquiry is structured around three experimental questions:
controlled
1 Introduction
Natural languages are complex and structured sys- tems which humans learn without direct structural supervision. It is fundamental and open prob- lem in linguistics and cognitive science to under- stand the inductive biases that make such learning possible: what structural predispositions does a successful language learner need to start with? In this work, we shed light on this cognitive problem using artificial learners. Using our method of struc- tural injection, we causally intervene in transformer language models and manipulate their structural in- ductive biases, before training natural language.
Experiment How does an inductive bias for recursion compare to an inductive bias for context-sensitive crossing relationships? (Section 5)
Experiment Is a bias for pure constituent recursion better when mixed with a small amount of tokens that break context-free con- stituency structure? (Section 6)
Experiment 3: Does learner biased wards learning power-law Zipfian vocab- ulary distribution learn language more effec- tively? (Section 7)
We predispose transformers with three struc- tural biases from the cognitive literature: a bias for recursive processing, a bias for keeping track

---

mer
d'chind
Hans
es huus
lönd let
hälfe aastriiche help
The lawyer that the man that the dog bit hired was disbarred
we the children
Hans
the house
paint
(a) Center embedding English
(b) Cross-serial dependencies in Swiss German, example from Shieber (1985); "We let the children help Hans paint the house"
"I voted for him even though am negatively affected by his redistrihution policies" he said
(c) Crossing discourse anaphora links in English
Figure 2: Examples of recursive and comlext-sensitive structures in natural language.
In Experiment 1, we disentangle the effects of re- cursion and context-sensitiiity: both occur in lan- guage, but which one more useful as a sole learn- ing bias? Language is characterized by recursive structures in context-free constituent relationships like those in Figure 2a, and some linguistic theories posit that recursive processing is crucial (and per- haps the only) inductive bias that makes human lan- guage learning possible (Hauser al., 2002; Chom- sky, 1995). However, is widely hypothesized that human language is mildly context sensitive: while there is recursive structure, there are also non- context-free relationships between tokens, both in syntactic structure (Figure 2h) and in the structure of meaning and discourse relationships (Figure 2c) (Joshi al., 1990; Stabler, 2010; Shieber, 1985; Steedman, 1990; Frank and Hunter, 2021; Joshi, 1985). We compare the two principles of recursion and non-context- inductive bias for non-context-free crossing depen- dencies is better for downstream language learning, However, both inductive biases greatly outperform random and regular language controls.
guage is structured not only how tokens relate, but also in the structure of the distribution that tokens are drawn from, a cognitive bias that es- pecially significant for memory-based theories of grammar (Shufaniya and Arnon, 2022; Piantadosi, 2014; Ellis and O' O'Donnell, 2012). We show that Zipfian pretraining bias makes models better downstream language learning even when there no correspondence between the pretraining and fine-tuning vocabularies.
While our experiments work with computational models, the question that we are examining is about humans: what are the possible inductive biases that make human language learning possible without ex- plicit structural supervision? Using computational models, we investigate this question through empirically manipulating the inductive bias of a language learner - a causal experimental route that is not possible when working with humans. Results from such experiments can act as firstly as proofs of concept, showing how language learning is or isn't possible with different inductive biases. Secondly, such experiments help with hypothesis generation: with structural injection, can test arbitrary inductive biases in theory-independent way (see Baroni, 2022; Portelance, 2022; Lappin, 2021; Wilcox al., 2023; Kauf et al., 2023, for further discussions on the role neural NLP mod- els cognitive science). Section we discuss how our method of leveraging artificial learners to causally investigate inductive biases adds new direction to the rich prior literature inductive bias in neural learners.
relationships, and find that an
In Experiment 2, we combine recursion and context-sensitivit and ask: is recursive induc- tive bias better with slight sontext-sensitivvtty? We show that there is a significant improvement in downstream language learning if we add just 1 1% of a bias for crossing dependencies, breaking the constituent structure of the other 99% recursive bias we give the learner. Even when learners are mostly biased towards recursion, they learn lan- guage faster when their bias includes constituent- breaking context-sensitive structures.
In summary, our findings indicate that biases for complex token-token interactions, whether these involve recursion or not, form a powerful inductive bias for learning natural language. Crucially, our results are compatible with and can inform a wide
In Experiment 3, we test the effect of induc- tive biases in vocabulary distribution: does a bias towards a human-like Zipfian vocabulary dis- tribution (Zipf, 1936) help language learning? Lan-

---

variety of cognitive architectures: models in which structural inductive biases in humans arise from prior statistical learning (Elman, 1996), models in which they arise from other aspects of cognition or communication (Lieven and Tomasello, 2008; Hahn et al., 2020; Gibson et al., 2019), and models in which they are presumed to be innate (Hauser et al., 2002).
states that the ability for such constituent recursion crucial linguistic inductive bias, constitutes the underlying structural bias for the faculty of lan- guage, and that recursion is what distinguishes lan- guage from animal communication (Hauser et al., 2002; Chomsky, 1995).
2.2 Context-sensitivity
Although context-free grammars allow recursive structure. they are not complex enough to model many attested linguistic effects, which require non-context-free (i.e., at least context-sensitive) grammar (Joshi et al., 1990; Stabler, 2010; Shieber, 1985; Steedman, 1990; Frank and Hunter, 2021; Joshi, 1985). For example, Shieber (1985) proves that a grammar modeling the unbounded cross- serial dependency structures possible in Swiss Ger- man (Fig. 2b) must be non-context-free, and Steed- man (1990) shows that gapping effects (sentences like "Harry eats beans, and Fred potatoes") simi- larly cannot be analyzed by context-free grammars. Looking beyond syntax, reference and discourse structures (Fig. 2c) as well as meaning-based inter- actions do not abide by context-free restrictions.
2
Background: Structural inductive bias in humans
In order to use our experiments as a window into hypotheses about human inductive biases, we look at three families of structural bias from the linguis- ties and cognitive science literature.
2.1 Recursion
One very prominent hypothesis for linguistic in- ductive bias focuses on recursion: the ability for hierarchically-sttrcturrdd constituents to contain other constituents of the same type, and as such allow for potentially infinitely deep hierarchical structure.
Recursive structures can be described in terms of context-free language: a grammar with rules of the form A + 3, where a non-terminal node A consists of a string 3 that can contain both terminal and non-terminal nodes. Such a phrase-structure grammar is recursive when non-terminal can con- tain string that includes another non-terminal of the same type. For example, one rule describing the language of well-nested parentheses is
While it is broadly hypothesized that both recur- sion and con-context-freedt freeness exist in natural lan- guage, our experiments empirically disentangle the two hypotheses. We compare two inductive biases: an inductive bias for recursive processing, and an inductive bias for non-context-fres structures that do not have recursion. Non-context-free structures allow for unbounded and crossing interactions be- tween tokens, while recursive grammars only allow tokens to influence each other in specific subtree relationships. To disentangle these biases, we test the effect of an inductive bias where there very minimal (and nnn-tree-structureed restrictions where related tokens can appear (see Section 4.2 and Fig. 3b for details of our formalization).
S (S)
where any well-formed sentence can make an- other well-formed sentence S it inserted into a pair of parentheses. Rules of this type allow for infinite nesting.
canonical linguistic example of recursion is center embedding. In the center embedding exam- ple in Figure 2a, noun phrase ('the man that the part of another noun-
2.3 Vocabulary distribution
We next investigate structural biases in the lexicon: how a cognitive bias towards vocabulary distribu- tions (i.e., which tokens are more likely to appear in a corpus of language) affects learning. We aim to answer: does a bias towards a Zipfian vocabu- lary distribution (Shufaniya and Arnon, 2022; Pi- antadosi, 2014) act structural bias that aids language learning?
dog bit") can be used phrase ("the lawyer that [the man that the dog bit] hired"), and this can carry on recursively. Such self- embedding structures, which are attested in human language, are possible with recursive grammars but not in finite-state languages (Chomsky, 1959). The recursion hypothesis for linguistic inductive bias
A feature pervasive across human languages is the unbalanced nature of vocabulary distributions. In human languages, some words are very common
'Code and instructions for running our ex- periments is at https://githur http:///github..cm/rrozz//// injecting-structte waw.uctturalhinss....

---

5
1.
34
54,
225,
225,
1)
248
103,
123,
123,
103,
248,
1. 54 225; 1, 54, 225) 248; 248, 123, 103; 128, 103,
(a) The recursive NEST language. Every S node represents well-formed NEST substring.
(b) The context-sensitive CROSS language. matching parentheses.
Edges connect
499 472 300 345 272 309 17 15 329 233 9 267 122
499 472 300 345 272 499 472 300 345 272 309 17 15
(c) The random language, RAND
(d) The regular REP language, with repetition length & = 5. The repeated block is circled. For our experiments, we use k = 10.
pretrain models to give them different structural inductive biases. For simplicity, we represent NEST and CROSS as 250 open tokens with 250 close tokens, while we present RAND and REP with 500 tokens. The vocabulary distributions are same between languages we concatenate the open and close tokens: token 1 in the parentheses languages is equivalent to token 251 in RAND and REP.
Figure 3: The formal languages
(like "the" or "a" in English) and most words are used very rarely, roughly following plwer-law distribution. sipf"s law states that the rth most common word has frequency proportional to
ciple. After sufficient pretraining on such a cor- pus, we have a transformer model which has never seen human language, but has successfully learnt to model a type of structure. We can then train this model on natural language data, taking its pretrain- ing to be a structural inductive bias. We fine-tune each model three different languages (English, Japanese, and Basque). The final performance on each language, measured in perplexity, an indica- tor how learnable the raw language data is when a learner starts with the specific inductive bias.
(1)
(r + 3)a
with a R 1 and 3 ~ 2.7 providing a good em- pirical estimate for human languages (Zipf, 1936; Mandelbrot, 1953). The literature on explaining and deriving this empirical fact is rich and varied (see Piantadosi, 2014, for a thorough review), while experimental and theoretical works examine the learnability of Zipfian-distributee data (Ellis and O' Donnelll 2012; Lavi-Rotbain and Amnon, 2022; Schuler et al., 2017; Chan et al., 2022). In our work, we examine the effect of Zipfian inductive on learning language (Experiment 3).
3.1 Implementation Details
For each experiment, we randomly initialize a GPT- 2-small model with a max sequence length of 512, and train on one formal language. The specific formal languages we use are described in Section 4. We use a batch size of 512 to train all models, and train for 5,000 steps with 1,000 steps of warmup. Each formal language corpus consists of billion tokens, which comes out to 3,81 14 batches. There- fore, in training 5,000 steps the model sees each corpus approximately one and half times.
3 Method Overview: Pretraining as structural inductive bias
We set up experiments where we control the induc- tive bias of a language model learner, and examine how inductive biases influence language learning. Our experiments consist of two steps: 1) pretrain- ing a GPT-2-sized model on formal language to inject structural bias and 2) fine-tuning on lim- prior
We fine-tune each pretrained model on three sep- arate languages to measure final test perplexity. We use the the wikitext- 103 English dataset (Merity et al., 2017, 103 million tokens), Papadimitriou and Jurafsky (2020)'s Japanese Wikipedia (129 million tokens) and Basque Wikipedia (63 million tokens) corpora. We fine-tune with two epochs on the En-
ited natural language data (drawing methods in Papadimitriou and Jurafsky (2020), Figure for depiction). To instill an inductive bias in an artificial language learner, we pretrain untrained transformer language model on synthetic data which exhibits a formal structural prin-
We use GPT-2-small to refer to the model config accessed using Hugging Face AutoConfig with key gpt2: 12 trans- former layers with 12 attention heads per layer, and a hidden size of 768

---

glish and Japanese datasets, and 4 epochs on the Basque datasets. Though the three corpora are not controlled for size and train-test difficulty, we do not compare results in order to analyze differences between languages and draw conclusions from this. We instead use the three languages to verify that any claims about language learning and inductive bias that we make are likely cross-linguistic and not English-specifie.
will never be any crossing ares in how we connect parentheses. The nested parenthesis grammar is context-free. An example of this language shown in Fig. 3a.
4.2 Non-context-free: the crossing parentheses language CROSS
The Crossing Parentheses language CROSS is a parentheses language with the same vocabulary of open and close tokens the NEST language, but where parentheses do not have be well-nested, but just have to be balanced: opened parentheses must close. As such, pairs of parentheses can in- terleave and cross. Whereas the NEST language imposes strict limitations to how different tokens can interact (a parenthesis can only close in the space between its parent opening and closing, there is no ability to see past the top of the stack), in the CROSS token can have its pair in any lo- cation. In order to control the CROSS language to be like the NEST language as much as possible (ex- cept for the dependent variable of non-context-free crossing) we add another structural constraint: the distribution of distances between open and close tokens is matched to the empirical dependency dis- tance distribution of the NEST language. This way, any difference in the inductive bias is not due to the lengths of dependencies, but due to the structural biases in the pretraining data. This dependency link distribution is what gives the CROSS language struc- tural information for a language model to learn: for every opened parenthesis, there are positions when it is more likely to be closing than others, and modeling this language involves modeling those probabilities.
Our pretraining languages all have vocabulary size of 500, which has no correspondences with the 50,257-size vocabulary of GPT-2. To accomodate this new tokenizer, we have to initialize new word embedding matrix, with 50,257 rows for the model to learn in fine- tine-tuning We initialize the embedding matrix by randomly sampling with replacement from the rows of the old embedding matrix with 500 rows, following Wu et al. (2023), who show that initializing an embedding matrix for transfer learning by sampling from the old embedding ma- trix leads to far better transfer learning than random reinitialization even for unrelated vocabularies (see also Hewitt, 2021). To account for the effeet of hav- ing to relearn the vocabulary, also add a control case where take the pretrained GPT-2 model and randomly resample the rows of the embedding matrix. To re-learn the embedding matrix, we fine- tune on the natural language corpora. This control appears on the right of our results graphs.
4 Pretraining languages
Our experimental method rests on pretraining lan- guage models on well-chosen formal languages. Here, we describe the four families of formal lan- guages that we use for pretraining, and present examples of each of the languages in Fig. 3.
The crossing parentheses language is not context free. Though every NEST string is legitimate (but exponentially unlikely) CROSS string, the CROSS language can include other strings, like arbitrarily long cross-serial dependency structures that are known to be non-context-free (Shieber, 1985);
4.1 Recursive and context free: the nested parentheses language NEST
The first language that we pretrain on a language of matching nested parentheses, also known as the Dyck language. The vocabulary of this language consists of a set of matched open and closed tokens, and vocabulary size can be set to any even number (for our experiments, we used a total vocabulary size of 500 for all pretraining language). Writing the language from left to right, at each step we randomly pick between two choices: either 1) open a parenthesis (p = 0.49) or 2) close the last opened parenthesis (p = 0.51). Since we only ever close the most recently-opened parenthesis token, there
(1 (2 (a (N ) )2 )a ... >>
In the NEST language, arbitrarily deep nesting (and arbitrarily long dependencies) are in the limit pos- sible, though in practice unlikely, and similarly in the CROSS language arbitrarily long dependencies are possible but unlikely. This roughly anal- ogous to human language, where processes like center embedding and cross-serial dependencies can in theory happen infinitely but in practice are

---

Japanese fine-tuning
English fine-tuning
120
Perplexity on Japanese
60
Perplexity on English (lowerr better)
(lower is better)
40-
20-
Random
Rendom. Repeal
Nesting Farons
Grossed Depenteencess
Control: Premained GP1-2
Rendom
Random.
Nesting
Crossed Dependencies
Control
Repeat every
every 10
Pretrained Model
Pretrained Model
Basque fine-tuning
Perplexity on Basque
300
[lower is belter)
0-
Rendom
Rendom,
Nesting
Crossed
Control:
Ropisat
Parans
Dependencies
Pretrained Model
Figure 4: Results for Experiment 1, CROSS is better inductive bias than NEST Each model pretrained with the formal language on the x-axis is evaluated on a wikipedia test set after natural language fine-tuning, Error bars represent 95% confidence intervals over fine-tuning compare perplexities between different test sets, we can only compare the ranking of the test conditions. The rank (CROSS better than NEST is better than REP is better than RAND) is consistent across English, Japanese, and Basque.
with different random seeds. Since cannot directly
limited. This probabilistic weighting doesn't af- feet our proof of non con-context-rremmsss the Shicber (1985) proof is independent of the probabilities of these constructions.
two tokens. This way, we can better measure the utility of more complex structural inductive biases. Note that REP is not the same as the 'copy language" (the set of arbitrarily long strings followed by their repetition), which is famously context-sensitive. In the case of REP, we're restricting any copied chunk to be exactly in length, which makes the language regular, indeed subregular.
4.3 Baselines: the Random language and the regular Repetition language
The Random language RAND For our RAND baseline, we take the vocabulary of our experimen- tal languages NEST and CROSS, and create a dataset by sampling each token randomly from the vocabu- lary distribution without any structural limitations.
Experiment 1: Disentangling recursive and context-sensitive inductive biases
The Repetition language REP We also test slightly stronger baseline than RAND, the repetition baseline. In the REP language, tokens are placed randomly, like in RAND, but every 10 random to- kens are then immediately repeated. Using the repetition language as a baseline lets us control for the effect of there being any structure that connects
For our first experiment, we compare the fine- tuning perplexity of models pretrained the NEST language and the CROSS language each other and to random and regular controls. Our fine-tuning results are shown in Fig. 4. From the fine-tuning perplexity that we get with different structural in- ductive biases, we can present two findings:

---

Japanese fine-tuning
English fine-tuning
Perplexity on Japanese (lower is better)
Perplexity on English
60-
(lower is beiter)
30+
20-
10
Nested
Mildd
Mioed, 10N Cross
Crossing Dependencess
Nested
Mixed,
Crossing
146 Cross
Cross
10% Cross
Pretrained Model
Pretrained Model
Basque fine-tuning
Perplexity on Basque (lower better)
50
25*
0-
Nestod
Mixad, Cross
Mixed, Pretrained Model
Crassing Decendencies
Figure 5: Results for Experiment 2, even small amounts of non-context-free structures in the inductive bias cause significant improvement in downstream perplexity. Mixing 1% and 10% of the CROSS language with the NEST language, which breaks the recursive constituency structure of NEST, causes significant downstream language learning improvements over plain NEST Error bars represent 95% confidence intervals over fine-tuning runs with different random seeds
Any structure has a significant effect as an in- ductive bias All of the models with structural pretraining language fare significantly better than the model with the random, unstructured pretrain- ing language. This especially surprising in the case of the REP language, which has very simple structure that similar to RAND. However, our experiments also show that the shallow structure of REP is worse than both of our languages with more complex structural organization. Comparing the performance of the NEST and the CROSS language leads to next finding:
CROSS language is most helpful for, which is an especially fruitful question for future work since context-sensitive linking structures arise in syn- tax, semantics, and discourse. Understanding the role of different complex structures in influencing language learning an in-vitro paradigm such ours shines light on the properties of language as a learnable system under different cognitive struc- tural assumptions.
Experiment 2: Mixing context-fiee and non-context-free structures
Context sensitivity acts as a better inductive bias than recursion Our experiments disentangle the effects of recursive structure and non-context-free linking structure, and show that the non-context- free CROSS language acts as stronger inductive bias for language learning than the recursive NEST language. Our current methodology does not us- certain which aspects of language learning the
In Experiment 1, we saw that the non-context- free inductive bias of the CROSS language is more beneficial for downstream language learning than the constituent recursion of the NEST language. However, neither of these extremes encompass the more widely agreed nature of linguistic syntactic structure: likely a recursive grammar with mildly context-sensitive elements. To examine the

---

effects of non-context-free elements in otherwise context-free grammars, create the NEST-MIX-P languages. These languages follow the NEST grammar, except that with probability p% an 'open' token does not follow the NEST grammar and instead follows the CROSS grammar. This means that we sample dependency distance and place
English
Japanese
Basque
100
(Lower is better)
75
Perplexity
200
400
50
200
25
"close" token for it without taking into account the constituent structure of the NEST language. The NEST example from Figure 3a, with one CROSS to- ken added and shown in green, would look like this:
01
Uniform Ziptian
Uniform Ziotian Pretrained Model
Uniform Ziotian
Figure 6: Results for Experiment 3, a Zipfian induc- tive bias improves downstream learning, even when the pretraining and fine-tuning vocabularies are unre- lated. We pretrain models with random tokens sampled either from uniform Zipfian distribution.
1 54, 64, 225, 225, 123, 1, 248, 103, 123, 103, 248) We test two such languages: NEST-MIX-I, with I % CROSS tokens, and WEST-MIX-1M 10, with 10% CROSS tokens. Our results are shown in Figure 5. Adding 1% CROSS tokens to NEST language significantly better inductive
experiments show that pretraining on random Zipfian corpus biases a model for better language learning than a uniform corpus, and that a struc- tural bias for vocabulary distribution is encoded beyond the word embedding matrix and more ab- stractly in network parameters. Due to the lack of any structure beyond vocabulary distribution in the random corpora, neither of the test cases lead to very good downstream language performance. Results around combining Zipfian vocabulary with other structural biases like those Experiment are mixed, and we do not have clear evidence that Zipfian vocabulary has a structural effect that can add on the effect of structure in data. We discuss these results in more detail in Appendix A.
causes it to bias, with an average improvement perplexity points, while adding 10% CROSS tokens causes an average downstream improvement of perplexity points. Our results show that even small amounts of context-sensitive inductive bias have big effect on language learning, aligning with theoretical linguistics results about human language being mildly context sensitive.
7 Experiment 3: Zipfian structural bias
How does bias towards a particular vocabulary distribution affect language learner? We test the effect of a uniform vocabulary distribution versus a Zipfian vocabulary distribution a pretrained inductive bias. As shown in Figure 6, a Zipfian pre- training distribution random tokens predisposes a model with a better language-learning bias.
Related Work: Inductive bias
Our experiments use transformer models in order to create a controllable testbed for understanding human language learning: by manually varying the inductive bias through pretraining, the effect of different inductive biases in making learner fit for language learning. Our focus is different, though related, to the focus of much of the work about inductive bias in transformers. While we set the inductive bias of our learners through pretraining, the built-in structural induc- tive biases of untrained deep neural network archi- tectures unknown and difficult to assess (Ba- roni, 2022: Warstadt and Bowman. 2022). rich line of past experiments work towards defining this inductive bias of neural models through em- pirical methods, looking at how much transform- ers are biased towards learning hierarchical rules
Crucially, there is no connection or correspon- dence between the pretraining and fine-tuning vo- cabularies: the pretraining vocabulary is made up of 500 parentheses tokens (whose frequency rank- ing is randomly ordered order make Zipfian distribution the Zipfian case), whereas the GPT- 2 tokenizer's fine-tumin ane-ttnnng vocabulary over 50K tokens. No token correspondence of specific to- ken information is transferred between pretraining and fine-tuning the tokens are unrelated and we re-sample the rows of the embedding matrix.
test
Since there is no correspondence, our vocabu- lary distribution ablations measure the effect of an abstract bias towards one type of vocabulary distribution, rather than the effect knowing the specific distribution of tokens in a language. Our

---

(Kharitonov and Chaabouni, 2021; Petty and Frank, 2021; Mulligan al., 2021) or tree-structured pro- cessing (Murty et al., 2023). or biases towards dif- ferent typological linguistic structures (White and Cotterell, 2021; Ravfogel et al., 2019; Ravishankar and Nivre, 2022), and how inductive bias changes with language modeling pretraining (Mueller et al.. 2022; Warstadt et al., 2020).
influence (rather than analyzing properties and uni- versals of language) we can test hypotheses that are not dependent on linguistic theory-building We can therefore explore outside the well-studied hy- potheses in the linguistics literature.
Our experiments address questions about the learnability of human languages under different structural inductive biases which have been pro- posed in the linguisties and cognitive science liter- ature as bases for language. However, experiments such as ours cannot directly prove how the human language system structurally biased. What such experiments can achieve is to shine light on lan- guage as a learnable system as a whole, and pro- duce hypotheses about the relevant structures that could be guiding human language learning. In our case, find that recursion is indeed strong in- ductive bias for language learning. However, our experiments also serve to point out that it's not nesting constituent, structurally-recuriive proper- ties that necessurily most help language learning. Using the CROSS language, we show that when we take away recursion from the inductive bias, but keep the overall framework of paired tokens that connect in non-trivial ways, we get better inductive learning bias. Thus, our work serves to showcase possible alternatives the Hauser et al. (2002) hypothesis that recursion is necessary bias: other atructurally-compl
We build on these results by working to in- fluence, rather than measure, the inductive bias of transformers. Our experimental paradigm re- lies on structural transfer between pretraining and fine-tuning data in unrelated modalities, an effect demonstrated by past research. We showed in pre- vious work how the structures of non-linguistic modalities like music and code can predispose learners for language learning (Papadimitriou and Jurafsky, 2020), work that has been reproduced (Chiang and Lee, 2022; Ri and Tsuruoka, 2022), extended to other modalities like amino acid se- quences (Chiang and Lee, 2020), and between lan- guage and multiple symbolic tasks (Lu et al., 2022). In similar line of inquiry, Krishna et al. (2021) show structural transfer from the abstract task struc- ture of summarizing random data to summarizing real natural language passages. The paradigm of structural transfer lets us study the fundamental cognitive issue of inductive bias from an exciting angle: through causal experiments where we con- trol the inductive bias of language learners.
ways of relating tokens may be just as strong. Differences such as those be- tween recursion and non-recursive complex struc- tures are hard to disentangle theoretically. As such, an experimental paradigm such as ours is a helpful step in widely exploring the hypothesis space in questions around human language cognition.
9 Discussion
Our experiments and methodology provide new view into the biases that make language learn- ing possible in both human and artificial learn- ers. We use transformer language learners and perform causal interventions that alter their induc- tive learning biases before training them on natural languages. Our findings show the relative strengths of different structural inductive biases: simple, reg- ular structure is far outperformed by both context- free and non-context-ie structural relationships, but a recursive structure is not necessarily the best such complex bias.
Artificial models of language can never defini- tively prove facts about human language process- ing. The models pretrained and fine-tuned in this paper are products of opaque optimization pro- cesses, and the results that we get do not neces- sarily match human learning. Nevertheless, our work serves to identify and examine different hy- potheses about human inductive learning biases. We hope that such work can help enrich the hypoth- esis space over which theoretical argumentation, well as human subject experiments, are conducted, ultimately leading to a richer understanding of the human language learning process.
We obtain our results from experimenting on artificial language learners, rather than working with humans or analyzing human language. In- ductive bias experiments on artificial learners let us easily identify possible hypotheses of structural inductive biases in human cognition. More impor- tantly, since we are working with systems we can

---

Limitations
Methodologies such as ours, especially applied more narrow definitions of learning than language learning, could become harmful if they are taken to provide actionable proof about human learning patterns.
As discussed throughout the paper, our method uses language models to better understand language learning and cognitive inductive bias in humans, and this comes with all of the limitations of using computational schematic models of cognition to understand the complex and intractable problem of human cognition. Though strong artificial language learners give us a tool with which run control- lable language learning experiments, experiments on artificial learners do not provide any proof about the actual cognitive processes that happen in hu- mans, and our experiments cannot be taken to pro- vide any such proof. Our results serve to showcase the properties of language under different inductive bias constraints. and can inform hypotheses and theories that are worked out and evaluated on human language learning and human subjects.
Acknowledgements
We would like to thank Michael Hahn, Philip Resnik, John Hewitt, the members of the NYU CAP Lab, and the EMNLP anonymous reviewers for enlightening discussions and comments this paper, well as all the members of the Jurafsky lab for their consistent and inspiring research support. This research was funded in part by NSF award number IIS-2128145.
learnable system
References
Marco Baroni. 2022. On the proper role of linguisticallyy oriented deep net analysis in linguistic theorising. In Algebraic Structures in Natural Language, pages I-
A limitation of our experiments is that we only evaluate models on natural language perplexity. Perplexity is coarse-grained measure, and to ex- pand these epxeriments plan to further examine which specific parts of language production dif- ferent models are doing better in. Understanding which inductive biases lead to which acquisition
16. CRC Press.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. Data distribu- tional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems, volume 35, pages 18878- 18891. Curran Associates, Inc.
patterns, especially with respect to the acquisition of semantic and syntactic patterns, would be a very interesting addition to these findings.
Cheng-Han Chiang and Hung-yi Lec. 2020. Pre- training a language model without human language. arXiv preprint 2012: 11995.
Lastly, our experiments evaluate language learn- ing by fine-tuning on modeling wikitext, which is not similar to the learning environment of hu- man language. Our methodological contribution is independent from the actual data we use, and one way our method could be applied would be in more realistic language acquisition situations, us- ing datasets of child-directed speech for fine-tuning (Warstadt et al., 2023). Understanding the effects of inductive biases in scenarios closer to human language acquisition would be great next step for this paradigm.
Cheng- Han Chiang and Hung-yi Lec. 2022. On the transferability of pre-traines language mod- els: A study from artificial datasets. Proceedings of the AAAI Conference on Artificial Intelligence, 10118-0055181555
Noam Chomsky. 1959. On certain formal properties of grammars. Information and Control, 2(2):137-167.
Noam Chomsky. 1995. The minimalist program. MIT press.
Nick C Ellis and Matthew Brook O'Donnell. 2012. Sta- tistical construction learning: Does a zipfian problem space ensure robust language learning. In Rebuschat Patrick and Williams John N., editors. Statistical learning and language acquisition, number Vol. in Studies in Second and Foreign Language Educa- tion. De Gruyter Mouton.
Ethics Statement
Our experiments address cognitive question about language learning by running in-vitro experiments on language models. Such methodologics do not provide definitive proof about any human cognitive processes, Instead, such experiments use language models in the way that computational models of cognition are generally used: order to surface and experiment on interesting cognitive hypotheses.
Jeffrey L Elman. 1996. Rethinking innateness: A con- nectionist perspective development, volume 10. MIT press.
Variation in mild Robert Frank and Tim Hunter. 2021.
sensitivinnt Evolutionary Linguistic Theory.

---

Edward Gibson, Richard Futrell. Steven Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in cognitive sciences, 23(5):389- 407.
Benoit Mandelbrot. 1953, An informational theory of the statistical structure of language. Communication theory, 84:486-502.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In International Conference on Learning Repre- sentations.
Michael Hahn, Dan Jurafsky, and Richard Futrell, 2020. Universals of word order reflect optimization of gram- mars for efficient communication. Proceedings of the National Academy of Sciences, 17(5):2347-2353.
Aaron Mueller, Robert Frank, Tal Linzen, Luheng Wang. and Sebastian Schuster. 2022. Coloring the blank slate: Pre-training imparts hierarchical inductive bias to sequence-to-sequee -eseueence models. In Findings of the Association for Computational Linguisties: ACL 2022, pages 1352-1368.
Mare D. Hauser, Noam Chomsky, and W. Tecumsch Fitch. 2002. The faculty of language: What is it, who has it, and how did it evolve? Science, 298(5598): 1569-15999999999
John Hewitt. 2021. Initializing new word embeddings for pretrained language models.
Karl Mulligan, Robert Frank, and Tal Linzen. 2021. Structure here, bias there: Hierarchical generaliza- tion by jointly learning syntactic transformations. In Proceedings of the Society for Computation in Lin- guisties 2021, pages 125-135, Online. Association for Computational Linguisties.
Aravind K Joshi. 1985. Tree adjoining grammars: How much sontext-sensitivity is required to provide rea- sonable structural descriptions? Natural language parsing: Psychological, computational and theoreti- cal perspectives, pages 206-250.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. 2023. Characterizing intrin- sie compositionality in transformers with tree projec- tions. In The Eleventh International Conference on Learning Representations.
Aravind K Joshi, K Vijay Shanker, and David Weir. 1990. The convergence of mildly sensext-sensitive grammar formalisms. Technical Reports (CIS), page 539.
Isabel Papadimitriou and Dan Jurafsky. 2020. Learn- ing music helps you read: Using transfer to study linguistic structure in language models. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6829-6839.
Carina Kauf, Greta Tuckute, Roger Levy, Jacob An- dreas, and Evelina Fedorenko. 2023. Lexical se- mantic content, not syntactic structure, is the main contributor ann-brain similarity of fMRI responses in the language network. bioRxiv.
Eugene Kharitonov and Rahma Chaabouni. 2021. What they do when in doubt: a study of inductive biases in seq2seq learners. In International Conference on Learning Representations.
Jackson Petty and Robert Frank. 2021. Trans- formers generalize linearly. arXiv preprint arXiv:2109.12336
Steven T Piantadosi. 2014. Zipf's word frequency law in natural language: critical review and future di- rections. Psychonomic bulletin & review, 21:1112- 1130.
Kundan Krishna, Jeffrey Bigham, and Zachary Lip- ton. 2021. Does pretraining for summarization re- quire knowledge transfer? In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 3178-3189 Punta Cana, Dominican Republic. Association for Computational Linguistics.
Eva Portelance. 2022. Neural Network Approaches to the Study of Word Learning. Stanford University.
Shalom Lappin. 2021. Deep Learning and Linguistic Representation. Routledge.
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of NAACL-HLT, pages 3532 3542.
Ori Lavi- and Inbal Amonn 2022. The Iearnabl-- ity consequences of zipfian distributions language. Cogmition, 223: 105038.
Vinit Ravishankar and Joakim Nivre. 2022. The ef- fects of corpus choice and morphosyntax on multilin- gual space induction. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 41 4130-4139.
Elena Lieven and Michael Tomasello. 2008. Children's 's first language acquistion from a usage-based perspec- tive. In Handbook of cognitive linguistics and second language acquisition, pages 178-206. Routledge.
Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with artificial language: Studying transferable knowl- edge in language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 7302-
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mor- datch. 2022. Frozen pretrained transformers as uni- versal computation engines. Proceedings the AAAI Conference on Artificial Intelligence, 36(7):7628 7636.
7315.

---

Kathryn D. Schuler, Patricia Reeder, Elissa L. New- port, and Richard N. Aslin. 2017. The effect of zip- fian frequency variations category formation in adult artificial language learning. Language Learn- ing and Development, 1344)33573374
A Combining syntactic structure and vocabulary distribution
In Experiment 3 (Section 7), we showed that Zipfian vocabulary distribution provides stronger structural inductive bias than uniform distribution. Here, present results in combining a Zipfian dis- tribution with the grammatical structural biases in Experiment The results are shown in Figure 7, and do not definitively point one way or the other regarding combining grammatical structure and vo- cabulary distribution: Zipfian pretraining distri- bution creates a stronger bias in some cases and a weaker bias in others. More well-powered and con- trolled experiments, looking at carefully-chosen grammatical structure combine with vocabulary distribution, in order understand the interactions between these two aspects structural information.
Stuart M. Shieber. 1985. Evidence against the context- freeness of natural language. In Jack Kulas, James H. Fetzer, and Terry L. Rankin, editors, Philosophy, Lan- guage, and Artificial Intelligence: Resources for Pro- cessing Natural Language, pages 79-89. Springer Netherlands, Dordrecht.
Amir Shufaniya and Inbal Arnon. 2022. A Cognitive Bias for Zipfian Distributions? Uniform Distributions Become More Skewed via Cultural Transmission. Journal of Language Evolution, 7(1):59-80.
Edward Stabler. 2010. Computational perspectives on minimalism. In Cedric Boeckx, editor, Oxford Handbook of Linguistic Minimalism, pages Oxford University Press.
Mark Steedman. 1990.
Gapping constituent dination. Linguistics and Philosophy, 13:207-263.
Mueto Dissooue
Uniem
Doten
A. Warstadt and Samuel R. Bowman. 2022. What artificial neural networks can tell us about human language acquisition. In Shalom Lappin and Jean- Philippe Bernady, editors, Algebraic Structures in Natural Language. CRC Press.
Engi sh
Sasque
Jupancse
Perpleett (Lower is bester)
Alex Warstadt, Leshem Choshen, Aaron Mueller, Ad- ina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023. Call for papers- the babylm challenge: Sample- efficient pretraining on developmentally plausible corpus. arXiv preprint arXiv:2301.
10
Nertig
Cineting
Coonning Pretrained Model
Neoting
Keating
Crossing
Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. Learning which fea- tures matter: RoBERTa acquires a preference for linguistic generalizations (eventually). Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)), pages 217-235, Online. Association for Computational Lin- guistics.
Figure 7: Results for downstream perplexity in all three languages when combining vocabulary distribution bi- ases with ).
Jennifer White and Ryan Cotterell. 2021. Examining the inductive bias neural language models with ar- tificial languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1thh International Joint Conference on Natural Language Processing. volume 1, pages 463. Association for Computational Linguistics.
Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2023. Using Computational Models to Test Syntactic Leamability. Linguistic Inquiry, pages 1-44.
Zhengxuan Wu, Alex Tamkin, and Isabel Papadimitriou. 2023. Oolong: Investigating what makes transfer learning hard with controlled studies. In "Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)"".
George Kingsley Zipf. 1936. The Psychobiology Of Language. Routledge.


Reviews:
[{"review_id": "wWa1wVUMrU", "reviewer": "Reviewer_wiKh", "summary": "", "questions": "1. Could you please re-explain the control case from line 353 to me? Is the pretrained GPT-2 model the one pretrained by you (in which case… isn’t the control case the same as the actual experiments you are running?) or the “real” GPT-2? Or are the embeddings in the control case randomly sampled from e.g. a  normal distribution? I found this explanation quite confusing. (based on the graphs I think it’s the original GPT-2, but please explain nonetheless)\n2. The Wikitext test set is an iid test set, I am assuming. Recursion has been widely discussed as potentially allowing generalisation to more complex structures in compositional generalisation or simply generalisation to longer lengths. Have you performed any experiments using non-iid test sets like that? I’d be very curious about the effect of CROSS and NEST in a setup like that. After all, humans don’t learn language in an iid train-test setup.\n3. Could you elaborate on why you think CROSS outperforms the remaining languages?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for this review! It’s really thorough and your comments are very useful to think through, and encouraging about the direction of this work. Below we discuss some of the main points brought up:\n\n**COMMENT 1 No further downstream tasks are considered, have you performed any experiments using non-iid test sets?**\n\nWe definitely agree that perplexity is a coarse-grained measure, and for follow-up papers we are working on more fine-grained evaluations. But we believe that our inquiry into inductive bias needs to be done in two steps.  The first step, inthis paper, is to see what kinds of structures aid language learning, using perplexity on naturally occurring text. Our experiments in this step use a small (for language-modeling-scale) fine-tuning dataset to ensure that the learning differences come from the starting inductive bias. Our current paper showcases both expected and unlikely results on the importance of a multifaceted set of grammatical and distributional structures.\n\nThe second step, which we are doing in follow-up work, is to **study a broad set of phenomena to understand _where_ the improvements that we see in this first paper come from**. Looking at the multifaceted ways in which language is structured, we want to distinguish the effects that different inductive biases have in downstream linguistic features: syntactic structure, semantic and lexical cohesion, and discourse structure. Previous work in the field thinking about structure in language learning focuses largely on syntactic structure, but structure plays a key role in many linguistic levels (the lexicon, coreference, discourse coherence, etc) so   understanding the linguistic causes of  results requires a broader paradigm which we believe to be a suitable second paper. \n\nIn the current paper, we look at perplexity on naturally occurring language, and **we’ve also performed further experiments since submission to look at a simple syntactic phenomenon: agreement**, which we can add to the camera-ready. Performance on simple subject-verb agreement (where we just check if the models have learnt the grammatical system of agreeing verbs to the subject when there aren’t other nouns to confuse them) is very similar to our perplexity results, with CROSS > NEST > REP > RAND.\n\n\n| Model | Accuracy |\n|-----------|------------|\n| Random | 74.8 |\n| Regular | 75.5 |\n| Nest | 86.9 |\n| Cross | 90.0 |\n| Control: GPT resample embs | 79.3 |\n| Control: off-the-shelf GPT-2 | 95.8 |\n\nFollow-up work looking at complex linguistic abilities would require a slightly different paradigm:  we would need a much larger fine-tuning dataset in order to induce consistent performance on the finer points of grammaticality. With our current fine-tuning size, we ran some experiments testing more complex syntactic phenomena from SyntaxGym, and all models (including the off-the-shelf GPT-2 control with shuffled embeddings) perform close to chance, and so we cannot draw meaningful conclusions. The failure of our control case suggests that the fine-tuning dataset size is insufficient to test such subtleties, as the lexicon is not developed enough after wikitext-103 training to be able to showcase underlying structural subtleties in difficult natural environments. A follow-up paper with a much larger fine-tuning set will include experiments on more sophisticated grammatical, semantic, and discourse phenomena, and would also need ways to differentiate the structure inserted in pretraining from structure learned during fine-tuning,  and we believe this large set of experiments to be out of scope for this first paper. \n\n(we also wrote this to reviewer MCqz, who had a similar question)\n\n**COMMENT 2 testing the Zipfian distribution seems a little out of place**\n\nThanks for this comment, we will try to change the framing a bit to make the change seem less jarring.  Grammatical structure and vocabulary distribution are the two most widely-studied linguistic structural biases in the literature. We include the Zipf section and experiments in order to make our results more complete, and to deal with cognitive linguistic inductive biases broadly and at many levels. We believe that it’s important to investigate both of these levels of structural biases in order to truly investigate the possibility of what biases aid language learning, and doing so doesn’t take away from the more traditional grammatical structure experiments.  \n\n\n**COMMENT 3 Could you please re-explain the control case from line 353?**\n\nIt’s as you suggest, our baseline is taking the real GPT-2 from the huggingface hub, but re-sampling the embedding mataric. We use this model as a baseline because we can assume this model has close to the best possible inductive bias for learning English – it already knows English. We shuffle the rows of the embedding matrix of this model in order to get a baseline for how well we can do with a very good inductive bias but no lexical information, which gives us an upper bound for how much a synthetic bias (with no lexical relation to language) can help. This baseline showcases the importance of vocabulary – there is only so much that training on a small dataset can give a model since the vocabulary is sparsely updated. \nYou’re right, reading over our text it’s clear that this should have been phrased more clearly and we’ll take care to amend it.\n\n**COMMENT 4 Could you elaborate on why you think CROSS outperforms the remaining languages?**\n\nNEST-style recursion is a well-studied aspect of syntax, underlying many grammatical structures. We hypothesize that the success of CROSS-style intersecting dependencies lie in that they are relevant in many aspects of language, both in syntax but also in semantic and discourse coherence: a word with certain semantics appearing somewhere changes the probability distribution of what words will appear around it, the same discourse referents keep being mentioned by pronouns and name, a conjunction signals a direct correspondence between phrases on both sides, and so many more. Our experiments show the possible importance in having an inductive bias that serves language learning across many levels, beyond syntax. Follow-up work will focus on disentangling the effect of different pretraining structures on different levels of language, from syntactic structure to semantic cohesion and discourse and reference.   \nThanks a lot for your smaller comments as well, they are very useful and we’ll take care to fix up the paper in those directions. And thank you very much for the thoughtfulness and thoroughness of your review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a method to test inductive biases for language learning that uses formal languages with desirable properties (being context-free & having recursion vs. being context sensitive), and exposes a language model to those formal languages. If pretraining on the formal language improves performance on natural language afterwards, that suggests that these are inductive biases for language learning (and potentially even language learning in humans). The paper successfully confirms that (given their experimental setup) introducing recursion and context-sensitivity improves language learning, which is an interesting find to the subcommunity of *ACL conferences interested in (psycho)linguistics.\n- The paper also confirms that biasing models towards Zipfian token distributions improves perplexity on natural language, which is similarly an interesting find.\n- Apart from the interesting findings, the paper's formal languages and data could be a useful resource for future work.", "reasons_to_reject": "- [Major] Even though the introduction of the languages and the pretraining procedure are appealing, the experimentation with the resulting models is quite minimal. Language learning is equated to perplexity in three languages. No further downstream tasks are considered, and no challenging train-test deviations are discussed. Particularly given the prominent role recursion is supposed to play in humans' capability to treat language in a systematic and productive way (which is not very well evaluated using just the perplexity on the WikiText test set), this limits the conclusions that can be drawn about the atypical finding that CROSS outperforms NEST(-MIX-p). Additionally, the results are just based on one model architecture, so the reader is left wondering how robust these results are.\n- [Minor] The 3rd experiment, testing the Zipfian distribution, seems a little out of place, considering that it evaluates structure over a vocabulary instead of grammatical structure. The paper could present a stronger argument by solely focusing on exp 1 & 2 but extending the experimentation or the discussion of the results.\n- [Minor] The article is a bit out-of-balance in general, as it discusses the background almost too elaborately. What recursion is, what context-free / context-sensitive languages are, and what a Zipfian distribution is, is quite well-known in the *ACL community and that text could be shortened significantly. As is, the article might be better suited to a different venue, such as a workshop.", "questions_to_authors": "1. Could you please re-explain the control case from line 353 to me? Is the pretrained GPT-2 model the one pretrained by you (in which case… isn’t the control case the same as the actual experiments you are running?) or the “real” GPT-2? Or are the embeddings in the control case randomly sampled from e.g. a  normal distribution? I found this explanation quite confusing. (based on the graphs I think it’s the original GPT-2, but please explain nonetheless)\n2. The Wikitext test set is an iid test set, I am assuming. Recursion has been widely discussed as potentially allowing generalisation to more complex structures in compositional generalisation or simply generalisation to longer lengths. Have you performed any experiments using non-iid test sets like that? I’d be very curious about the effect of CROSS and NEST in a setup like that. After all, humans don’t learn language in an iid train-test setup.\n3. Could you elaborate on why you think CROSS outperforms the remaining languages?", "paper_topic_and_main_contributions": "This article examines three types of inductive biases that potentially aid language learning in humans, by exposing language models to data capturing those biases, and measuring whether that, in fact, has a positive contribution to language learning. The three biases examined are recursion, context-sensitive language and a Zipfian distribution over the vocabulary.\n\nAfter an extensive background section (2) discusses the inductive biases, section 3 goes into the experimental setup: a GPT-2 sized model is pretrained on a formal language meant to inject the inductive bias (focusing on only one principle at a time), and afterwards the model is briefly finetuned on natural language data in 1 of 3 languages. Special care is taken to ensure that the embeddings of the natural language are of adequate quality: they are initialised using randomly sampled pretraining tokens.\n\nSection 4 details 4 formal languages used in pretraining:\n1. NEST: a recursive and context-free Dyck language\n2. CROSS: using brackets similar to 1, but with crossing dependencies. All opening brackets require a corresponding closing bracket.\n3. RAND: randomly sampled brackets without any limitations.\n4. REP: randomly sampled strings of 10 tokens where after each 10 tokens, they are repeated immediately\n\nThen, 3 experiments are conducted:\n1. Experiment 1 compares the perplexity on English/Japanse/Basque of models pretrained on those 4 languages, observing that CROSS < NEST < REP < RAND. Hence, context sensitivity acts as a better inductive bias than recursion.\n2. Experiment 2 introduces a third formal language, NEST-MIX-p that mixes NEST with some crossed dependencies, leading to improved performance compared to NEST (but not CROSS). The authors conclude that this aligns with natural language being mildly context-sensitive.\n3. Experiment 3 compares a uniform token distribution to a Zipfian token distribution, suggesting that the latter leads to strong decreases in perplexity for English, Japanese and Basque.\n\nIn particular, CROSS being the most advantageous language is an interesting finding, because it suggests recursion is not necessary for a strong inductive bias.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Section 6, I was a bit confused about why you did not discuss the paradoxical result of CROSS being better than NEST-MIX-p. Assuming that natural language is mildly-context sensitive but does use recursion quite a bit, that is an interesting/odd result. You do comment on that in the discussion, so perhaps put a reference to that discussion in Section 6? (“We will further elaborate on [...] in Section X”)\n2. In terms of presentation, I don’t understand why some of the Japanese and Basque results are in the appendix. With a simple adaptation of your current figures, you could include those results in the main paper, e.g. by using a pointplot with different colours per language. If you’re worried about the y-axes not matching you could even visualise perplexity as perplexity relative to the RAND baseline.\n3. Line 160 \"isn't\" --> \"is not\"\n4. Line 325: 2x \"the\"\n5. Line 605: \"it's\" --> \"it is\", and this sentence as a whole needs rephrasing, it is very hard to parse."}}, {"review_id": "F2zSvs76Ob", "reviewer": "Reviewer_A6Y3", "summary": "", "questions": "A. Is it possible to say more precisely what you mean by \"structure\" (line 463)? It's especially jarring to say that RAND has no structure (466) but REP has a structure similar to RAND (469).\n\nB. In Experiments 1 and 2, how many pairs of parentheses do NEST and CROSS use?\n\nC. In Experiment 3, are the parentheses nested and/or balanced? I would think neither, but if so, I can't think of any reason why the alphabet consists of parentheses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the encouraging review and the interesting points you bring up! It’s been a very helpful read for framing our paper.\n\n**COMMENT 1  The paper does not make much effort to investigate why the effects occur, or even offer any guesses**\n\nThanks for your comment, this is a question that we’re very interested in! With Experiment 2, we run a first  investigation to understand why CROSS may aid language learning. Our hypothesis that we want to test is that the CROSS performance is caused by the fact that the language is context-sensitive, rather than other artifacts that arise because of the specific structure of the CROSS language. The NEST-mix-p languages test this hypothesis by adding occasional non-context-free parenthesis pairs to the NEST language, creating a context-sensitive language that has different commonly-seen patterns than the CROSS language. Our results show that the addition of such pairs causes a significant increase in performance, showing that the context-sensitivity of CROSS can explain a large part of the increase in performance. We’ll work to make the purpose and hypothesis testing of Experiment 2 more clear, as we realize that it’s not very well signaled in the text currently. \n\nIn terms of more broad reasons or guesses: we believe that context-sensitive structures are abundant in language, though many may not exactly parallel the specific structure of the CROSS language. For example, a word with certain semantics appearing somewhere changes the probability distribution of what words will appear around it, the same discourse referents keep being mentioned by pronoun and name, a conjunction signals a direct correspondence between phrases on both sides, and so many more. The CROSS language equips a model to learn these language structures at many levels, and that may be more useful than the NEST model that more reflects syntax. In follow-up work, we plan to set up a paradigm to distinguish between the different pre-training languages and the kinds of linguistic processes they help with. \n\n**COMMENT 2 Why would we say that the REP language has structure similar to RAND?**\n\nThis was a slip-up in phrasing, thanks for catching this! By saying this, we meant that the REP language had very minimal structure, and so was most similar to the RAND language which had none. We would broadly define structure as the complex and simple ways in which tokens in a language are not independent, both in their position and in their identity. For example, a Dyck language like NEST has a complex structure where one pair of parentheses can dictate where many other ones can close, because the language’s structure prevents intersection.\n\n**COMMENT 3 In Experiments 1 and 2, how many pairs of parentheses do NEST and CROSS use?**\n\nThe vocabulary consists of 250 different types of parentheses. We mention this in a caption but not in the main text, which is an omission of ours – we’ll be sure to fix this\n \n**COMMENT 4  In Experiment 3, are the parentheses nested and/or balanced?**\n\nThe results in Figure 6 are comparing Zipfian and Uniform versions of the RAND language. Further results combining the structural languages of Experiment 1 with Zipfian distribution effects are in Figure 9 in the appendix. We’ll make this more clear in the prose, because now you’re right that it’s not well-signaled. We use the same vocabulary for all experiments, so the same tokens can act like matching pairs in the languages that have matching, and just like tokens in the languages that don’t require it. \n\nThanks a lot for this review, and for helping us clarify our points, we’ll take care to take your comments into account!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The findings that NEST is better than RAND and that the Zipfian distribution helps are interesting and expected; the finding that CROSS is better than NEST is interesting and unexpected.\n\n", "reasons_to_reject": "Although the findings are interesting, the paper does not make much effort to investigate why the effects occur, or even offer any guesses about why CROSS is better than NEST.\n\nNot reasons to reject, but things that can be improved:\n\nline 369: Please clarify whether the alphabet can contain multiple pairs of parentheses (sounds like yes).\n\nline 416: The fact that this language contains cross-serial dependencies isn't enough to prove that it is not context-free. ($\\Sigma^*$ contains everything, but is context-free.) You have to intersect the language with the regular language ${(_1}^* {(_2}^* {)_1}^* {)_2}^*$ to obtain $\\{ {(_1}^m {(_2}^n {)_1}^m {)_2}^n \\mid m,n\\ge 0 \\}$, which is not context-free.\n\n\n", "questions_to_authors": "A. Is it possible to say more precisely what you mean by \"structure\" (line 463)? It's especially jarring to say that RAND has no structure (466) but REP has a structure similar to RAND (469).\n\nB. In Experiments 1 and 2, how many pairs of parentheses do NEST and CROSS use?\n\nC. In Experiment 3, are the parentheses nested and/or balanced? I would think neither, but if so, I can't think of any reason why the alphabet consists of parentheses.", "paper_topic_and_main_contributions": "This paper experiments with pretraining a transformer language model on a formal language, then fine-tuning on a natural language. The formal pretraining languages are NEST (= the Dyck language), and CROSS, the language of parentheses that are balanced but not nested. The natural languages are English, Japanese, and Basque. The experiments show that NEST is better than random training data, and CROSS is better than NEST. Furthermore, pretraining with a Zipfian distribution also helps.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uvG7Jbnriy", "reviewer": "Reviewer_MCzq", "summary": "", "questions": "I may have missed something, but the perplexity for the ZIpfian pretraining experiments is much higher than for the earlier, structural experiments - e.g. see the difference between Figure 5 and Figure 6. Why is that?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**COMMENT 1: it's clearly not the case that nothing is learnable without the structural inductive biases proposed in this paper.**\n\nThis is certainly true, and we did not mean to frame our inductive biases by saying that they give a model a linguistic advantage which the untrained models do not have (we’ll take care to make this clearer in the text). Instead, **we view our inductive bias injection as a way of _controlling_ inductive bias**. Baroni (2021) argues that untrained models, through their architecture and initialization, have inductive biases towards learning some structures over others and are not blank-slate statistical learners (the recent advances in NLP in fact show us that transformers likely have very good linguistic-leaning inductive biases). Our experiments simply put models in a state where we know a great deal about the structures that they’re biased towards learning, rather than knowing nothing, and so we can run controlled experiments like we do. We’ll take care to make this theoretical approach more clear.\n\n**COMMENT 2: The argument would be much stronger the evaluation focused on determining if the model acquired specific linguistic phenomena**\n\nWe definitely agree that perplexity is a coarse-grained measure, and for follow-up papers we are working on more fine-grained evaluations. But we believe that our inquiry into inductive bias needs to be done in two steps.  The first step, inthis paper, is to see what kinds of structures aid language learning, using perplexity on naturally occurring text. Our experiments in this step use a small (for language-modeling-scale) fine-tuning dataset to ensure that the learning differences come from the starting inductive bias. Our current paper showcases both expected and unlikely results on the importance of a multifaceted set of grammatical and distributional structures.\n\nThe second step, which we are doing in follow-up work, is to **study a broad set of phenomena to understand _where_ the improvements that we see in this first paper come from**. Looking at the multifaceted ways in which language is structured, we want to distinguish the effects that different inductive biases have in downstream linguistic features: syntactic structure, semantic and lexical cohesion, and discourse structure. Previous work in the field thinking about structure in language learning focuses largely on syntactic structure, but structure plays a key role in many linguistic levels (the lexicon, coreference, discourse coherence, etc) so   understanding the linguistic causes of  results requires a broader paradigm which we believe to be a suitable second paper. \n\nIn the current paper, we look at perplexity on naturally occurring language, and **we’ve also performed further experiments since submission to look at a simple syntactic phenomenon: agreement**, which we can add to the camera-ready. Performance on simple subject-verb agreement (where we just check if the models have learnt the grammatical system of agreeing verbs to the subject when there aren’t other nouns to confuse them) is very similar to our perplexity results, with CROSS > NEST > REP > RAND.\n\n\n| Model | Accuracy |\n|-----------|------------|\n| Random | 74.8 |\n| Regular | 75.5 |\n| Nest | 86.9 |\n| Cross | 90.0 |\n| Control: GPT resample embs | 79.3 |\n| Control: off-the-shelf GPT-2 | 95.8 |\n\nFollow-up work looking at complex linguistic abilities would require a slightly different paradigm:  we would need a much larger fine-tuning dataset in order to induce consistent performance on the finer points of grammaticality. With our current fine-tuning size, we ran some experiments testing more complex syntactic phenomena from SyntaxGym, and all models (including the off-the-shelf GPT-2 control with shuffled embeddings) perform close to chance, and so we cannot draw meaningful conclusions. The failure of our control case suggests that the fine-tuning dataset size is insufficient to test such subtleties, as the lexicon is not developed enough after wikitext-103 training to be able to showcase underlying structural subtleties in difficult natural environments. A follow-up paper with a much larger fine-tuning set will include experiments on more sophisticated grammatical, semantic, and discourse phenomena, and would also need ways to differentiate the structure inserted in pretraining from structure learned during fine-tuning,  and we believe this large set of experiments to be out of scope for this first paper. \n\n(we also wrote this last section to reviewer Wikh, who had a similar question) \n\n\n**COMMENT 3 Why is the Zipfian perplexity higher than the other experiments?**\n\nThe Zipfian results that we report in the main paper are with RAND languages, where the only information comes from the distribution that the tokens are being randomly sampled from. So they should be compared to the RAND language perplexity in Experiment 1. We also test the effect of combining distribution and structure and report these results in the Appendix, Figure 9. We can promote this and some further discussion to the main text in the camera-ready\n\nThanks for all of the smaller comments as well, it’s very valuable to see your view on the presentation of the work and we will work to adopt a lot of this in rewriting!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The methodology of giving models an inductive bias through pretraining on a formal language is simple and convincing. The specific formal languages that the authors use are well-motivated from linguistics, there are good control languages, and the experiments are repeated for three typologically diverse natural languages. The experiments are elegant and clearly presented.", "reasons_to_reject": "I am not sure I agree with the current framing - e.g. in the first paragraph, the authors suggest that the study can answer the question: \"what structural predispositions does a learner need in order for language to be learnable?\". In fact, though, what the experiments show is that certain inductive biases help a transformer learn predict the next word better, but it's not clear that those biases are *necessary* to make language learnable, as there's no criterion defined in the paper for when a language has been learned successfully. In fact, we know from other work that LMs trained from scratch on the same amount of data as used in this paper can learn a lot about language (e.g. syntax), so it's clearly not the case that nothing is learnable without the structural inductive biases proposed in this paper. \n\nThe argument would be much stronger the evaluation focused on determining if the model acquired specific linguistic phenomena instead of just measuring its perplexity. Indeed, any sort of error analysis or a more qualitative insights into what in particular pretraining on the each formal language helps with would have improved the paper, and perhaps would have helped explain why even the repetition language is so helpful. The repetition language finding suggests that perhaps copying / \"induction heads\" are playing a central role here. It would be great to see a test of this hypothesis and in general more analysis of the pretrained models, either behavioral (e.g. evaluate them on strings that involve copying) or representationl (e.g. looking at attention heads).\n\nAlso because of the coarse grained perplexity-based evaluation it's not so clear what the inductive biases is that the models trained on formal languages end up acquiring (those may not be exactly the same as the inductive biases the authors intend the models to end up with). I find the results in Figure 4, for example, to be quite surprising: how is it that even after training on 100 million tokens of WikiText there is such an enormous effect of the pretraining corpus? ", "questions_to_authors": "I may have missed something, but the perplexity for the ZIpfian pretraining experiments is much higher than for the earlier, structural experiments - e.g. see the difference between Figure 5 and Figure 6. Why is that?", "paper_topic_and_main_contributions": "In this work, the authors pretrain transformers on various synthetic languages, and test for transfer to natural language by measuring perplexity. They find that pretraining on formal languages with any amount of structure, including even repetition of sequences of 10 tokens, has a very large on perplexity after fine tuning on the natural corpus, but pretraining on structure that is more linguistically relevant (in particular, cross-serial dependency) is particularly helpful. Likewise, pretraining on a corpus where the vocabulary follows a Zipfian frequency distribution reduces perplexity after fine tuning.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The introduction needs to be reorganized in my view - it currently doesn't have a clear structure and goes back and forth between laying out the argument and providing \"related work\" style lists of references. As I mentioned above I was also missing a crisper motivation for the experiments and what exactly (if anything) they can teach us about how humans learn language."}}]

Author rebuttal:

Meta-review: The paper is sound and addresses problems at the core of this track.
Decision: Accept-Findings