Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models
Authors: James Michaelov
Catherine Arnett
Tyler A. Chang
Ben Bergen
Venue: EMNLP 2023 Main (2023)
Keywords: abstraction
representation
multilingual language models
psychlinguistics
linguistic structure
TL;DR: Multilingual language models exhibit both within-language and across-language structural priming effects

Abstract: Abstract grammatical knowledge—of parts of speech and grammatical patterns—is key to the capacity for linguistic generalization in humans. But how abstract is grammatical knowledge in large language models? In the human literature, compelling evidence for grammatical abstraction comes from structural priming. A sentence that shares the same grammatical structure as a preceding sentence is processed and produced more readily. Because confounds exist when using stimuli in a single language, evidence of abstraction is even more compelling from crosslingual structural priming, where use of a syntactic structure in one language primes an analogous structure in another language. We measure crosslingual structural priming in large language models, comparing model behavior to human experimental results from eight crosslingual experiments covering six languages, and four monolingual structural priming experiments in three non-English languages. We find evidence for abstract monolingual and crosslingual grammatical representations in the models that function similarly to those found in humans. These results demonstrate that grammatical representations in multilingual language models are not only similar across languages, but they can causally influence text produced in different languages.

Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models
James A. Michaelova
Catherine * Tyler A. Chang
Benjamin K. Bergen
*Department of Cognitive Science,
"Department of Linguistics,
University of California San Diego
{jlmichae, ccarnett, tachang, bkbergen)@ucsd.com edu
Abstract
representations are abstract enough be shared across languages both humans (Heydel and Mur- ray, 2000; Hartsuiker et al., 2004; Schoonbacrt et al., 2007) and language models (Conneau et al., 2020b,a; Jones et al., 2021).
Abstract grammatical knowledge of parts of speech and grammatical patterns-is key to the capacity for linguistic generalization in humans. But how abstract is grammatical knowledge in large language models? In the human literature, compelling evidence for grammatical abstrac- tion comes from structural priming. A sentence that shares the same grammatical structure as a preceding sentence is processed and produced more readily. Because confounds exist when using stimuli in a single language, evidence of abstraction is even more compelling from crosslingual structural priming, where use of syntactic structure in one language primes an analogous structure in another language. We measure crosslingual structural priming in large language models, comparing model behavior to human experimental results from eight crosslin- gual experiments covering six languages, and four monolingual structural priming experi- ments in three non-English languages. We find evidence for abstract monolingual and crosslin- gual grammatical representations in the models that function similarly to those found in hu- mans. These results demonstrate that grammat- ical representations in multilingual language models are not only similar across languages, but they can causally influence text produced in different languages.
The strongest evidence for grammatical abstrac- tion in humans comes from structural priming, widely used and robust experimental paradigm. Structural priming is based on the hypothesis that grammatical structures may be activated during language processing. Priming then increases the likelihood of production or increased ease of pro- cessing of future sentences sharing the same gram- matical structures (Bock, 1986; Ferreira and Bock, 2006; Pickering and Ferreira, 2008; Dell and Fer- reira, 2016; Mahowald et al., 2016; Branigan and Pickering. 2017). For example, Bock (1986) finds that people are more likely to produce an active sen- tence (e-g. one of the fans punched the referee) than a passive sentence (c.g. the referee was punched by one of the fans) after another active sentence. This has been argued (Bock, 1986; Heydel and Murray, 2000; Pickering and Ferreira, 2008; Reit- ter et al., 2011; Mahowald et al., 2016; Branigan and Pickering, 2017) to demonstrate common ab- stractions generalized across all sentences with the same structure, regardless of content.
Researchers have found evidence that structural priming for sentences with the same structure oc- curs even when the two sentences are in different languages (Loebell and Bock, 2003; Hartsuiker et al., 2004; Schoonbaert et al., 2007; Shin and Christianson, 2009; Bernolet et al., 2013; van Gom- pel and Arai, 2018; Kotzochampou and Chondro- gianni, 2022). This crosslingual structural priming takes abstraction one step further. First, avoids any possible confounding effects of lexical rep- etition and lexical priming of individual words within a given language, sentences with the same structure often share function words (for discus- sion, see Sinclair et al., 2022). More fundamentally.
1 Introduction
What do language models learn about the structure of the languages they are trained on? Under both more traditional generative (Chomsky, 1965) and cognitively-inspiee usage-based theories of lan- guage (Tomasello, 2003; Goldberg, 2006; Bybee, 2010), the key to generalizable natural language comprchension and production is the acquisition of grammatical structures that are sufficiently abstract account for the full range of possible sentences in a language. In fact, both theoretical and experimen- tal accounts of language suggest that grammatical
"qqual contribution.

---

crosslingual structural priming represents an extra degree of grammatical abstraction not just within a language, but across languages.
are causally involved in the text predicted or gener- ated by the model (Voita and Titov, 2020; Sinclair et al., 2022). The structural priming paradigm al- lows researchers to evaluate whether grammatical representations generalize across sentences in lan- guage models, and whether these representations causally influence model-generated text. Further- more, structural priming is agnostic to the specific language model architecture and does not rely on direct access internal model states.
We apply this same logic to language models in the present study. While several previous stud- ies have explored structural priming in language models (Prasad et al., 2019; Sinclair et al., 2022; Frank, 2021; Li et al., 2022; Choi and Park, 2022), to the best of our knowledge, this is the first to look at crosslingual structural priming in Transformer language models. We replicate eight human psy- cholinguistic studies, investigating structural prim- ing in English, Dutch (Schoonbaert et al., 2007; Bernolet et al.. 2013), Spanish (Hartsuiker et al.. 2004), German (Loebell and Bock, 2003), Greek (Kotzochampou and Chondrogianni, 2022), Polish (Fleischer et al., 2012), and Mandarin (Cai al., 2012). We find priming effects in the majority of the crosslingual studies and all of the monolingual studies, which we argue supports the claim that multilingual models have shared grammatical rep- resentations across languages that play a functional role in language generation.
However, the structural priming paradigm has not been applicd to modern multilingual language models. Previous work has demonstrated that mul- tilinuaal language models encode grammatical fea- tures in shared subspaces across languages (Chi et al., 2020; Chang et al., 2022: de Varda and Marelli, 2023), largely relying on probing meth- ods that do not establish causal effects on model predictions. Crosslingual structural priming would provide evidence that the abstract grammatical rep- resentations shared across languages in the models have causal effects on model-generated text. It would also afford a comparison between grammati- cal representations in multilingual language models and human bilinguals. These shared grammatical representations may help explain crosslingual trans- fer abilities in multilingual models, where tasks learned in one language can be transferred to an- other (Artetxe et al., 2020; Conncau et al., 2020a,b; K et al., 2020; Goyal et al., 2021; Ogueji et al., 2021; Armengol-EEstaéé et al., 2021, 2022; Blevins and Zettlemoyer, 2022; Chai et al., 2022; Muen- nighoff et al., 2023; Wu et al., 2022; Guarasci et al.. 2022; Eronen al., 2023).
2 Background
Structural priming effects have been observed in humans both within a given language (Bock, 1986; Ferreira and Bock, 2006; Pickering and Ferreira, 2008; Dell and Ferreira, 2016; Mahowald et al., 2016; Branigan and Pickering, 2017) and crosslin- gually (Loebell and Bock, 2003; Hartsuiker et al.. 2004; Schoonbaert al., 2007; Shin and Chris- tianson, 2009; Bernolet et al., 2013; van Gompel and Arai, 2018; Kotzochampou and Chondrogianni, 2022). In language models, previous work has demonstrated structural priming effects in English (Prasad et al., 2019; Sinclair et al., 2022; Choi and Park, 2022), and initial results have found prim- ing effects between English and Dutch in LSTM language models (Frank, 2021). As these studies ar- gue, the structural priming approach avoids several possible assumptions and confounds found in previ- ous work investigating abstraction in grammatical learning. For example, differences language model probabilities for individual grammatical vs. ungrammatical sentences may not imply that the models have formed abstract grammatical represen- tations that generalize across sentences (Sinclair et al., 2022); other approaches involving probing (e.g. Hewitt and Manning, 2019; Chi et al., 2020) often do not test whether the internal model states
Thus, this study presents what is our knowl- edge the first experiment testing for crosslingual structural priming in Transformer language models. The findings broadly replicate human structural priming results: higher probabilities for sentences that share grammatical structure with prime sen- tences both within and across languages.
3 Method
We test multilingual language models for structural priming using the stimuli from eight crosslingual and four monolingual priming studies in humans. Individual studies are described in $4.
3.1 Materials
All replicated studies have open access stimuli with prime sentences for different constructions ($3.3).

---

Where target sentences are not provided (because participant responses were manually coded by the experimenters), we reconstruct target sentences and verify them with native speakers.
Of- and S-Genitives represent two different ways of expressing possessive mean- ing. an of-genítive, the possessed thing fol- lowed by a preposition such as of and then the possessor, e.g., the scarf of the boy is yellow. In s-genitives in the languages we analyze (English and Dutch), the possessor is followed by word or attached morpheme such as 's which is then followed by the possessed thing, e-g-, the boy's scarf is yellow (Bernolet et al., 2013). Researchers compare the proportion of -g-eiiitvve or s-genitive sentences produced by experimental participants following an of oe-niiiie or s-genitive prime.
3.2 Language Models
We test structural priming in XGLM 4.5B (Lin et al., 2022), a multilingual autoregressive Trans- former trained on data from all languages we study in this paper, namely, English, Dutch, Spanish, Ger- man, Greek, Polish, and Mandarin. To the best of our knowledge, this the only available pretrained (and not fine-tuned) autoregressive language model trained on all the aforementioned languages. To avoid drawing any conclusions based on the id- iosyncrasies of a single language model, we also test number of other multilingual language mod- els trained on most of these languages, namely the other XGLM models, i.e., 564M, 1.7B, 2.9B, and 7,5B, which are trained on all the languages ex- cept for Dutch and Polish; and PolyLM 1.7B and 13B (Wei et al., 2023), which are trained on all the languages except for Greek.
3.4 Testing Structural Priming in Models
In human studies, researchers test for structural priming by comparing the proportion of sentences (targets) of given types produced following primes of different types. Analogously, for each experi- mental item, we prompt the language model with the prime sentence and compute the normalized probabilities of each of the two target sentences. We illustrate our approach computing these nor- malized probabilities below.
3.3 Grammatical Alternations Tested
First, consider the example dative alternation stimulus sentences from Schoonbaert et al. (2007):
We focus on structural priming for the three alter- nations primarily used in existing human studies.
(1) (a) DO prime:
The cowboy shows the pirate an apple.
Dative Alternation (DO/PO) Some languages permit multiple orders of the direct and indirect objects in sentences. In PO (prepositional object) constructions, e.g., the chef gives a hat to the swim- mer (Schoonbaert et al., 2007), the direct object a hat immediately follows the verb and the indirect object is introduced with the prepositional phrase to the swimmer. In DO (double object) constructions, c.g., the chef gives the swimmer a hat, the indirect object the swimmer appears before the direct ob- ject a hat and neither is introduced by a preposition. Researchers compare the proportion of DO or PO sentences produced by experimental participants following a DO or PO prime.
(b) PO prime: The cowboy shows an apple
to the pirate.
(c) DO target: The chef gives the swimmer
a hat.
(d) PO target: The chef gives a hat to the swimmer.
We can use language models to calculate the proba- bility of each target following each prime by taking the product of the conditional probabilities of all to- kens in the target sentence given the prime sentence and all preceding tokens in the target sentence. In practice, these probabilities are very small, but for illustrative purposes, we can imagine these have the probabilities in (2).
Active/Passive In active sentences the syntactic subject is the agent of the action, while in pas- sive sentences the syntactic subject is the patient or theme of the action. E.g., the taxi chases the truck is active, and the truck chased by the taxi is passive (Hartsuiker et al., 2004). Researchers com- pare the proportion of active or passive sentences produced by experimental participants following an active or passive prime.
(a) P(PO Target DO Prime) 0.03
(b) P(DO Target DO Prime) = 0.02
(c) P(PO Target PO Prime) 0.04
(d) P(DO Target PO Prime) = 0.01
We then normalize these probabilities by calculat- ing the conditional probability of each target sen- tence given that the model response is one of the two target sentences, as shown in (3).

---

(3) (a) (NO DO) 0.03/(0.03++0.0))) 0.60
4.1 Crosslingual Structural Priming
(b) (DO DO) 0.02/0.03+0.02)) 0.40
We test whether eight human crosslingual struc- tural priming studies replicate in language models. These studies cover structural priming between En- glish and Dutch (Schoonbaert et al., 2007; Bernolet et al., 2013), Spanish (Hartsuiker et al., 2004), Ger- man (Loebell and Bock, 2003), Greek (Kotzocham- pou and Chondrogianni, 2022), and Polish (Fleis- cher al., 2012). For each experiment, we show the original human probabilities and the normalized probabilities calculated using each language model, as well as whether there is a significant priming effect (Figure 1). The full statistical results are reported in Appendix B.
(c) PPO PO) = 0.04/0..040.001 0.80
(d) (DO PO) = 0.01/(0.04+0.001 0.20
Because the normalized probabilities of the two tar- gets following a given prime sum to one, we only consider the probabilities for one target type in our analyses (comparing over the two different prime types). For example, to test for priming effect, we could either compare the difference between (PO PO) and (PO DO) the difference between (DO POO) and (DO DO). We fol- low the original human studies in the choice of which target construction to plot and test.
4.1.1 Schoonbaert al. (2007):
We run statistical analyses, testing whether ef- fects are significant for each language model on each set of stimuli. To do this, we construct linear mixed-effeee model predicting the target sentence probability (e.g. probability of PO sentence) for each item. We include random intercept for ex- perimental item, and we test whether prime type (e.g. DO vs. PO) significantly predicts target struc- ture probability. All reported p-values are corrected for multiple comparisons by controlling for false discovery rate (Benjamini and Hochberg, 1995). All stimuli, data, code, and statistical analyses are provided at https://ost io/2vjw66/
Dutch Dutch-+Englis *English
Schoonbaert et al. (2007) prime 32 Dutch-English bilinguals with 192 Dutch sentences with either prepositional (PO) or dative object (DO) construc- tions. Schoonbaert et al. (2007) find that exper- imental participants produce more PO sentences when primed with PO sentence than when primed with DO sentence (see Figure IA). We see the same pattern with nearly all the language models (Figure 1A). With the exception of XGLM 1.7B, where the effect is only marginally significant after correction for multiple comparisons, all language models predict English PO targets to be signifi- cantly more likely when they follow Dutch PO primes than when they follow Dutch DO primes.
4 Results
In reporting whether the structural priming effects from human experiments replicate in XGLM lan- guage models, we primarily consider the direction of each effect in the language models (e.g. whether PO constructions are more likely after PO DO primes) rather than effect sizes or raw probabilities. The mean the relative probabilities assigned by language models to the different constructions in each condition may not be directly comparable to human probabilities of production. Humans are sensitive to contextual cues that may not be avail- able to language models; notably, in these tasks, hu- mans are presented with pictures corresponding to events in the structural priming paradigm. Further- more, construction probabilities in language mod- els may be biased by the frequency of related con- structions in any the many languages on which the models are trained. Thus, we focus only on whether the language models replicate the direc- tion of the principal effect in each human study.
4.1.2 Schoonbaert al. (2007): English+Dutte
Schoonbaert et al. (2007) also observe DO/PO structural priming from English Dutch (32 par- ticipants; 192 primes). As seen in Figure IB, all language models show a significant priming effect.
4.1.3 Bernolet et al. (2013): Dutch- Bernolet et al. (2013) conduct a Dutch- English structural priming experiment with 24 Dutch- English bilinguals on 192 prime sentences, and they find that the production of s-genitives is signif- icantly more likely after an s-genitive prime than after an -f-giiiivve prime. We also observe this in all of the language models, as seen in Figure IC.
4.1.4 Hartsuiker et al. (2004):
Spanish->Englins >English Hartsuiker et al. (2004) investigate structural priming with

---

IN odsorsaat u.E. (amm) Dediviengt
(Bi Sccondaent di I2b0T) Engten ipuch Procesiiy
RO
ca--
60
60
121121: Englun
Momuladd Produciit S demieo
Nomnalaed Piccucally Pasaaoo
Haran
n64M
IGG M
A
PoW
XGI GMM
M 1:70
Jin pan
Poyt
Normalzzd Prozeti
Norralmee Protbsblly PO Targen
Normalie Procasity Teigets
Haman
XGLM 5047
NGLVI 1.75
XOLMARD
MYNLA 1:71
XGLM SUAM
JDLM 1.70
NOLM 2011
IOLM 7:30 PayLM
MoyLM
7
Narma
Pasoiv Targes
Nomalee Prctanalyy
Priddis trgees
Itaman
XGUM 5049
NGUII 1.70
ACLM 4:0
XCUUI 7:00
haeua 1:10
Pegual 120
Tuman
XCLM SCOM
XILM 1770
XGGL 2.00
3D 4.50
MOLW
PayEM
PayLM
n.
F4
Adle Fairr
Figure I: Human and language model results for crosslingual structural priming experiments.
Sytvebber (PSS7) -uuth-fuuhh Normalme Thotmaellyy targen
Koerodt (P000) Greetoe wie Narmalaed Procuailyy thadae Targets
Harin
XOUM s649!
MOLVI1
X0LM445
XOUM
IMKUM 1.78
PueLMr 1281
Human
XOLM 1.8
NOun zoe vos un WOLV 7.50 PayLM 1.78
PayLS ree
ci--
co6-
Romaked Probabiity
25
60
60
KiW Falre
<dwe male
ACli Faire
Nom Proboiilly DO Taryets
Nommltee Probitiyy DO Passio Targeto
XGLM 30W9
PAK-M 1:01
PoAu 130
Lumon
XGLMISSM
XGLM 1m
VOLM 2.00
MGLU 2.50
PayEM 10
Raye
60
20
20
2
60
60
60
2
60
60
55
10
Prina Type
Figure 2: Human and language model results for within-language structural priming experiments.
24 Spanish-Fnglish bilinguals on 128 prime sentences, finding a significantly higher proportion of passive responses after passive primes than active primes. As shown in Figure ID, this effect is replicated by XGLM 564M, 2.9B, and 7.5B as well as PolyLM 13B, with XGLM 4.5B showing marginal effect (p- 0.0565).
numerical effect in the correct direction, the effect is only significant for XGLM 7.5B.
4.1.6 Loebell and Bock (2003):
English-Germann
Loebell and Bock (2003) also test 48 German- English bilinguals for dative alternation (DO/PO) priming effect from English primes to German tar- gets (32 prime sentences), finding small but sig- nificant priming effect. As we show in Figure 1F, the models are relatively varied direction of nu- merical difference. However, only XGLM 2.9B and PolyLM 13B display a significant effect, and in both cases the effect is in the same direction that found with human participants.
4.1.5 Loebell and Bock (2003):
GGerman-Ennglinn
Loebell and Bock (2003) find a small but signifi- cant priming effect of dative alternation (DO/PO) from German to English with 48 German-Fnglish bilinguals on 32 prime sentences. As can be seen in Figure IE, while all language models show

---

4.1.7 Kotzochampou and Chondrogianni (2022): Greek-English
(N=25) using 48 primes. As shown in Figure 2B, this effect replicated by all language models.
Kotzochampou and Chondrogianni (2022) find ac- tive/passive priming from Greek to English in 25 Greek-English bilinguals. Participants are more likely to produce passive responses after passive primes (48 prime sentences). As shown in Fig- ure 1G), all XGLMs display this effect, while the PolyLMs, which are not trained on Greek, do not.
4.2.3 Cai et al. (2012): Mandarin Manddainn-Mandarinn Using two separate sets of stimuli, Cai al. (2012) find within-languaae DO/PO priming effects in na- tive Mandarin speakers (N=28, N=24). As seen in Figure 2C and 2D, all language models show significant effects for both sets of stimuli (48 prime sentences in their Experiments 1 and 2, and 68 prime sentences in their Experiment 3).
4.1.8 Fleiseher et al. (2012): Polish-EEnglih
Similarly, Fleischer et al. (2012) find active/passive priming from Polish English in 24 Polish- English bilinguals on prime sentences. As we see Figure HH, while models show a numeri- cal difference in the correct direction, the effect is only significant for XGLM 564M, 2.9B, and 7.5B, and for PolyLMI 1.7B.
4.3 Further Tests of Structural Priming
We have now observed within-language structural priming multilingual language models for lan- guages other than English. In $4.1, we found robust English Dutch structural priming (Schoonbaert et al., 2007) but only limited priming effects for targets in German. Although there are no human sults for the non- non-English targets in the other studies in $4.1, we can still evaluate crosslingual structural priming with non- non-English targets the language models by switching the prime and target sentences in the stimuli. Specifically, we test structural prim- ing from English to Dutch (Bernolet et al., 2013), Spanish (Hartsuiker et al., 2004), Polish (Fleischer et al., 2012), and Greek (Kotzochampou and Chon- drogianni, 2022).
4.2 Monolingual Structural Priming
In the previous section, we found crosslingual prim- ing effects in language models for the majority of crosslingual priming studies in humans. However, six of the eight studies have English target sen- tences. Our results up to this point primarily show an effect of structural priming on English targets. While both previous work (Sinclair et al., 2022) and our results in $4.1 may indeed demonstrate the effects of abstract grammatical representations on generated text in English, we should not assume that such effects can reliably be observed for other languages. Thus, we whether multilingual lan- guage models exhibit within-language structural priming effects comparable to those found in hu- man studies for Dutch (Schoonbaert et al., 2007), Greek (Kotzochampou and Chondrogianni, 2022), and two studies in Mandarin (Cai et al., 2012).
All models show significant effect the versed Bernolet et al. (2013) stimuli (Figure 3A; English-DDuthh), and all models but PolyLM 1.7B show the same for the reversed Hartsuiker al. (2004) stimuli (Figure 3B; The other results are less clear-cut. While XGLM 564M, 2.9B, and 4.5B and the PolyLMs show numerical effect in the correct direction for the reversed Fleischer al. (2012) stimuli (English->Polishy Polishh Figure 3C), only PolyLM 1.7B shows a significant effect. For the reversed Kot- zochampou and Chondrogianni (2022) stimuli (English++Greek; Figure 3D), all the XGLMs and PolyLM 13B show a numerical tendency in the correct direction, but only XGLM 564M and 4.5B show a significant effect.
4.2.1 Schoonbaert et al. (2007):
Dutch->Dutch
Using Duteh prime and target sentences (192 primes), Schoonbaert al. (2007) find that Dutch- English bilinguals (N=32) produce PO sentences at a higher rate when primed by a PO sentence com- pared to a DO sentence. As we see in Figure 2A, all language models display this effect.
'The original study tests the effect of variants of DO/PO primes (topicalized DO/PO and Ba-DO; see Cai et al., 2012). To unify our analyses across studies, we only look at structural priming following the canonical DO and PO primes used in both Experiments I and 2 of the original study, well as those used in Experiment 3.
4.2.2
Kotzochampou and Chondrogianni (2022): Brreek>Grreek
In their Greek-Grred priming experiment, Kot- zochampou and Chondrogianni (2022) find ac- tive/passive priming effect in native Greek speakers

---

(A) Berncii al (am1:) Engleh -Dushh
Har budler (2004) Eng -Sommit
No mostreeed Probshall Si-min larges
horms Hormettred Protbcnilly Passive largets
XGLLM
XOLM
XGLMM
Normalized Probabilty
erdoo
Nabva Maluow
Caloo Maiivo
Adw Pidko
Jalvo Moloo
Padko
(G) Pleadwrrit ol 1201 Englih -Pulsh Normaleed Probabilly Passive Targets
(DI Keladdancou (2002) Englidi -G -Grask Targeta
Normalled Probanllyy Passive Targets
XXLM
XOLM
XGLM
XOLM
XOLM 1.78
XOLM
XGLM
XGLM
PulAM1177
n.B.
Primo Type
Figure 3: Language model results for structural priming experiments with no human baseline.
5 Discussion
same way that humans find it easier to process words that have recently been mentioned (Rugg. 1985, 1990; Van Petten et al., 1991; Besson et al., 1992: Mitchell et al., 1993; Rommers and Feder- meier, 2018), language models may predict that previously-mentioned words are more likely to oc- cur again (a familiar phenomenon in the case of repeated text loops; see Holtzman et al., 2020; See et al., 2019; Xu et al., 2022) even if they not trained on the words explicitly. This would explain the results for the Kotzochampou and Chondro- gianni (2022) stimuli, as the Greek passive stimuli always include the word azóó
We find structural priming effects at least one language model on each set of stimuli (correcting for multiple comparisons). Moreover, we observe a significant effect in all models with the mono- lingual stimuli, and in the majority of the models for 8 of the 12 crosslingual stimuli. In line with previous work (Hewitt and Manning, 2019; Chi et al., 2020), this supports the claim that language models learn generalized, abstract, and multilin- gual representations of grammatical structure. Our results further suggest that these shared grammat- ical representations are causally linked to model output.
Such an explanation could also account for the performance of XGLM 564M, 1.7B, 2.9B, and 7.5B on the Dutch and Polish stimuli. Despite not being intentionally trained Dutch or Pol- ish, we see robust crosslingual Dutch- Beachh-Englhnn English and English~>Dutch structural priming, well as Polish Fnggish structural priming, three of these models. However, as discussed previously, crosslingual structural priming avoids the possi- ble confound of lexical overlap. For these results, therefore, more likely explanation is language contamination. In contemporaneous work, we find that training fewer than IM tokens in a second language is sufficient for structural priming effects to emerge (Arnett al., 2023); our estimates of the amount of language contamination in XGLM 564M, 1.7B, 2.9B, and 7.5B range from 1.77M tokens of Dutch and 1. 1.46M tokens of Polish at the most conservative to 152.5M and 33.4M tokens respectively at the most lenient (see Appendix A). The smaller amount of Polish contamination. as well as the fact that Polish is less closely lated English, may explain the less consistent
5.1 Differences between models
In some ways, we expected patterns across mod- els. For For example, for the XGLMs trained 30 lan- guages (XGLM 564M, 1.7B, 2.9B, and 7.5B), the larger models tend to display larger effect sizes than the smaller models, line with the idea that model performance can scale with number of parame- ters (Brown et al., 2020; Kaplan et al., 2020; Rae et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023). Additionally, the PolyLMs, which are not trained on Greek, do not show crosslingual struc- tural priming for Greek (neither Greek English nor English-&Greek).
On the other hand, one surprising finding that despite not being trained on Greek, the PolyLMs are able to successfully model monolingual struc- tural priming in Greek. The most likely expla- nation for this is what Sinclair et al. (2022) re- fer to 'lexical overlap"- the overlap of func- tion words between primes and targets substan- tially boosts structural priming effects. In the

---

Ponlsh-tEnglish structural priming effects and the virtually non-existent English-+Polish effects in these models, but as will be discussed in $5.2, there may be other reasons for this latter pattern.
ing and humans' dependence on prime language experience for structural priming would suggest that there are key differences between the models and humans in how grammatical representations function in comprehension and production.
5.2 Null Effects and Asymmetries
An alternative reason for the absence of crosslingual structural priming effects for the English-PPolish and English+Greek stimuli is combination of model features and features of the languages themselves. For example, structural priming effects at the syntactic level may overall be stronger for English targets. English a lan- guage with relatively fixed word order, and thus, competence in English may require a more explicit representation of word order than other languages. In contrast to English, Polish and Greek are mor- phologically rich languages, where important in- formation is conveyed through morphology (e.g. word inflections), and word orders are less fixed (Tzanidaki, 1995; Siewierska, 1993). Thus, struc- tural priming effects with Polish and Greek targets would manifest as differences in target sentence morphology. However, contemporary language models such as XGLM have a limited ability to deal with morphology. Most state-of-the-an the-art mod- els WordPiece (Wu al., 2016) or Sentence- Piece (Kudo and Richardson, 2018) tokenizers, but other approaches may be necessary for morpho- logically rich languages (Klein and Tsarfaty, 2020; Park et al., 2021; ; Soulos et al., 2021; Nzeyimana and Niyongabo Rubungo, 2022; Seker et al., 2022),
More theoretically interesting is the question of why some language models fail to display crosslin- gual structural priming on some sets of stimuli, even when trained on both languages. For ex- ample, in the Loebell and Bock (2003) replica- tions, only XGLM 7,5B shows a significant ef- fect of German Frrman-+Englisie EEgllish structural priming, and only XGLM 2.9B and PolyLM 13B show a signifi- cant effect of English >German structural priming. This may be due to the grammatical structures used in the stimuli (DO/PO). While the original study does find crosslingual structural priming effects, the effect sizes are small; the authors suggest that this may partly be because "the prepositional form is used more restrictively in German" (Loebell and Bock, 2003, p. 807).
We also see an asymmetry in the crosslingual structural priming effects between some languages. While the effects in the Dutch--Engiih (Bernolet et al., 2013) and Spanish->English (Hartsuiker et al., 2004) studies mostly remain when the di- rection of the languages is reversed, this is not the case for the FniishhEnggiihh (Fleischer al.. 2012) and Greek FEnglshh (Kotzochampou and Chondrogianni, 2022) results. This may be due to the smaller quantity of training data for Polish and Greek compared to Spanish in XGLM. While XGLM is only trained on slightly more Dutch than Polish, Dutch is also more similar to English in terms of its lexicon and morphosyntux, it may benefit from more effective crosslingual transfer (Conncau al., 2020b; Gerz et al., 2018; Guarasci et al., 2022; Winata et al., 2022; Ahuja et al., 2022; Oladipo et al., 2022; Eronen et al., 2023).
Thus, while humans are able to exhibit crosslin- gual structural priming effects between languages when the equivalent structures do not share the same word orders (Muylle et al., 2020; Ziegler et al., 2019; Hsich, 2017; Chen et al., 2013), this may not hold for contemporary language models. Specifically, given the aforementioned limitations of contemporary language models, it would be un- surprising that structural priming effects are weaker for morphologically target languages with rel- atively free word order such as Polish and Greek.
If it is indeed the case that structural priming effects in language models are weaker when the tar- get language is less trained on, this would contrast with human studies, where crosslingual structural priming appears most reliable when the prime is in participants' native or primary language (L1) and the target is in their second language (I.2). The reverse case often results in smaller effect sizes (Schoonbaert et al., 2007) or effects that are not sig- nificant at all (Shin, 2010). Under this account, lan- guage models' dependence on target language train-
5.3 Implications for Multilingual Models
The results reported here seem bode well for the crosslingual capacities of multilingual language models. They indicate shared representations of grammatical structure across languages (in linc with Chi et al., 2020; Chang et al., 2022; de Varda and Marelli, 2023), and they show that these repre- sentations have causal role in language generation.

---

The results also demonstrate that crosslinguistic transfer can take place at the level of grammatical structures, not just specific phrases, concepts, and individual examples. Crosslinguistic generaliza- tions can extend at least to grammatical abstrac- tions, and thus learning grammatical structure in one language may aid in the acquisition of its homologue in a second language.
language comprehension (e.g, Oh and Schuler, 2023; Michaclov al., 2022; Wilcox al., 2021; Hollenstein et al., 2021; Kuribayashi et al., 2021; Goodkind and Bicknell, 2018), and while the de- gree to which the the mechanisms involved in com- prehension and production differ in humans is a matter of current debate (Pickering and Garrod, 2007, 2013; Hendriks, 2014; Meyer et al., 2016; Martin et al., 2018), our results show that human studies of language production can also be repro- duced language models used for text generation.
How do language models acquire these abstrac- tions? As Contreras Kallens al. (2023) point out, language models learn grammatical knowl- edge through exposure. To the degree that similar outcomes for models and humans indicate shared mechanisms, this serves to reinforce claims of usage-based (i.e. functional) accounts of language acquisition (Tomasello, 2003; Goldberg, 2006; By- bee, 2010), which argue that statistical, bottom-up learning may be sufficient account for abstract grammatical knowledge. Specifically, the results of our study demonstrate the in-principle viability of learning the kinds of linguistic structures that are sensitive to structural priming using the statistics of language alone. Indeed, under certain accounts of language (e.g. Branigan and Pickering, 2017), it is preciscly the kinds of grammatical structures that be primed that are the abstract linguistic repre- sentations that we learn when we acquire language. Our results are thus in line with Contreras Kallens et al.'s (2023) argument that it may be possible to use language models as tests for necessity in theories of grammar learning. Taking this further, future work might use different kinds of language models to test what types of priors or biases, if any, are required for any learner to acquire abstract linguistic knowledge.
Conclusion
Using structural priming, we measure changes in probability for target sentences that do do not share structure with prime sentence. Analogously to humans, models predict that similar target structure generally more likely than a different one, whether within or across languages. We ob- serve several exceptions, which may reveal features of the languages in question, limitations of the mod- els themselves, or interactions between the two. Based on results, argue that multilingual au- toregressive Transformer language models display evidence of abstract grammatical knowledge both within and across languages. Our results provide evidence that these shared representations are not only latent in multilingual models' representation spaces, but also causally impact their outputs.
Limitations
To ensure that the stimuli used for the language models indeed elicit structural priming effects people, we only use stimuli made available by the authors of neviously-publisher studies on struc- tural priming in humans. Thus, our study analyzes only a subset of possible grammatical alternations and languages. All of our crosslingual structural priming stimuli involve English one of the lan- guages, and all other languages included are, with the exception of Mandarin, Indo- Ando-European lan- guages spoken in Europe. All are also moderately or highly-resouren et al., 2020). Thus, our study is not able to account for the full diversity of human language.
In practical terms, the structural priming paradigm an innovative way to probe whether a language model has formed an abstract representa- tion of a given structure (Sinclair et al., 2022), both within and across languages. By testing whether a structure primes homologous structure in an- other language. we can assess whether the model's representation for that structure is abstract enough generalize beyond individual sentences and has a functional role in text generation. As language models are increasingly used text generation sce- narios (Lin et al., 2022) rather than fine-tuning rep- resentations (Conneau et al., 2020a), understanding the effects of such representations on text genera- tion is increasingly important. Previous work has compared language models to human studies of
in the NLP literature (Joshi
Additionally, while psycholinguistic studies of- ten take crosslingual structural priming indicate shared representations, there are alternate interpre- tations. Most notably, because structurally similar sentences are more likely to occur in succession than chance, it is possible that increased proba-

---

bility for same-structure target sentences reflects likely co-occurrence of distinct, associated repre- sentations, rather than a single, common, abstract representation (Ahn and Ferreira, 2023). While this is a much more viable explanation for mono- lingual than crosslingual priming. the presence of even limited code-switchie in training data could in principle lead to similar effects across languages.
4323, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Rami Al-Rfou. 2020. Gcld3: CLD3 is a neural network model for language identification.
Jordi Armengol-Estape, Casimiro Pio Carrino, Carlos Rodriguez- Penaagss Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, and Marta Villegas. 2021. Are Multilin- gual Models the Best Choice for Moderately Under- resourced Languages? A Comprehensive Assessment for Catalan, In Findings of the Association for Com- putational Linguisties: ACL-IJCNLP 2021, pages 4933-4946, Online. Association for Computational Linguisties.
Ethics Statement
Our work complies with the ACL Ethics Policy, and we believe that testing how well language models handle languages other than English is an important avenue of research to reduce the potential harms of language model applications. We did not train any models for this study; instead, we used the pre-trained XGLM (Lin et al., 2022) and PolyLM (Wei et al., 2023) families of models made available through the transformers Python package (Wolf et al., 2020). All analyses were run on an NVIDIA RTX A6000 GPU, running for a total of 4 hours.
Jordi Armengol- Estapé, Ona de Gibert Bonet, and Maite Melero. 2022. On the Multilingual Capabilittes of Very Large-Scale English Language Models. In Pro- ceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3056 3068, Marseille, France. European Language Resources Association.
Catherine Amett, Tyler A. Chang, James A. Michaclov, and Benjamin K. Bergen, 2023. Crosslingual Struc- tural Priming and the Pre-Training Dynamics of Bilin- gual Language Models.
Acknowledgements
Mikel Artetxe. Sebastian Ruder, and Dani Yogatama. 2020. On the Cross-lingual Transferability of Mono- lingual Representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticx, pages 4623 Online, Association for Computational Linguistics.
We would like to thank Sarah Bernolet, Kathryn Bock, Holly P. Branigan, Zhenguang G. Cai, Vasi- liki Chondrogianni, Zuzanna Fleischer, Robert J. Hartsuiker, Sotiria Kotzochampou, Helga Loe- bell, Janet F. McLean, Martin J. Pickering, Sofie Schoonbaert, and Eline Veltkamp for making their experimental stimuli available; and Nikitas Angele- tos Chrysaitis, Pamela D. Rivière Ruiz, Stephan Kaufhold, Quirine van Engen, Alexandra Taylor, Robert Slawinski, Felix J. Binder, Johanna Meyer, Tiffany Wu, Fiona Tang, Emily Xu, and Jason Tran for their assistance in preparing them for use in the present study. Models were evaluated using hard- ware provided by the NVIDIA Corporation as part of an NVIDIA Academic Hardware Grant. Tyler Chang is partially supported by the UCSD HDSI graduate fellowship.
Yoav Benjamini and Yosef Hochberg. 1995. Control- ling the False Discovery Rate: A Practical and Pow- erful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289-300
Sarah Bernolet, Robert J. Hartsuiker, and Martin J. Pick- ering. 2013. From language-speeific shared syntac- tic representations: The influence of second language proficiency syntactic sharing in bilinguals. Cogni- rion, 127())287-336.
Mireille Besson, Marta Kutas, and Cyma Van Petten. 1992. An Event-Related Potential (ERP) Analysis of Semantic Congruity and Repetition Effects in Sen- tences. Journal of Cognitive Neuroscience, 4(2):132- 149.
References
Terra Blevins and Luke Zettlemoyer. 2022. Language Contamination Helps Explains the Cross-lingual Ca- pabilities of English Pretrained Models. Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563-3574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics
Danbi Ahn and Victor S Ferreira. 2023. Shared vs sepa- rate structural representations: Evidence from cumu- lative cross- languaa structural priming. Quarterly Journal of Experimental Psychology.
Kabir Ahuja, Sunayama Sitaram, Sandipan Dandapat, and Monojit Choudhury. 2022. On the Calibration of Massively Multilingual Language Models. Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 4310-
J. Kathryn Bock. 1986. Syntactic persistence in lan- guage production. Cognitive Psychology, 18(3):355- 387.

---

Holly P. Branigan and Martin J. Pickering. 2017. An experimental approach to linguistic representation. Behavioral and Brain Sciences, 40se282.
Proceedings of 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Leamerr. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Alexis Conneau, Shijic Wu, Haoran Li, Luke Zettle- moyer, and Veselin Stoyanov. 2020b. Emerging Cross-lingual Structure in Pretrained Language Mod- els. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022 6034, Online. Association for Computational Linguisties.
Pablo Contreras Kallens, Ross Deans Kristensen- McLachlan, and Morten H. Christiansen. 2023. Large Language Models Demonstrate the Potential of Statistical Learning in Language. Cognitive Science 47(3)e13256.
Joan Bybee. 2010. Language, Usage and Cognition. Cambridge University Press, Cambridge.
Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffer- nan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault. Gabriel Mejia Gonzalez. Prangthip Hansanti. John Hoffman, Semarley Jarreu, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Saftyyah Saleem, Holger Schwenk, and Jeff Wang- 2022. No Language Left Behind: Scaling Human- Centered Machine Translation.
Zhenguang G. Cai, Martin Pickering. and Holly P. Branigan. 2012. Mapping concepts to syntax: Evi- dence from structural priming in Mandarin Chinese. Journal of Memory and Language 66(4):833-849.
Yuan Chai, Yaobo Liang, and Nan Duan. 2022. Cross- Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Lingwistics ( (Volume 1: Long Pa- pers), pages 4702-4712, Dublin, Ireland. Association for Computational Linguistics.
Tyler Chang, Zhuowen Tu, and Benjamin Bergen, 2022. The geometry of multilingual language model repre- sentations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 119-136, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Andrea Gregor de Varda and Marco Marelli. 2023. Data- driven Cross-lingual Syntax: An Agreement Study with Massively Multilingual Models. Computational Linguistics. pages 1-39.
Gary S. Dell and Victor S. Ferreira. 2016. Thirty years of structural priming: An introduction to the special issue. Journal of Memory and Language, 91:1-4.
Baoguo Chen, Yuefang Jia, Zhu Wang. Susan Dunlap, and Jeong-Ah Shin. 2013. Is word-order similar- ity necessary for cross-linguistic structural priming? Second Language Research, 29(4)3375-88,
Juuso Eronen, Michal Ptaszynski, and Fumito Masui. 2023. Zero-shot cross-lingual transfer language se- lection using linguistic similarity. Information Pro- cessing & Management, 60(3):003200.
Ethan A Chi, John Hewitt, and Christopher D Manning. 2020. Finding universal grammatical relations in multilingual bert. Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages $564-5577 Online. Association for Computational Linguistics.
Victor S. Ferreira and Kathryn Bock. 2006. The func- tions of structural priming. Language and Cognitive Processes, 21 1-11110111--099
Zuzanna Fleischer, Martin J. Pickering, and Janet F. McI.ean. 2012. Shared information structure: Evi- dence from cross- linguisic priming. Bilingualism: Language and Cognition, 15(3):668-577
Sunjoo Choi and Myung-Kwan Park. 2022. Syntac- tic priming in the L2 neural language model. The Journal of Linguistic Science, 103:81-104.
Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT Press, Cambridge, MA, USA.
Stefan Frank. 2021. Cross-language structural priming in recurrent neural network language models. Pro- ceedings of the Annual Meeting of the Cognitive Sci- ence Society, 43(43).
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020a. Unsupervised Cross-lingual Representation Learning at Scale. In
Daniela Gerz, Ivan Vulié, Edoardo Maria Ponti, Roi Re- ichart, and Anna Korhonen. 2018. On the Relation between Linguistic Typology and (Limitations of)

---

Multilingual Language Modeling. Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing, pages 316-327, Brussels, Belgium. Association for Computational Linguistics.
Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Sys- tems.
Adele Goldberg. 2006. Constructions at Work: The Nature of Generulization in Language. Oxford Uni- versity Press, Oxford, New York.
Nora Hollenstein, Federico Pirovano, Ce Zhang, Lena Jäger, and Lisa Beinborn. 2021. Multilingual lan- guage models predict human reading behavior. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies. pages 106-123, Online. Association for Computa- tional Linguistics.
Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Com- putational Linguistics (CMCL 2018), pages 10-18. Salt Lake City, Utah. Association for Computational Linguisties.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case Neural Text Degeneration. In International Conference on Learn- ing Representations.
Naman Goyal, Jingfei Du, Myle Ott, Giri Ananthara- man, and Alexis Conneau. 2021. Larger-Seale Trans- formers for Multilingual Masked Language Model- ing. In Proceedings of the 6th Workshop on Represen- tation Learning for NLP (RepL4NLP-2021] pages 29-33, Online. Association for Computational Lin- guistics.
Yufen Hsich. 2017. Structural priming during sentence comprehension in Chinese English bilinguals. Ap- plied Psycholinguistics, 38(3):657-678.
A Alexander Jones, William Yang Wang, and Kyle Ma- howald. 2021. A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 5833- 5847, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Raffaele Guarasci, Stefano Silvestri, Giuseppe De Pietro, Hamido Fujita, and Massimo Esposito. 2022. BERT syntactic transfer: A computational experiment on Italian, French and English languages. Computer Speech & Language, 71:101261.
Robert J. Hartsuiker, Martin J. Pickering, and Eline Veltkamp. 2004. Is Syntax Separate or Shared Be- tween Languages?: Cross-Linguistic Syntactic Prim- ing in Dpanish-English Bilinguals. Psychological Science, (56)):4094444
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguistics.
Petra Hendriks. 2014. Asymmetries between Language Production and Comprehension, volume 42 of Stud- ies in Theoretical Psycholinguistics. Springer Nether- lands, Dordrecht.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pa- pers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019. A Structural Probe for Finding Syntax in Word Repre- sentations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4129-49 4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Karthikeyan K. Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-Lingual Ability of Multilingual BERT: An Empirical Study, In International Confer- ence on Learning Representations.
Maren Heydel and Wayne S. Murray, 2000. Concep- tual Effects in Sentence Priming: A Cross- Linguuiine Perspective. In Marica De Vincenzi and Vincenzo Lombardo, editors, Cross-Lànguistic Perspectives on Language Processing, Studies in Theoretical Psy- cholinguistics, pages 227-254. Springer Netherlands, Dordrecht.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scout Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models.
Stav Klein and Reut Tsarfaty. 2020. Getting the ##life out of living: How Adequate Are Modelling Complex Morphology? In Proceedings of the 17th SIGMORPHION Workshop on Computa- rional Research in Phonetics, Phonology, and Mor- phology, pages 204-209, Online. Association for Computational Linguistics.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai. Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bog- dan Damoc, Aurelia Guy, Simon Osindero, Karen
for

---

Sotiria Kotzochampou and Vasiliki Chondrogianni. 2022. How similar are shared syntactic represen- tations? Evidence from priming of passives in Greek-English bilinguals. Bilingualism: Language and Cognition, (565)772-7738
James A. Michaelov, Seana Coulson, and Benjamin K. Bergen. 2022. So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements. IEEE Transac- lions on Cognitive and Developmental Systems.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Penny F. Mitchell, Sally Andrews, and Philip B. Ward. 1993. An event-related potential study of seman- tic congruity and repetition in a sentence-reading task: Effects of context change. Psychophysiology. 30(5):49--509
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai- ley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji. Khalid Almubarak, Samuel Al- banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual Generaliza- tion through Multitask Finetuning. Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15991-16111, Toronto, Canada, Association for Computational Linguistics.
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021.
Lower Perplexity is Not Always Human-Like. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5203- 5217. Online. Association for Computational Lin- guistics.
Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022. Neural reality of argument struc- ture constructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 7410-7423, Dublin, Ireland. Association for Computational Lin- guistics.
Merel Muylle, Sarah Bernolet, and Robert J. Hartsuiker. 2020. The Role of Case Marking and Word Order in Cross-Linguistic Structural Priming Late L2 Acquisition. Language Learning, 70($2):194-200
Antoine Nzeyimana and Andre Niyongabo Rubungo. 2022. KinyaBERT: A Morphology-awwe Kin- yarwanda Language Model. Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 5347-5363, Dublin, Ireland. Association for Compu- tational Linguistics.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Oit, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot Learning with Multilingual Generative Language Models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9019- 9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small Data? No Problem! Exploring the Viabil- ity of Pretrained Multilingual I Language Models for Low-resourced Languages. In Proceedings of the Ist Workshop on Mudtilingual Representation Learning, pages I 126.126 Punta Cana, Dominican Republic. Association for Computational Linguistics.
Helga Loebell and Kathryn Bock. 2003. Structural priming across languages. Linguistics, 41(5):791- 824.
Byung-Doh Oh and William Schuler. 2023. Why Does Surprisal From Larger Transformer-Basse Language Models Provide a Poorer Fit to Human Reading Times? Transactions of the Association for Com- putational Linguistics, 1 11:36-350.
Kyle Mahowald, Ariel James, Richard Futrell. and Ed- ward Gibson. 2016. A meta-analysis of syntactic priming in language production. Journal of Memory and Language, 91:5-27.
Akintunde Oladipo, Odunayo Ogundepo, Kelechi Ogueji, and Jimmy Lin. 2022, An Exploration of Vocabulary Size and Transfer Effects in Multilingual Language Models for African Languages. In 3rd Workshop on African Natural Language Processing.
Clara D. Martin, Francesca M. Branzi, and Moshe Bar. 2018. Prediction is Production: The missing link between language production and comprehension. Scientific Reports, 8(1):1079.
Chanjun Park, Sugycong Eo, Hyeonseok Moon, and Heuiseok Lim. 2021. Should we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Antje S. Meyer, Falk Huettig, and Willem J.M. Levelt. 2016. Same, different, or closely related: What is the relationship between language production and comprehension? Journal of Memory and Lunguage, 19:1-7.

---

Technologies:
Industry Papers, pages 97-104, Online, Association for Computational Linguistics.
Michael D. Rugg. 1990. Event-related brain poten- tials dissociate repetition effects of high-and low- frequency words. Memory & Cognition, 18(4):367- 379.
Martin J. Pickering and Victor Ferreira. 2008, Struc- tural priming: A critical review, Psychological Bul- letin, 134:427-459.
Sofie Schoonbaert, Robert J. Hartsuiker, and Martin J. Pickering. 2007. The representation of lexical and syntactic information in bilinguals: Evidence from syntactic priming. Journal of Memory and Language. 56(2):153-171
Martin J. Pickering and Simon Garrod. 2007. Do peo- ple use language production to make predictions dur- ing comprehension? Trends in Cognitive Sciences, 11(3):105-13)
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning, 2019. Do Massively Pretrained Language Models Make Better Storytellers? In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 843-861, Hong Kong. China. Asso- ciation for Computational Linguistics.
Martin J. Pickering and Simon Garrod. 2013. An inte- grated theory of language production and comprehen- sion. Behavioral and Brain Sciences, 36(4):329-347,
Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). pages 66-76, Hong Kong, China. Association for Computational Linguistics.
Amit Scker, Elron Bandel, Dan Bareket, Idan Brusilovsky. Refael Greenfeld, and Reut Tsarfaty. 2022. AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Papers), pages 46-56, Dublin, Ireland. Asso- ciation for Computational Linguistics.
Jack W. Rac, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini. Laurent Sifre. Lena Martens, Xiang Lorraine Li. Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Baptiste Lespiau. Maria Tsim- poukelli. Nikolai Grigorev. Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, 7hitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li. Tayfun Terzi. Vladimir Mikulik. Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukeuoglu, and Geoffrey Irving. 2022. Seal- ing Language Models: Methods, Analysis & Insights from Training Gopher.
Jeong-Ab -A Shin. 2010. Structural priming and L2 pro- ficiency effects on bilingual syntactic processing in production. Korean Journal of English Language and Linguistics, 10(3)499-511.
Jeong-Ah Shin and Kiel Christianson. 2009. Syntac- tic processing in Korean-Finglish bilingual produc- tion: Evidence from linguiing structural prim- ing. Cognition, 1 11551)175-1
Anna Siewierska. 1993. Syntactic weight informa- tion structure and word order variation in Polish. Journal of Linguisties, 29()):333222)33322
Arabella Sinclair, Jaap Jumelet, Willem Zuidema, and Raquel Fernández. 2022. Structural Persistence in Language Models: Priming as a Window into Ab- stract Language Representations. Transactions of the Association for Computational Linguistics, 10:1031- 1050.
Paul Soulos, Sudha Rao, Caitlin Smith, Eric Rosen, Asli Celikyilmaz, R. Thomas McCoy, Yichen Jiang, Coleman Halcy, Roland Fernandez, Hamid Palangi, Jianfeng Gao, and Paul Smolensky. 2021. Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages. In Proceed- ings of the 4th Workshop on Technologies for MT of Low Resource Languages (LoResMT2021), pages 52-67, Virtual, Association for Machine Translation in the Americas.
David Reitter, Frank Keller, and Johanna D. Moore. 2011. I. A Computational Cognitive Model of Syntac- tie Priming. Cognitive Science, 35(4)5587-637
Joost Rommers and Kara D. Federmeier. 2018. Pre- dictability's 's aftermath: Downstream consequences of word predictability as revealed by repetition effects. Cortex, 16-:16-30.
Michael Tomasello. 2003. Constructing a Language: A Usage-Based Theory of Language Acquisition. Har- vard University Press.
Michael D. Rugg. 1985. The Effects of Semantic Prim- ing and Word Repetition on Event-Related Potentials. Psychophysiology, 2246)664

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models.
Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Dimitra Irini Tzanidaki. 1995. Greek word order: to- wards new approach. UCL Working Paper in Lin- guistics, 7:247-277.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Eukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young. Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals. Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's Neural Machine Translation Sys- tem: Bridging the Gap between Human and Machine Translation.
Roger P. G. van Gompel and Manabu Arai. 2018. Struc- tural priming in bilinguals. Bilingualism: Language and Cognition, 21444448-455
Cyma Van Petten, Marta Kutas, Robert Kluender, Mark Mitchiner, and Heather Mclsaac. 1991. Fractionating the Word Repetition Effect with Event-Relates Poten- tials. Journal of Cognitive Neuroscience, 3(2))1131 150.
Elena Voita and Ivan Titov. 2020. Information- Theoretic Probing with Minimum Description Length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183-196, Online. Association for Computational Linguistics.
Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin. 2022. Oolong: Investigating What Makes Crosslin- gual Transfer Hard with Controlled Studies.
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation. Advances in Neural Information Pro- cessing Systems, 35:3082-3095.
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023. PolyLM: An Open Source Polyglot Large Language Model.
Linting Xue, Noah Constant, Adam Roherts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-toTTex. Transformer. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pages 483- 498, Online. Association for Computational Linguis- tics.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. CCNet: Ex- tracting High Quality Monolingual Datasets from Web Crawl Data. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 4003-4012 Marseille, France. European Language Resources Association.
Ethan Wilcox, Pranali Vani, and Roger Levy. 2021. targeted assessment of incremental processing in neu- ral language models and humans. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 9322, Online. Association for Computational Linguistics.
Jayden Ziegler, Rodrigo Morato, and Jesse Snedeker. 2019. Priming semantic structure in Brazilian Por- tuguese. Journal of Cultural Cognitive Science, 3(1):25-37.
A Language Contamination in Multilingual Language Models
Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro 2022. Cross- lingual Few-Shot Learning Unseen Languages. In Proceedings of the 2nd Conference of the Asia- Puccific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 777-791, Online only. Association for Computational Linguistics.
In this section, we estimate language contamina- tion in CC-100-XL the dataset used to train the XGLM models. While the dataset itself is not made available by Lin et al. (2022), the procedure used for language identification is similar to CC-100 (Conneau et al., 2020a: Wenzek et al., 2020).
While there are some differences in the ap- proaches used for filtering languages to ensure high- quality data, both corpora are based on Common- Crawl snapshots and are divided into languages
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz,

---

Dutch
Polish
Language ID Tool
Proportion
Estimated Tokens
Proportion
Estimated tokens
cld3
0.03051 %
152,528.079
0.00668%
33,418.112
Test
0.00212%
10,595,403
0.00157%
7,841,824
Consensus (cld3 fastText)
0.00035%
1,774,765
0.00029%
1,456,856
Table I: Estimated Dutch and Polish contamination in the training data XGLM 564M, 1.7B, 2.9B, and 7.5B,
based on language identification using cld3 and fastText, only considering tokens that both language identification models predict to be Dutch or Polish.
using the TastText language identification model (Joulin al., 2017). Both CC-100 and CC-100- XL also involve further language identification step. For CC-100, an unnamed internal tool is also used for language identification; for CC-100-XL. an additional step of language identification takes place where text language is also identified at the paragraph level.
divide paragraphs into sentences by splitting each period character, and we run each sentence through both the aforementioned latest version of the fastText language identification model (Costa- jusså et al., 2022; Joulin et al., 2017) and the cld3 language identifier (Xue et al., 2021) as provided in the gcld3 python package (AI-Rfou, 2020). We use stricter threshold of 0.9 (as recommended for high-resoures languages; Costa- fosta-jussà et al., 2022) for the former and use the default threshold of 0.7 for the latter.²
To test for Dutch and Polish contamination, sample roughly 100M tokens (based on the XGLM 7.5B tokenizer) of all languages in the replicated CC-100 dataset? that XLGM 564M, 1.7B, 2.9B, and 7.5B are trained on. We only consider lan- guages that have 100M or more tokens in CC-100 and that either use the Latin alphabet (Spanish, French, Italian, Portuguese, Finnish, Indonesian, Turkish, Vietnamese, Catalan, Estonian, Swahili, Basque), are Slavic (Russian, Bulgarian), or both (English, German). Specifically, we sample from each of these languages until we have enough doc- uments that the number of tokens in each language is at least 100M. Thus, our sample of CC-100 in- cludes roughly 1.6B tokens.
To estimate the total amount of contamination each of these languages, we calculate the propor- tion of each language sample that includes Dutch or Polish. We then multiply this by the number of tokens in each language, which we estimate by mul- tiplying the proportions given in Figure of Lin et al. (2022) by 500B, the total number of tokens. We first provide two estimates of contamination for Dutch and Polish Table 1: the amount of con- tamination as identified by the fastText language identification model, and the amount identified by cld3. We also provide a third, more conservative estimate, that only includes the tokens that both language identification models identify as either Dutch Polish. We note that because we only look at data from 16 of the 30 training languages, these numbers are likely to substantially underesti- mate the amount of language contamination in the XGLM pre-training data.
To replicate the additional filtering of CC-100 XL, we split all documents by paragraph and run language identification on them using the latest sion of the Tast language identification model released as part of the "No Language Left Behind" project (Costa-jussà et al., 2022). set the identi- fication threshold to 0.5, which the authors find to be effective for lower-resource languages (which some of our sampled languages are among). We note that this is newer and likely more accurate version of the language identification model than that used to create CC-100-XL, and thus is even less likely include data from languages other than those intended. We only analyze the data from paragraphs identified to be the same language as the document label.
B Statistical Tests
We provide the full results of the statistical tests for XGLM 4.5B (Table 2), the PolyLMs (Table 3), and the remaining XGLMs (Table 4).
To identify Dutch and Polish in these paragraphs,
SSee https://github http://gthuubbbcoggoggl///////// com/google//cdd//boob mester/sschmme__nnussedddddddd
and
Pht //tata//a/aaa//////eea http//daaa..tttttttt..mmm statmt.com wwwtttorg/cc-ie 100/

---

Language Model
Study
Language Pair
F
df
dfg
XGLM 4.5B
Bernolet (2013)
Duich-<Englis__Tae Target
151.98
144
<0.0001
Bernolet ct al. (2013)
English-+Dutch Target
24.00
141
<0.0001
Cai el. (2012) Experiment 1/2
Mandarin-Mandarty Target
192.37
24
<0.0001
Cai (2012) Experiment
Mandarin-Mandarin, Target
419.66
<0.0001
Fleischer et al. (2012)
English->Polise Target
1.35
31
0.2955
Fleischer et al. (2012)
Polish Polish->English Target
0.96
32
0.3704
Hartsuiker (2004)
English--pannihh Target
112
0.0056
Hartsuiker et al. (2004)
4.33
0.0558 112
Kotznchampou et al. (2022)
English-GGrrekk Target
7.28
24
0.0201
Kotzochampou (2022)
Greck-+Englis_TTnne Target
5.05
0.0485
Kotzochampou et al. (2022)
Greek~>Greck Target
8.40
24
0.0132
Lochell al. (2003)
English Target
0.13
0.7462 16
Lochell (2003)
German--Fnglish, Target
0.7647
Schoonbaert et al. (2007)
Dutch->Dutch_ Target
385.71
144
<0.0001
Schoonbaert et al. (2007)
Ouich-+English Target
57.28
144
<0.0001
Schoonbaert et (2007)
English-DDucc_
134.53
137
<0.0001
Table 2: Statistical tests of structural priming for XGLM 4.5B.
Language Model
Study
Language Pair
F
df,
df,
PolyI.M 1.7B
Beroleet (2013)
Dutch-+Fnglish, _arget
116.87
144
<0.0001
Bernolet et al. (2013)
English--Duthh_Tare
18.80
144
<0.0001
Cai al. (2012) Experiment 1/2
Mandarin-Mandarin. Target
164.45
24
<0,0001
Cai et al. (2012) Experiment 3
Mundarin-Maandrrn.. Target
228.25
<0.0001
Fleischer et al. (2012)
P Target
7.50
0.0165
Fleischer et al. (2012)
Polish Baanhh-Englhi_Taaee Target
7.34
32
0.0174
Hartsuiker al. (2004)
English-Spanish Target
2.47
112
0.1498
Hartsuiker et al. (2004)
Spanish--Englih__THT TTagget
1.76
112
0,2280
Kotzochampou et al. (2022)
Smelish--Greek_TTagee
0.13
24
0,7462
Kotzochampou (2022)
Greck-+English_ Target
0.13
0.7462
Kotzachampou et (2022)
Greck->Greck_aaaree
8.50
0.0128
Loebell al. (2003)
English-GGermn__Taaa Target
1.39
16
0,2955
Loebell (2003)
Genlan--English Target
2.66
16
0.1525
Schoonbaert et al. (2007)
Dutch-+Dutch Target
105.51
144
<0.0001
Schoonbaert et al. (2007)
Dutch- Target
55.84
144
<0.0001
Schoonbaert al. (2007)
English->Dutee Target
140.97
<0.0001 144
PolyLM 13B
Bernolet et al. (2013)
Dutch- Target
193.43
144
<0.0001
Bernolet al. (2013)
English-PDutch Target
16.73
0.0002 144
et al. (2012) Experiment
Mandario-Mandarinn. Target
141.67
<0.0001
Cai et (2012) Experiment 3
Mundarin-Manndrin.. Target
257.28
32
<0.0001
Fleischer et al. (2012)
Bnglish-PPolish__
2.45
32
0.1570
Fleiseher al. (2012)
Polish-Faglish_TTR Target
0.29
0.6275
Hartsuiker et al. (2004)
English-Spannih_TBaae Target
21.87
112
<0.0001
Hartsuiker et al. (2004)
Spanish Target
41.60
112
<0.0001
Kotzochampou (2022)
English-GGreek Target
0.70
0.4481
Kotzochampou et al. (2022)
Greek Cneek->English_TTaree
0,54
24
0,5062
Kotzochampou et (2022)
Greek->Greek Target
9.03
24
0.0106
Loebell (2003)
English-~German, Target
5.36
0.0485
Loebell et al. (2003)
German--Engliih_Taaae
1.51
16
0.2794
Schoonbaert et al. (2007)
Duich-tDutch_arrg
260.25
144
<0.0001
Schoonbaert al. (2007)
Dutch-+English, Target
129.76
144
<0.0001
Schoonbaert et al. (2007)
Fnglish-DDutch,
58.52
144
<0.0001
Table 3: Statistical tests of structural priming for PolyLM 1.7B and 13B.

---

Language Model
Study
Language Pair
df.
dfg
P
XGLM 564M
Bernolet ct al. (2013)
-utch-+English Target
12.89
144
0.0010
Bemolet ct al. (2013)
Fnglish++Dutch_ae Target
16.39
144
0.0002
Cai ct al. (2012) Experiment 1/2
Mandarin--Mundarin Target
301.39
24
<0.0001
Cai ct al. (2012) Experiment 3
Mandarin-+Mandarin, Taare
1006.36
32
<0.0001
Fleischer et al. (2012)
English-PPolish_Tare -arget
1.05
32
0.3497
Fleischer et al. (2012)
Soiish--Ennliih_TTaggee
10.30
32
0.0056
Hartsuiker et al. (2004)
Euglish-tSpanish Target
0.51
112
0.5076
Hartsuiker et al. (2004)
Spunish--English Target
4.72
112
0.0471
Kotzuchampou al. (2022)
English-GGreek_
5.90
24
0.0352
Kotzochampou al. (2022)
Greck->English Target
11.25
24
0.0051
Kotzochampou al. (2022)
Greek~>Greekk_en Target
10,80
24
0.0056
Loebell et al. (2003)
English-+German Target
Target
3.65
16
0.1001
Lochell et al. (2003)
German-FFnglih_Taaae Target
2.76
16
O. 0.1494
Schoonbaert al. (2007)
Dutch-sDuthhTaarel
545.14
144
<0.0001
Schoonbaert al. (2007)
Ducch>FFnglish_Taap
5.66
144
0.0291
Schoonbaert al. (2007)
English--Duuch_Tange
$5.69
144
<0,0001
XGLM 1.7B
Remolet al. (2013)
Dutch->FnglishTTa Target
17.64
144
0.0001
Bemolet al. (2013)
English-:Dutch_ Target
32.57
144
<0.0001
Cai (2012) Experiment 1/2
Mandarin-=Mandarin, Target
751.15
<0.0001
Cai et (2012) Experiment
Mandaain--Maandarin,_ Targer
1519.71
<0.0001
Fleischer (2012)
English-=Polis_TTare
0.7761
Fleischer et (2012)
Polish-iEnglish_Taare
0.4481
Hartsuiker (2004)
English--Spanish Target
112
0.0467
Hartsuiker (2004)
Spanish-tEnglish Target
3.19
112
0.1026
Kotzochampou
(2022)
English- Target
0.1502
Kotzochampou al. (2022)
Greek->FnglihhTTa TTagge
11.20
0.0051
Kotzochampou et al. (2022)
Greek->Greek_Targee
18.49
0.0005
Locbell et al, (2003)
English-+German Target
1.80
0,2358
Loebell et al. (2003)
Dermann-Enllis_TTarge
3.13
O. 1247
Schoonbaerl et al. (2007)
Dutch-;Dutch Tarch
312,38
144
<0.0001
Schoonbaert al. (2007)
Dutch-Engglish_TTe
3.72
144
0.0770
Schoonbaert et al. (2007)
English-+Dutch Target
55.88
134
<0.0001
XGLM 2.9B
Bernolet et al. (2013)
Dutch-Eogglihh_argg
47.12
144
<0,0001
Berolet ct al. (2013)
Aiglish-DDuchTTargee
27.25
144
<0.0001
Cai al. (2012) Experiment 1/2
Mandaain--Manllrioonnna Target
427.12
24
<0.0001
Cai al. (2012) Experiment 3
Mandarin--Mandaria Target
1363.62
32
<0.0001
Fleischer et al. (2012)
Eoglish- Target
1.31
32
0.2988
Fleischer et al. (2012)
Target
12.11
0.0031 32
Hartsuiker al. (2014)
Fnglish-sSpamish Target
4.61
112 0.0489
Hartsuiker et al. (2004)
Spanish-HEnglish Target
10.42
112
0.0033
Kotzochampou al. (2022)
Fnglish-<Greek__aage _arger
3.58
0.0966 24
Kotzochampou al. (2022)
Greck-EEnglihh_Taage
12.26
24
0.0036
Kotzochampou al. (2022)
Greok->Girek_Tarre Target
16.05
0.0011 24
Loebell et al. (2003)
English-rGerman__Tan Target
6.22
16
0.0362
Loebell et al. (2003)
German--nglish Target
1.11
0.3485
Schoonbaert al. (2007)
Dutch->Dutch_ Darget
327.66
144
<0.0001
Schoonbaert (2007)
Dutch->English Target
21.01
144
<0.0001
Schoonhaert (2007)
English-:Dutch Target
90.89
<0.0001 144
XGLM 7.5
Beroolet et al. (2013)
Butch->Engiist Target
37.88
144
<0.0001
Bernolet et al. (2013)
English-tDutch_ Taage
21.46
144
<0.0001
Cai et (2012) Experiment 1/2
Mandarin--Manddrin Target
402.46
<0.0001
Cai et (2012) Experiment
Mandarin--Mundarin Target
1193.10
<0.0001
Fleischer et al. (2012)
English-PPolish Target
0.7761
Fleischer et al. (2012)
Dolish--inglihhhTargg
8.96
0.0093
Hartsuiker et al. (2004)
English--SSamish_TTagg
16.41
112
0.0002
Hartsuiker et al. (2004)
Spanish-FFnglihh_Taagg
17.28
112
0.0002
Kotzochampou et al. (2022)
English-rGreek_TTage
3.10
24
0202
Kotzochampou et al. (2022)
12,33
24
0.0036
Kotzochampou et al. (2022)
Greek=>Greek_Targe
9.47
24
0.0092
Loebell et al. (2003)
Euglish-GGerman Target
1.86
16
0,2310
Locbell et al. (2003)
-Englh- Target
6.84
16
0.0291
Schoonbaert et al. (2007)
Dutch-;Dutch Target
402.81
144
<0.0001
Schoonhaert et al. (2007)
Dutch->English_T __argetrrtctt
43.84
144
<0.0001
Schoonbaert al. (2007)
Bnutish-DDuch_Taaage
83.09
144
<0.0001
Table Statistical tests of structural priming for XGLM 564M, 1.7B. 2.9B, and 7.5B.


Reviews:
[{"review_id": "7bEAJYkA8P", "reviewer": "Reviewer_HSdV", "summary": "", "questions": "- Can you elaborate on the advantage of (crosslingual) structural priming over various forms of probing to investigate abstract linguistic representations of neural language models?\n- Have you compared monolingual and multilingual language models against monolingual structural priming paradigms to test whether multilingual training reinforces abstract linguistic representations of multilingual language models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address specific comments below:\n\n*\"This paper seems to over-interpret the results as 'reinforc[ing] the claims of usage-based (i.e. functional) accounts of language acquisition'.\"*\n\nWe thank the reviewer for alerting us to the possible overly-broad interpretation of this statement. By mentioning that the results 'reinforce the claims of usage-based (i.e. functional) accounts of language acquisition', we simply mean that they demonstrate the in-principle viability of learning the kinds of abstract linguistic representations that are sensitive to structural priming using the statistics of language alone. We will clarify this in the revised version of the paper.\n\n*\"Can you elaborate on the advantage of (crosslingual) structural priming over various forms of probing to investigate abstract linguistic representations of neural language models?\"*\n\nAs noted only briefly on L130-142, one limitation of probing studies is that they often do not demonstrate causal effects of representations on model outputs. Even if a model represents similar structures across languages similarly, the model may not necessarily use the representations in functionally the same way across languages. In some cases, the representational similarity may be something learned by the probe but not the models themselves (Belinkov, 2022: Probing Classifiers: Promises, Shortcomings, and Advances). Multilingual structural priming demonstrates higher-order abstractions across languages with causal effects on model outputs, and it does not require access to internal model states or any assumptions about model architecture. \n\nIn addition, one limitation of using monolingual structural priming to assess language models is that because humans demonstrate structural priming, language models may learn associations between specific words such that they appear to show effects of structural priming without learning any abstract patterns. By using crosslinguistic structural priming, we mitigate this effect for language models that are not trained on large amounts of data involving code-switching. Discussion of this point will also be included in the revised draft.\n\n*\"Have you compared monolingual and multilingual language models against monolingual structural priming paradigms to test whether multilingual training reinforces abstract linguistic representations of multilingual language models?\"*\n\nWe agree that this is an interesting question. However, because current pretrained monolingual and multilingual models are not trained on the same datasets (or even on datasets of the same size), and because there is evidence that ostensibly 'monolingual' pretrained language models may often be exposed to large amounts of data in other languages (Blevins and Zettlemoyer, 2022: Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models), this would require training our own models to run these analyses, and thus is beyond the scope of the present study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper comprehensively investigates abstract linguistic representations of multilingual language models via structural priming based on 12 (8 crosslingual + 4 monolingual) structural priming paradigms in total.", "reasons_to_reject": "- This paper seems to over-interpret the results as \"reinforc[ing] the claims of usage-based (i.e. functional) accounts of language acquisition\".", "questions_to_authors": "- Can you elaborate on the advantage of (crosslingual) structural priming over various forms of probing to investigate abstract linguistic representations of neural language models?\n- Have you compared monolingual and multilingual language models against monolingual structural priming paradigms to test whether multilingual training reinforces abstract linguistic representations of multilingual language models?", "paper_topic_and_main_contributions": "The main contribution of this paper is to demonstrate via crosslingual structural priming that multilingual language models have abstract linguistic representations within and across languages.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cTwqY6SiL5", "reviewer": "Reviewer_1oDq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Specific comments are addressed below:\n\n*\"My concern with this paper is whether the effect on probability given the intervention is sufficient evidence to conclude that LMs have abstract grammatical representations (AGRs). The majority of the experiments where the effects or structural priming are seen are between relatively closely related languages, where similar grammatical structures might result in surface-level similarities, like word order, which may account for the effects seen in the experiments. The phenomenon discussed in the paragraphs at L509 and L529 would seem to support this hypothesis. In my view, this paper would be stronger if there was more discussion of alternative hypotheses to AGRs that would explain their results and perhaps reasons to reject them.\"*\n\nWe thank the reviewer for noting this concern, and we agree that a revision should address alternate hypotheses and the nuanced interpretation of results more directly. We note that while many of the language pairs studied are related, there is no overlap in the words or morphemes used to construct each structure across any pair of languages. From this we can infer that even if a given word order primes a homologous word order in a related language, this still demonstrates abstraction at least at the level of word order. Whether word order can be considered to be a form of abstract grammatical representation is hotly debated. For instance, under theories of the cognitive representation of language such as that proposed by Branigan and Pickering (2017), abstract grammatical representations are precisely defined as those  structures that can be structurally primed (such as specific word orders). Other theories disagree. We will clarify the implications of the study's findings for different definitions of abstract grammatical representations.\n\n*\"'We include a random intercept for experimental item' does not make sense to me.\"*\n\nBy this we mean that the linear mixed-effects regression includes a random intercept for each experimental item. This accounts for effects of individual experimental items, independent of prime condition.\n\n*\"'DO (double object)' should be dative I believe.\"*\n\nIt is indeed true that the DO is a dative, but in this case we are distinguishing between two specific kinds of dative: the double object (DO) dative and the prepositional object (PO) dative.\n\n*\"Sections 4.{1,2}.\\* should have been a table in my opinion. It reads as highly repetitive.\"*\n\nWe thank the reviewer for this suggestion, and will consider this for the next version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has a compelling experimental design, and generally I am excited about comparing results from human linguistic experiments to results from the same experiments on LMs.", "reasons_to_reject": "My concern with this paper is whether the effect on probability given the intervention is sufficient evidence to conclude that LMs have abstract grammatical representations (AGRs). The majority of the experiments where the effects or structural priming are seen are between relatively closely related languages, where similar grammatical structures might result in surface-level similarities, like word order, which may account for the effects seen in the experiments. The phenomenon discussed in the paragraphs at L509 and L529 would seem to support this hypothesis. In my view, this paper would be stronger if there was more discussion of alternative hypotheses to AGRs that would explain their results and perhaps reasons to reject them.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors apply methods from human structural priming studies to language models (LMs) and find that models that are primed with a grammatical structure favor (in terms of probability) utterances with the same structure. They demonstrate this effect even when the priming example and target utterance are in different languages. The authors present this as evidence that these models have abstract grammatical representations.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "228: \"We include a random intercept for experimental item\" does not make sense to me.\n\n253: \"DO (*double* object)\" should be *dative* I believe.\n\nSections `4.{1,2}.*` should have been a table in my opinion. It reads as highly repetitive."}}, {"review_id": "ucXw1EdzZl", "reviewer": "Reviewer_ACY8", "summary": "", "questions": "It is not entirely clear to me how the Priming Effect is being computed. You state around L197 that in psycholinguistic studies researchers often investigate the proportion of sentences of a particular structure being **produced**. The distinction between production and comprehension is important within priming, and when transferring these paradigms to LLMs it is important to be clear about what is being investigated. \n\nIn your case you are not letting the models generate a target open-endedly, but you only compare the probabilities of the two structural alterations. I suggest you address this more explicitly; having a concrete example of your setup in Section 3.2 would improve the clarity.\n\n------\n\nContrary to Sinclair et al. (2022), you compare model probabilities of two different *target* sentences. In Sinclair et al., the authors purposely opted to only alternate the *prime* sentences, and compare the probability of the same target sentence. This ensures that length differences between alternating targets play role, or prior probabilities needing to be corrected for (as in Prasad et al. (2019), see also Fig. 1 in Sinclair et al. (2022)). From the paragraph around L224 it is not entirely clear to me if your mixed-effects model fully addresses these concerns: could these factors play a role in your results?\n\n------\n\nShould I interpret your priming effect measure as 0.5 having no significant impact at all? For example, both PO/DO yielding the same probability, as explained in the paragraph around L197?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address specific comments below:\n\n*\"It is not entirely clear to me how the Priming Effect is being computed. You state around L197 that in psycholinguistic studies researchers often investigate the proportion of sentences of a particular structure being produced. The distinction between production and comprehension is important within priming, and when transferring these paradigms to LLMs it is important to be clear about what is being investigated.\nIn your case you are not letting the models generate a target open-endedly, but you only compare the probabilities of the two structural alterations. I suggest you address this more explicitly; having a concrete example of your setup in Section 3.2 would improve the clarity.\"*\n\n*\"As stated in one of my questions in the field above, having a concrete example of a model + prime/target + model probabilities + final priming effect would help a lot with interpreting the experimental results.\"*\n\nWe thank the reviewer for noting that the exact way in which the priming effect is calculated is not clearly described in the submitted version of the manuscript, and agree that including this will help to clear up some of the methodological details and results. We will include a concrete example of how the priming effect is calculated in the updated manuscript. A sketch of this is given below:\n\nConsider the following two primes and the following two targets:\n\nDO prime: The nun shows the prisoner a hat.    \nPO prime: The nun shows a hat to the prisoner.     \nDO target: The chef gives the swimmer a hat.    \nPO target: The chef gives a hat to the swimmer.\n\nWe can then use language models to calculate the probability of each target following each prime. As an example, the language models can provide probabilities as follows:\n\nP(PO Target | DO Prime) = 0.30    \nP(DO Target | DO Prime) = 0.20    \nP(PO Target | PO Prime) = 0.40    \nP(DO Target | PO Prime) = 0.10\n\nWe then normalize the probabilities of each target given its prime (i.e. restrict the probabilities to the two possible targets). In the submitted version of the paper, these normalized probabilities were referred to as 'relative probabilities'. In the final version of the paper, we will change 'relative probabilities' to 'normalized probabilities'. \n\nNormalized P(PO Target | DO Prime) = 0.60    \nNormalized P(DO Target | DO Prime) = 0.40    \nNormalized P(PO Target | PO Prime) = 0.80    \nNormalized P(DO Target | PO Prime) = 0.20\n\nBecause the probabilities of the two targets following a given prime now sum to one, we only need to consider the probabilities for one target type (e.g. PO Targets). We then run our analyses by comparing the normalized probabilities for that target type (similar to Sinclair (2022)). For example, if we were to compare the relative probabilities of PO Targets, we would compare the following two probabilities:\n\nNormalized P(PO Target | DO Prime) = 0.60    \nNormalized P(PO Target | PO Prime) = 0.80\n\nIn this case, we see a clear structural priming effect—the presence of the PO prime increases the relative probability of a PO Target by 0.20. Our statistical models take multiple such instances to see whether this is a general trend across stimuli. Because they are symmetric, we only calculate the priming effect for one target type (e.g. the PO Target above), following the lead of the original study in determining which target type to use to calculate the effect. It is the normalized probabilities of these targets that are plotted in the graphs in the paper.\n\n*\"Contrary to Sinclair et al. (2022), you compare model probabilities of two different target sentences. In Sinclair et al., the authors purposely opted to only alternate the prime sentences, and compare the probability of the same target sentence. This ensures that length differences between alternating targets play role, or prior probabilities needing to be corrected for (as in Prasad et al. (2019), see also Fig. 1 in Sinclair et al. (2022)). From the paragraph around L224 it is not entirely clear to me if your mixed-effects model fully addresses these concerns: could these factors play a role in your results?\"*\n\nIt is indeed true that the prior probabilities might be biased towards one structure (e.g. the PO target is always more probable than the DO target in the example above, regardless of prime type). We hope that the description above has clarified how we account for this—like Sinclair et al. (2022), our priming effect is based on the *difference* between the probability of a target given each prime. The main difference is that to make our results more comparable with many structural priming studies where the proportions of each target response are compared, we use the difference between the normalized probabilities to operationalize the effect rather than the difference between the un-normalized log-transformed probabilities.\n\n*\"Should I interpret your priming effect measure as 0.5 having no significant impact at all? For example, both PO/DO yielding the same probability, as explained in the paragraph around L197?\"*\n\nWe hope that the previous description has helped to make our priming effect measure clearer. To clarify: if a PO and DO target have the same probability given a specific prime (e.g., a PO prime), then their normalized probability will each be 0.5. In this case, the priming effect will be calculated as the difference between this and the normalized probability of the same target given a DO prime. This is different from a priming effect of 0.5, which would mean a 0.5 difference between the normalized probability of a target given each prime (e.g. the difference between the normalized probability of the PO target given a PO prime and the normalized probability of the PO target given a DO prime).\n\n*\"Some typological investigations into shared, abstract language representations in MLMs could be added, for example:\nInvestigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology -- Choenni & Shutova (2022)\"*\n\nWe thank the reviewer for bringing this paper to our attention. We will include a brief discussion about possible effects of linguistic typology on shared abstract grammatical representations in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a highly interesting addition to the priming paradigm within LLM interpretability research, with the introduction of a multilingual approach that has played a major role within psycholinguistics research.\n- The paper engages deeply with the existing psycholinguistics literature on the subject, which results in a rich and broad experimental setup tackling a wide range of languages.", "reasons_to_reject": "- The procedure for measuring priming effect is not explained in enough detail, and could raise some concerns (see my Question below).", "questions_to_authors": "It is not entirely clear to me how the Priming Effect is being computed. You state around L197 that in psycholinguistic studies researchers often investigate the proportion of sentences of a particular structure being **produced**. The distinction between production and comprehension is important within priming, and when transferring these paradigms to LLMs it is important to be clear about what is being investigated. \n\nIn your case you are not letting the models generate a target open-endedly, but you only compare the probabilities of the two structural alterations. I suggest you address this more explicitly; having a concrete example of your setup in Section 3.2 would improve the clarity.\n\n------\n\nContrary to Sinclair et al. (2022), you compare model probabilities of two different *target* sentences. In Sinclair et al., the authors purposely opted to only alternate the *prime* sentences, and compare the probability of the same target sentence. This ensures that length differences between alternating targets play role, or prior probabilities needing to be corrected for (as in Prasad et al. (2019), see also Fig. 1 in Sinclair et al. (2022)). From the paragraph around L224 it is not entirely clear to me if your mixed-effects model fully addresses these concerns: could these factors play a role in your results?\n\n------\n\nShould I interpret your priming effect measure as 0.5 having no significant impact at all? For example, both PO/DO yielding the same probability, as explained in the paragraph around L197?", "paper_topic_and_main_contributions": "The paper investigates *structural priming* in the context of multi-lingual language models. Structural priming is an important paradigm within psycholinguistics, which over the past 4 decades has been one of the main tools for uncovering the presence of abstract, linguistic structure within human language processing.\n\nRecently, structural priming has been deployed as a tool for uncovering linguistic structure within language model representations. The authors build on this work, and introduce a cross-lingual experimental setup in which a language model is primed in 1 language, and tested in another language. The authors pose that this setup is able to uncover abstract structure even better than in a monolingual setting, since the caveat of token overlap between prime- and target sentences is no longer present.\n\nThe authors leverage a wide range of cross-lingual experiments from prior psycholinguistic work, and find that the MLM they consider (XGLM) generally follows similar patterns as have been reported for humans.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some typological investigations into shared, abstract language representations in MLMs could be added, for example:\n- *Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology* -- Choenni & Shutova (2022)", "Typos_Grammar_Style_and_Presentation_Improvements": "Very well written!\n\nAs stated in one of my questions in the field above, having a concrete example of a model + prime/target + model probabilities + final priming effect would help a lot with interpreting the experimental results."}}]

Author rebuttal:

Meta-review: The paper applies structural priming, a method in psycholinguistics for studying the presence of linguistic structure in human language processing, to multilingual language models. A cross-lingual setup for priming is followed, testing multiple grammatical alterations. The results show that language model behaviour correlates with those of human subjects. The reviewers expressed some concerns about an overly broad interpretation of the results, however the study still makes a valuable contribution to the question studied.
Decision: Accept-Main