Emergent Word Order Universals from Cognitively-Motivated Language Models (2024)

Abstract: The world’s languages exhibit certain so-called typological or implicational universals; for example, Subject-Object-Verb (SOV) languages typically use postpositions. Explaining the source of such biases is a key goal of linguistics. We study word-order universals through a computational simulation with language models (LMs). Our experiments show that typologically-typical word orders tend to have lower perplexity estimated by LMs with cognitively plausible biases: syntactic biases, specific parsing strategies, and memory limitations. This suggests that the interplay of cognitive biases and predictability (perplexity) can explain many aspects of word-order universals. It also showcases the advantage of cognitively-motivated LMs, typically employed in cognitive modeling, in the simulation of language universals. https://github.com/kuribayashi4/ word-order-universals-cogLM

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14522–14543 August 11-16, 2024 ©2024 Association for Computational Linguistics
https://github.com/kuribayashi4/ word-order-universals-cogLM

heading: 1 Introduction
text: There are thousands of attested languages, but they exhibit certain universal tendencies in their design. For example, Subject-Object-Verb (SOV) word order often combines with postpositions, while SVO order typically employs prepositions (Greenberg et al., 1963). Researchers have argued that such implicational universals are not arbitrary but shaped by their advantage for humans (Hawkins, 2004; Culbertson et al., 2012, 2020).
Such language universals have been recently studied through neural-based computational simulation to elucidate the mechanisms behind the universals (Lian et al., 2023). The languages which emerge, however, have typically not been humanlike (Chaabouni et al., 2019a,b; Rita et al., 2022; Ueda et al., 2022). Such mismatch arguably stems from the lack of human-like cognitive biases in neural agents (Galke et al., 2022), but injecting
cognitive biases into systems and showing their benefits has proved challenging (Lian et al., 2021).
In this study, expanding on a study of wordorder biases in language models (LMs: White and Cotterell (2021)), we demonstrate the advantage of cognitively-motivated LMs, which can simulate human cognitive load during sentence processing well (Hale et al., 2018b; Futrell et al., 2020a; Yoshida et al., 2021; Kuribayashi et al., 2022), and thus predict many implicational word-order universals in terms of their inductive biases (Figure 1). Specifically, we train various types of LMs in artificial languages with different word-order configurations (§3). Our experiments show that perplexities estimated by cognitively-motivated LMs (§5) correlate better with frequent word-order configurations in attested languages than standard LMs (§6). This confirms that such biases are a potential source of the word-order universals as well as demonstrate the plausibility of cognitively-motivated LMs as models of human language processing.
14522

heading: 2 Related research

heading: 2.1 Impossible languages and LMs
text: Generative linguistic theory has focused on delineating the impossible from possible languages in terms of universal grammar. Chomsky et al. (2023) has recently argued that neural LMs cannot distinguish possible human languages from impossible, unnatural languages, based on the experiments by Mitchell and Bowers (2020), and are, therefore, of no interest to linguistic theory. Kallini et al. (2024) challenge this claim, demonstrating that a standard transformer-based autoregressive model (GPT-2) assigns higher perplexity and greater surprisal to a range of artificially-generated, unattested, and thus putatively impossible candidate languages when compared to English. In this work, by contrast, we explore the ability of a variety of neural LMs to distinguish typologically rare combinations of word orders from the common attested combinations as predicted by Greenberg’s implicational universals (Greenberg et al., 1963).

heading: 2.2 The Chomsky hierarchy and LMs
text: We test how easy it is to learn a specific artificial language (with a specific word-order configuration) for certain LMs. Such exploration is related to the investigation of the capabilities of neural LMs to generate formal, artificial languages in a specific class of the Chomsky hierarchy, such as irreducibly context-free (e.g., the Dyck languages) or mildly context-sensitive (e.g., anbncn) languages (Weiss et al., 2018; Suzgun et al., 2019; Hewitt et al., 2020; Deletang et al., 2022). While this line of research can elucidate whether specific models (LSTMs, transformers, etc.) are capable in principle of expressing and generalizing such languages, and thus also generating their putative analogs in natural language, in this work we focus on artificial languages which are more human language-like in that they exhibit a range of attested construction types, a more realistic vocabulary, and are less marked in terms of features like average sentence length, at least compared to the formal languages adopted in this line of research (App. A).

heading: 2.3 Word order preferences of LMs
text: Researchers have asked what kind of language is hard to language-model (Cotterell et al., 2018; Mielke et al., 2019), motivated by concern over whether the current language-modeling paradigm is
equally suitable for all languages. However, experiments using only attested language corpora made it difficult to single out impactful factors since they differ from each other in multiple dimensions (Cotterell et al., 2018; Mielke et al., 2019). Thus, prior studies adopted the use of artificially controlled language(-like) data as a lens to quantify the inductive bias of models (Wang and Eisner, 2016; White and Cotterell, 2021; Hopkins, 2022). Specifically, White and Cotterell (2021) pointed out some differences between LM’s word-order preferences and common attested word orders (typological markedness). We expand on their research by exploring which models, including cognitively-motivated ones, exhibit preferences more aligned with common typological patterns (Figure 1).

heading: 2.4 Cognitively-motivated LMs
text: Cognitively-motivated LMs is a reference to a particular class of language model, typically examined and preferred in the cognitive modeling field. In brief, in the cognitive modeling field, the nextword probability and human reading behavior are compared (Beinborn and Hollenstein, 2023), based on surprisal theory (Levy, 2008; Smith and Levy, 2013), to reverse-engineer what humans compute during language processing. Cognitively-motivated LMs typically yield a better fit to human behaviors than standard LMs. In Section 5.2, we specifically focused on three aspects which are well established in the cognitive modeling literature: (i) syntactic bias (Hale et al., 2018b), (ii) parsing strategy (Resnik, 1992; Yoshida et al., 2021), and (iii) memory limitations (Futrell et al., 2020a; Kuribayashi et al., 2022). Our interest is that if language design is shaped by cognitive biases, such cognitively-motivated LMs should simulate the typological distribution of natural languages.

heading: 3 Preliminary
text: We explain the assumptions behind this research.

heading: 3.1 Need for computational simulation
text: This research aims to determine whether/how attested word order universals can stem from cognitive biases underlying human language processing. Perhaps the strongest evidence would be obtained through counterfactual ablations. That is, researchers should compare word orders that emerged in different, controlled scenarios: one emerged in our world with humans, and the other
one emerged in the same world except for having non-human agents. If the attested word order universals can emerge only in the scenario with humans, this can be evidence that the attested word order universal is related to human-like cognitive biases. Nevertheless, unfortunately, such a controlled ablation is not feasible in the real world. Instead, this study adopts computational simulation to explore the source of word order universals. That is, we virtually simulate several scenarios, each involving neural agents with different inductive biases, including human-like ones, and investigate under which biases our world’s word order universals can emerge.

heading: 3.2 Likning hypothesis
text: We further explain the assumptions underlying our computational simulations of the emergence of word order universals.
Assumption 1: Given the theory that language has evolved to promote its processing efficiency (Hawkins, 2004; Gibson et al., 2019), we posit that word orders frequently adopted in the world should be easily learned for human-like agents, and then, they can process it efficiently. That is, we assume that the frequency of word order o within the attested languages should correlate with its (negative) learning/processing cost:
frequency(o) ∝ −cost(o) . (1)
See §4.1 for word order configurations o. See §4.2 for the frequency(o) term. See §4.3 and assumption 2 for the cost(o) term. See §4.4 for the correlation metrics ∝.
Assumption 2: We further posit that humans continuously predict the upcoming word during language processing, based on expectation-based theory (Levy, 2008; Smith and Levy, 2013). That is, we assume that the costθ(o) for a particular model θ is determined by how the word order o is difficult to learn and then incurs worse predictability through next-word prediction. Such a processing difficulty is measured as the average processing cost required to process sentences with word order o for a particular model θ trained over a certain period of time of language acquisition (language modeling). This can be quantified by perplexity (PPL),1 the geometric mean of word predictability,
1Using the average surprisal − 1|xo| ∑ i log p(x o i |xo<i) (entropy) instead of PPL is more aligned with surprisal the-
of a corpus with tokens xo following word order o under a learner θ:
costθ(o) ∼ ∏
xoi∈xo pθ(x
o i |xo<i)
− 1|xo| , (2)
:= PPLθ(x o) . (3)
Here, pθ(xoi |xp<i) is a probability of the i-th word in context for a model θ that is trained on a corpus with word order o. We analyze which LM θ computes PPLθ(wo) that well correlates with its typological frequency as expected in Eq.1.
See §5 for LMs θ we examine. See §6.1 and §6.2 for main results. See §6.3 for other variants of PPLs. See §8 for the connection to the bi-dimensional view of communicative efficiency.

heading: 4 Problem settings

heading: 4.1 Word-order configurations
text: Branching directionality ∈ {L, R}, the concept of whether a dependent phrase is positioned left (L) or right (R) of its head in a particular constituent, is a key component of the typological theory. We define six word-order parameters to classify attested languages as shown in Table 1. For example, the parameter S determines the order of the subject NP and the VP. Then, the combination of word order parameter assignments defines a word order configuration o = ( S , VP , PP , NP , Rel , Case ) ∈ O := {L, R}6, denoted by a sequence of L/R. For example, LRLLLR ∈ O is the configuration where all phrases, except for VP and Case, are left-branching. Such parameter combinations result in 26 = 64 word order configurations.

heading: 4.2 Frequencies of word order
text: The 64 word-order configurations are not uniformly distributed among attested languages (blue points in Figure 2). This distribution is estimated by the frequency of word orders in The World Atlas of Language Structures (WALS: Dryer and Haspelmath (2013)),2 which is also denoted as a vector f =
ory (Smith and Levy, 2013), but such a logarithmic conversion did not alter our findings (§7). Thus, we use PPL, following White and Cotterell (2021).
2We used the word order statistics of 1,616 languages, out of 2,679, where at least one parameter is annotated. If a particular parameter is missing or non-binary (X), one count
[freq(LLLLLL), freq(LLLLLR), ..., freq(RRRRRR)]. Notably, particular configurations, typically with harmonic (consistent) branching-directionality, e.g., LLLLLL, LRRRRR, are common; such a skewed distribution (typological markedness or word-order universals) has been studied from multiple perspectives typically tied with human cognitive biases (Vennemann, 1974; Gibson et al., 2000; Briscoe, 2000; Levy, 2005; Christiansen and Chater, 2008; Culbertson et al., 2012; Temperley and Gildea, 2018; Futrell and Levy, 2019; Futrell et al., 2020b).

heading: 4.3 Processing costs of word order
text: Artificial languages: We quantify which word orders are harder for a particular LM. Here, we adopt3 the set of artificial languages created by White and Cotterell (2021) as a lens to quantify the LMs’ biases. These languages share the same default probabilistic context-free grammar (PCFG) and differ from each other only in their word-order configuration o ∈ O (§4.1) overriding the word order rules in the default grammar, resulting in 26 = 64 corpora with different word order o. Note that the 64 corpora generated have the same probabilities under the respective grammar and gold parser; thus, differences in language-modeling difficulties can only stem from the model’s biases. See App. A for the detailed configurations of artificial languages.
is distributed between its compatible word orderings, e.g., LLLLLR, LLLLRR, LRLLLR, and LRLLRR each gets a 1/4 count for LXLLXR. See App. B for the details of the WALS.
3We introduce the Case parameter determining the position of case marker, while White and Cotterell (2021) fixed it to be L. We omitted the Comp switch controlling the complementizer position, e.g., “that,” due to the lack of large-scale statistics on its order. We experimented with prepositional and postpositional complementizer settings in each of the 64 settings and used the average perplexities of the two settings.
Quantifying perplexities: We train an LM on each corpus with word order o and measure the PPL of tokens xo in the respective held-out set. Repeatedly conducting the training/evaluation across the 64 corpora produces a PPL score vector, PPLθ = [PPLθ(x LLLLLL),PPLθ(x LLLLLR),PPLθ(x
LLLLRL), ...,PPLθ(x
RRRRRR)], which indicates the wordorder preferences of an LM.

heading: 4.4 Frequency–perplexity correlations
text: Global correlation: We measure the Pearson correlation coefficient r(·, ·) between word order frequencies f (§4.2) and their negative PPLθ (§4.3), considering lower PPL is better. We call r(f ,−PPLθ) global correlation. A high global correlation indicates that the LMs’ word-order preferences reflect typological markedness.
Local correlation: White and Cotterell (2021) reported that simulating the word-order distribution among subject, object, and verb (SOV≻SVO≻VOS≻OVS), which is determined by the first two parameters of S and VP , is challenging. Therefore, we assess how easy it is to simulate the markedness of the other parameters’ assignments. Specifically, we measure a relaxed version of the correlation ignoring the subject, object, and verb order (local correlation), which is defined by the averaged correlation within each base wordorder group: SOV ( S = L, VP = L), SVO ( S = R, VP = L), OVS ( S = L, VP = R), and VOS ( S = R, VP = R).
1 4 (r(f SOV,−PPLSOVθ ) + r(f SVO,−PPLSVOθ )
+ r(f OVS,−PPLOVSθ ) + r(f VOS,−PPLVOSθ )) . (4)
Here, f X and PPLXθ are the list of frequencies and perplexities, limited to the languages with the assignments of S and VP corresponding to X. If this relaxed correlation is high and the global correlation is low, the ordering of subject, object, and verb remains challenging, and indeed, this is the case (see §7.1).

heading: 5 Models
text: We examine 23 types of uni-directional LMs, which adopt subwords split by byte-pairencoding (Sennrich et al., 2016) and are trained for 10 epochs. For each type of LM, we train/evaluate five models with different random seeds. In each run, 20K sentences (train:dev:test is 8:1:1) are generated from PCFG to train/evaluate the model. See App. C for model details.

heading: 5.1 Standard LMs
text: We test the PPL estimated by a Transformer (Vaswani et al., 2017), LSTM (Hochreiter and Schmidhuber, 1997), simple recurrent neural network (SRN) (Elman, 1990), and N-gram LMs.4 See App. C.2 for the model details.

heading: 5.2 Cognitively-motivated LMs
text: We further test cognitively-motivated LMs employed in cognitive modeling and incremental parsing, mentioned in §2.4. We target three properties: (i) syntactic inductive/processing bias, (ii) parsing strategy, and (iii) working memory limitations, following recent works in cognitive modeling research (Dyer et al., 2016; Hale et al., 2018b; Resnik, 1992; Oh et al., 2021; Yoshida et al., 2021; Futrell et al., 2020a; Kuribayashi et al., 2022).
Syntactic LMs and parsing strategy: We begin with syntactic LMs to explore the cognitivelymotivated LMs. Under a particular word order configuration o, they jointly predict tokens xo and their syntactic structures yo by incrementally predicting parsing actions ao, such as “NT(S) NT(NP) GEN(I) REDUCE(NP)...”:
pθ(x o,yo) =
∏
t
pθ(a o t |ao<t) . (5)
Here, we examine two commonly-adopted strategies to convert the (xo,yo) into the actions ao: topdown (TD) and left-corner (LC) strategies (Kun-
4Neural LMs are trained with the fairseq toolkit (Ott et al., 2019). N-gram LMs are trained with the KenLM toolkit (Heafield, 2011) with Kneser-Ney smoothing.
coro et al., 2017).5 The LC strategy is theoretically expected to estimate more human-like cognitive loads than the TD (Abney and Johnson, 1991; Resnik, 1992) in a sense that LC parsers can estimate the cognitive load of center-embedding constructions, typically with disharmonic word order configurations, which tend to be avoided by humans (Miller and Isard, 1964).6
Memory limitation: Humans generally have limited working memory (Miller, 1956) and struggle with processing long-distance dependencies during sentence processing (Hawkins, 1994; Gibson et al., 2019; Hahn et al., 2021). We thus expect that model architectures with more severe memory access, e.g., in the order of SRN≻LSTM≻Transformer, have such human-like biases and exhibit higher correlations with the word-order universals. We also apply such a memory limitation to syntactic LMs (next paragraph). We expect that memory-limited syntactic LMs can softly reflect the stack-depth, a standard concept of working memory demand (Abney and Johnson, 1991; Noji and Miyao, 2014), since the requirement of more stack depth needs the model to maintain longer parsing histories, presumably incurring a high cost for memory-limited syntactic LMs (Jin and Schuler, 2020).
Implementations: We train (i) Parsing-asLanguage-Model (PLM) (Choe and Charniak, 2016) and (ii) recurrent neural network grammar (RNNG) (Dyer et al., 2016; Kuncoro et al., 2017), using the syntactic structures obtained during generating the corpus from PCFG (§4.3). PLMs are the same as standard LMs expect that they are trained on the action sequences a. Four PLMs with different architectures (Transformer, LSTM, SRN, and N-gram) are tarined. RNNGs also predict the action sequences, but they have an explicit composition function to compute phrase representations. We use the stack-only RNNG implementation by Noji and Oseki (2021), and we newly introduce its memory-limited version (simple recurrent neural network grammar; SRNNG), where (Bi)LSTMs are replaced by SRNs. Note that the cognitive plausibility of RNNGs has
5TD and LC are also denoted as pre-order and in-order traversals, respectively.
6We adopted the arc-standard LC strategy, following Kuncoro et al. (2018) and Yoshida et al. (2021). Strictly speaking, an arc-eager LC strategy is cognitively plausible, and an arc-standard one has similar characteristics with bottom-up traversal (Resnik, 1992). That is, our LC results may be biased towards L assignments.
been reported in both cognitive modeling (Hale et al., 2018a; Yoshida et al., 2021) and language generalization test (Kuncoro et al., 2018; Wilcox et al., 2019). Henceforth, syntactic LMs refer to the PLMs and (S)RNNGs.
PPL computation: We measure the PPL over action sequences in each word order o when quantifying the word-order preference of syntactic LMs (§4.3): PPLθ(xo,yo) := ∏ t pθ(a o t |ao<t) 1 |ao| . We also examine a token-level predictability PPLθ(x
o) in §8 and App. D.1, but such variations did not alter the conclusions.

heading: 5.3 Baselines
text: We set two baselines: (i) a chance rate with random assignments of perplexities (gray lines in Figure 3), and (ii) perplexities estimated by pre-trained LLaMA2 (7B) (Touvron et al., 2023), a representative of the large language models (LLMs), prompted with several example sentences (blue lines in Figure 3) (App. C.3) as a naive baseline.

heading: 6 Experiments
text: We compare the LM’s word-order preferences with attested word-order distributions (§6). Then, we
further analyze what kind of word-order combinations LMs prefer (§7).

heading: 6.1 Results
text: Figure 3 shows global and local correlations (see
App. D for the full results). The TD and LC variations of the Transformer, LSTM, SRN, and N-gram LMs correspond to the PLMs with their respective architecture. We expect syntactic LMs with the LC strategy to exhibit higher correlations than the LMs without syntactic biases ( > ) and those with cognitively unmotivated TD syntactic bias ( > ).
Most LMs beat the chance rate: Overall, most global and local correlations were higher than the random baseline, reproducing the general trend that common word orders induce lower PPL (Hahn et al., 2020).7 As a sanity check, we also observed that the LLaMA-2 exhibited weaker correlations than cognitively-motivated LMs; the current success of LLMs is orthogonal to our results.
7With a one-sample, one-sided t-test, models except for LSTM LM, TD SRN PLM, TD RNNG, LC RNNG, TD SRNNG yielded global correlations significantly larger than zero, and models except for TD RNNG yielded local correlations significantly larger than zero.
Syntactic biases and parsing strategies: First, the LC syntactic LMs generally outperformed the standard LMs ( > ) in each setting except for SRNs. This indicates the advantage of cognitivelymotivated syntactic biases in simulating the word-order universals. Second, LMs with the LC strategy tend to exhibit higher correlations than TD syntactic LMs ( > ), especially in terms of local correlation. That is, the cognitively-motivated LC parsing strategy better simulates the wordorder implicational universals. Note that RNNGs, on average, exhibited low correlations, although they are often claimed to be cognitively plausible LMs.
Memory limitation: The results show that memory-limited models tend to exhibit better correlations, with the exception of PLMs. In particular, RNNGs typically benefited from memory limitations (SRNNG≻RNNG), while PLMs did not (SRN≺Transformer). This implies a superiority of RNNGs’ memory decay over hierarchical tree encoding to PLMs’ simple linear memory decay.

heading: 6.2 Statistical test
text: Settings: We statistically test the advantage of cognitively-motivated factors toward higher correlations. Specifically, we train the following regression model to predict the global or local correlation scores obtained in the experiment (§6.1):8
r(f ,−PPLθ) ∼ ModelClass(θ) +MemLim(θ) + Syntax(θ) + LC(θ) . (6)
Here, ModelClass denotes the coarse type (e.g., neural model or not) of the model θ yielding the respective correlation score, MemLim denotes its strength of context limitation (higher is severer, e.g., SRN≻LSTM≻Transformer), Syntax denotes whether the model is syntactic LMs (1 for syntactic LMs; otherwise 0), and LC denotes whether the model uses the LC strategy (1 for LC syntactic LMs; otherwise 0). Positive coefficients for these features indicate their contribution to higher correlations. See App. E for the details of the regression.
Results: We observe that the coefficients for the Syntax and LC features were significantly larger than zero with one-sample, two-sided t-test in both cases of predicting global and local correlations.9
8We used the statsmodels (Seabold and Perktold, 2010). 9p = 0.07 for the Syntax and p < 0.05 for the LC in the case of global correlation. p < 0.01 for the Syntax and p < 0.01 for the LC in the case of local correlation.
The coefficient for the MemLim feature was not significantly larger than zero when targeting all the models (p > 0.1); however, when PLMs were excluded, the coefficient of the MemLim feature was also significantly larger than zero with onesample, two-sided t-test (p < 0.001 in both global and local correlations) as suggested in §6.1. To sum up, these corroborate the findings in §6.1.

heading: 6.3 Linking functions
text: The experiments so far have assumed the linearity between PPL and word-order frequency (Eq. 1)— did this choice bias our results? We investigated various linking functions between PPL and word order frequency: the perplexity of order k and logarithmic PPL (entropy). Our findings robustly hold with other linking functions (full results are in Appendix D.2). Figure 4 illustrates LMs’ local correlations under differently converted perplexities.

heading: 7 Analyses: what kind of word order is particularly (un)preferred?

heading: 7.1 {S,O,V} word-order biases
text: Observation: White and Cotterell (2021) reported that LMs could not show the subject, object, verb word-order biases attested in natural language (SOV≻SVO≻VOS≻OVS). Even our cognitivelymotivated LMs did not overcome this limitation, based on the global correlations being consistently lower than the local ones (§6.1; Figure 3). This tendency is visualized in Figure 5; within each base group (SOV, SVO, OVS, VOS), common word orders tend to obtain high predictability (i.e., lower PPL; bigger circles are at the top) except OVSorder’s high predictability and SVO-order’s low predictability. This made it clear that predictability generally explains word-order universals, but the markedness of word orders among subject, object, and verb must stem from additional factors.
Implication: This finding is consistent with: (i) humans arguably have an agent-first bias in event cognition, which could be the source of the subjectinitial word order (Wilson et al., 2022); and (ii) agent-preference in human sentence processing cannot be explained by surprisal estimated by neural LMs in N400 amplitudes modeling (Huber et al., 2024). Our findings corroborate that cognitivelymotivated LMs still lack such a human-like bias. Orthogonally, the artificial language ignores some important linguistic aspects, such as information structure (Gundel, 1988; Verhoeven, 2015; Ranjan et al., 2022), which may explain subject-object order; refining the artificial data would also be one direction to explore in future work.

heading: 7.2 Branching directionality preferences
text: Settings: Human languages, on average, do not favor either left- or right-branching (Dryer and Haspelmath, 2013). Given this, we measure how strongly a model prefers a specific branching directionality. We calculate the Pearson correlation (L-pref.) between negative PPL and the number of L assignments #L(·)10 of the word order:
r(l,−PPLθ) , (7) l := [#L(LLLLLL),#L(LLLLLR), ...,#L(RRRRRR)] .
(8)
As a sanity check, the word-order frequency distribution of attested languages, indeed, is weakly correlated (0.11) with the left-branching directionality. Thus, LMs are not expected to have an extremely high or low L-pref. score. Notably, the branching
10For example, #L(LLLRLL) = 5.
bias of LMs/parsers has been of interest in the NLP research (Li et al., 2020a,b; Ishii and Miyao, 2023).
Results: Figure 6 shows the results of branching preferences. LMs with the TD strategy are theoretically expected to have a lower L-pref. score (favoring right-branching) than the LC models (Resnik, 1992) ( < ). While the PLMs faithfully reflect such characteristics, RNNGs, surprisingly, exhibited opposite trends, suggesting the challenge in controlling the inductive bias of neural syntactic LMs. We also observed architecture-dependent branching preference; Transformers prefer leftbranching, while LSTMs prefer right-branching as suggested by Hopkins (2022). Such architecturedependent biases seem to have more impact on the branching preferences than the parsing strategies in PLMs.

heading: 8 Connection to predictability and parsability
text: Background: Human language is arguably designed to minimize complexity (how unpredictable symbols are) while maintaining informativity (how easy it is to extract a message from symbols) (i Cancho and Solé (2003); Piantadosi et al. (2012); Kemp and Regier (2012); Frank and Goodman (2012); Kirby et al. (2015); Kanwal et al. (2017); Gibson et al. (2019); Xu et al. (2020); Hahn et al. (2020); inter alia). We revisit this bi-dimensional view. Concretely, Hahn et al. (2020) showed that both PPL of an LM and parsability for a (notcognitively-motivated) parser (Kiperwasser and Goldberg, 2016) explain word-order universals.
Oveview: We demonstrate that predictability (PPL)11 of syntactic LMs entails parsability. That is, they can provide a more concise informationtheoretic measurement of word-order universals (syntactically-biased predictability).
Settings: We decompose the performance of syntactic LMs into two parts: token-level perplexity PPLθ(xo) (predictability) and parsing performance parseθ(x
o,yo) (parsability), using wordsynchronous beam-search (Stern et al., 2017). When computing the token-level predictability PPLθ(x
o), next-word probability is computed while predicting the upcoming partial syntactic
11Predictability is typically measured as entropy, but again, the choice of entropy or PPL did not substantially change the correlation scores (See §6.3 and App. D.2).
trees.12 We measure parseθ(x o,yo) as the F1score of the top-1 parse found with the beam search.13 Then, we test whether the parsability factor contributes to explaining the frequency of word order o in addition to PPL, using the following nested regression models:
Base: freq(o) ∼ PPLθ(xo) , +Parse: freq(o) ∼ PPLθ(xo) + parseθ(xo,yo) .
Results: The increase in log-likelihood scores of the +Parse model over the Base model is not significant with the likelihood-ratio test (p > 0.1) in all the RNNG settings ({TD, LC}×{SRNNG, RNNG}×{seeds}).14 That is, at least in our setting, we cannot find an advantage of parsability over predictability in explaining word-order universals. This may be because the next-word prediction for the syntactic LMs is explicitly conditioned by the parsing states, which might sufficiently bias the predictability measurements to reflect parsability.
Figure 7 also illustrates the predictability and parsability estimated by the LC SRNNG. Here, the predictability identifies more types of word orders as typologically marked (left small circles) than the parsability does (bottom small circles). This is contrary to the picture of both predictability and parsability as complementary factors explaining word-order universals (Hahn et al., 2020). Nevertheless, our artificial data could be easier to parse
12PPLθ(x o) := ∏ t pstx(x o t |xo<t) 1 |xo| . pstx(xot |xo<t) :=∑
y′∈Y(xo<t) p(xot , y ′|xo<t) Here, given a context x<t, a set of its upcoming compatible partial syntactic trees Y(x<t) is predicted. Next word xot is predicted by each candidate parse y′ ∈ Y(xo<t), then such predictions are merged over Y(xo<t).
13Evalb (https://nlp.cs.nyu.edu/evalb/) was used. 14We only tested RNNGs given the limited availability of batched beam-search implmentations (Noji and Oseki, 2021).
than natural language in the sense that our data have less structural ambiguity; thus, future work should explore this using more real artificial language.

heading: 9 Conclusions
text: We have shown that cognitively-motivated LMs better led to the emergence of word-order universals than standard LMs. From the linguistic typology perspective, we provide computational evidence that the universals emerge from cognitive biases, which has been challenging to demonstrate in previous work (Lian et al., 2021; Galke et al., 2022). From the cognitive modeling perspective, our results demonstrate that cognitively-motivated LMs have human-like biases that are sufficient to replicate some human word-order universals. Our results also clarified what typological patterns can still not be modeled by cognitively-motivated LMs (agent-first bias), as well as opened the direction to relate LMs in the traditional linguistic theory of delineating possible language from the impossible.
Limitations
Artificial language: We used artificial data to quantify the LMs’ inductive/processing biases for word-order configurations. While the use of artificial languages has typically been adopted to conduct controlled experiments (§2.2), such artificial data lack some properties of natural languages, such as the semantic relationships between the linguistic constituents (§7.1). In future work, we hope to devise further artificially controlled languages that exhibit some of these properties.
In addition, our PCFG implementation of artificial languages and word-order configurations unreasonably limited the candidate word-order variations; for example, the production rules (i) S → NPsubject VP, and (ii) VP → NPobject Verb could not produce variations with V and O apart (VSO and OSV) (Table 12). Exploring more flexible ways to create and cover artificial langauge variations will be another interesting direction.
Furthermore, our used data (White and Cotterell, 2021) is relatively small scale, which might incur unintended bias in LM performance, although there is a perspective to analyze inductive bias via measuring training efficiency (Kharitonov and Chaabouni, 2021; Warstadt et al., 2023).
Estimating word-order distribution: The word order frequency estimates derived from WALS might also be biased; for example, Indo-European languages tend to have richer meta-linguistic information in WALS, although our study takes the statistics from as many as 1,616 languages into account. A richer estimation of missing parameter information is desirable. As a more general concern, the frequency of a word-order configuration can be estimated in various ways, such as the number of native speakers or the number of language families adopting a particular word order. Furthermore, word order variation can be inherently non-binary (Levshina et al., 2023). Our study, as an initial foray, relied on the number of unique languages, a commonly used metric in linguistic typology research (Dryer and Haspelmath, 2013; Hammarström, 2016), considering that other approaches raise additional complications, such as an estimation of the speaker numbers or language family variability.
Ethical Considerations
We only used artificial language, which does not have information with potential risks, e.g., human privacy data. One concern is the bias in our word order frequency estimates; this might have led to biased conclusions, e.g., diminishing the impact of minority languages, although we used the largest publicly available data (WALS) to date. We used AI assistance tools within the scope of “Assistance purely with the language of the paper” described in the ACL 2023 Policy on AI Writing Assistance.

heading: Acknowledgements
text: We are grateful to Ana Brassard, Benjamin Heinzerling, Tatsuro Inaba, Go Inoue, and Yova Kementchedjhieva for their insightful comments on an early version of this study. This work was supported by JST PRESTO Grant Number JPMJPR21C2, Japan.

heading: A Details of artificial languages
text: Table 1215 shows the default grammar to create artificial language data, which adopts the configuration of White and Cotterell (2021). The “rele-
vant parameter” column indicates the parameter that controls the order of right-hand items in the respective production rule (the order is swapped when the parameter assignment is R). Note that the subcategory of non-terminal symbols (e.g., VP_S) was only used for generating the data; in the final data for training/evaluating syntactic LMs, these subcategories are omitted (e.g., VP_S should be VP), and the resulting uninformative edge in the syntactic structure (e.g., VP → VP) was also removed. Table 2 shows the example of a sentence with different word-order configurations. Different word order parameters yield a syntactic structure with different branching directionalities; for example, the constituency tree of LLLLLL extends to the bottom left (left-branching). The average sentence length was 11.8 tokens, and the average tree depth was 9.1. The vocabulary size of pseudowords is 1,314 same as White and Cotterell (2021).

heading: B WALS data statistics
text: Table 3 shows the statistics of the WALS data and the details about word order parameters. Out of the 2,679 languages listed in the WALS, 1,616 languages are involved, and approximately 2/3 of their word order parameters were annotated in the WALS; the missing values are completed as explained in §4.1 (footnote 1). As a sanity check, we observe the ratio of the assignments of the first
two parameters ( S and VP ), which controls the order of subject, object, and verb; these approximately replicate the ratio reported in Dryer (2013f), e.g., SOV and SVO orders occupy over 80% of languages.

heading: C Model details
text: The license of the used models/data is listed in Table 4; all of them are used under their intended use. All the models were trained/tested with a single NVIDIA A100 GPU (40GB). All of the experiments were done within approximately 600 GPU hours. The LLaMA-2 (7B) model was used via the hugging face toolkit (Wolf et al., 2020).
15The corresponding Table is positioned in the later part of the Appendix for readability.
C.1 Parsing strategies
Figure 8 shows the parsing actions converted with different strategies (TD and LC). PLMs are trained to predict such a sequence of parsing actions in a left-to-right manner.
C.2 Hyperparamteres
Tables 13 and 14 show the hyperparameters of LMs,15 which basically follow their default settings. Standard LMs and PLMs use the same hyperparameters. Their vocabulary size is set to 512.
C.3 Word order preference of LLaMA-2
We employed few-shot settings instead of fullfinetning with the limits in computational cost. Specifically, we create a prompt consisting of the instruction “The below sentences are written in an artificially created new language:” and ten example sentences extracted from the respective training set. The probability of each test sentence is computed conditioned with this prompt, and aggregating these probabilities results in the PPL of an entire corpus.

heading: D Results
text: The full results of the experiment (§6) and analysis (§7) are shown in Table 6. This also shows the top-3 preferred word orders by each LM, which demonstrates the model-dependent differences in their word-order preferences. We also include the baseline of average stack depth required to process sentences for each parsing algorithm in each word order as a standard measurement of memory cost.
D.1 Beam-search in RNNG/SRNNGs
Table 7 shows the results of RNNG/SRNNGs with and without word-synchronous beam search (Stern et al., 2017). The settings without beam-search are adopted in §6, and the advantages of memory limitation (SRRNG) and the LC strategies were replicated even with beam-search, where the tokenlevel perplexity PPL(x) is used (§8).
D.2 Results with different linking functions
Tables 8, 9, 10, and 11 show the detailed results with different linking functions (PPL1/2, PPL2, PPL3, log PPL) between model-computed complexity measurements and word order frequencies. Experiments with different linking hypotheses did not alter the conclusions. This supports the generality of our findings.

heading: E Details of regression analysis
text: We explored which factor impacts the global/local correlation scores obtained by various LMs θ. As explained in §6.2, we train a regression model to predict the correlation score obtained by a particular LM θ, given the features characterizing the LMs:

heading: Future Work
text: In future work, we hope to devise further artificially controlled languages that exhibit some of these properties. In addition, our PCFG implementation of artificial languages and word-order configurations unreasonably limited the candidate word-order variations; for example, the production rules (i) S → NPsubject VP, and (ii) VP → NPobject Verb could not produce variations with V and O apart (VSO and OSV) (Table 12). Exploring more flexible ways to create and cover artificial langauge variations will be another interesting direction. Furthermore, our used data (White and Cotterell, 2021) is relatively small scale, which might incur unintended bias in LM performance, although there is a perspective to analyze inductive bias via measuring training efficiency (Kharitonov and Chaabouni, 2021; Warstadt et al. , 2023). Estimating word-order distribution: The word order frequency estimates derived from WALS might also be biased; for example, Indo-European languages tend to have richer meta-linguistic information in WALS, although our study takes the statistics from as many as 1,616 languages into account. A richer estimation of missing parameter information is desirable. As a more general concern, the frequency of a word-order configuration can be estimated in various ways, such as the number of native speakers or the number of language families adopting a particular word order. Furthermore, word order variation can be inherently non-binary (Levshina et al. , 2023). Our study, as an initial foray, relied on the number of unique languages, a commonly used metric in linguistic typology research (Dryer and Haspelmath, 2013; Hammarström, 2016), considering that other approaches raise additional complications, such as an estimation of the speaker numbers or language family variability. Ethical Considerations
We only used artificial language, which does not have information with potential risks, e. g. , human privacy data