Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models
Authors: Qing Yao
Kanishka Misra
Leonie Weissweiler
Kyle Mahowald
Venue: COLM 2025 (2025)
Keywords: linguistics
dative alternation
indirect evidence
language learning
cognitive science
linguistic constructions
TL;DR: Through manipulating word-order preferences in datives and non-datives in the training sets of language models, we find that they acquire dative alternation preferences from both direct and indirect evidence.

Abstract: Language models (LMs) tend to show human-like preferences on a number of syntactic phenomena, but the extent to which these are attributable to direct exposure to the phenomena or more general properties of language is unclear. We explore this with the English dative alternation (DO: "gave Y the X" vs. PO: "gave the X to Y"), using a controlled rearing paradigm wherein we iteratively train small LMs on systematically manipulated input. We focus on two properties that affect the choice of alternant: length and animacy. Both properties are directly present in datives but also reflect more global tendencies for shorter elements to precede longer ones and animates to precede inanimates. First, by manipulating and ablating datives for these biases in the input, we show that direct evidence of length and animacy matters, but easy-first preferences persist even without such evidence. Then, using LMs trained on systematically perturbed datasets to manipulate global length effects (re-linearizing sentences globally while preserving dependency structure), we find that dative preferences can emerge from indirect evidence. We conclude that LMs' emergent syntactic preferences come from a mix of direct and indirect sources.

Published as a conference paper at COLM 2025
Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models
Qing Yao, Kanishka Misra, T Leonie Weissweiler, Kyle Mahowald
Department of Linguistics, The University of Texas at A
Department of Linguistics and Philology, Uppsala University"
qyao@utexas. edu, kmisra@uteexss. edu, leonie. wwissweillrrllingfiii uu kyle@utexas.edt edu
Abstract
Language models (LMs) tend to show human-like preferences on a number of syntactic phenomena, but the extent to which these are attributable to direct exposure the phenomena or more general properties of language is unclear. We explore this with the English dative alternation (DO: gave Y the X vs. PO: gave the X to Y), using a controlled rearing paradigm wherein we iteratively train small LMs on systematically manipulated input. We focus on two properties that affect the choice of alternant: length and animacy. Both properties are directly present in datives but also reflect more global tendencies for shorter elements to precede longer ones and animates to precede inanimates. First, by manipulating and ablating datives for these biases in the input, we show that direct evidence of length and animacy matters, but easy-first preferences persist even without such evidence. Then, using LMs trained on systematically perturbed datasets to manipulate global length effects (re-linearizing sentences globally while preserving dependency structure), we find that dative preferences can emerge from indirect evidence. We conclude that LMs' emergent syntactic preferences come from a mix of direct and indirect sources.
1 Introduction
Consider the dative alternation. Roughly the same real-world event is conveyed by She gave the boy who signed up for class and was excited it (using the Double Object DO form) and She gave it to the boy who signed up for class and was excited (the Prepositional Object or PO form). Yet most people would prefer the latter. The reason that users of English typically show "short-first" preference in the dative alternation. That is, if the theme is longer than the recipient they prefer the DO. If the theme shorter, the PO. See an illustration Fig. 1. Besides length, there are a host of other factors that influence the choice of alternation used, such as animacy and pronominality (Bresnan et al., 2007; de Marneffe et al., 2012).
this in
double (DO) She gave the bay who signed up for class and was excited long-first
prep. object (PO) She short-fist
the boy who signed for class and excited,
mone
double object (DO) She gave
him
the book that everyone was excited to read.
short-first
recipiint
thone
object (PO)
gave the book everyone excited read
long-first
chonee
Figure 1: The dative alternation allows for either PO or DO realization. We highlight the arguments (theme and recipient) that are relevant to our feature analysis.
1

---

Published as a conference paper at COLM 2025
These preferences in the dative alternation are learned well by even relatively small LMs (Hawkins et al., 2020; Misra & Kim, 2024). But what has to be true of the training data for these preferences to emerge? Recent work has considered the role of indirect evidence in language learning, asking whether LMs learn syntactic phenomena by memorizing only the most similar instances or whether they rely on structurally similar instances in order to generalize. Evidence so far has been mixed, with some work arguing for generalization from indirect evidence. Misra & Mahowald (2024) find that somewhat rare syntactic structure is learned, in part, from indirect evidence from more common related structures. Patil et al. (2024) find that grammatical generalizations are possible even when never encountered. Oba et al. (2024), though, find that indirect evidence is less important than direct evidence. Houghton et al. (2025) find that binomial preferences (e.g. "peanut butter and jelly" vs. "jelly and peanut butter") in models largely depend on memorization, with little evidence of generalization to human-like preferences for novel binomial pairs. Jumelet et al. (2024) find that language models can make some generalization about adjective order preference (e.g. "big black boxes" vs "black big boxes") on adjective pairs not seen in training.
We posit that the dative alternation is an elegant testing ground for exploring these questions, since it is linguistically well-studied and shown to be sensitive to a variety of factors that are not only directly observable in extant dative data, but which also depend on more general constraints in the language. One possibility (direct evidence) is that LMs prefer dative constructions that are highly similar to those observed in training. That is, if She gave it to the boy who signed up for class and was excited is attested in the training but She gave the boy who signed up for class and was excited it is not, then LMs might assign higher probability to strings that are very similar to the former. A second possibility (indirect evidence) is that the observed dative preferences are emergent from more general features of the input data. Specifically, we hypothesize that the LMs' preference for "short-first" and "animate-firss' dative utterances might follow not from memorizing instances of such datives but from those preferences existing more generally in the data (Behaghel, 1932; MacDonald, 2013) - e.g,, Tur et al. (2025) find contemporary LMs to follow the general sshort-before-long ordering across a variety of syntactic structures.
To adjudicate between these possibilities, we use a controlled rearing paradigm (Jumelet et al., 2021; Frank, 2023; Misra & Mahowald, 2024; Feng et al., 2024; Patil et al., 2024; Leong & Linzen, 2024; Kallini et al., 2024; Xu et al., 2025), whereby we train small models on carefully controlled input data and then test whether the predicted preferences emerge. We first train a default model on a roughly 88M-word corpus and test whether these preferences emerge in the dative. We then manipulate the input, balancing length and animacy-relatee related statistics in the training set to remove any order bias in the instances of the dative construction. We do this by creating a matching PO for every DO, and matching DO for every PO. We show that dative preferences still persist-althouge just like default t except all datives are inserted in the reverse alternant, such that the direct evidence is now the opposite of what it normally would be. We find that this neutralizes the standard dative preferences. Then, we train models which do not have direct exposure to datives, and see that length preferences still emerge.
they get weaker. So we train another model
The above results show that direct evidence matters but also that human-like trends emerge even without direct evidence, especially for length. 1 To ascertain whether these trends come from more global statistical patterns, we turn to manipulating global preferences directly. We do this by systematically reordering all constituents in training sentences based on their length, in order to create input corpora with systematically controlled length effects. We find that, while the usual dative length preferences emerge in the "short-first" corpus, the preferences are gradually lost as training sentences increasingly become more "long- first". We take this as evidence that more general properties of the language non-trivially contribute the observed dative preference in LMs. As such, this work joins a larger body of work suggesting the emergence of abstract linguistic behavior in LMs from statistical regularities in their training (see Futrell Mahowald, 2025, for a summary).
"ode and data for the experiments are available at http:///gghuubbcommmooee//////
2

---

Published as a conference paper at COLM 2025
2 Data, Methods, and Models
2.1 Corpus
Our training data is a subset of the 2023 BabyLM corpus with 100M words (Warstadt et al., 2023), which we choose for its tractable and human-scale size. We exclude the QED portion, since it includes special symbols and lacks proper sentence breaks, making dative detection difficult. This leaves our initial training set at around 88M words.
2.2 Defining and Detecting Datives
dobj
Our approach crucially depends on identifying dative utterances in the corpus, so that we can manipulate them during training and also create separate test set of dative utterances. Defining datives not straightforward and has been han- dled in different ways in the literature (Levin, 1993; Bresnan & Nikitina, 2009; Liu & Morgan, 2020; Liu & Wulff, 2023). In particular, delineat- ing datives requires making choices like whether to include benefactives (baked me cake) or non- alternating verbs ("dedicated the mayor the park).
doby dative
verb
recipient
theme
fo: dative/ppeee
dative
dobj
pobj
verb theme
toffor
recipient
Figure 2: spaCy parses of possible DO da- tives (top) and PO datives (bottom).
First, we obtained dependencies parses from spaCy (Honnibal al., 2020) for all utterances in the BabyLM dataset, on which the following definition of datives in the main paper is based. Then, we label an utterance as DO dative if contains a verb whose dependency contains a direct and an indirect object. For PO datives, we consider utterances that have a verb with dependency object (the recipient) of the following form: 1) to prepositional phrase under a dative or prep dependency; 2) a for prepositional phrase under a dative dependency. See Fig. for an illustration.
direct object (the theme), and dependencies
an indirect
This definition is intentionally broad, with intent cover all constructions of interest, but may include false positives, such as the PO-like burning it to the ground. Precision is less important than recall in controlled rearing experiments, since recall is key for controlling dative exposure in our models. More details, including an alternative (stricter) method for detecting datives and a demonstration that our results do not depend heavily on the precise operationalization of datives, can be found in App. F.
Recall Error Estimate and Artificial Pollution To confirm that no datives slipped through, we spot-check a random sample of 4000 utterances classified as non-dative of token length greater than 5 by hand. We find just one dative entry, namely they gave the poor professor divers and sundry medicines, due to spaCy interpreting professor divers as a compound noun rather than archaic spelling of diverse. From this, we estimate the recall rate to be 0.025% (compare with 2.57% of all entries in BabyLM being detected as datives under our loose but high-recall criteria). Considering that our training sets consists of around 9M non-dative utterances, we estimate there are 2250 false negative datives, as a conservative upper bound.
Following Misra & Mahowald (2024), we artificially "pollute" our data with counterbalanced data intended to offset the effect of the undetected datives. These undetected datives could, in principle, provide direct evidence for human-like dative preferences. To counteract them, we insert datives, in roughly equal number, that violate the standard preferences. Specifically, for each model where we manipulate just the dative exposure, we insert 2250 counterfactual datives (swapping the structure of attested datives) to minimize the effects of the estimated 2250 false negative datives. We estimate that the undetected datives are twice as likely to be DO than a PO, so 1500 counterfactuals are inserted in the DO form (when attested was PO) and 750 in PO form (when attested was DO). For each model, will discuss if counterfactual insertion is done as they are introduced. See Table 7 in the Appendix for a breakdown of the number of controlled dative exposures, estimates of unaccounted-for datives, and insertions of counterfactuals by model.
3

---

Published as a conference paper at COLM 2025
Exp.
Model
Pre-manipulation
Post-manipulation
All default
N/A
N/A
gave the dog a bone
la balanced
gave the dog a bone
gave bone to the dog
+gave seavve-the-eepe the--dege---eee gave bone to the dog
1b
awapped-datives
gave the dog a bone
gave the dog a bone
gave the dog a bone
no-datives
He eats the green melon from
He eats the green melon from the shop with fork
1c
shop with fork
gave the dog a bone He eats the green melon from
gave the dog bone He the green meloon fromthh shop with fork
no-2postverbal
the shop with fork
short-first
He fork cat the green
[he] uses [a fork] [[to] eat [[the] [green] melon [from the shop]])
melon from the shop
randon-first
He uses fork eat the green [[to] eat [[the] [green] melon [from the shop]] uses (a fork] [he]
melon from shop
long-first
He uses furk to cat the green
[[[from the shop] [the] melon [green] cat [to]] uses [a fork) [he]
melon from shop
He uses fork to eat the green [[[the shop from] [the] [green] melon] [to] eat] [a fork] [The] uses
melon from shop
Table 1: An overview of the manipulations for our experiments. The default model is the same across all experiments. All models in Experiment have no entries containing verbs
with two postverbal arguments. Heads are in bold for the Long-first-headff model.
2.3 Model Architecture
We use the OPT-125M architecture (Zhang et al., 2022). See App. A for specific training details. Each of our LMs is trained using the transformers library (Wolf et al., 2020). We train each LM with three random seeds to account for randomness in weight initialization. For models with exposure datives, we ensure that the total number of datives (including estimated unaccounted-for datives and counterfactual insertions) is equal across models, so that we can fairly compare dative preferences across models.
2.4 Measuring Dative Preference
In all our subsequent analyses, we will specifically study our trained LMs' preferences between the two possible realization of a dative construction given dative verb and its arguments, following previous works (Hawkins et al., 2020; Misra & Kim, 2024). Given pair of DO and PO realizations of the same dative construction, we compute the preference of the LM as the difference between its log probabilities for the DO and the PO sentence, each normalized by its sentence length, to account for the extra to lo/for the PO sentences. All length-normalized log prsbabiiiiiiess are computed using minicons (Misra, 2022).
2.5 A Test Set for Testing Preferences by Feature
To quantify our LMs' dative preferences across sentences with varying length and animacy features, we construct a test set consisting of pairs of dative-alternating sentences sampled from the LabyLM test corpus (again excluding the QED portion). We sample 1000 sentences detected as a DO dative and 1000 detected as PO from the test corpus. The recipients and themes are labeled for pronominality using spaCy, and manually labeled for animacy by us. Both features are considered to be binary in our analysis.
We manually ensure that each instance is a dative, and that the verb usage considered to alternate in general, even though the constructed unattested alternant may not sound natural. This is because we are interested in isolating the differences in the models' preferences according to differences in the features of interest (e.g., length and animacy), so the inclusion of potentially unnatural utterances is precisely what we are interested in.
To create DO forms from attested PO forms, remove the preposition and swap the position of the recipient and theme. Creating PO sentences from attested DO datives is more complicated since consider both lo-datives and benefactives (i.e., with for) to be
4

---

Published as a conference paper at COLM 2025
part of our PO dative set. While some dative verbs have fixed preposition usage, some can use either preposition depending on indirect object (recipient/ segiient/beneficiaryy beneficiary) and theme: bring that letter to me vs. bring napkin for me. Taking this into account, we take the following steps to create our PO sentences: first, we check the verb occurs as a bisiiv///beefaaiivee verb in Levin (1993). If it does, then we check if it alternates, and based on whether its alternation is listed as a to-dative or benefactive, we decide the preposition (to vs. for). For the remaining verbs not classified as datives in Levin (1993) not listed as alternating, we use the Llama-3.2 3B model (Grattafiori et al., 2024) to decide between the appropriate surface form by creating sentences in both forms, and then selecting the one which has the higher log- rog-probability according to the model. Using this method gives us 2000 pairs of datives in our test set, with total of distinct verb lemmata.
Feature Encoding For future regression analyses, we compute a length difference score and an animacy difference score for each entry in the test set. Length difference defined as the log difference in length between the recipient and theme. Because animacy is binary- coded, their difference scores take on values -1,0, or 1. An animacy difference of 0 means that either the theme and recipient are both animate or both inanimate, a difference of means that the recipient is animate and theme is inanimate, and difference of means that the recipient inanimate and the theme is animate.
The noun phrases in the attested datives are approximately in the natural distribution of features in DO and PO datives since they are randomly sampled. The average log difference in length between the recipient and theme is -0.88 in DOs and 0.199 in POs, consistent with the easy-first preference. Animates are strongly aligned with the recipient position in either alternant (recipients were animate in 95.1% of DOs, and 71% of POs), consistent with the fact that inanimates are bad recipients (Beavers, 2011).
3 Preliminary Experiment: Does a BabyLM-trained LM learn human-like dative preferences?
While many dative verbs alternate freely be- tween the DO and the PO, human alternation behavior for these verbs is graded (Bresnan Nikitina, 2009). As preliminary experiment, we test the extent to which our BabyLM-trainee LMs capture human-like dative preferences in general. Specifically, we train an LM on our subset of the BabyLM corpus (see $2.1), which we will refer to as default. The training set is filtered to consist of 66,822 datives in both DO and PO forms (and possibly undetected datives). The model should have roughly equal exposure to each alternant.
Classification Alternating
Non-alemmting
deffult
GPT-2
DO preference
41.5-
-1.0-
-1.5
0.2
0.4
0.6
0.2
0.4
Human Judgment
Figure 3: Pearson's correlation between human judgments and predictions from default and GPT-2 LMs, both scored, across the 155 verbs DAIS (Hawkins et al., 2020) that are present in the BabyL.M corpus. GPT-2 predic- tions are taken directly from DAIS.
For testing, we use the DAIS dataset from Hawkins et al. (2020), which contains 5000 differ- ent pairs of sentences the DO and PO forms, spanning 200 different verbs. In addition, it also contains behavioral results from LMs such as GPT-2 and other models. We only select the sentences for which the dative verb appears in a dative construction in the training set, giving us 155 unique verbs and 3672 pairs of sentences. For each instance, the dataset also has a measure of the human judgments of preference for DO over PO sentences, analogous to our main measure of DO preference (see $2.4). To compare our model's 's preferences to followed Hawkins et al. and compared the average Human DO
those from humans, preference for the 155 target verbs to the average (z-scored) LM DO preference by computing their Pearson correlation.
5

---

Clear Preferences Emerge We compute the correlations with human judgments obtained from our model and, for reference, we also re-compute the correlation obtained for GPT-2 (Radford et al., 2019) for the subset the verbs we included our analysis. As seen in Fig. our model (with seed selected by best evaluation loss) obtains correlation of 0.50 with the human ratings on DAIS. Not only is this positive, it is not far off from the correlation corpus that is Published as a conference paper at COLM 2025
swapped datives
default
balanced
no-datives
no 2postverbal
r= 0.43
r= -0.33
0.03
== -0.4
= -0.2
0.5
0.0
DO Preference
-0.5
-1.0
2
2
2
0
2
0
2
0
0
2
Log Difference Recipient and Theme Lengths
0.2
044
-06
0.8
L
M
E
M
M
Recipient Less, Equally, or More animate
Figure 4: Correlation between DO preference and length difference (top) and animacy difference (bottom) by model on instances in the test set (N = 2000). A negative correlation indicates short-first preference, while greater DO preference for recipients being animate indicates unimate-firs preference. The seeds with best evaluation loss are shown for length, whereas all three seeds are shown for animacy. Models shown here are for Experiment 1.
obtained by GPT-2smal1 (r 0.59), even though several orders of magnitude smaller than that of GPT-2smal1-
model was trained
To measure the effect of length on dative preference, we compute the log- difference in length between the theme and recipient (positive value means recipient is shorter), and examine its correlation with the DO preference. As expected, observe negative correlation between the length difference and the DO preference: y -0.43 for the default model with the best loss-ie, the model prefers DO more when the recipient is shorter than the theme. We also found an animacy-firy trend, such that DO preference was highest when the recipient was more animate than the theme.
We fit a mixed-effect model predicting DO preference based on fixed effects of length difference and animacy difference. We include random intercepts for verb lemma and seed with maximal random effect slope structures, excluding correlations to help convergence, following Barr et al. (2013). We compute p-values using nested model comparisons. For the default model, we found significant effects of length (B = - .174, p < .0001) but not for animacy (B = .065, p .1).. On further inspection, we found that the observed animate-firs first effects seem be driven by specific v.rbs-e.g, leave me over leave me it, but write me letter over write letter to me. We include a discussion about verb-specific effects in App. E. We show length and animacy effects for default (and subsequent models) in Fig.
4 Experiment 1: Balancing, Swapping, and Removing Datives
Having established that our small models show clear dative preferences and, in particular, show strong preference for length, key question emerges: are these preferences result of memorization from similar dative sentences, or do they emerge from a more general property of the input? To address this, we series of controlled rearing experiments, in which we systematically modify the evidence that the model receives and examine the persistence of the length and animacy effects.

---

Published as a conference paper at COLM 2025
In Experiment la, we manipulate the dative sentences in training to be perfectly balanced, effectively removing all length and animacy effects from dative constructions exposed to the model. In Experiment 1b, use the same training set as for default, except all datives are in swapped-ordes (i.e. recipient and theme are "out-of-order") This means that the model will receive the opposite direct evidence that expect in the default case. In Experiment 1c, we then entirely remove dative constructions, as well as all constructions with two postverbal arguments, so that the models do not receive any direct evidence. We also artificially pollute (see $2.2) the training set to approximately neutralize the effect of undetected datives in Experiments la and 1c (skipping in Experiment 1b since the vast majority of datives in its training set are in reversed order anyway).
Broadly, we assess the significance of our manipulations in two ways. The first is that we fit a mixed effect regression separately for each model, predicting DO preference based on length and animacy (as described above). This tells us whether, in each model, we observe length and animacy effects significantly different from 0 (Table 2 for all experiments). The second broad analysis is a global mixed effect model predicting DO preference across all ex- periments, based on fixed effects of length and animacy and the interact of each with model, treating def fault as baseline, as described in App. C. The interaction of each term (length and animacy) with model tells us whether our manipulations result in congth/animan effects significantly different from the fefault model (Table 4).
4.1 Experiment 1a: Effect of Balancing Dative Alternating Forms
The goal of our first manipulation is to balance the datives by matching the PO and DO frequencies and then creating a PO utterance to match every DÒ utterance. A model trained on such manipulated data should not observe any direct associations between word order and the features of the dative verb or recipient and theme. In particular, this means that the dataset will have no preference for the short argument to come first, and no preference for animates to come first (since we balance every "short-first sentence with the counterfactual "long-firee first""i""""" sentence, and similarly for any animacy bias).
We balance the dataset by creating alternate forms for each occurrence of dative, using the method outlined in $2.5. For instance, for the DO sentence I gave the dog bone create its PO form I gave a bone to the dog and then add both sentences in the training set (see Table 1). We ensure that the LMs trained on this manip- ulated corpus still have roughly the same num- ber of datives in total as def default after artificial pollution by balancing 32,850 attested DOs and 32,850 attested POs with their unattested alter- nants. By design, this model should observe no word-order preferences in datives. We refer to this model as balanced. As can be seen in Fig. 4, the length effect is weakened in the balanced re- sults but still very much present: the correlation goes from r 0.43 in the default to -033. As shown in Table 4, our global regression model showed that the effect of balancing sig- nificantly weakened the length and animacy effects relative to default. But, as shown in Table 2, using the same per-model regression as before but focusing on the balanced model's data, we again found a significant effect of length (but not for animacy). We also once again found verb-specific animacy effects.
Exp.
Model
Length
Animacy
All
defaule
-0.174
0.065
1a
balanced
00001***
0,032
1b
swappad-datives
0.012
0.186
1c
no-datives
no-2postverbal
0.056**
0.048*
0.035
0.007
short-first
-.064**
0.004
randon=first
-0.017
0.007
fing-first
0.010
0.068
0.00
0.004
Table 2:
Regression coefficients from mixed effect models, fit separately for each LM's DO preference, with for p < .05, for p < 0.01, and *es for p < .001. Length and animacy are difference scores, as described in $2.5.
4.2 Experiment 1b: Effect of Reversing All Datives
We see that when model receives preference-neutral dative input in training, easy-first association still arises, albeit to a lesser extent. Taking this a step further, we train a swapped-datives model where all datives shown to the model are out-of-order from their
7

---

Published as a conference paper at COLM 2025
natural order. Its training set is identical to that of default, except the controlled 66,822 DO and PO datives are inserted their unattested alternant forms. Extrapolating from the previous models' preferences, we should expect swapped-datives to have even less short-first preferences, if not reversed.
This is indeed the case: swapped-dative has roughly neutral length effect of r 0.03. Animacy effects are reversed, suggesting swapping direct evidence has an effect on dative preferences. swapped-datives showed significantly smaller length and animacy effects than the default LMs (see Table 4). Length and animacy coefficients were not significant by our per-condition regression analysis (see Table 2). Thus the swapped-datives models, as expected, finds effects in the opposite direction of default, but the magnitude is smaller.
4.3 Experiment 1c: Effect of Removing Dative and Ditransitive-like Constructions
We have shown that dative preferences persist even when datives are balanced in training, and are only roughly neutralized when all dative exposure are counterfactuals. This suggests that non-dative sentences are contributing to the learned preferences. If the models had no exposure to datives whatsoever, can they still recover dative preferences? This case is crucially different from the balanced case: in the balanced case, the models get direct evidence that there is no length or animacy effect in datives. In the case where all datives are removed, there no evidence either way, so any preferences must come from elsewhere. To test this, we train an LM on a version of our BabyLM corpus without datives, which we refer to as no-datives.
Additionally, it is possible that our LM extracts dative preferences from preferences within constructions that share the structure with the dative. For example, other verb alternations such as spray/load (Levin, 1993, Section 2.3): Jack sprayed the paint on the wall vs. Jack sprayed the wall with the paint and creation creation/transforot http://traoooooooonnnnoooo (Levin, 1993, Section 2.4): Martha carved toy out of the piece of wood vs. Martha carved the piece of wood into toy could all be subject to similar easy-first biases, and the model can plausibly identify them with datives. For this reason, we also train an LM which has no exposure to any ditransitives cases where the verbs take a prepositional object (not restricted to toffor prepositions) as the second object-which
refer to no-2postverbal. i.e., cases with two postverbal arguments. To perform this ablation, we remove all entries which meet our dative criterion but in addition consider all prepositions instead of only considering tolfor.
As with balanced datives, the length and animacy effects are reduced compared to the default model. As shown in Fig. 4 and Table 2, we again find a persistent length effect: r == -0.24 for no-datives and r -0.2 for no- -postverbal While there is not significant evidence for a global animacy-first preference, we again observe verb-specific effects that are interestingly similar to the effects in the default case (see Fig. 8 in App. E).
4.4 Interim Discussion
Overall, two different methods of changing the input data (balancing and removing datives) fail to remove the length effect, although in all cases, the effect is unsurprisingly weaker than in the default model. Only when all the input datives are in reversed order do length effects become closer to neutral (though interestingly they are not reversed). The effects of animacy are more nuanced. Here, we do not find a global "animate-first" preference- instead, this preference seems to depend on the verb (see App. E). For instance, when the recipient is more animate, leave and sell are strongly preferred in the PO, while bring and write show strong DO preferences. At the same time, the verb-specific effects are similar between the default and no-datives condition (Fig. 8), suggesting some higher order effect of indirect evidence. We leave deeper exploration of this to future work.
Experiment 2: Effect of Global Length Manipulation
Having shown that length effects, in particular, persist even when we remove datives or balance them, natural question emerges: Where do these effects come from? One
8

---

Published as a conference paper at COLM 2025
long-first headfinal
default
short-first
random- mandom-first
long-first
r= 0.43
r= -023
== -0.12
r= -.04
r= 0.06
0.5
0.0
DO Preference
-0.5
-1.0
2
2
0
2
0
2
0
0
2
Log Difference Recipient and Theme Lengths
0.2
0.4
-06
0.8
L
M
E
M
M
Recipient Less, Equally, or More animate
Figure 5: Correlation between DO preference and length difference (top) and animacy difference (bottom) by model on instances in the test set (N = 2000). Models shown here are for Experiment 2.
hypothesis is they come from a more global linguistic preference for short constituents to appear earlier in English. In this experiment, we then ask whether more global properties of the input affect length preferences by systematically manipulating the training corpus to either favor placing short syntactic constituents first or long ones first, or- as a control- place either long or short-first randomly.
To perform our global length manipulations, we operate on our no-2postverbal corpus that excludes datives and related constructions. For each sentence, the children of any node in its dependency representation
ordered by length (either short-first long-firsty Since long-first preference has been observed in head-final languages (Yamashita Chang, 2001; Futrell et al., 2020), we also create corpus where constituents are arranged by decreasing length and every head appears in the final position. See App. for pseudocode. We train fhortt long-first a control, we also train a model on corpus where child-constituent balancing is randomly chosen frandom-first) The training sets for these four models are identical apart from constituent ordering, and so no counterfactual pollution is used.
and firg-first--i wfisst-eeadfiiii headfinal LM on these corpora, respectively. As
The training set of no- 2postverbal is already strongly short-first roughly 70% of entries are short-first in their original form and 44% long-first. Disregarding simple sentences where no head has more than one child (since these are automatically choot/long-foe 53% of sentences are already short-first compared to 16% for long-first. This suggests that preferences learned by the short-firs model should be similar that of no-2postverbal. We provide more detailed analysis of how short-first the original corpus is in App. I.
The
LMs have no exposure to any cases with two postverbal argu- ments, and in addition to that, they observe biased version of English word order with consistent length properties. Therefore, if global length properties have an effect, we should expect gradient such that the dative length effect is greatest in the short-first manip- ulation, minimal in the random-first manipulation, and non-existent or reversed in the long-first-headfr manipulation.
5.1 Results
As shown in Fig. 5, short-first, random-first, long-first, and Long-first-headfinal show a length-effect gradient, appearing the expected order. To assess significance of these results, ran another regression (similar to our global regression, App. C for details and Table 5 for results), focused on just the length-manipulated conditions and with the random-firse condition baseline. As expected, the cong-first and

---

Published as a conference paper at COLM 2025
long- feee-fiist-hhee herst-headfinal,
long first
Length Effect
0.0
0.0
swapped-datee
expected heve seraker preference
expected have stealeer shortsfins preference
no- no-2postverbal.
-0.2
-0.2
randon- Soeee---fee first
no datiaiives
short-first
-0.4
default
balanced
stronger shott-list pedtenence
atrunger
enpected
preference
50
55
60
65
70
50
60
70
80
90
100
Perplexity on Validation Set
Figure 6: Correlation of DO preference score with log-difference in length (y-axis) vs. the geometric mean of perplexities (x-axis) on fixed sample of 10,000 sentences in the validation set (with appropriate length manipulations if applicable). Left: Models without length manipulations; Right: Models with length manipulations. The more golden model's color, the more we expect its training set manipulation to favor short-first
Song-first-headinaa models showed significantly weakened length preferences com- pared to random-first, but short-first showed significantly stronger length preferences than random- firss.. Therefore, when no direct evidence is available, the learned dative preferences tend towards preferences in the hypothesized indirect evidence.
Since the training data in many of our experiments are heavily manipulated, it is possible that observed length effects are due to higher perplexities from less good models overall. That is, maybe some models just don't learn dative preferences very well because they don't learn amything very well. To test this, we compute the geometric mean of perplexities 10,000 sentences from the validation set with corresponding length manipulations for every model. These perplexity values and length effect coefficients per seed are shown in Fig. 6. The observed perplexities do not fully explain the patterns in data. For example, the long-first models have lower perplexity on the validation set than random- candom-first models, and yet they show weaker (less negative) length effects. We see that within groups of models with or without length manipulation, the more blue or red models have weaker short-first preference regardless of perplexities. This suggests that length effects are due to fundamental changes in global constituent ordering, beyond those captured by perplexity.
6 Conclusion
Bespoke small models learn reasonably human-like preferences for the dative alternation. Length preference persistent even when we train models on datasets without direct evidence for such preference, suggesting that short-first preference is more general property of English. By then training models on counterfactual languages where the indirect evidence becomes decreasingly "short-first", we show that the length effect is gradually lost, indicative the role of indirect evidence in acquiring that preference. The story for animacy is more subtle. Instead of global unimate-first effect, we found verb-specific differences where other factors such as the theme might also play a role. At the same time, these verb-specific effects were persistent even in the absence of direct evidence, and oftentimes did not flip in ablations that flipped the direct evidence.
Overall, take these results to show that LMs' dative preferences emerge from more general properties of the language, not just those directly observable in dative constructions. As such, we join growing consensus that key property of the success of LMs is the consistency of linguistic evidence in the input.
Acknowledgments
We thank John Beavers, Sasha Boguraev, Richard Futrell, and the UT Computational Lin- guistics research community for discussions and feedback. Qing Yao thanks Simon Todd for
10

---

Published as a conference paper at COLM 2025
his mentorship during his undergraduate thesis, which extended into this work. Qing Yao is supported by the Donald D. Harrington Fellowship at UT Austin. Kanishka Misra is sup- ported by the Donald D. Harrington Faculty Fellowship at UT Austin. Leonie Weissweiler was supported by a postdoctoral fellowship of the German Academic Exchange Service (DAAD). Kyle Mahowald was supported by NSF CAREER grant 2339729 from the Director of STEM Education (EDU) Division of Research on Learning in Formal and Informal (DRL).
References
Dale J Barr, Roger Levy, Christoph Scheepers, and Harry Tily, Random effects structure for confirmatory hypothesis testing: Keep maximal. Journal of memory and language, 68(3): 255-278, 2013.
John Beavers. An Aspectual Analysis of Ditransitive Verbs of Caused Possession in English. Journal of Semantics, 28(1):1-54, 2011.
Otto Behaghel. Deutsche Syntax: Eine geschichtliche Darstellung, Band IV: Wortstellung, Winter, Heidelberg, 1932. Behaghel's Law of Increasing Constituents (short before long) is formulated here.
Joan Bresman and Tatiana Nikitina. The Gradience of the Dative Alternation. Reality exploration and discovery: Pattern interaction in language and life, pp. 161-184, 2009.
Joan Bresnan, Anna Cueni, Tatiana Nikitina, and Harald Baayen. "Predicting the Dative Alternation". In Gerlof Bouma, Irene Krämer, and Joost Zwarts (eds.), Cognitive Foun- dations of Interpretation, pp. 69-94. Royal Netherlands Academy of Arts and Sciences, 2007.
Catie-Catherine de Marneffe, Scott Grimm, Inbal Arnon, Susannah Kirby, and Joan Bresnan. A statistical model of the grammatical choices in child production of dative sentences. Language and Cognitive Processes, 27(1):55-61, 2012. doi: 10.1080 (00000/01699955520005222222
URL https: org/ 1080 09909656
690965. 2010. 542651.
Steven Y. Feng, Noah Goodman, and Michael Frank. Is child-directed speech effective training data for language models? In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22055-22071, Miami, Florida, USA, November 2024. Associ- ation for Computational Linguistics. doi: 10.18653/vi /2024.emmll-main. 1231. URL attps://aclanthol//
2rgg/224.emmlpmmaai.... monlp-main.1222
Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 451-4452, 2023.
Richard Futrell and Kyle Mahowald. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences, pp. 1-98, 2025. doi: 10.1017/ 50140525x2510112x
Richard Futrell, Roger Levy, and Edward Gibson. Dependency locality as an explanatory principle for word order. Language, 96(2):371-412, 2020.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Koreney, Arthur Hinsvark, et al. The llama 3 herd of models, 2024. 21783.
URL
Robert Hawkins, Takateru Yamakoshi, Thomas Griffiths, and Adele Goldberg, Investigating representations of verb bias in neural language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4653-4663, Online, November 2020. Association for Computational Linguistics. doi: 10.18653 /0003//112000 http//0eeemm/mmmai...... URL https:
/ //aclanthor hoc.lanthloog.orgg//oo org.202.. emnlp-main 3766
11

---

Published as a conference paper at COLM 2025
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-ttrength natural language processing in python, 2020.
Zachary Nicholas Houghton, Kenji Sagae, and Emily Morgan. The role of abstract representations and observed preferences in the ordering of binomials in large lan- guage models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Associalion for Computational Linguistics (Volume 2: Short Papers), pp. 695-702, Vienna, Austria, July 2025, Association for Computational Linguistics. ISBN 979-8-89176-252-7. URL https://aclantholoog org/ org/2025. acl short.55.
Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert- Threlkeld. Language models use monotonicity assess NPI licensing. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Com-
putational Linguistics: ACL-IJCNLP 2021, pp. 4958 4969, Online, August 2021. Associ- ation for Computational Linguistics. doi: 10.18653/v1, (2653////2221....// htt..22...innnns--a4443333 URL https://aclanthoooog..or//// .ttp//inn.....ing...s.... acl. 439/.
Jaap Jumelet, Lisa Bylinina, Willem Zuidema, and Jakub Szymanik. Black big boxes: Do language models hide a theory of adjective order?, 2024. URL htttps://axiii org/abs/
02136.
Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. Mission: Impossible language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14691-14714, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/vl
2024.acl-long.777 787. URL
787.
Cara Su-Yi Leong and Tal Linzen. Testing learning hypotheses using neural networks by manipulating learning data, 2024. URL https::/aarxv..orgaabb////// 04593..
Beth Levin. English Verb Classes and Alternations: A Preliminary Investigation. University
of Chicago Press, Chicago, IL, September 1993. ISBN 978-0-22---775333-2 URL https:
//press. ucchicago.eduuuup/bboooooo
Zoey Liu and Emily Morgan. drepeencyydependeent regularization in constituent ordering preferences. In Annual Meeting of the Cognitive Science Society, 2020. URL h/tps:
semanticscholes .aeer.og///ooo /CCoppusDD:2317920411
Zoey Liu and Stefanie Wulff. The development of dependency length minimization in early child language: A case study of the dative alternation. In Owen Rambow and François Lareau (eds.), Proceedings of the Seventh International Conference on Dependency Linguistics (Depling, CURT/Syntaxeest 2023), pp. 1-8, Washington, D.C., March 2023. Association for Computational Linguistics. URL ht hatp:://aclanthhlggg..o orgg 222//2223.. dep
Maryellen C. MacDonald. How language production shapes language form and compre- hension. Frontiers in Psychology, 4, 2013. ISSN 1664-1078. doi: 10.3389/fpsyg 00226. URL http:://www.rrottiee www.:roontterrn.ooogo/oooaaaa//s htt.://wwr....ooooooooo//o////oe 3389/ fpsyg. 2013.0022..
Kanishka Misra. minicons: Enabling flexible behavioral and representational analyses of transformer language models, 2022. URL http:/:///aarv..og/a//////// 13112.
Kanishka Misra and Najoung Kim. Generating novel experimental hypotheses from language models: A case study on cross-dative generalization, 2024. URL https:
//arxiv.org/aaee wtt.or///b////..... 05880
Kanishka Misra and Kyle Mahowald. Language models learn rare phenomena from less rare phenomena: The case of the missing AANNs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 913-929, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1 2024.emmmp-mmmn.55.. URL https:
www.aaathooog.org///oo ennlp-main.53/
12

---

Published as a conference paper at COLM 2025
Miyu Oba, Yohei Oseki, Akiyo Fukatsu, Akari Haga, Hiroki Ouchi, Taro Watanabe, and Saku Sugawara. Can language models induce grammatical knowledge from indirect evidence? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20591-20603, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 1111663/v1/2224.mmlppp 1146/.
URL https: https://aclanttie /aaaaaaaaaaaaaaoyaaaaaaaa org/2024. www....n11..1mmemmmmmmmm smnlp-main.
Abhinav Patil, Jaap Jumelet, Yu Ying Chiu, Andy Lapastora, Peter Shen, Lexie Wang, Clevis Willrich, and Shane Steinert-Threlkedd. Filtered corpus training (FiCT) shows that language models can generalize from indirect evidence. Transactions of the Association for Computational Linguistics, 12:1597-1615, 2024. doi: 10.1162 h/t////all.// a 00720. URL https: //aclanthology. o22/2024. tacl-1.877
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019. URL
clw.ffroot..oooooooooomm net/bbeterrllanuggg--oeee mooella....uunspprrrssssmmiitiis
Ada Tur, Gaurav Kamath, and Siva Reddy. Language models largely exhibit human- like constituent ordering preferences. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), pp. 1498-25211 Albuquerque, New Mexico, April 2025. Association for Computa- tional Linguistics. ISBN 979-8-99176-189-6 doi: 10.18653. (1166///v1222..maallllmm https://aclanthologgy org 2025. naacl-long 126/.
URL
Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.), Proceedings of the BabyLM Challenge the 27th Conference on Computational Natural Language Learning, pp. 1-34, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1. 11111166611/222..... http:/0002..olll-aall....m URL https://aclantholll org/ org/2023..2.
1/.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, et al. Transformers: State-of-the-arr natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 38-45, Online, October 2020. Association for Computational Linguistics. doi: 010663//1/220eemmlp-emmeee URL https://aclannthologg. org/ org/2020 ennlp-demos. 6/.
Tianyang Xu, Tatsuki Kuribayashi, Yohei Oseki, Ryan Cotterell, and Alex Warstadt. Can language models learn typologically implausible languages?, org/abs/2502.1 12005022..111.... 12317.
[anguuggell 2025. URL https://arxiv.
Hiroko Yamashita and Franklin Chang. "long before short" preference in the production of head-final language. Cognition, 81(2):B4-BB55 September 2001, ISSN 0010-0277. doi: 10.1016 0000000002200000002111111
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models, 2022.
13

---

Published as a conference paper at COLM 2025
A
Training Details
Our models are trained largely following the procedure in Misra & Mahowald (2024). We have total of eleven model-types, of which nine are discussed in the main text (with two following more strict definition of datives discussed in App. D), and for each type, we conducted three training runs with different seeds to account for variation due to initialization. In total, this amounts to 33 LM training runs. See Table 3 for a summary of training details- haalllss-hpppppramttettt compute hardware, etc.
B
List of Models on Huggingface
Huggingface paths to three seeded runs of each model and their corresponding training sets shown below, where xelstrict_default, loose desaule strict serrit_balanced loose balanced, swapped-datives, no-datives, no-pposveaall short-first, random-first, long-first, long-first- headfinal).
Model path: q2ng- 62)
Dataset path: tiiasstts/iinnyooo/daativeee///
Hyperparameter
Value
Architecture
OPT (Zhang et al., 2022)
Embed size
768
FFN dimension
3072
Num. layers
12
Attention heads
12
Vocab size
32768
Max seq. length
256
Warmup steps
32000
Total parameters
110M
Compute
1x NVIDIA A40
Training time
11 hours
Table 3: Training details
C Regression Models
C.1 Per-model Regressions
For each LM, we treat length difference and animacy difference as predictors for predicting DO preference scores across three seeds, and run the following regression model to obtain B coefficients for length and animacy.
The logic of this regression, which we run separately for each model, is to ask whether length and animacy effects are significantly different from
1_ full score Length_differenee
mnimacy_differentt (1 length_dii fifereence + animacy di fference verb_lemma) +
(1 length fference animacy di fference seed)
Here, length difference and animacy difference are fixed effects. Verb lemma has random intercepts and slopes for both fixed effects, with correlations, and seed has random intercepts only.
We compute significance scores for length difference and animacy difference in each LM is via a nested model comparison, using the command: anova mova(1_full, l_reduced), where l_ reduced excludes the fixed effects related to the predictor.
14

---

Published as a conference paper at COLM 2025
C.2 Global Regression Models
We also fit a global model, in which we treat default baseline and print its output below. In this model, we used random intercepts for verb lemma and random seed, with random slopes for length difference and animacy difference. We excluded other random slopes and correlations for convergence.
The logic of this regression is to ask whether length and animacy effects, for each model, differ significantly from the length and animacy effects the default model. We use the 1merTest package to compute significance (Satterthwaite's method).
In Table 4, the interaction coefficients (e.g,, balanced : Length Difference) can be thought of as telling how much the Length or Animacy difference differs in the given condition, opposed to the default baseline.
1_global <- score ~ www.riiowwenengtdiieeeeeeeeeeeee
condi www.tiiinnnnmmcniiiirirree
(1 Length_differeee + animacy_di _iffferencefeee ||
verb_lenma)
(1 dength_different animacy_di fference ||
seed)
Predictor
Fatimate
Intercept
-0.45ge4
balanced
0.0
no-datives
no-Rpostverbal
0.087xx
swapped-dative
0,121
short-first
0,127
0.13g
long-firat
0.24gex4
long-first-ie kong-First-headfinal
0.248
Length Difierence
0.18gre
Animacy Difference
0.091 **
balanced: Length Difference
0.105 ***
no-datives enggt Difference
no-2postverball Length Difference
0.140**
snappad-datives Length Difference
0.227
short-first: Length Difference
0,131
first: Length Difference
0.187
long-first: Length Difference
0.225
long-firat-haadfl Length Difference
0.266***
balanced: Animaacy Difierence
0.117'ex
no-datives: Animacy Difference
-0.10geee
no-2postverbal: Animacy Difference
-0.10geee
swapped datives: Animacy Difference
-0.1
Animacy
00.07)
randon-first:: Animacy Difference
0.066'*x
long-first: Animacy Difference
0.034
Long-färat-headfinn Animacy
Table 4: Summary of estimates from model comparing default with significance levels indicated by asterisks (***p 0.001, "'p p < 0.01, "p < 0.05).
For Experiment 2, we fit another regression (same as the global regression above) to just the length-manipulated data in Experiment 2, with random-first as the baseline. The logic of this regression is to test whether the length-manipulated models are significantly different from the random-first models in length effects. (We also compute animacy effects here, but have no predictions about them for this set of models.)
15

---

Published as a conference paper at COLM 2025
Predictor
Estimate
Intercept
0.307
short-first
0.012
long-first
0.110
long-first-hen firssttheahafinl
0.109
Length Difference
0.005
Animacy Difference
0.002
short-first Length difference
0.055
long-first: Length Difference
0.039
Long-firstt-headfinal Length Difference
0.080**
short-first: Animacy Difference
0.004
long-first: Animacy Difference
0.032*
long-first-hheadfinal Animacy Difference
0.060
Table Regression output focusing on just the length manipulations, with the random-first condition as baseline (*** p < 0.001, "pp < 0.01, < 0.05).
D Strict-dative default and balanced Results
default
balanced
r
-0.43
r -0.34
10-
0.5
0.0-
Exp.
Model
Length
Animacy
DO Preference
-05
All
default - 0..174** 0.062
-1.
Log Difference in Recipient and Theme Lengths
1a
balanced
- =0.072***
0.029
Table Regression coefficients from mixed effect model, fit separately for strict default and balanced predicting DO preference, with for p < .001.
M
B
M
8
Recipient More, Equally, or Less animate
Figure 7: Correlation between DO preference and length difference and animacy difference for strict default and balanced.
We refer to the datives defined in the main paper loose-datives, and introduce a stricter notion of datives, which we term strict-datives. These are proper subsets of loose-datives. Here we take stricter approach and only consider loose-datives whose verbs satisfy the following conditions, based on the classification laid out by Levin (1993): 1) it is classified as to-dative benefactive verb; and 2) its usage in the detected entry is consistent with its alternation behavior (e.g. verb is classified as PO only but detected in a DO, verb is classified to be benefactive but is used in to-dative).
This definition results in 125,415 DOs and 66,822 POs, compared to 139,249 DOs and 118,040 POs under the loose definition.
We now show that the choice between strict and loose classifications has little effects on our analysis by repeating Experiment 1a under the strict notion.
16

---

Published as a conference paper at COLM 2025
defuutt
01
02
03
Random Slopes for Animacy
01
43
00
43
43
Sattier
03
00
41
03
Figure Verb-specific random slope-difference (differences from the global random slope) for the animacy term in our linear mixed-effects model analysis. Positive value indicates relatively greater DO preference for animate recipients, while negative ones indicate lower DO preference for inanimate recipients.
Both the regression coefficients in Table 6 and the length and animacy effects in Fig. for the strict models are similar to their loose counterparts. We therefore focus just the loose notion in the main paper, so that the ablated models are more aggressively removing direct evidence.
Verb-specific Random Slopes for Animacy
Random slopes by verb lemma for default, balanced, swapped-datives, and no-datives shown in Fig. We see that animacy effects for certain verbs are heavily affected by the manipulations in balanced and swapped-dative of these verbs are more tied to direct evidence. However, no-datives recovers similar, albeit bit weaker, animacy effects as in default. This suggests that usages of dative verbs in the indirect evidence can inform an LM's dative animacy preferences.
indicating that the animacy preferences
F Detecting Datives and 2postverbals
Using spaCy, first detect all utterances that could contain DO PO dative. For DOs, we search for utterances containing verb which has dative and a dobj dependency or two dobj dependencies. For POs, we look for utterances containing a verb with a dobj dependency, and with either to prepositional phrase under a dative prep dependency or a for prepositional phrase under a dati ve dependency. From these entries, we perform sanity check to ensure that the noun phrases come after the verb, and the recipient and theme have appropriate parts of speech.
These entries are then loose-datives. From these entries, we check if the verb lemma is classified as a dative or benefactive verb in Levin (1993). If so, we check if it in the permissible form (e.g. if the verb is classified as benefactive only, the entry cannot be to-PO, or the verb is classified as DO only, the entry cannot be a PO). If a loose-dative satisfies the above, it is also strict-dative.
17

---

Published as a conference paper at COLM 2025
The entries which are not detected loose-datives are not necessarily non-datives, since no small set of definitions based on dependency parses can capture all datives. To ensure high recall, the non-dative set is obtained by removing ambiguous entries from the set of non-loose-dativer Via a manual check, we found that undetected datives generally include verbs having both dobj and clausal complement dependency (as in So you' re telling me this is over?) and entries with verbs having a dobj and a prep arc to for with a pobj dependency (as in He played 38 games for Japan until 1971). The remaining non-loose-dative entries after this procedure are considered non-datives.
Lastly, we obtain the set of non-2postverbal from the of non-datives by further removing entries containing verb with two direct objects of any kind, and entries containing verb with dobj and prep-pobj of any kind.
An implementation of this procedure is in the code repository under src/detect_datives,
G Artificial Pollution
The counts of dative exposure by model are listed in Table 7.
Type
default
balanced
no-datives
no-2postverbal
DO
66822
65700
0
0
Controlled datives
PO
66822
65700
0
0
DO
1500
1500
1500
1500
False negatives (estimate)
750
750
750
750
DO
1500
1500
1500
Counterfactuals
PO
0
750
750
750
Total (estimate)
DO
68322
68700
3000
3000
PO
67572
67200
1500
1500
% of total utterances
1.55
1.55
0.05
0.05
Table 7: Number of dative exposures by model. False negatives are estimated based on an error rate of 1 /40000. sength-manipulated models are the same no-2posterrbal apart from constituent ordering, and wrapped-dativee is the same as default apart from controlled datives being in their unnatural alternants.
H Manipulating Constituent Ordering by Length
We use the following algorithm to reorder all constituents of sentence to short/long cooott/longfirss first using spaCy dependency parses. Sentences for the random-first model are created by randomly choosing short/long-fire when sorting the children of each node.
18

---

Published as a conference paper at COLM 2025
Algorithm 1 Length Manipulation in Syntax Tree
1: function (tokee,, visited, short first, head final)
2:
if token E visited then
3: return /ookee
4: end if
5: Recursively process child nodes
6: if short_first then
7: Sort children by constituent length ascending
8:
else
9:
Sort children by constituent length descending end if
10:
11:
Reassign original positions to maintain relative ordering if head final then
13:
Set current node's constituent as the concatenation of its children constituents, with the node itself the end else
14:
15:
Set current node's constituent as the concatenation of its children constituents end if
16:
17:
Store node in visited dictionary
18:
return node
19:
end function
These algorithms can be found in the code repository under src/utils.pp py.
I Measuring the Short-Firstness of English
We provide a more fine-grained measurement of the degree to which English sentences short-first or long-first in terms of the number of adjacent swap-operations required to rearrange the sentence in length-sorted order.
For each sentence, compute the inversion number corresponding to head that has n 2 dependents as the number of out-of-order pairs (w.rt. to short-first or long-first). The inversion number of head precisely equal to the number of adjacent swaps to sort its dependents. As such, head with π dependents has an inversion number at most (")).
We then add up the inversion numbers of each head within the sentence, and normalize by the maximum possible sum of inversion numbers (attained when dependents of every head are maximally out-of-order) obtain a measurement of poauttllon-ffsstmme
For entries where constituent rearrangements were possible (i.e. there exists one head with two or more dependents), it takes 23.8% of the maximal number of swaps to reach short-first, compared to 53.2% for long-first. This means that, on average, it takes more than twice the number of adjacent-swap operations to sort by long-first compared to short-first.
19


Reviews:
[{"review_id": "l890sbQFSU", "reviewer": "Reviewer_tLzp", "summary": "The paper sets out to investigate whether the preferences observed in language models on a number of syntactic phenomena are directly attributable to exposure to the phenomenon or to general properties of language. \n\nThey apply this investigation to the dative alternation (I gave a book to Bill/I gave Bill a book), concentrating on length (it is ok to say 'I gave him a book' but much less good to say 'I gave the boy that you met yesterday and did not like a book', preferred option would be 'I gave a book to the boy that you met yesterday and did not like' and animacy  effects (animate recipients is much preferred as first argument).\n\n\nThe results are that a mixture of construction specific (for example, verb-specific) effects and general effects are at play. \n\nBoth the (rather vague, in my opinion) question and the linguistic phenomenon are much studied, as are the properties here investigated (order and animacy). The results are rather unsurprising, and confirm a lot of previous evidence on dependency length minimisation and animacy effects.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The controlled rearing  methodology is interesting and supports very well controlled experimentation\n- The experimental set up is  very careful and controlled", "reasons_to_reject": "I am afraid I found the paper rather thought-un-provoking, testing rather intuitive hypotheses.\nI am afraid I think that the same results could have been gleaned by carefully analyzing attested dependency length effects and other work on datives\n\nI don't feel I learnt much, despite the interesting method.", "questions_to_authors": "Perhaps I can suggest exploring the more verb-dependent biases and try to develop methods to study semantic roles? Length and animacy (as expressed by pronouns, a property of English) have been widely studied. Studying deeper abstract properties might use the interesting method to greater insight.", "extra_scores": {}}, {"review_id": "MXvBwogB6i", "reviewer": "Reviewer_BYEk", "summary": "This paper uses the controlled rearing methodology to assess the degree to which LMs rely on direct and indirect evidence when choosing to output double object or prepositional object forms of dative sentences. That is, they train LMs on various different datasets, manipulating the extent and types of direct / indirect evidence available at training time; they then evaluate LM preferences. They find that even when all direct evidence concerning when one or another form of dative should be output has been removed from the training corpus, LMs still exhibit a preference for shorter constituents first (animacy effects are weaker). They then re-train on data where length ordering is altered, and find that constituent length ordering evidence does influence LMs' preferences re: dative alternation. They conclude that models rely on both direct and indirect evidence.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In general, I think this is a strong paper, for the following reasons:\n- The controlled rearing methodology is one of the best ways of getting causal evidence that a certain phenomenon in the training data led to a given behavior on the part of the LM. It takes quite a bit of effort to do this, both in terms of curating the training data and training the actual models, and the authors have trained multiple seeds for each LM.\n- This paper seems well grounded in the human psycholinguistics literature, although I'm not an expert on this particular phenomenon.\n- This paper is in general quite well-polished and readable", "reasons_to_reject": "- Experiment 2 is, in my view, less convincing than experiment 1. This is because the corpora in experiment 2 are unnatural / composed of lots of data that is not grammatically correct English. As a result, one wonders if the effects observed are really only a result of this rather dramatic shift in the corpus, as opposed to the more subtle length ordering rules that exist in English in reality. I think the ideal experiment here would be to reorder only those constituents that are allowed to be reordered. That is, in the long-first case, \"He uses a fork to eat the green melon from the shop\" could become \"To eat the green melon from the shop, he uses a fork\", but not \"from the shop the melon green eat to uses a fork he\". I would expect effect sizes to shrink somewhat. That said, I think this sort of complex reordering would be quite difficult to do, so I understand why the current experiment was done instead.\n- The framing / motivation for this project is a bit weak. \n    - The papers cited in favor of the \"direct evidence only\" view are not that convincing from a methodological point of view, and also not very influential. I mention the latter point because I think there are more influential voices arguing that LLMs' abilities come in large part from copying / direct evidence. I realize that these are murky waters to wade into, but engaging with higher-level debates around LLMs could help emphasize what is at stake here. \n    - Similarly, I expected to see some sort of discussion of these results in relation to humans, and was a bit disappointed to find that missing.\n    - Relatedly, there have been a few papers now using controlled rearing to show how LLMs use direct and indirect evidence to learn various constructions. I agree that this is now a \"growing consensus\", but I'm also left wondering what the bigger picture is. I'm now left wondering what the bigger picture is - why does this matter, and what next?", "questions_to_authors": "- 1b and 1c are swapped in Table 1\n- 186: Section 2.1, did you mean?\n- Figure 3 is a bit tough on readers, as it is referenced a couple of times later in the text as well. Maybe split it up, but have full version in the appendix for comparison?\n- 342-343: This would be clearer if written in the active voice", "extra_scores": {}}, {"review_id": "cbF5ElyWWb", "reviewer": "Reviewer_M82d", "summary": "This paper explores the behavior of LMs on the English dative alternation. First, the author ensures that a \"babyLM\" shows roughly the same behavior as larger LMs on this alternation. They then go on to do controlled rearing experiments, to manipulate the evidence, i.e., the datives seen during training, to explore the preference of the model, and the impact of length and animacy preferences, as well as indirect evidence. The paper is very well written and clear, with well-motivated and interesting experiments, shedding some light on the dative alternation, as well as on the impact of general language preferences.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper explores interesting questions of language preferences in a sound setup, leading to interesting conclusions. I especially find it interesting that you manage to investigate both indirect and direct evidence for the different preferences.\n\nThe controlled rearing method is interesting, since it can provide causal evidence. The use in this paper is sound and interesting, and could potentially provide inspiration for future studies of other phenomena.", "reasons_to_reject": "No major reason to reject\n\nA couple of minor issues (which should not affect acceptance, but which I did not find a good place to put):\n\n* In the conclusion, you draw general conclusions about language, even though you just experimented on English. Please rephrase (it is clear that everything is for English in the rest of the paper, though).\n* Please try to make the figures and tables larger in the final paper. They are now hard to read.\n* Since you manually verified 2000 samples in 2.5, it would have been preferable also to report how accurate this sample was", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MgT524rWzg", "reviewer": "Reviewer_BPUy", "summary": "This draft investigates the process of learning English dative construction by means of language models. The draft looks into the alternation between double object and prepositional object in relation to two features: length and animacy of the arguments. Many experiments are presented, and the main finding is that preferences emerge from a mix of direct exposure to the phenomena and more general properties of the language.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Experiments have been carefully designed and discussed. \nInteresting and novel results on an already established line of research.", "reasons_to_reject": "I do not see any reason to reject.", "questions_to_authors": "l.126: Each of the our LMs", "extra_scores": {}}]

Author rebuttal:

Meta-review: Dative sentences in English usually sound better if the first argument is shorter than the second (regardless of whether the first argument is the goal or the theme). This paper shows that small language models show this preference as well when trained on a natural corpus, and then explores ablations of the corpus to investigate the source of this preference. It finds that this preference emerges even when datives are excluded from training, suggesting that the short-first preference is language-wide.

This is a clear and methodologically well-executed paper, with clever controls (e.g., balancing the effect of possible failures to exclude particular constructions by "polluting" the corpus with noise in the opposite direction). I share the concerns of reviewer tLzp about the scientific conclusions, which seems fairly modest. It's not clear to me if anyone (from Behagel onwards) has argued that the short-before-long principle is specific to datives, so the experiments aren't contributing to a live debate. In the discussion, the authors respond to these concerns by framing the paper as showcasing a new methodology, but that doesn't seem convincing to me as the methodology is quite similar to Mishra and Mahowald (2024) and a cluster of similar papers (cited in this manuscript). I also thought reviewer BYEk's concerns with Experiment 2 were valid. As a minor point, I wonder if it is fair to say that models show human-like preferences when their correlation with human judgments is only r = 0.5.

Overall, although this paper is somewhat incremental, it is generally well executed and I don't see a risk in accepting it.
Decision: Accept