Language Models as an Alternative Evaluator of Word Order Hypotheses:
A Case Study in Japanese
Tatsuki Kuribayashi1,3 , Takumi Ito1,3 , Jun Suzuki1,2 , Kentaro Inui1,2
1
Tohoku University 2 RIKEN 3 Langsmith Inc.
{kuribayashi, t-ito, jun.suzuki, inui} @ecei.tohoku.ac.jp

generation
probabilities

Abstract
We examine a methodology using neural language models (LMs) for analyzing the word
order of language. This LM-based method has
the potential to overcome the difficulties existing methods face, such as the propagationof preprocessor errors in count-based methods. In this study, we explore whether the LMbased method is valid for analyzing the word
order. As a case study, this study focuses on
Japanese due to its complex and flexible word
order. To validate the LM-based method, wetest (i) parallels between LMs and human word
order preference, and (ii) consistency of theresults obtained using the LM-based methodwith previous linguistic studies. Through ourexperiments, we tentatively conclude that LMs
display sufficient word order knowledge for usage as an analysis tool. Finally, using the LMbased method, we demonstrate the relationshipbetween the canonical word order and topicalization, which had yet to be analyzed by largescale experiments.

1

Introduction

Speakers sometimes have a range of options forword order in conveying a similar meaning. A
typical case in English is dative alternation:(1) a. A teacher gave a student a book.
b. A teacher gave a book to a student.
Even for such a particular alternation, several studies (Bresnan et al., 2007; Hovav and Levin, 2008;
Colleman, 2009) investigated the factors determining this word order and found that the choice is
not random. For analyzing such linguistic phenomena, linguists repeat the cycle of constructing hypotheses and testing their validity, usually throughpsychological experiments or count-based methods.
However, these approaches sometimes face difficulties, such as scalability issues in psychologicalorder! : 質に影響をquality-DAT effect-ACC

order" : 影響を質に

effect-ACC quality-DAT

与えた.
gave.

与えた.

0.0002LM
0.0000001

gave.

(∅!" affected the quality.)order1 is more likely.

Figure 1: LM-based method for evaluating the canonicality of each word order considering their generation
probabilities.

experiments and the propagation of preprocessor
errors in count-based methods.
Compared to the typical approaches for evaluating linguistic hypotheses, approaches using LMs
have potential advantages (Section 3.2). In thisstudy, we examine the methodology of using LMs
for analyzing word order (Figure 1). To validate
the LM-based method, we first examine if there is aparallel between canonical word order and generation probability of LMs for each word order. Futrelland Levy (2019) reported that English LMs havehuman-like word order preferences, which can be
one piece of evidence for validating the LM-based
method. However, it is not clear whether the above
assumption is valid even in languages with more
flexible word order.
In this study, we specifically focus on theJapanese language due to its complex and flexible
word order. There are many claims on the canonicalword order of Japanese, and it has attracted considerable attention from linguists and natural language
processing (NLP) researchers for decades (Hoji,
1985; Saeki, 1998; Miyamoto, 2002; Matsuoka,
2003; Koizumi and Tamaoka, 2004; Nakamoto
et al., 2006; Shigenaga, 2014; Sasano and Okumura, 2016; Orita, 2017; Asahara et al., 2018).
We investigated the validity of using JapaneseLMs for canonical word order analysis by conducting two sets of experiments: (i) comparing word
order preference in LMs to that in Japanese speakers (Section 4), and (ii) checking the consistency

Notation
Typical particle
Related section

Topic

Time

Location

Subject(Adverb)

Indirect object

Direct object

Verb

TOP
“は” (wa)6

TIM“に” (ni)
5.2

LOC“で” (de)5.2

NOM“が” (ga)
5.2

5.3

DAT
“に” (ni)
5.1

ACC“を” (o)5.1

5.1

Table 1: Overview of the typical cases in Japanese, their typical particles, and the sections where the correspondingcase is analyzed. The well-known canonical word order of Japanese is listed from left to right.

between the preference of LMs with previous linguistic studies (Section 5). From our experiments,
we tentatively conclude that LMs display sufficient
word order knowledge for usage as an analysis tool,and further explore potential applications. Finally,
we analyzed the relationship between topicalization
and word order of Japanese by taking advantage of
the LM-based method (Section 6).
In summary, we:• Discuss and validate the use of LMs as a tool
for word order analysis as well as investigate
the sensitivity of LMs against different word
orders in non-European language (Section 3);• Find encouraging parallels between the results
obtained with the LM-based method and thosewith the previously established method on various hypotheses of canonical word order ofJapanese (Sections 4 and 5); and• Showcase the advantages of an LM-based
method through analyzing linguistic phenomena that is difficult to explore with the previous data-driven methods (Section 6).

2

Linguistic background

This section provides a brief overview of the linguistic background of canonical word order, somebasics of Japanese grammar, and common methodsof linguistic analysis.
2.1

On canonical word order

Every language is assumed to have a canonicalword order, even those with flexible word order (Comrie, 1989). There has been a significant
linguistic effort to reveal the factors determining
the canonical word order (Bresnan et al., 2007;
Hoji, 1985). The motivations for revealing the
canonical word order range from linguistic interests to those involved in various other fields—it
relates to language acquisition and production in
psycholinguistics (Slobin and Bever, 1982; Akhtar,
1999), second language education (Alonso Belmonte et al., 2000), and natural language generation (Visweswariah et al., 2011) or error cor-

rection (Cheng et al., 2014) in NLP. In Japanese,
there are also many studies on its canonical word
order (Hoji, 1985; Saeki, 1998; Koizumi andTamaoka, 2004; Sasano and Okumura, 2016).
Japanese canonical word order The word order of Japanese is basically subject-object-verb(SOV) order, but there is no strict rule except placing the verb at the end of the sentence (Tsujimura,2013). For example, the following three sentences
have the same denotational meaning (“A teacher
gave a student a book.”):あげた.
(2) a. 先生が
生徒に
本を
:::::
..............
teacher-NOM student-DAT book-ACC gave.
b. 先生が
本を
生徒に
あげた.
:::::
..............
teacher-NOM book-ACC student-DAT gave.
c. 本を
生徒に
先生が
あげた.
:::::
..............
book-ACC student-DAT teacher-NOM gave.

This order-free nature suggests that the positionof each constituent does not represent its semanticrole (case). Instead, postpositional case particles indicate the roles. Table 1 shows typical constituentsin a Japanese sentence, their postpositional particles, their canonical order, and the sections of thispaper where each of them is analyzed. Note that
postpositional case particles are sometimes omitted
or replaced with other particles such as adverbial
particles (Section 6). These characteristics complicate the factors determining word order, which
renders the automatic analysis of Japanese word
order difficult.
2.2

On typical methods for evaluating word
order hypotheses and their difficulties

There are two main methods in linguistic research:human-based methods, which observe human reactions, and data-driven methods, which analyze text
corpora.
Human-based methods A typical approach of
testing word order hypotheses is observing the reaction (e.g., reading time) of humans to each word
order (Shigenaga, 2014; Bahlmann et al., 2007).
These approaches are based on the direct observation of humans, but this method has scalability

issues. There are also concerns that the participants
may be biased, and that the experiments may not
be replicable.

Data-driven methods Another typical approachis counting the occurrence frequencies of the targeted phenomena in a large corpus. This countbased method is based on the assumption that there
are parallels between the canonical word order andthe frequency of each word order in a large corpus.
The parallel has been widely discussed (Arnon andSnider, 2010; Bresnan et al., 2007), and many studies rely on this assumption (Sasano and Okumura,
2016; Kempen and Harbusch, 2004). One of theadvantages of this approach is suitability for largescale experiments. This enables considering a large
number of examples.
In this method, researchers often have to identify the phenomena of interest with preprocessors(e.g., the predicate-argument structure parser usedby Sasano and Okumura (2016)) in order to countthem. However, sometimes, identification of the targeted phenomena is difficult for the preprocessors,which limits the possibilities of analysis. For example, Sasano and Okumura (2016) focused only on
simple examples where case markers appear explicitly, and only extract the head noun of the argument
to avoid preprocessor errors. Thus, they could notanalyze the phenomena in which the above conditions were not met. The above issue becomes
more serious in low-resource languages, where thenecessary preprocessors are often unavailable.
In this count-based direction, Bloem (2016) used
n-gram LMs to test the claims on the German twoverb clusters. This method is closest to our proposed approach, but the general validity of using
LMs is out of focus. This LM-based method alsorelies on the assumption of the parallels between
the canonical word order and the frequency.
Another common data-driven approach is totrain an interpretable model (e.g., Bayesian linear mixed models) to predict the targeted linguistic phenomena and analyze the inner workings ofthe model (e.g., slope parameters) (Bresnan et al.,2007; Asahara et al., 2018). Through this approach,researchers can obtain richer statistics, such as thestrength of each factor’s effect on the targeted phenomena, but creating labeled data and designingfeatures for supervised learning can be costly.

3LM-based method

3.1

Overview of the LM-based method

In the NLP field, LMs are widely used to estimate
the acceptability of text (Olteanu et al., 2006; Kann
et al., 2018). An overview of the LM-based method
is shown in Figure 1. After preparing several word
orders considering the targeted linguistic hypothesis, we compare their generation probabilities inLMs. We assume that the word order with the highest generation probability follows their canonicalword order.
3.2

Advantages of the LM-based method

In the count-based methods mentioned in Section 2.2, researchers often require preprocessors
to identify the occurrence of the phenomena ofinterest in a large corpus. On the other hand, researchers need to prepare data to be scored by LMs
to evaluate hypothesis in the LM-based method.
Whether it is easier to prepare the preprocessor or
the evaluation data depends on the situation. For example, the data preparation is easier in the situation
where one wants to analyze the word order trends
when a specific postpositional particle is omitted.
The question is whether Japanese speakers prefer
the word order like in Example (3)-a or (3)-b.1(3) a. 生徒に本をあげた.
..............
student-DAT book(-ACC) gave.
生徒に
あげた.
b. 本を
..............
book(-ACC) student-DAT gave.

While identifying the cases (ACC in Example (3))
without their postpositional particle is difficult, creating the data without a specific postpositional particle by modifying the existing data is easier suchas creating Example (4)-b from Example (4)-a.
(4) a. 生徒に本をあげた.
..............
student-DAT book-ACC gave.
b. 生徒に本をあげた.
..............
student-DAT book(-ACC) gave.

Thus, in such situation, the LM-based method canbe suitable.
The human-based method is more reliable given
an example. However, it can be prohibitively costly.
While the human-based method requires an evaluation data and human subjects, the LM-based
method only requires the evaluation data. Thus,
the LM-based method can be more suitable for estimating the validity of hypotheses and considering
1

Omitted characters are crossed out. (e.g., を)many examples as exhaustively as possible. In addition, the LM-based method can be replicable. The
suitable approach can be different in a situation,and broadening the choice of alternative methodologies may be beneficial to linguistic research.
Nowadays, various useful frameworks, language
resources, and machine resources required to train
LMs are available,2 which support the ease of implementing the LM-based method. Moreover, we
make the LMs used in this study available.3
3.3

Strategies to validate the use of LMs toanalyze the word order

The goal of this study is to validate the use of
LMs for analyzing the canonical word order. The
canonical word order itself is still a subject of research, and the community does not know all about
it. Thus, it is ultimately impossible to enumerate
the requirements on what LMs should know about
the canonical word order and probe the knowledgeof LMs. Instead, we demonstrate the validity of
the LM-based method by showcasing two types ofparallels: (i) word order preference of LMs showing parallels with that of humans, and (ii) the results obtained with the LM-based method and thosewith previous methods being consistent on various
claims on canonical word order. If the results of
LMs are consistent with those of existing methods,
the possibility that LMs and existing methods havethe same ability to evaluate the hypotheses is supported. If the LM-based method is assumed to bevalid, the method has the potential to streamline theresearch on unevaluated claims on word order. In
the experiment sections, we examine the properties
of Japanese LMs on (i) and (ii).
3.4

CAUTION – when using LMs forevaluating linguistic hypotheses

Even if LMs satisfy the criteria described in 3.3,
there is no exact guarantee that LM scores will reflect the effectiveness of human processing of specific constructions in general. Thus, there seems
to be a danger of confusing LM artifacts withlanguage facts. Based on this, we hope that researchers use LMs as a tool just to limit the hypothesis space. LM supported hypotheses should thenbe re-verified with a human-based approach.
2
For example, one can train LMs with fairseq (Ott et al.,2019) and Wikipedia data on cloud computing platforms.
3
https://github.com/kuribayashi4/LM_
as_Word_Order_Evaluator.

Furthermore, since there is a lot of hypotheses
and corresponding research, we cannot check all
the properties of LMs in this study. This study
focuses on intra-sentential factors of Japanese caseorder, and it is still unclear whether the LM-based
method works properly in linguistic phenomena
which are far from being the focus of this study.
This is the first study where evidence is collected on
the validity of using LMs for word order analysis
and encourages further research on collecting such
evidence and examining under what conditions thisvalidity is guaranteed.
3.5

LMs settings

We used auto-regressive, unidirectional LMs withTransformer (Vaswani et al., 2017). We used twovariants of LMs, a character-based LM (CLM) anda subword-based LM (SLM). In training SLM, theinput sentences are once divided into morphemesby MeCab (Kudo, 2006) with a UniDic dictionary,4 and then these morphemes are split into
subword units by byte-pair-encoding. (Sennrichet al., 2016)5 . 160M sentences6 randomly selectedfrom 3B web pages were used to train the LMs.
Hyperparameters are shown in Appendix A.
Given a sentence s, we calculate its generation−−−
probability p(s) = →p (s) · ←p (s), where →p (·) and←−p (·) are generation probabilities calculated by aleft-to-right LM and a right-to-left LM, respectively. Depending on the hypothesis, we compare
the generation probabilities of various variants ofs with different word orders. We assume that theword order with the highest generation probability
follows their canonical word order.

4

Experiment1: comparing human andLMs word order preference

To examine the validity of using LMs for canonicalword order analysis, we examined the parallels between the LMs and humans on the task determining
the canonicality of the word order (Figure 2). First,
we created data for this task (Section 4.1). We thencompared the word order preference of LMs andthat of humans (Section 4.2).
4https://unidic.ninjal.ac.jp/
Implemented in sentencepiece (Kudo and Richardson,2018) We set character coverage to 0.9995，and vocab size
to 100,000.
6
14GB in UTF-8 encoding. For reference, Japanese
Wikipedia has around 2.5 GB of text. Because the focus of thisstudy has context-independent nature, the sentences order isshuffled to prevent learning the inter-sentential characteristicsof the language.
5

corpusorder! :

彼が

流行りの

本を

買った

he-NOM

popularbook-ACC

gave.

LM

compareoriginal order

order" :order1 is more likely

流行りの

本を

彼が

買った

popularbook-ACC

he-NOMgave.

scrambled order

order1 is more natural

humansorder1order1Figure 2: Overview of the experiment of comparing human and LMs word order preference. First, we createddata for the task of comparing the appropriateness of
the word order (left part), then we compare the preference of LMs and humans through this task (right part).

4.1

Human annotation

Data We randomly collected 10k sentences from3B web pages, which are not overlapped with theLM training data. To remove overly complex sentences, we extracted sentences that must: (i) haveless than or equal to five bunsetsu and one verb,(ii) have bunsetsu with a sibling relationship in its
dependency tree, and they accompany a particle oradverb, (iii) not have special symbols such as parentheses, and (iv) not have a backward dependency
path. For each sentence, we created its scrambled
version.7 The scrambling process is as follows:
1. Identify the dependency structure by using
JUMAN8 and KNP9 .
2. Randomly select a bunsetsu with several children.
3. Shuffle the position of its children along with
their descendants.
Annotation We used the crowdsourcing platform Yahoo Japan!10 . For our task, we showed
crowdworkers a pair of sentences (order1 , order2 ),where one sentence has the original word order,
and the other sentence has a scrambled word order.11 Each annotator was instructed to label thepair with one of the following choices: (1) order1 is
better, (2) order2 is better, or (3) the pair contains asemantically broken sentence. Only the sentences(order1 , order2 ) were shown to the annotators, andthey were instructed not to imagine a specific context for the sentences. We filtered unmotivatedworkers by using check questions.12 For each pair
7

When several scrambled versions were possible for agiven sentence, we randomly selected one of them.
8
http://nlp.ist.i.kyoto-u.ac.jp/EN/
index.php?JUMAN
9
http://nlp.ist.i.kyoto-u.ac.jp/EN/
index.php?KNP
10
https://crowdsourcing.yahoo.co.jp/
11
Crowdworkers did not know which sentence was the original sentence.
12
We manually created check questions considering theJapanese speakers’ preference in trial experiments in advance.

instance, we employed 10 crowdworkers. In total,
756 unique, motivated crowdworkers participatedin our task.
From the annotated data, we collected only thepairs satisfying the following conditions for our
experiments: (i) none of 10 annotators determinedthat the pair contains a semantically broken sentence, and (ii) nine or more annotators preferred
the same order. The majority decision is labeledin each pair; the task is binary classification. We
assume that if many workers prefer a certain word
order, then it follows its canonical word order, andthe other one deviates from it. We collected 2.6k
pair instances of sentences.
4.2

Results

We compared the word order preference of LMs
and that of the workers by using the 2.6K pairs created in Section 4.1. We compared the word order
preference of LMs and that of the workers by using
the 2.6K pairs created in Section ??. Given a pairof word order, the accuracy of selecting the order
preferred by crowd worker is 94.7 for CLM and
95.0 for SLM. This indicates that LMs exhibit surprisingly well parallel with humans in word order
preferences. Note that such a direct comparison ofword order is difficult with the count-based methods because of the sparsity of the corpus.

5

Experiment2: consistency withprevious studies

This section examines whether LMs show word
order preference consistent with previous linguistic
studies. The results are entirely consistent, which
support the validity of the LM-based methods inJapanese. Each subsection focuses on a specificcomponent of Japanese sentences.
5.1

Double objects

The order of double objects is one of the most controversial topics in Japanese word order. Examplesof the possible order are as follows:(5) DAT-ACC: 生徒に本をあげた
student-DAT book-ACC gave.
ACC-DAT: 本を
生徒に
あげた
:::::
book-ACC student-DAT gave.

Henceforth, DAT-ACC / ACC-DAT denotes theword order in which the DAT / ACC argument precedes the ACC / DAT argument. We evaluate the1

ACC-DAT rate

ACC-DAT rate (our results)

1

0.5

0.5

0

0
0

0.5

ACC-DAT rate (S&O 2016)

0

1

0.5

1

RDAT-only(a) Each verb’s ACC-DAT (b) Relationship betweenv
rate.
each verb’s RDAT-onlyandthe ACC-DAT rate.

ACC-DAT rate

1

-1

CLM
SLM

0.5

S&O 2016
Linear approx. (CLM)

0

-0.5

0

ΔNPMI

0.5

1

Linear approx. (SLM)
Linear approx. (S&O 2016)(c) Relationship between the degree of co-occurrence of verb
and arguments, and the ACC-DAT rate in each example. For
the results of LMs, the ACC-DAT rate of each example isregarded as 1 if LMs prefer ACC-DAT order, otherwise weregard the example as 0.

Figure 3: Overlap of the results of Sasano and Okumura (2016) and that of LMs. In figures (a) and (b),each plot corresponds to each verb. In figure (c), eachplot corresponds to each example. The legend of figure (a) and (b) is the same as in figure (c). “S&O 2016”refers to Sasano and Okumura (2016).

claims Sasano and Okumura (2016) focused onwith the data they collected.13
Word order for each verb First, we analyzed
the trend of the double object order for each verb.
We analyzed 620 verbs following Sasano and Okumura (2016).14 For each set of examples S v corresponding to a verb v, we: (i) created an instancewith the swapped order of ACC and DAT for eachexample, and (ii) compared the generation probabilities of the original and swapped instance. Ŝ v isv
the set of examples preferred by LMs. RACC-DAT
iscalculated as follows:v
NACC-DATv
RACC-DAT= v,
v
NACC-DAT + NDAT-ACC
v
v
where NACC-DAT/ NDAT-ACC
is the number of examples with the ACC-DAT / DAT-ACC order inŜ v .
Figure 3-(a) shows the relationship betweenv
RACC-DAT determined by LMs and one reported in a
13

We filtered the examples overlapping with the training
data of LMs in advance. As a result, we collected 4.5M
examples.
14
We removed verbs for which all examples overlap withthe data for training the LMs.

previous count-based study (Sasano and Okumura,2016). These results strongly correlate with thePearson correlation coefficient of 0.91 and 0.88, in
CLM and SLM, respectively. In addition, “canonical word order is DAT-ACC” (Hoji, 1985) is unlikely to be valid because there are verbs wherev
RACC-DAT
is very high (details in Appendix B.1).
This conclusion is consistent with Sasano and Okumura (2016).
Word order and verb types In Japanese, there
are show-type and pass-type verbs (details in Appendix B.2). Matsuoka (2003) claimed that theorder of double objects differs depending on theseverb types. Following Sasano and Okumura (2016),
we analyzed this trends.
We applied the Wilcoxon rank-sum test bev
tween the distributions of RACC-DAT
determined
by LMs in the two groups (show-type and passtype verbs). The results show no significant difference between the two groups (p-value is 0.17
and 0.12 in the experiments using CLM and SLM,
respectively). These results are consistent withthe count-based (Sasano and Okumura, 2016) andthe human-based (Miyamoto, 2002; Koizumi andTamaoka, 2004) methods.
Word order and argument omission Sasano
and Okumura (2016) claimed that the frequently
omitted case is placed near the verb. First, wev
calculated RDAT-onlyfor each verb v as follows:v
RDAT-only=v
NDAT-onlyvv
NDAT-only+ NACC-only,vv
where NDAT-only/ NACC-only
denotes the number ofexamples in which the DAT / ACC case appears, andv
the other case does not in S v . A large RDAT-onlyscore indicates that the DAT argument is less frequently omitted than the ACC argument in S v . Wev
analyzed the relationship between RDAT-only
andv
RACC-DAT for each verb.
Figure 3-(b) shows that the regression lines
from the LM-based method and Sasano and Okumura (2016) corroborate similar trends. The Pearv
son correlation coefficient between RDAT-only
andv
RACC-DAT
is 0.404 for CLM and 0.374 for SLM.
The results are consistent with Sasano and Okumura (2016), where they reported that the correlation coefficient was 0.391.

Word order and semantic role of the dative argument Matsuoka (2003) claimed that the canonical word order differs depending on the semanticrole of the dative argument. Sasano and Okumura

CLM
SLM
Count

TIM<LOCTIM<NOM

LOC<NOMModel

M ODALT IME

M ANNERR ESULTIVE

.757
.708
.686

.642
.632
.666

.604
.615
.681

CLM
SLM

1.
1.

1
0.5

0.5
1.

1.
0.5

Table 2: The columns a < b show the score o(a < b),which indicates the rate of case a being more likely to
be placed before b. The row “Count” shows the countbased results in the dataset we used.

Table 3: The scores denote the rank correlation between the preference of each adverb position in LMs
and that reported in (Koizumi and Tamaoka, 2006).

5.2(2016) evaluated this claim by analyzing the trend
in the following two types of examples:(6) Type-A: 本を
学校に
返した
:::::
book-ACC school-DAT returned.
Type-B: 先生に
本を
返した
:::::
teacher-DAT book-ACC returned.

Type-A has an inanimate goal (school) as the DAT
argument, while Type-B has an animate processor(teacher). It was reported that Type-A is likely tobe the ACC-DAT order, while Type-B is likely to be
the DAT-ACC order. Following Sasano and Okumura (2016), we analyzed 113 verbs.15 For eachverb, we compared the ACC-DAT rate in its type-A
examples and the rate in its type-B examples.
The number of verbs where the ACC-DAT orderis preferred in Type-A examples to Type-B examples is significantly larger (a two-sided sign testp < 0.05). This result is consistent with that of
Sasano and Okumura (2016); Matsuoka (2003) and
implies that the LMs capture the animacy of thenouns. Details are in Appendix B.3.
Word order and co-occurrence of verb and arguments Sasano and Okumura (2016) claimed
that an argument that frequently co-occurs with theverb tends to be placed near the verb. For eachexample, the LMs determine which word order(DAT-ACC or ACC-DAT) is appropriate. Each example also has a score ∆NPMI (definition in Appendix B.4). Higher ∆NPMI means that the DATnoun in the example more strongly co-occurs withthe verb in the example than the ACC noun.
Figure 3-(c) shows the relationship between∆NPMI and the ACC-DAT rate in each example.
∆NPMI and the ACC-DAT rate are correlated with
the Pearson correlation coefficient of 0.517 and
0.521 in CLM and SLM, respectively. These results
are consistent with Sasano and Okumura (2016).
15

Among the 126 verbs used in Sasano and Okumura(2016), 113 verbs with data that do not overlap with the LM
training data were selected.

Order of constituents representing time,location, and subject information

Our focus moves to the cases closer to the beginning of the sentences. The following claim is awell-known property of Japanese word order: “Thecase representing time information (TIM) is placed
before the case representing location information(LOC), and the TIM and LOC cases are placed before the NOM case” (Saeki, 1960, 1998). We examined a parallel between the result obtained with theLM-based and count-based methods on this claim.
We randomly collected 81k examples from 3B
web pages.16 To create the examples, we identified
the case components by KNP, and the TIM and
LOC cases were categorized with JUMAN (details
in Appendix C). For each example s, we created all
possible word orders and obtained the word orderwith the highest generation probability (ŝ). GivenŜ a set of ŝ, we calculated a score o(a < b) for
cases a and b as follows:o(a < b) =Na<b,
Na<b + Nb<awhere Nk<l is the number of examples where thecase k precedes the case l in Ŝ. Higher o(a < b)indicates that the case a is more likely to be placed
before the case b. The results with the LM-based
methods and the count-based method are consistent(Table 2). Both results show that o(TIM < LOC)
is significantly larger than o(LOC > TIM) (p <0.05 with a two-sided signed test), which indicatesthat the TIM case usually precedes the LOC case.
Similarly, the results indicate that the TIM case andthe LOC case precedes the NOM case.
5.3

Adverb position

We checked the preference of the adverb positionin LMs. The position of the adverb has no restriction except that it must be before the verb, which
is similar to the trend of the case position. However, Koizumi and Tamaoka (2006) claimed that
“There is a canonical position of an adverb depend16

Without overlap with the training data of LMs.

Modellong precedes short

short precedes long

CLM
SLM

5,640
5,757

3,754
3,914

Table 4: Changes in the position of a constituent withthe largest number of chunks.

of examples where the longest constituent moves
closer to the beginning of the sentence. This result
is consistent with existing studies and supports thetendency for longer constituents to appear before
shorter ones.
5.5

ing on its type.” They focus on four types of adverbs: MODAL, TIME, MANNER, and RESULTIVE.
We used the same examples as Koizumi andTamaoka (2006). For each example s, we created
its three variants with a different adverb position as
follows (“A friend handled the tools roughly.”):(10) ASOV: 乱暴に 友達が道具を 扱ったroughly friend-NOM tools-ACC handled.
SAOV: 友達が乱暴に 道具を 扱った
friend-NOM roughly tools-ACC handled.
SOAV: 友達が道具を 乱暴に 扱った
friend-NOM tools-ACC roughly handled.

where the sequence of the alphabet such as “ASOV”
denote the word order of its corresponding sentences. For example, “ASOV” indicates the order:
adverb < subject < object < verb. “A,” “S,” “O,”and “V” denote “adverb,” “subject,” “object,” and“verb,” respectively.
Then, we obtained the preferred adverb position
by comparing their generation probabilities. Finally, for each adverb type and its examples, we
ranked the preference of the possible adverb positions: “ASOV,” “SAOV,” and “SOAV.” Table 3
shows the rank correlation of the preference of theposition of each adverb type. The results show similar trends of LMs with that of the human-based
method (Koizumi and Tamaoka, 2006).
5.4

Long-before-short effect

The effects of “long-before-short,” the trend thata long constituent precedes a short one, has beenreported in several studies (Asahara et al., 2018;
Orita, 2017)． We checked whether this effect canbe captured with the LM-based method. Amongthe examples used in Section 5.2, we analyzed
about 9.5k examples in which the position of theconstituent with the largest number of chunks17
differed between its canonical case order18 and theorder supported by LMs.
Table 4 shows that there are significantly (p <0.05 with a two-sided signed test) large numbers

We found parallels between the results with theLM-based method and that with the previously established method on various properties of canonicalword order. These results support the use of LMs
for analyzing Japanese canonical word order.

6

Analysis: word order andtopicalization

In the previous section, we tentatively concluded
that LMs can be used for analyzing the intrasentential properties on the canonical word order.
Based on this finding, in this section, we demonstrate the analysis of additional claims on the properties of the canonical word order with the LMbased method, which has been less explored bylarge-scale experiments. This section shows theanalysis of the relationship between topicalizationand the canonical word order. Additional analyses
on the effect of various adverbial particles for theword order are shown in Appendix F.
6.1

Topicalization in Japanese

The adverbial particle “は” (TOP) is usually used
as a postpositional particle when a specific constituent represents the topic of the sentence (Heycock, 1993; Noda, 1996; Fry, 2003). When a casecomponent is topicalized, the constituent moves
to the beginning of the sentence, and the particle“は” (TOP) is added (Noda, 1996). Additionally,
the original case particle is sometimes omitted,19
which makes the case of the constituent difficult to
identify. For example, to topicalize “本を” (bookACC) in Example (8)-a, the constituent moves tothe beginning of the sentence, and the original accusative case particle “を” (ACC) is omitted. Similarly, “先生が” (teacher-NOM) is topicalized in
Example (8)-b. The original sentence is enclosed
in the square brackets in Example (8).
(8) a. 本をは [先生が本をあげた.]book-TOP teacher-NOM book-ACC gave.
b. 先生がは [先生が本をあげた.]teacher-TOP teacher-NOM book-ACC gave.

17

chunks were identified by KNP.
18
In this section, canonical case order is assumed to be
TOM<LOC<NOM<DAT<ACC.

Summary of the results

19

The particles “を” (ACC) and “が” (NOM) are omitted.

With the above process, we can easily create a sentence with a topicalized constituent. On the other
hand, identifying the original case of the topicalized case components is error-prone. Thus, theLM-based method can be suitable for empiricallyevaluating the claims related to the topicalization.
6.2

Experiments and results

By using the LM-based method, we evaluate thefollowing two claims:(i) The more anterior the case is in the canonicalword order, the more likely its component is
topicalized (Noda, 1996).
(ii) The more the verb prefers the ACC-DAT order,
the more likely the ACC case is topicalized
than the DAT case.
The claim (i) suggests that, for example, the NOM
case is more likely to be topicalized than the ACC
case because the NOM case is before the ACC casein the canonical word order of Japanese. The claim(ii) is based on our observation. It can be regarded
as an extension of the claim (i) considering theeffect of the verb on its argument order. We assume
that the canonical word order of Japanese is TIM <LOC < NOM < DAT < ACC in this section.
Claim (i) We examine which case is more likely
to be topicalized. We collected 81k examples fromJapanese Wikipedia (Details are in Appendix C).
For each example, a set of candidates was createdby topicalizing each case, as shown in Example (8).
Then, we selected the sentences with the highest
score by LMs in each candidate set. We denote theobtained sentences as Ŝ topic . We calculated a scoreta|b for pairs of cases a and b.
ta|b =

Na|b
Na|b + Nb|awhere Na|b is the examples where the case a andb appear, and case a is a topic of the sentence inŜ topic . The higher the score is, the more the case a
is likely to be topicalized than the case b is.
We compared ta|b and tb|a among the pairs ofcases a and b, where the case a precedes the caseb in the canonical word order. Through our experiments, ta|b was significantly larger than tb|a(p < 0.05 with a paired t-test) in CLM and SLM
results, which supports the claim (i) (Noda, 1996).
Detailed results are shown in Appendix E.
Claim (ii) The canonical word order of double
objects is different for each verb (Section 5.1).
Based on this assumption and the claim (i), we

hypothesized that the more the verb prefers the
ACC-DAT order, the more likely the ACC case of
the verb is topicalized than the DAT case.
We used the same data as in Section 5.1. For
each example, we created two sentences by topicalizing the ACC or DAT argument. Then we compared their generation probabilities. In each set ofexamples corresponding to a verb v, we calculated
the rate that the sentence with the topicalized ACC
argument is preferred rather than that with the topiv
calized DAT argument. This rate and RACC-DAT
issignificantly correlated with the Pearson correlation coefficient of 0.89 and 0.84 in CLM and SLM,
respectively. This results support the claim (ii).
Detailed results are shown in Appendix E.

7Conclusion and Future work

We have proposed to use LMs as a tool for analyzing word order in Japanese. Our experimental
results support the validity of using Japanese LMs
for canonical word order analysis, which has thepotential to broaden the possibilities of linguistic
research. From an engineering view, this study supports the use of LMs for scoring Japanese word
order automatically. From the viewpoint of thelinguistic field, we provide additional empirical evidence to various word order hypotheses as well as
demonstrate the validity of the LM-based method.
We plan to further explore the capability ofLMs on other linguistic phenomena related toword order, such as “given new ordering” (Nakagawa, 2016; Asahara et al., 2018). Since LMs arelanguage-agnostic, analyzing word order in anotherlanguage with the LM-based method would also be
an interesting direction to investigate. Furthermore,
we would like to extend a comparison between machine and human language processing beyond theperspective of word order.

8

Acknowledgments

We would like to offer our gratitude to Kaori
Uchiyama for taking the time to discuss our paper and Ana Brassard for her sharp feedback on
English. We also would like to show our appreciation to the Tohoku NLP lab members for theirvaluable advice. We are particularly grateful to Ryohei Sasano for sharing the data for double objects
order analyses. This work was supported by JST
CREST Grant Number JPMJCR1513, JSPS KAKENHI Grant Number JP19H04162, and Grant-inAid for JSPS Fellows Grant Number JP20J22697.

References
Nameera Akhtar. 1999. Acquiring basic word order:
Evidence for data-driven learning of syntactic structure. Journal of child language, 26(2):339–356.
Isabel Alonso Belmonte et al. 2000. Teaching English
Word Order to ESL Spanish Students. A Functional
Perspective.
Inbal Arnon and Neal Snider. 2010. More than words:
Frequency effects for multi-word phrases. Journalof Memory and Language, 62(1):67–82.
Masayuki Asahara, Satoshi Nambu, and Shin-Ichiro
Sano. 2018. Predicting Japanese Word Order in
Double Object Constructions. In Proceedings ofthe Eight Workshop on Cognitive Aspects of Computational Language Learning and Processing, pages
36–40, Melbourne. Association for Computational
Linguistics.
Jörg Bahlmann, Antoni Rodriguez-Fornells, Michael
Rotte, and Thomas F Münte. 2007. An fMRI studyof canonical and noncanonical word order in German. Human brain mapping, 28(10):940–949.
Jelke Bloem. 2016. Testing the Processing Hypothesis of word order variation using a probabilistic
language model. In Proceedings of the Workshop
on Computational Linguistics for Linguistic Complexity (CL4LC), pages 174–185, Osaka, Japan. The
COLING 2016 Organizing Committee.
Joan Bresnan, Anna Cueni, Tatiana Nikitina, and
R Harald Baayen. 2007. Predicting the dative alternation. In Cognitive foundations of interpretation,
pages 69–94. KNAW.
Shuk-Man Cheng, Chi-Hsin Yu, and Hsin-Hsi Chen.
2014. Chinese Word Ordering Errors Detection andCorrection for Non-Native Chinese Language Learners. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics:
Technical Papers, pages 279–289, Dublin, Ireland.
Dublin City University and Association for Computational Linguistics.
Timothy Colleman. 2009. Verb disposition in argument structure alternations: a corpus study of thedative alternation in Dutch. Language Sciences,31(5):593–611.

Édouard Grave, Armand Joulin, Moustapha Cissé,David Grangier, and Hervé Jégou. 2017. Efficient
softmax approximation for GPUs. In Proceedingsof the 34th International Conference on Machine
Learning, volume 70 of Proceedings of Machine
Learning Research, pages 1302–1310, International
Convention Centre, Sydney, Australia. PMLR.
Caroline Heycock. 1993. Syntactic predication in
Japanese.
Journal of East Asian Linguistics,
2(2):167–211.
Hajime Hoji. 1985. Logical form constraints and configurational structures in Japanese. PHD Thesis.
University of Washington.
Malka Rappaport Hovav and Beth Levin. 2008. The
English dative alternation: The case for verb sensitivity. Journal of linguistics, 44(1):129–167.
Katharina Kann, Sascha Rothe, and Katja Filippova.
2018. Sentence-Level Fluency Evaluation: References Help, But Can Be Spared! In Proceedings ofthe 22nd Conference on Computational Natural Language Learning, pages 313–323, Brussels, Belgium.
Association for Computational Linguistics.
Gerard Kempen and Karin Harbusch. 2004. A corpus
study into word order variation in German subordinate clauses: Animacy affects. Multidisciplinary approaches to language production, pages 173–181.
Masatoshi Koizumi and Katsuo Tamaoka. 2004. Cognitive processing of Japanese sentences with ditransitive verbs. Gengo Kenkyu (Journal of the Linguistic Society of Japan), 2004(125):173–190.
Masatoshi Koizumi and Katsuo Tamaoka. 2006. The
Canonical Positions of Adjuncts in the Processingof Japanese Sentence. Cognitive Studies: Bulletin of
the Japanese Cognitive Science Society, 13(3):392–
403.
Taku Kudo. 2006.
Mecab:
Yet anotherpart-of-speech and morphological analyzer.
http://mecab.sourceforge.jp.
Taku Kudo and John Richardson. 2018. SentencePiece:
A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.
In Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing: System
Demonstrations, pages 66–71, Brussels, Belgium.
Association for Computational Linguistics.

Bernard Comrie. 1989. Language universals and linguistic typology: Syntax and morphology. University of Chicago press.

Mikinari Matsuoka. 2003. Two Types of Ditransitive
Consturctions in Japanese. Journal of East Asian
Linguistics, 12(2):171–203.

John Fry. 2003. Ellipsis and wa-marking in Japaneseconversation. Taylor & Francis.

Edson T Miyamoto. 2002. Sources of difficulty in theprocessing of scrambling in Japanese. Sentence processing in East Asian languages, pages 167–188.

Richard Futrell and Roger P Levy. 2019. Do RNNs
learn human-like abstract word order preferences?
In Proceedings of the Society for Computation in
Linguistics (SCiL) 2019, pages 50–59.

Natsuko Nakagawa. 2016. Information structure in
spoken japanese: Particles, word order, and intonation.

Keiko Nakamoto, Jae-ho Lee, and Kow Kuroda. 2006.
Preferred Word Orders Correlate with “Sentential”
Meanings That Cannot Be Reduced to Verb Meanings: A New Perspective on “Construction Effects” in Japanese. Cognitive Studies: Bulletin of
the Japanese Cognitive Science Society, 13(3):334–
352.
Hisashi Noda. 1996. Wa to ga [Wa and ga]. Kurosio
Publishers.
Marian Olteanu, Pasin Suriyentrakorn, and Dan
Moldovan. 2006. Language models and reranking
for machine translation. In Proceedings of the Workshop on Statistical Machine Translation, pages 150–
153, New York City. Association for Computational
Linguistics.
Naho Orita. 2017. Predicting japanese scrambling in
the wild. In Proceedings of the 7th Workshop on
Cognitive Modeling and Computational Linguistics,
pages 41–45.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela
Fan, Sam Gross, Nathan Ng, David Grangier, andMichael Auli. 2019. fairseq: A Fast, Extensible
Toolkit for Sequence Modeling. In Proceedings ofthe 2019 Conference of the North American Chapter of the Association for Computational Linguistics(Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
Tetsuo Saeki. 1960. Gendaigo ni okeru gojun no keikō– iwayuru hogo no baai [The trend of word order inmodern writing– in so-called complements]. Gengoseikatsu [Language life], (111):56–63.
Tetsuo Saeki. 1998. Yōsetsu Nihongo no Gojun [Essentials of Japanese word order]. Kurosio Publishers.
Ryohei Sasano and Manabu Okumura. 2016. A
Corpus-Based Analysis of Canonical Word Order ofJapanese Double Object Constructions. In Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 2236–2244, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural Machine Translation of Rare Words
with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1715–
1725, Berlin, Germany. Association for Computational Linguistics.
Yasumasa Shigenaga. 2014. Canonical Word Order ofJapanese Ditransitive Sentences: A Preliminary Investigation through a Grammaticality Judgment Survey. Advances in Language and Literary Studies,5(2):35–45.
Dan I Slobin and Thomas G Bever. 1982. Children use
canonical sentence schemas: A crosslinguistic studyof word order and inflections. Cognition, 12(3):229–
265.

Natsuko Tsujimura. 2013. An introduction to Japanese
linguistics. John Wiley & Sons.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz
Kaiser, and Illia Polosukhin. 2017. Attention is All
you Need. In I. Guyon, U. V. Luxburg, S. Bengio,
H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
Karthik Visweswariah, Rajakrishnan Rajkumar, Ankur
Gandhe, Ananthakrishnan Ramanathan, and Jiri
Navratil. 2011. A Word Reordering Model for Improved Machine Translation. In Proceedings of the2011 Conference on Empirical Methods in Natural
Language Processing, pages 486–496, Edinburgh,
Scotland, UK. Association for Computational Linguistics.

A

Hyperparameters and implementationof the LMs

We used the Transformer (Vaswani et al., 2017)
LMs implemented in fairseq (Ott et al., 2019). Table 5 shows the hyperparameters of the LMs. The
adaptive softmax cutoff (Grave et al., 2017) is only
applied to SLM. We split 10K sentences for dev set.
The left-to-right and right-to-left CLMs achieved
a perplexity of 11.05 and 11.08, respectively. The
left-to-right and right-to-left SLMs achieved a perplexity of 28.51 and 28.25, respectively. Note thatthe difference in the perplexities between CLM and
SLM is due to the difference in the vocabulary size.

B

Details on Section 5.1 (double objects)

B.1

Word order for each verb

It is considered that different verbs have different
preferences in the order of their object. For example, while the verb “例える” (compare) prefers the
ACC-DAT order (Example (9)-a), the verb “表する” (express) prefers the DAT-ACC order (Example (9)-b).
(9) a. 人間を色に例えた.
person-ACC color-DAT compared.
(φI compared a person to color.)b. 店主に敬意を表した.
shopkeeper-DAT respect-ACC expressed.
(φI expressed a respect to a shopkeeper.)Table 6 shows the verbs with the top five and thevfive worst RACC-DAT.
B.2

Word order and verb types

There are two types of causative-inchoative alternating verbs in Japanese: show-type verbs and passtype verbs. The verb types are determined by the
subject of the sentence where the corresponding
inchoative verb is used. For the show-type verbs,
the DAT argument of a causative sentence becomes
the subject in its corresponding inchoative sentence(Example (10)). On the other hand, the ACC argument of a causative sentence becomes the subject
in its corresponding inchoative sentence for the
pass-type verbs (Example (11)).
(10) Causative: 生徒に本を見せた
student-DAT book-ACC showed.
(φI showed a student a book.)
見た
Inchoative: 生徒がstudent-NOM saw.
(A student saw φsomething .)(11) Causative: 生徒に本を渡した
student-DAT book-ACC showed.
(φI passed a student a book.)
Inchoative: 本が渡った
book-NOM passed.
(A book passed to φsomething .)Matsuoka (2003) claims that the show-type verb
prefers the DAT-ACC order, while the pass-typeverb prefers the ACC-DAT order.
v
Table 7 shows RACC-DATof the show-type andpass-type verbs. The results show no significant
difference in word order trends between show-type
and pass-type verbs, which are consistent with thatof Sasano and Okumura (2016).

B.3

Word order and semantic role of thedative argument

As described in Section 5.1, Sasano and Okumura(2016) reported that type-A examples prefer the
ACC-DAT order and type-B examples prefer the
DAT-ACC order. We used the same examples
as Sasano and Okumura (2016) used. We analyzed
the difference in the trend of argument order between type-A and type-B examples in each verb.
Table 8 shows the verbs, which show a significant
change in the argument order between type-A andtype-B examples (p < 0.05 in a two-proportion
z-test). In the experiment using CLM, 31 verbs
show the trend that type-A examples more prefer
the ACC-DAT order to type-B, and 17 verbs show
contrary trends. In the experiment using SLM, 38
verbs show the trend that type-A examples more
prefer the ACC-DAT order to type-B, and 11 verbs
show contrary trends. These results show that the
number of verbs, where the ACC-DAT order is preferred by type-A examples rather than type-B, issignificantly larger (p < 0.05 with a two-sided signtest). This experimental design follows Sasano andOkumura (2016).

B.4

Word order and co-occurrence of verb
and arguments

We evaluate the claim that an argument frequently
co-occurring with the verb tends to be placed near
the verb. We examine the relationship between
each example’s word order trend and ∆NPMI.
∆NPMI is calculated as follows:Fairseq modelarchitecture
adaptive softmax cut off

Optimizer

algorithm
learning rates
momentumweight decay
clip norm

Learning rate schedulertransformer lm
50,000, 140,000
Nesterov accelerated gradient (nag)
1e-5
0.99
0
0.1

type
warmup updates
warmup init lrarning rate
max learning rate
min learning ratet mult (factor to grow the length of each period)
learning rate period updates
learning rate shrink
batch size
epochsTraining

cosine16,000
1e-7
0.1
1e-9
2
270,000
0.75
4608 tokens
3

Table 5: Hyperparameters of the LMs.
ACC-DAT is preferredv
RACC-DAT

Model

Verb

CLM

“例える” (compare)“換算する” (convert)“押し出す” (extrude)“見立てる” (liken)“変換” (convert)

SLM“例える” (compare)“押し出す” (extrude)“監禁” (confine)“役立てる” (help)“帰す” (attribute)

DAT-ACC is preferredv
RACC-DAT

S&O

Verb

S&O

0.993
0.992
0.979
0.994
0.975

0.945
0.935
0.923
0.919
0.898

“表する” (express)“澄ます” (listen)“煮やす” (cook inside)“瞑る” (close the eyes)“竦める” (shrug)

0.001
0.000
0.000
0.001
0.002

0.013
0.017
0.019
0.021
0.022

0.993
0.979
0.885
0.933
0.838

0.926
0.914
0.912
0.904
0.903

“喫する” (drink)“表する” (express)“澄ます” (listen)“抜かす” (leave out)
“踏み入れる” (step into)

0.003
0.001
0.000
0.002
0.002

0.018
0.018
0.021
0.022
0.025

v
in each LM. The “S&O” columns show the
Table 6: The verbs with the top five and the worst five RACC-DATACC-DAT rate reported in Sasano and Okumura (2016).

∆NPMI = NPMI(nDAT , v)
− NPMI(nACC , v) ,PMI(nc , v)
where NPMI(nc , v) =,−log(p(nc , v))
p(nc , v)
PMI(nc , v) = log,p(nc )p(v)where, v is a verb and nc (c ∈ DAT, ACC) is its
argument.

C

Data used in Section 5.2, Section 6,
and Appendix FFirst, we randomly collected 50M sentences from3B web pages. Note that there is no overlap between the collected sentences and the training dataof LMs. Next, we obtained the sentences that satisfy the following criteria:• There is a verb (placed at the end of the sentence) with more than two arguments (accompanying the case particle ga, o, ni, or de),where dependency distance between the verb
and arguments is one.
• Each argument (with its descendant) has fewer
than 11 morphemes in the argument.
In each example, the verb (satisfying the above
condition), its arguments, and the descendants of
the arguments are extracted. Example sentences
are created by concatenating the verb, its argument,
and the descendants of the arguments with preserving their order in the original sentences.
In the experiments in Section 5.2, we analyzed
the word order trend of the TIM and LOC constituents. We regard the constituent (argument andits descendants) satisfying the following condition
as the TIM constituent:• Accompanying the postpositional case particle “に” (DAT).

Show-type

Pass-type

Verb

CLM

SLM

S&O

Verb

CLM

SLM

S&O

Verb

CLM

SLM

S&O

“知らせる” (notify)“預ける” (deposit)“見せる” (show)“被せる” (cover)“教える” (teach)“授ける” (give)“浴びせる” (shower)“貸す” (lend)“着せる” (dress)
-

.718
.426
.353
.240
.297
.101
.113
.253
.115
-

.754
.391
.429
.224
.293
.084
.121
.213
.109
-

.522
.399
.301
.256
.235
.186
.177
.118
.113
-

“戻す” (put back)“止める” (lodge)“包む” (wrap)“伝える” (inform)“乗せる” (place on)“届ける” (deliver)“並べる” (range)“ぶつける” (knock)“付ける” (attach)“渡す” (pass)“落とす” (drop)

.366
.638
.316
.419
.556
.364
.423
.333
.326
.349
.379

.395
.704
.356
.460
.498
.419
.485
.344
.329
.336
.397

.771
.748
.603
.522
.496
.491
.481
.436
.368
.362
.351

“漏らす” (leak)“浮かべる” (float)“向ける” (direct)“残す” (leave)“埋める” (bury)“混ぜる” (blend)“当てる” (hit)“掛ける” (hang)“重ねる” (pile)“建てる” (build)
-

.152
.387
.291
.323
.405
.336
.287
.285
.226
.117
-

.207
.406
.319
.318
.430
.276
.320
.288
.263
.099
-

.332
.255
.251
.238
.223
.200
.185
.108
.084
.069
-

Macro Avg.

.291

.291

.305

.347

.364

.361

Macro Avg.

• Containing time category morphemes20 .
We regard the constituent (argument and its descendants) satisfying the following condition as the
LOC constituent:• Accompanying the postpositional case particle “で”.
• Containing location category morphemes20 .
81k examples were created. The averaged number of characters in a sentence was 45.1 characters.
The number of occurrences of each case is shownin Table 9. The scrambling process conducted in
the experiments (Sections 5.2 and 6) is the same as
described in Section 4.

D

Details on Section 5.3 (adverb)

Table 10 shows the correlation between the resultof LMs and that of Koizumi and Tamaoka (2006).
The column “Canonical” shows the position, which
is significantly preferred over the other positions.
“A,” “S,” “O,” and “V” denote “adverb,” “subject,”“object,” and “verb,” respectively. The sequence of
the alphabets corresponds to their order; for example, “ASOV” indicates the order: adverb < subject< object < verb. Following Koizumi and Tamaoka(2006), we examined the three candidate positionsof the adverb: “ASOV,” “SAOV,” and “SOAV.” Thescore r denotes the Pearson correlation coefficientof the preferred ranks of each adverb position tothat reported in Koizumi and Tamaoka (2006).

E

Details on Section 6.2 (topicalization)

We topicalized a specific constituent by moving
the constituent to the beginning of the sentence and20

identified by JUMAN

rate of the examples where
ACC is more likely to be topicalized

Table 7: Overlap of the results of LMs and that of Sasano and Okumura (2016) on the relationship of the ACC-DAT
rate and verb types. Each score corresponding to a verb denotes its DAT-ACC rate. The “S&O” columns show the
ACC-DAT rate reported in Sasano and Okumura (2016). There is no significant difference between the distributionsof the DAT-ACC rate in two verb types.
1
0.8
0.6
0.4
0.2
0
0

0.2

0.4

0.6

0.8

1

ACC-DAT rate obtained from LMs

CLM

SLM

Figure 4: Correlation between the ACC-DAT rate andthe rate that the ACC argument is more likely to be topicalized than DAT for each verb. Each plot corresponds
to the result of each verb.

adding the adverbial particle “は” (TOP). Strictly
speaking, conjunctions are preferentially placed at
the beginning of the sentence rather than topicalized constituents. The examples we used do notinclude the conjunctions at the beginning of the sentence. The adverbial particle was added according
to the rules shown in Table 12.
Claim (i): Table 11 shows the ta|b for each pair of
the case a (row) and b (column). The results showthat the more anterior the case a is and the moreposterior the case b is in the canonical word order,
the larger the ta|b is.
Claim (ii): Figure 4 shows that the more a verb
prefers the ACC-DAT order, the more ACC casetends to be topicalized. The X-axis denotes the
ACC-DAT rate of the verb, and the Y-axis denotes
the trend that ACC is more likely to be topicalized
than DAT.

ModelVerbs whose type-A examples prefer the ACC-DAT order

Verbs whose type-B examples prefer the ACC-DAT order

CLM“預ける” (deposit), “置く” (place), “持つ” (have), “入れる” (put in), “納める” (pay), “郵送” (mail), “供給” (supply), “出す” (put out), “運ぶ” (transport), “流す” (shed), “掛ける” (hang), “飾る” (decorate), “広げる” (spread), “移す” (transfer), “残す” (leave), “配送” (deliver), “送る” (send), “投げる” (throw), “送付” (send), “返却” (return), “届ける” (send), “戻す”
(return), “着ける” (wear), “上げる” (increase), “落とす” (drop), “載せる” (publish), “変更” (change), “納入” (deliver), “卸す” (unload), “掲載” (publish), “通す” (get X through)“配 布” (distribute), “渡 す” (pass), “プ レ ゼ ン ト”
(present), “合わせる” (match), “見せる” (show), “提供” (offer), “与える” (give), “当てる” (hit), “回す”
(turn), “追加” (add), “貸す” (lend), “展示” (exhibit),“据える” (lay), “依頼” (request), “挿入” (insert), “纏める” (collect), “請求” (claim)

SLM“預ける” (deposit), “置く” (place), “頼む” (ask), “入れる” (put in), “納める” (pay), “郵送” (mail), “出す” (put out), “運ぶ” (transport), “流す” (shed), “掛ける” (hang), “広げる” (spread), “移す” (transfer), “残す” (leave), “リクエスト” (request), “配送” (deliver),“送る” (send), “投げる” (throw), “送付” (send), “求める” (ask), “提出” (submit), “届ける” (deliver), “要求” (request), “戻す” (return), “寄付” (donate), “寄贈”(donation), “着ける” (wear), “乗せる” (place), “上げる” (increase), “落とす” (drop), “貼る” (stick), “分ける” (divide), “ばらまく” (scatter), “はめる” (fit), “支払う” (pay), “配達” (deliver), “卸す” (unload), “纏める” (collect), “通す” (get X through)“プレゼント” (present), “持つ” (have), “合わせる”(match), “見せる” (show), “向ける” (point), “提供”
(offer), “装備” (equip), “追加” (add), “展示” (exhibit),“据える” (lay), “採用” (adopt)

Table 8: The verbs which show a significant change in the argument order trend depending on the semantic roleof its dative argument. Type-A corresponds to the examples with an inanimate goal dative argument. Type-B corresponds to the examples with an animate processor dative argument. The number of type-A verbs is significantly
larger than that of type-B verbs.

Case#occurrence

TIM
LOC
NOM
DAT
ACC

11,78015,54455,23056,24357,823

Table 9: The number of occurrence for each case in thedata used in Section 5.2, Section 6, and Appendix FF

Additional analysis: adverbial
particles and their effect for wordorder

The adverbial particles We can add supplementary information with adverbial particles. The adverbial particle “は” (TOP) is the typical one. In
Example (12), the adverbial particle “も” (also),
instead of “を” (ACC), implies that there is anotherthing the teacher gave to the student (“a teacher
gave not only φ but also a book to a student.”).
(12) 生徒に
本を も
あげた.
:::::
student-DAT also book-ACC gave.

Experiments A constituent accompanying theadverbial particle “は” (TOP) is moved to the beginning of the sentence (Noda, 1996). However, it

is not clear whether other adverbial particles also
have the above property. In this section, we evaluate the following claim: a different adverbial particle shows different degrees of the effects for theword order.
For each example s ∈ S collected from Japanese
Wikipedia, we replaced the postpositional particle
with a specific adverbial particle, following therules in Table 12. We used four typical adverbial
particles: “は” (TOP), “こそ” (emphasis), “も”(also), and “だけ” (only). Two variants of word order, Non-moved, and Moved were created for eachexample. Example (13) is an example focusing on
the ACC case with the particle “も” (also).
(13) Original:

生徒に
本を
あげた.
::::
student-DAT book-ACC gave.

Non-moved:

Moved:

生徒に
本をも
あげた.
::::
student-DAT also book-ACC gave.

本をも
生徒に
本を
あげた.
::::
also book-ACC student-DAT book-ACC gave.

We compared the generation probabilities between
the Non-moved and Moved orders. We calculated
the rate that the Moved order is preferred in eachcombination of the case types and the adverbial
particles.

ModelM ODAL
Canonical r

CLM
SLM
Koizumi(2016)

ASOV
ASOV
ASOV

1.
1.
-

T IME
Canonicalr

M ANNER
Canonicalr

ASOV, SAOVSAOVASOV, SAOV

1.
0.5
-

SAOV, SOAVSAOV, SOAVSAOV, SOAV

0.5
1.
-

R ESULTIVE
Canonicalr
SAOV, SOAVSOAV
SAOV, SOAV

1.
0.5
-

Table 10: Overlap of the preference of the adverb position of LMs and that of Koizumi and Tamaoka (2006). The
column “Canonical” shows the adverb position, which is significantly preferred over the other positions. The scorer denotes the Pearson correlation coefficient of the preferred rank of three possible adverb positions obtained from
LMs to that of Koizumi and Tamaoka (2006).

TIM
PLC
NOM
DAT
NOM

TIM

PLC

NOM

DAT

NOM

Original case particle

.510
.671
.280
.302

.490
.516
.252
.258

.329
.484
.196
.148

.720
.748
.804
.464

.698
.742
.852
.536
-

After the adverbial particle “は” (TOP) is added

が (TOP)に (TIM, DAT)を (ACC)
で (LOC)

がは
には
をは
では

(a) CLM

TIM
PLC
NOM
DAT
NOM

TIM

PLC

NOM

DAT

NOM

.462
.598
.324
.289

.538
.447
.243
.251

.402
.553
.226
.166

.676
.757
.774
.448

.711
.749
.834
.552
-

(b) SLM

Table 11: The scores denote ta|b . The row corresponds
to the case a, the column corresponds to b. Higher ta|bsuggests the trend that the case a is more likely to be
topicalized than the case b.

Results The results are shown in Table 13. Whenusing “は” (TOP) as a postpositional particle, theMoved order is preferred to Non-moved, which isconsistent with the well-known characteristics of
topicalization described in Section 6. In addition,
the degree of preference between Moved and Nonmoved differs depending on the adverbial particles.
Furthermore, the results indicate that the anterior
case in the canonical word order is likely to moveto the beginning of the sentence by the effect of theadverbial particle.
Additional experiments and results We analyzed the trend of double object order when a specific case accompanies an adverbial particle. Figure 5 shows the result when the ACC argument
accompanies an adverbial particle, and Figure 6
shows the result when the DAT argument accompanies an adverbial particle. The left parts of thesefigures show the result of CLM, and the right partof these figures shows the result of SLM. The Xaxis denotes the ACC-DAT / DAT-ACC rate of theverb when both of the arguments do not accom-

Table 12: Rules of deleting the original case particle
when the adverbial particle “は” (TOP) is added. This
rule is also applied when adding the other adverbial particles (Appendix F).

pany an adverbial particle. The Y-axis denotes the
ACC-DAT / DAT-ACC rate when a specific case accompanies an adverbial particle. The results showthat the case accompanying an adverbial particle islikely to be placed near the beginning of the sentence. In addition, the degree of the above trend
depends on the adverbial particles. These results
suggest that some adverbial particles have a effectfor word order.

ModelToritate particle

TIM

LOC

NOM

DAT

ACC

Avg.

CLM

“は” (TOP)“こそ” (emphasis)“も” (also)“だけ” (only)

.715
.492
.560
.385

.777
.423
.557
.340

.675
.521
.458
.312

.624
.313
.343
.227

.623
.486
.271
.184

.683
.447
.438
.331

Avg.

.538

.525

.544

.377

.391

-

“は” (TOP)“こそ” (emphasis)“も” (also)“だけ” (only)

.667
.567
.511
.334

.751
.596
.531
.309

.635
.574
.457
.285

.565
.398
.292
.172

.580
.462
.259
.126

.640
.519
.410
.303

Avg.

.520

.547

.560

.357

.357

-

SLM

Table 13: The scores denote that the Moved order is preferred over the Non-moved order when the correspondingcase (column) accompanies the corresponding particle (row). The trend is different depending on the case andparticle.

ACCadv moves to the beginning of the sentence
1

0.9

0.9

0.8

adv -DAT rate (SLM)
ACC
ACC-DATrate (SLM)

adv -DAT rate (CLM)
ACC
ACC-DATrate (CLM)

ACCadv moves to the beginning of the sentence
1

0.7
0.6
0.5
0.4
0.3
0.2

0.8
0.7
0.6
0.5
0.4
0.3
0.2

0.1

0.1

ACCadv moves closer to the verb

0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

ACCadv moves closer to the verb

0

1

0

0.1

0.2

0.3

ACC-DAT(CLM)
ACC-DATraterate(CLM)
は (TOP)こそ (emphasis)

0.4

0.5

0.6

0.7

0.8

0.9

1

ACC-DAT
rate(SLM)
ACC-DATrate(SLM)

も (also)だけ (only)

は (TOP)こそ (emphasis)(a) CLMも (also)だけ (only)(b) SLM

Figure 5: Change of the ACC-DAT order when the ACC argument accompanies an adverbial particle. These results
indicate that the ACC argument with an adverbial particle (ACCadv ) is more likely to be placed before the DAT
argument. In addition, this trend differs for each particle.

DAT adv moves to the beginning of the sentence
1

0.9

0.9
adv -ACC rate (SLM)DAT
ACC-DATrate (SLM)

DAT adv -ACC
rate (CLM)
DAT-ACC
rate (CLM)

DAT adv moves to the beginning of the sentence
1

0.8
0.7
0.6
0.5
0.4
0.3

0.8
0.7
0.6
0.5
0.4
0.3
0.2

0.2
0.1

Research Seminar

0.1

DAT adv moves closer to the verb

019

DAT adv moves closer to the verb

0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

0.1

DAT-ACC
raterate(CLM)
DAT-ACC(CLM)
は (TOP)こそ (emphasis)(a) CLMも (also)

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

DAT-ACC(SLM)
ACC-DATrate
rate
(SLM)
だけ (only)

は (TOP)こそ (emphasis)

も (also)だけ (only)(b) SLM

Figure 6: Change of the DAT-ACC order when the DAT argument accompanies an adverbial particle. These results
indicate that the DAT argument with an adverbial particle (DATadv ) is more likely to be placed before the ACC
argument. In addition, this trend differs for each particle.