SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 3-(b) shows that the regression lines from the LM-based method and Sasano and Okumura (2016) corroborate similar trends. The Pearson correlation coefficient between  $R_{\mathrm{DAT-only}}^v$  and  $R_{\mathrm{ACC-DAT}}^v$  is 0.404 for CLM and 0.374 for SLM. The results are consistent with Sasano and Okumura (2016), where they reported that the correlation coefficient was 0.391.
Word order and semantic role of the dative argument Matsuoka (2003) claimed that the canonical word order differs depending on the semantic role of the dative argument. Sasano and Okumura
<sup>13</sup>We filtered the examples overlapping with the training data of LMs in advance. As a result, we collected 4.5M examples.
14 We removed verbs for which all examples overlap with the data for training the LMs.
TIM<LOC	TIM< NOM	LOC< NOM
CLM	.757	.642	.604
SLM	.708	.632	.615
Count	.686	.666	.681
(2016) evaluated this claim by analyzing the trend in the following two types of examples:
(6) Type-A: 本を学校に返した book-ACC school-DAT returned.
Type-B: 先生に 本を返んだ teacher-DAT book-ACC returned.
Type-A has an inanimate goal (school) as the DAT argument, while Type-B has an animate processor (teacher). It was reported that Type-A is likely to be the ACC-DAT order, while Type-B is likely to be the DAT-ACC order. Following Sasano and Okumura (2016), we analyzed 113 verbs. $^{15}$  For each verb, we compared the ACC-DAT rate in its type-A examples and the rate in its type-B examples.
The number of verbs where the ACC-DAT order is preferred in Type-A examples to Type-B examples is significantly larger (a two-sided sign test  $p < 0.05$ ). This result is consistent with that of Sasano and Okumura (2016); Matsuoka (2003) and implies that the LMs capture the animacy of the nouns. Details are in Appendix B.3.
Word order and co-occurrence of verb and arguments Sasano and Okumura (2016) claimed that an argument that frequently co-occurs with the verb tends to be placed near the verb. For each example, the LMs determine which word order (DAT-ACC or ACC-DAT) is appropriate. Each example also has a score  $\Delta$ NPMI (definition in Appendix B.4). Higher  $\Delta$ NPMI means that the DAT noun in the example more strongly co-occurs with the verb in the example than the ACC noun.
Figure 3-(c) shows the relationship between  $\Delta$ NPMI and the ACC-DAT rate in each example.  $\Delta$ NPMI and the ACC-DAT rate are correlated with the Pearson correlation coefficient of 0.517 and 0.521 in CLM and SLM, respectively. These results are consistent with Sasano and Okumura (2016).
Table 2: The columns  $a < b$  show the score  $o(a < b)$ , which indicates the rate of case  $a$  being more likely to be placed before  $b$ . The row "Count" shows the count-based results in the dataset we used.
Model	MODAL	TIME	MANNER	RESULTIVE
CLM	1.	1	0.5	1.
SLM	1.	0.5	1.	0.5
Table 3: The scores denote the rank correlation between the preference of each adverb position in LMs and that reported in (Koizumi and Tamaoka, 2006).
5.2 Order of constituents representing time, location, and subject information
Our focus moves to the cases closer to the beginning of the sentences. The following claim is a well-known property of Japanese word order: "The case representing time information (TIM) is placed before the case representing location information (LOC), and the TIM and LOC cases are placed before the NOM case" (Saeki, 1960, 1998). We examined a parallel between the result obtained with the LM-based and count-based methods on this claim.
We randomly collected 81k examples from 3B web pages. To create the examples, we identified the case components by KNP, and the TIM and LOC cases were categorized with JUMAN (details in Appendix C). For each example  $s$ , we created all possible word orders and obtained the word order with the highest generation probability  $(\hat{s})$ . Given  $\hat{S}$  a set of  $\hat{s}$ , we calculated a score  $o(a < b)$  for cases  $a$  and  $b$  as follows:
$$
o (a <   b) = \frac {N _ {a <   b}}{N _ {a <   b} + N _ {b <   a}},
$$
where  $N_{k < l}$  is the number of examples where the case  $k$  precedes the case  $l$  in  $\hat{S}$ . Higher  $o(a < b)$  indicates that the case  $a$  is more likely to be placed before the case  $b$ . The results with the LM-based methods and the count-based method are consistent (Table 2). Both results show that  $o(\mathrm{TIM} < \mathrm{LOC})$  is significantly larger than  $o(\mathrm{TIM} > \mathrm{LOC})$  ( $p < 0.05$  with a two-sided signed test), which indicates that the TIM case usually precedes the LOC case. Similarly, the results indicate that the TIM case and the LOC case precedes the NOM case.
5.3 Adverb position
We checked the preference of the adverb position in LMs. The position of the adverb has no restriction except that it must be before the verb, which is similar to the trend of the case position. However, Koizumi and Tamaoka (2006) claimed that "There is a canonical position of an adverb depend
15Among the 126 verbs used in Sasano and Okumura (2016), 113 verbs with data that do not overlap with the LM training data were selected.
Without overlap with the training data of LMs.
Model	long precedes short	short precedes long
CLM	5,640	3,754
SLM	5,757	3,914
Table 4: Changes in the position of a constituent with the largest number of chunks.
ing on its type." They focus on four types of adverbs: MODAL, TIME, MANNER, and RESULTIVE.
We used the same examples as Koizumi and Tamaoka (2006). For each example  $s$ , we created its three variants with a different adverb position as follows ("A friend handled the tools roughly"):
(10) ASOV: 乱暴に友達が 道具を 抹った roughly friend-NOM tools-ACC handled.
SAOV: 友達が 乱暴の 道具を 报った friend-NOM roughly tools-ACC handled.
SOAV: 友達が 道具を 乱暴に 抹った friend-NOM tools-ACC roughly handled.
where the sequence of the alphabet such as "ASOV" denote the word order of its corresponding sentences. For example, "ASOV" indicates the order: adverb  $<$  subject  $<$  object  $<$  verb. "A," "S," "O," and "V" denote "adverb," "subject," "object," and "verb," respectively.
Then, we obtained the preferred adverb position by comparing their generation probabilities. Finally, for each adverb type and its examples, we ranked the preference of the possible adverb positions: "ASOV," "SAOV," and "SOAV." Table 3 shows the rank correlation of the preference of the position of each adverb type. The results show similar trends of LMs with that of the human-based method (Koizumi and Tamaoka, 2006).
5.4 Long-before-short effect
The effects of "long-before-short," the trend that a long constituent precedes a short one, has been reported in several studies (Asahara et al., 2018; Orita, 2017). We checked whether this effect can be captured with the LM-based method. Among the examples used in Section 5.2, we analyzed about  $9.5\mathrm{k}$  examples in which the position of the constituent with the largest number of chunks<sup>17</sup> differed between its canonical case order<sup>18</sup> and the order supported by LMs.
Table 4 shows that there are significantly  $(\mathfrak{p} < 0.05$  with a two-sided signed test) large numbers
of examples where the longest constituent moves closer to the beginning of the sentence. This result is consistent with existing studies and supports the tendency for longer constituents to appear before shorter ones.
5.5 Summary of the results
We found parallels between the results with the LM-based method and that with the previously established method on various properties of canonical word order. These results support the use of LMs for analyzing Japanese canonical word order.
6 Analysis: word order and topicalization
In the previous section, we tentatively concluded that LMs can be used for analyzing the intrasentential properties on the canonical word order. Based on this finding, in this section, we demonstrate the analysis of additional claims on the properties of the canonical word order with the LM-based method, which has been less explored by large-scale experiments. This section shows the analysis of the relationship between topicalization and the canonical word order. Additional analyses on the effect of various adverbial particles for the word order are shown in Appendix F.
6.1 Topicalization in Japanese
The adverbial particle "は" (TOP) is usually used as a postpositional particle when a specific constituent represents the topic of the sentence (Heycock, 1993; Noda, 1996; Fry, 2003). When a case component is topicalized, the constituent moves to the beginning of the sentence, and the particle "は" (TOP) is added (Noda, 1996). Additionally, the original case particle is sometimes omitted,[19] which makes the case of the constituent difficult to identify. For example, to topicalize "本を" (book-ACC) in Example (8)-a, the constituent moves to the beginning of the sentence, and the original accusative case particle "を" (ACC) is omitted. Similarly, "先生が" (teacher-NOM) is topicalized in Example (8)-b. The original sentence is enclosed in the square brackets in Example (8).
(8) a. 本をは [先生が 本を あた.] book-Top teacher-NOM book=ACC gave.
b. 先生がは [先生が 本を あけた.] teacher-TOP teacher-NOM book-ACC gave.
17 chunks were identified by KNP. 18 In this section, canonical case order is assumed to be TOM<LOC<NOM<DAT<ACC.
19The particles “を” (ACC) and “が” (NOM) are omitted.
With the above process, we can easily create a sentence with a topicalized constituent. On the other hand, identifying the original case of the topicalized case components is error-prone. Thus, the LM-based method can be suitable for empirically evaluating the claims related to the topicalization.
6.2 Experiments and results
By using the LM-based method, we evaluate the following two claims:
(i) The more anterior the case is in the canonical word order, the more likely its component is topicalized (Noda, 1996).
(ii) The more the verb prefers the ACC-DAT order, the more likely the ACC case is topicalized than the DAT case.
The claim (i) suggests that, for example, the NOM case is more likely to be topicalized than the ACC case because the NOM case is before the ACC case in the canonical word order of Japanese. The claim (ii) is based on our observation. It can be regarded as an extension of the claim (i) considering the effect of the verb on its argument order. We assume that the canonical word order of Japanese is  $\mathrm{TIM} < \mathrm{LOC} < \mathrm{NOM} < \mathrm{DAT} < \mathrm{ACC}$  in this section.
Claim (i) We examine which case is more likely to be topicalized. We collected 81k examples from Japanese Wikipedia (Details are in Appendix C). For each example, a set of candidates was created by topicalizing each case, as shown in Example (8). Then, we selected the sentences with the highest score by LMs in each candidate set. We denote the obtained sentences as  $\hat{S}^{\mathrm{topic}}$ . We calculated a score  $t_{a|b}$  for pairs of cases  $a$  and  $b$ .
$$
t _ {a | b} = \frac {N _ {a | b}}{N _ {a | b} + N _ {b | a}}
$$
where  $N_{a|b}$  is the examples where the case  $a$  and  $b$  appear, and case  $a$  is a topic of the sentence in  $\hat{S}^{\text {topic }}$ . The higher the score is, the more the case  $a$  is likely to be topicalized than the case  $b$  is.
We compared  $t_{a|b}$  and  $t_{b|a}$  among the pairs of cases  $a$  and  $b$ , where the case  $a$  precedes the case  $b$  in the canonical word order. Through our experiments,  $t_{a|b}$  was significantly larger than  $t_{b|a}$  ( $p < 0.05$  with a paired t-test) in CLM and SLM results, which supports the claim (i) (Noda, 1996). Detailed results are shown in Appendix E.
Claim (ii) The canonical word order of double objects is different for each verb (Section 5.1). Based on this assumption and the claim (i), we
hypothesized that the more the verb prefers the ACC-DAT order, the more likely the ACC case of the verb is topicalized than the DAT case.
We used the same data as in Section 5.1. For each example, we created two sentences by topicalizing the ACC or DAT argument. Then we compared their generation probabilities. In each set of examples corresponding to a verb  $v$ , we calculated the rate that the sentence with the topicalized ACC argument is preferred rather than that with the topicalized DAT argument. This rate and  $R_{\mathrm{ACC - DAT}}^v$  is significantly correlated with the Pearson correlation coefficient of 0.89 and 0.84 in CLM and SLM, respectively. This results support the claim (ii). Detailed results are shown in Appendix E.
We have proposed to use LMs as a tool for analyzing word order in Japanese. Our experimental results support the validity of using Japanese LMs for canonical word order analysis, which has the potential to broaden the possibilities of linguistic research. From an engineering view, this study supports the use of LMs for scoring Japanese word order automatically. From the viewpoint of the linguistic field, we provide additional empirical evidence to various word order hypotheses as well as demonstrate the validity of the LM-based method.
We plan to further explore the capability of LMs on other linguistic phenomena related to word order, such as "given new ordering" (Nakagawa, 2016; Asahara et al., 2018). Since LMs are language-agnostic, analyzing word order in another language with the LM-based method would also be an interesting direction to investigate. Furthermore, we would like to extend a comparison between machine and human language processing beyond the perspective of word order.
We would like to offer our gratitude to Kaori Uchiyama for taking the time to discuss our paper and Ana Brassard for her sharp feedback on English. We also would like to show our appreciation to the Tohoku NLP lab members for their valuable advice. We are particularly grateful to Ryohei Sasano for sharing the data for double objects order analyses. This work was supported by JST CREST Grant Number JPMJCR1513, JSPS KAKENHI Grant Number JP19H04162, and Grant-in-Aid for JSPS Fellows Grant Number JP20J22697.
Nameera Akhtar. 1999. Acquiring basic word order: Evidence for data-driven learning of syntactic structure. Journal of child language, 26(2):339-356.
Isabel Alonso Belmonte et al. 2000. Teaching English Word Order to ESL Spanish Students. A Functional Perspective.
Inbal Arnon and Neal Snider. 2010. More than words: Frequency effects for multi-word phrases. Journal of Memory and Language, 62(1):67-82.
Masayuki Asahara, Satoshi Nambu, and Shin-Ichiro Sano. 2018. Predicting Japanese Word Order in Double Object Constructions. In Proceedings of the Eight Workshop on Cognitive Aspects of Computational Language Learning and Processing, pages 36-40, Melbourne. Association for Computational Linguistics.
Jörg Bahlmann, Antoni Rodriguez-Fornells, Michael Rotte, and Thomas F Munte. 2007. An fMRI study of canonical and noncanonical word order in German. Human brain mapping, 28(10):940-949.
Jelke Bloem. 2016. Testing the Processing Hypothesis of word order variation using a probabilistic language model. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CLALC), pages 174-185, Osaka, Japan. The COLING 2016 Organizing Committee.
Joan Bresnan, Anna Cueni, Tatiana Nikitina, and R Harald Baayen. 2007. Predicting the dative alternation. In Cognitive foundations of interpretation, pages 69-94. KNAW.
Shuk-Man Cheng, Chi-Hsin Yu, and Hsin-Hsi Chen. 2014. Chinese Word Ordering Errors Detection and Correction for Non-Native Chinese Language Learners. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 279-289, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Timothy Colleman. 2009. Verb disposition in argument structure alternations: a corpus study of the dative alternation in Dutch. Language Sciences, 31(5):593-611.
Bernard Comrie. 1989. Language universals and linguistic typology: Syntax and morphology. University of Chicago press.
John Fry. 2003. Ellipsis and wa-marking in Japanese conversation. Taylor & Francis.
Richard Futrell and Roger P Levy. 2019. Do RNNs learn human-like abstract word order preferences? In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 50-59.
Édouard Grave, Armand Joulin, Moustapha Cisse, David Grangier, and Hervé Jégou. 2017. Efficient softmax approximation for GPUs. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1302-1310, International Convention Centre, Sydney, Australia. PMLR.
Caroline Heycock. 1993. Syntactic predication in Japanese. Journal of East Asian Linguistics, 2(2):167-211.
Hajime Hoji. 1985. Logical form constraints and configurational structures in Japanese. *PHD Thesis*, University of Washington.
Malka Rappaport Hovav and Beth Levin. 2008. The English dative alternation: The case for verb sensitivity. Journal of linguistics, 44(1):129-167.
Katharina Kann, Sascha Rothe, and Katja Filippova. 2018. Sentence-Level Fluency Evaluation: References Help, But Can Be Spared! In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 313-323, Brussels, Belgium. Association for Computational Linguistics.
Gerard Kempen and Karin Harbusch. 2004. A corpus study into word order variation in German subordinate clauses: Animacy affects. Multidisciplinary approaches to language production, pages 173-181.
Masatoshi Koizumi and Katsuo Tamaoka. 2004. Cognitive processing of Japanese sentences with ditransitive verbs. Gengo Kenkyu (Journal of the Linguistic Society of Japan), 2004(125):173-190.
Masatoshi Koizumi and Katsuo Tamaoka. 2006. The Canonical Positions of Adjuncts in the Processing of Japanese Sentence. Cognitive Studies: Bulletin of the Japanese Cognitive Science Society, 13(3):392-403.
Taku Kudo. 2006. Mecab: Yet another part-of-speech and morphological analyzer. http://mecab.sourceforge.jp.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Mikinari Matsuoka. 2003. Two Types of Ditransitive Constructions in Japanese. Journal of East Asian Linguistics, 12(2):171-203.
Edson T Miyamoto. 2002. Sources of difficulty in the processing of scrambling in Japanese. Sentence processing in East Asian languages, pages 167-188.
Natsuko Nakagawa. 2016. Information structure in spoken Japanese: Particles, word order, and intonation.
Keiko Nakamoto, Jae-ho Lee, and Kow Kuroda. 2006. Preferred Word Orders Correlate with "Sentential" Meanings That Cannot Be Reduced to Verb Meanings: A New Perspective on "Construction Effects" in Japanese. Cognitive Studies: Bulletin of the Japanese Cognitive Science Society, 13(3):334-352.
Hisashi Noda. 1996. Wa to ga [Wa and ga]. Kurosio Publishers.
Marian Olteanu, Pasin Suriyentrakorn, and Dan Moldovan. 2006. Language models and reranking for machine translation. In Proceedings of the Workshop on Statistical Machine Translation, pages 150-153, New York City. Association for Computational Linguistics.
Naho Orita. 2017. Predicting japanese scrambling in the wild. In Proceedings of the 7th Workshop on Cognitive Modeling and Computational Linguistics, pages 41-45.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A Fast, Extensible Toolkit for Sequence Modeling*. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Tetsuo Saeki. 1960. Gendaigo ni okeru gojun no keikō - iwayuru hogo no baai [The trend of word order in modern writing- in so-called complements]. Gengo seikatsu [Language life], (111):56-63.
Tetsuo Saeki. 1998. Yosetsu Nihongo no Gojun [Essentials of Japanese word order]. Kurosio Publishers.
Ryohei Sasano and Manabu Okumura. 2016. A Corpus-Based Analysis of Canonical Word Order of Japanese Double Object Constructions. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2244, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Yasumasa Shigenaga. 2014. Canonical Word Order of Japanese Ditransitive Sentences: A Preliminary Investigation through a Grammaticality Judgment Survey. Advances in Language and Literary Studies, 5(2):35-45.
Dan I Slobin and Thomas G Bever. 1982. Children use canonical sentence schemas: A crosslinguistic study of word order and inflections. Cognition, 12(3):229-265.
Natsuko Tsujimura. 2013. An introduction to Japanese linguistics. John Wiley & Sons.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Karthik Visweswariah, Rajakrishnan Rajkumar, Ankur Gandhe, Ananthakrishnan Ramanathan, and Jiri Navratil. 2011. A Word Reordering Model for Improved Machine Translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 486-496, Edinburgh, Scotland, UK. Association for Computational Linguistics.
A Hyperparameters and implementation of the LMs
We used the Transformer (Vaswani et al., 2017) LMs implemented in fairseq (Ott et al., 2019). Table 5 shows the hyperparameters of the LMs. The adaptive softmax cutoff (Grave et al., 2017) is only applied to SLM. We split 10K sentences for dev set. The left-to-right and right-to-left CLMs achieved a perplexity of 11.05 and 11.08, respectively. The left-to-right and right-to-left SLMs achieved a perplexity of 28.51 and 28.25, respectively. Note that the difference in the perplexities between CLM and SLM is due to the difference in the vocabulary size.
B Details on Section 5.1 (double objects)
B.1 Word order for each verb
It is considered that different verbs have different preferences in the order of their object. For example, while the verb "例える" (compare) prefers the ACC-DAT order (Example (9)-a), the verb "表する" (express) prefers the DAT-ACC order (Example (9)-b).
(9) a. 人間を 色に 例えた.
person-ACC color-DAT compared.
$(\phi_I$  compared a person to color.)
b.店主の 敬意を 表た。
shopkeeper-DAT respect-ACC expressed.
$(\phi_{I}$  expressed a respect to a shopkeeper.)
Table 6 shows the verbs with the top five and the five worst  $R_{\mathrm{ACC-DAT}}^v$ .
B.2 Word order and verb types
There are two types of causative-inchoative alternating verbs in Japanese: show-type verbs and pass-type verbs. The verb types are determined by the subject of the sentence where the corresponding inchoative verb is used. For the show-type verbs, the DAT argument of a causative sentence becomes the subject in its corresponding inchoative sentence (Example (10)). On the other hand, the ACC argument of a causative sentence becomes the subject in its corresponding inchoative sentence for the pass-type verbs (Example (11)).
(10) Causative: 生徒に 本を 見せた
student-DAT book-ACC showed.
$(\phi_I$  showed a student a book.)
Inchoative: 生徒が 見た
student-NOM saw.
(A student saw  $\phi_{\mathrm{something}}$
(11) Causative: 生徒に 本を 渡た
$(\phi_I$  passed a student a book.)
Inchoative: 本が 渡った
book-NOM passed.
(A book passed to  $\phi_{\mathrm{something}}$
Matsuoka (2003) claims that the show-type verb prefers the DAT-ACC order, while the pass-type verb prefers the ACC-DAT order.
Table 7 shows  $R_{\mathrm{ACC-DAT}}^v$  of the show-type and pass-type verbs. The results show no significant difference in word order trends between show-type and pass-type verbs, which are consistent with that of Sasano and Okumura (2016).
B.3 Word order and semantic role of the dative argument