SlowGuess/ACL_2020_OCR huggingface dataset:
Tatsuki Kuribayashi $^{1,3}$ , Takumi Ito $^{1,3}$ , Jun Suzuki $^{1,2}$ , Kentaro Inui $^{1,2}$
$^{1}$ Tohoku University  $^{2}$ RIKEN  $^{3}$ Langsmith Inc.
{kuribayashi, t-ito, jun.suzuki, inui} @ecei.tohoku.ac.jp
We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of preprocessor errors in count-based methods. In this study, we explore whether the LM-based method is valid for analyzing the word order. As a case study, this study focuses on Japanese due to its complex and flexible word order. To validate the LM-based method, we test (i) parallels between LMs and human word order preference, and (ii) consistency of the results obtained using the LM-based method with previous linguistic studies. Through our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool. Finally, using the LM-based method, we demonstrate the relationship between the canonical word order and topicalization, which had yet to be analyzed by large-scale experiments.
Speakers sometimes have a range of options for word order in conveying a similar meaning. A typical case in English is dative alternation:
(1) a. A teacher gave a student a book.
b. A teacher gave a book to a student.
Even for such a particular alternation, several studies (Bresnan et al., 2007; Hovav and Levin, 2008; Colleman, 2009) investigated the factors determining this word order and found that the choice is not random. For analyzing such linguistic phenomena, linguists repeat the cycle of constructing hypotheses and testing their validity, usually through psychological experiments or count-based methods. However, these approaches sometimes face difficulties, such as scalability issues in psychological
Figure 1: LM-based method for evaluating the canonicality of each word order considering their generation probabilities.
experiments and the propagation of preprocessor errors in count-based methods.
Compared to the typical approaches for evaluating linguistic hypotheses, approaches using LMs have potential advantages (Section 3.2). In this study, we examine the methodology of using LMs for analyzing word order (Figure 1). To validate the LM-based method, we first examine if there is a parallel between canonical word order and generation probability of LMs for each word order. Futrell and Levy (2019) reported that English LMs have human-like word order preferences, which can be one piece of evidence for validating the LM-based method. However, it is not clear whether the above assumption is valid even in languages with more flexible word order.
In this study, we specifically focus on the Japanese language due to its complex and flexible word order. There are many claims on the canonical word order of Japanese, and it has attracted considerable attention from linguists and natural language processing (NLP) researchers for decades (Hoji, 1985; Saeki, 1998; Miyamoto, 2002; Matsuoka, 2003; Koizumi and Tamaoka, 2004; Nakamoto et al., 2006; Shigenaga, 2014; Sasano and Okumura, 2016; Orita, 2017; Asahara et al., 2018).
We investigated the validity of using Japanese LMs for canonical word order analysis by conducting two sets of experiments: (i) comparing word order preference in LMs to that in Japanese speakers (Section 4), and (ii) checking the consistency
Topic	Time	Location	Subject	(Adverb)	Indirect object	Direct object	Verb
Notation	TOP	TIM	LOC	NOM	-	DAT	ACC	-
Typical particle	“は” (wa)	“に” (ni)	“て” (de)	“が” (ga)	-	“に” (ni)	“を” (o)	-
Related section	6	5.2	5.2	5.2	5.3	5.1	5.1	5.1
Table 1: Overview of the typical cases in Japanese, their typical particles, and the sections where the corresponding case is analyzed. The well-known canonical word order of Japanese is listed from left to right.
between the preference of LMs with previous linguistic studies (Section 5). From our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool, and further explore potential applications. Finally, we analyzed the relationship between topicalization and word order of Japanese by taking advantage of the LM-based method (Section 6).
- Discuss and validate the use of LMs as a tool for word order analysis as well as investigate the sensitivity of LMs against different word orders in non-European language (Section 3);
- Find encouraging parallels between the results obtained with the LM-based method and those with the previously established method on various hypotheses of canonical word order of Japanese (Sections 4 and 5); and
- Showcase the advantages of an LM-based method through analyzing linguistic phenomena that is difficult to explore with the previous data-driven methods (Section 6).
2 Linguistic background
This section provides a brief overview of the linguistic background of canonical word order, some basics of Japanese grammar, and common methods of linguistic analysis.
2.1 On canonical word order
Every language is assumed to have a canonical word order, even those with flexible word order (Comrie, 1989). There has been a significant linguistic effort to reveal the factors determining the canonical word order (Bresnan et al., 2007; Hoji, 1985). The motivations for revealing the canonical word order range from linguistic interests to those involved in various other fields—it relates to language acquisition and production in psycholinguistics (Slobin and Bever, 1982; Akhtar, 1999), second language education (Alonso Belmonte et al., 2000), and natural language generation (Visweswariah et al., 2011) or error cor
rection (Cheng et al., 2014) in NLP. In Japanese, there are also many studies on its canonical word order (Hoji, 1985; Saeki, 1998; Koizumi and Tamaoka, 2004; Sasano and Okumura, 2016).
Japanese canonical word order The word order of Japanese is basically subject-object-verb (SOV) order, but there is no strict rule except placing the verb at the end of the sentence (Tsujimura, 2013). For example, the following three sentences have the same denotational meaning ("A teacher gave a student a book"):
(2) a. 先生が生徒に本をあけた teacher-NOM student-DAT book-ACC gave.
b. 先生が 本を 生徒に あた teacher-NOM book-ACC student-DAT gave.
c.本生徒先生が 书-ACC student-DAT teacher-NOM gave.
This order-free nature suggests that the position of each constituent does not represent its semantic role (case). Instead, postpositional case particles indicate the roles. Table 1 shows typical constituents in a Japanese sentence, their postpositional particles, their canonical order, and the sections of this paper where each of them is analyzed. Note that postpositional case particles are sometimes omitted or replaced with other particles such as adverbial particles (Section 6). These characteristics complicate the factors determining word order, which renders the automatic analysis of Japanese word order difficult.
2.2 On typical methods for evaluating word order hypotheses and their difficulties
There are two main methods in linguistic research: human-based methods, which observe human reactions, and data-driven methods, which analyze text corpora.
Human-based methods A typical approach of testing word order hypotheses is observing the reaction (e.g., reading time) of humans to each word order (Shigenaga, 2014; Bahlmann et al., 2007). These approaches are based on the direct observation of humans, but this method has scalability
issues. There are also concerns that the participants may be biased, and that the experiments may not be replicable.
Data-driven methods Another typical approach is counting the occurrence frequencies of the targeted phenomena in a large corpus. This count-based method is based on the assumption that there are parallels between the canonical word order and the frequency of each word order in a large corpus. The parallel has been widely discussed (Arnon and Snider, 2010; Bresnan et al., 2007), and many studies rely on this assumption (Sasano and Okumura, 2016; Kempen and Harbusch, 2004). One of the advantages of this approach is suitability for large-scale experiments. This enables considering a large number of examples.
In this method, researchers often have to identify the phenomena of interest with preprocessors (e.g., the predicate-argument structure parser used by Sasano and Okumura (2016)) in order to count them. However, sometimes, identification of the targeted phenomena is difficult for the preprocessors, which limits the possibilities of analysis. For example, Sasano and Okumura (2016) focused only on simple examples where case markers appear explicitly, and only extract the head noun of the argument to avoid preprocessor errors. Thus, they could not analyze the phenomena in which the above conditions were not met. The above issue becomes more serious in low-resource languages, where the necessary preprocessors are often unavailable.
In this count-based direction, Bloem (2016) used n-gram LMs to test the claims on the German two-verb clusters. This method is closest to our proposed approach, but the general validity of using LMs is out of focus. This LM-based method also relies on the assumption of the parallels between the canonical word order and the frequency.
Another common data-driven approach is to train an interpretable model (e.g., Bayesian linear mixed models) to predict the targeted linguistic phenomena and analyze the inner workings of the model (e.g., slope parameters) (Bresnan et al., 2007; Asahara et al., 2018). Through this approach, researchers can obtain richer statistics, such as the strength of each factor's effect on the targeted phenomena, but creating labeled data and designing features for supervised learning can be costly.
3 LM-based method
3.1 Overview of the LM-based method
In the NLP field, LMs are widely used to estimate the acceptability of text (Olteanu et al., 2006; Kann et al., 2018). An overview of the LM-based method is shown in Figure 1. After preparing several word orders considering the targeted linguistic hypothesis, we compare their generation probabilities in LMs. We assume that the word order with the highest generation probability follows their canonical word order.
3.2 Advantages of the LM-based method
In the count-based methods mentioned in Section 2.2, researchers often require preprocessors to identify the occurrence of the phenomena of interest in a large corpus. On the other hand, researchers need to prepare data to be scored by LMs to evaluate hypothesis in the LM-based method. Whether it is easier to prepare the preprocessor or the evaluation data depends on the situation. For example, the data preparation is easier in the situation where one wants to analyze the word order trends when a specific postpositional particle is omitted. The question is whether Japanese speakers prefer the word order like in Example (3)-a or (3)-b.
(3) a. 生徒に 本壺 あらた.
student-DAT book(-ACC) gave.
b. 本壺 生徒に あらた.
book(-ACC) student-DAT gave.
While identifying the cases (ACC in Example (3)) without their postpositional particle is difficult, creating the data without a specific postpositional particle by modifying the existing data is easier such as creating Example (4)-b from Example (4)-a.
(4) a. 生徒に 本を あた。
student-DAT book-ACC gave.
b.生徒に 本を た.
Thus, in such situation, the LM-based method can be suitable.
The human-based method is more reliable given an example. However, it can be prohibitively costly. While the human-based method requires an evaluation data and human subjects, the LM-based method only requires the evaluation data. Thus, the LM-based method can be more suitable for estimating the validity of hypotheses and considering
Omitted characters are crossed out. (e.g., 奎)
many examples as exhaustively as possible. In addition, the LM-based method can be replicable. The suitable approach can be different in a situation, and broadening the choice of alternative methodologies may be beneficial to linguistic research.
Nowadays, various useful frameworks, language resources, and machine resources required to train LMs are available, $^{2}$  which support the ease of implementing the LM-based method. Moreover, we make the LMs used in this study available. $^{3}$
3.3 Strategies to validate the use of LMs to analyze the word order
The goal of this study is to validate the use of LMs for analyzing the canonical word order. The canonical word order itself is still a subject of research, and the community does not know all about it. Thus, it is ultimately impossible to enumerate the requirements on what LMs should know about the canonical word order and probe the knowledge of LMs. Instead, we demonstrate the validity of the LM-based method by showcasing two types of parallels: (i) word order preference of LMs showing parallels with that of humans, and (ii) the results obtained with the LM-based method and those with previous methods being consistent on various claims on canonical word order. If the results of LMs are consistent with those of existing methods, the possibility that LMs and existing methods have the same ability to evaluate the hypotheses is supported. If the LM-based method is assumed to be valid, the method has the potential to streamline the research on unevaluated claims on word order. In the experiment sections, we examine the properties of Japanese LMs on (i) and (ii).
3.4 CAUTION - when using LMs for evaluating linguistic hypotheses
Even if LMs satisfy the criteria described in 3.3, there is no exact guarantee that LM scores will reflect the effectiveness of human processing of specific constructions in general. Thus, there seems to be a danger of confusing LM artifacts with language facts. Based on this, we hope that researchers use LMs as a tool just to limit the hypothesis space. LM supported hypotheses should then be re-verified with a human-based approach.
Furthermore, since there is a lot of hypotheses and corresponding research, we cannot check all the properties of LMs in this study. This study focuses on intra-sentential factors of Japanese case order, and it is still unclear whether the LM-based method works properly in linguistic phenomena which are far from being the focus of this study. This is the first study where evidence is collected on the validity of using LMs for word order analysis and encourages further research on collecting such evidence and examining under what conditions this validity is guaranteed.
3.5 LMs settings
We used auto-regressive, unidirectional LMs with Transformer (Vaswani et al., 2017). We used two variants of LMs, a character-based LM (CLM) and a subword-based LM (SLM). In training SLM, the input sentences are once divided into morphemes by MeCab (Kudo, 2006) with a UniDic dictionary, and then these morphemes are split into subword units by byte-pair-encoding. (Sennrich et al., 2016). 160M sentences randomly selected from 3B web pages were used to train the LMs. Hyperparameters are shown in Appendix A.
Given a sentence  $s$ , we calculate its generation probability  $p(s) = \overrightarrow{p}(s) \cdot \overleftarrow{p}(s)$ , where  $\overrightarrow{p}(\cdot)$  and  $\overleftarrow{p}(\cdot)$  are generation probabilities calculated by a left-to-right LM and a right-to-left LM, respectively. Depending on the hypothesis, we compare the generation probabilities of various variants of  $s$  with different word orders. We assume that the word order with the highest generation probability follows their canonical word order.
4 Experiment1: comparing human and LMs word order preference
To examine the validity of using LMs for canonical word order analysis, we examined the parallels between the LMs and humans on the task determining the canonicality of the word order (Figure 2). First, we created data for this task (Section 4.1). We then compared the word order preference of LMs and that of humans (Section 4.2).
4https://unidic.ninja.ac.jp/
<sup>5</sup>Implemented in sentencepiece (Kudo and Richardson, 2018) We set character coverage to 0.9995, and vocab size to 100,000.
$^{6}$ 14GB in UTF-8 encoding. For reference, Japanese Wikipedia has around 2.5 GB of text. Because the focus of this study has context-independent nature, the sentences order is shuffled to prevent learning the inter-sentential characteristics of the language.
2For example, one can train LMs with fairseq (Ott et al., 2019) and Wikipedia data on cloud computing platforms. 3https://github.com/kuribayashi4/LM_ asWord_Seor_Evaluator.
Figure 2: Overview of the experiment of comparing human and LMs word order preference. First, we created data for the task of comparing the appropriateness of the word order (left part), then we compare the preference of LMs and humans through this task (right part).
4.1 Human annotation
Data We randomly collected 10k sentences from 3B web pages, which are not overlapped with the LM training data. To remove overly complex sentences, we extracted sentences that must: (i) have less than or equal to five clauses and one verb, (ii) have clauses with a sibling relationship in its dependency tree, and they accompany a particle or adverb, (iii) not have special symbols such as parentheses, and (iv) not have a backward dependency path. For each sentence, we created its scrambled version.<sup>7</sup> The scrambling process is as follows:
1. Identify the dependency structure by using JUMAN and KNP.
2. Randomly select a clause with several children.
3. Shuffle the position of its children along with their descendants.
Annotation We used the crowdsourcing platform Yahoo Japan!10. For our task, we showed crowdworkers a pair of sentences (order  $_{1}$ , order  $_{2}$ ), where one sentence has the original word order, and the other sentence has a scrambled word order.11 Each annotator was instructed to label the pair with one of the following choices: (1) order  $_{1}$  is better, (2) order  $_{2}$  is better, or (3) the pair contains a semantically broken sentence. Only the sentences (order  $_{1}$ , order  $_{2}$ ) were shown to the annotators, and they were instructed not to imagine a specific context for the sentences. We filtered unmotivated workers by using check questions.12 For each pair
instance, we employed 10 crowdworkers. In total, 756 unique, motivated crowdworkers participated in our task.
From the annotated data, we collected only the pairs satisfying the following conditions for our experiments: (i) none of 10 annotators determined that the pair contains a semantically broken sentence, and (ii) nine or more annotators preferred the same order. The majority decision is labeled in each pair; the task is binary classification. We assume that if many workers prefer a certain word order, then it follows its canonical word order, and the other one deviates from it. We collected 2.6k pair instances of sentences.
We compared the word order preference of LMs and that of the workers by using the 2.6K pairs created in Section 4.1. We calculated the correlation of the decisions between the LMs and the workers; which word order is more appropriate order $_1$  or order $_2$ . The word orders supported by CLM and SLM are highly correlated with workers, with the Pearson correlation coefficient of 0.89 and 0.90, respectively. This supports the assumption that the generation probability of LMs can determine the canonical word order as accurately as humans do. Note that such a direct comparison of word order is difficult with the count-based methods because of the sparsity of the corpus.
5 Experiment2: consistency with previous studies
This section examines whether LMs show word order preference consistent with previous linguistic studies. The results are entirely consistent, which support the validity of the LM-based methods in Japanese. Each subsection focuses on a specific component of Japanese sentences.
5.1 Double objects
The order of double objects is one of the most controversial topics in Japanese word order. Examples of the possible order are as follows:
(5) DAT-ACC: 生徒に student-DAT 本を book-ACC gave.
ACC-DAT: 本を book-ACC student-DAT gave.
Henceforth, DAT-ACC/ACC-DAT denotes the word order in which the DAT/ACC argument precedes the ACC/DAT argument. We evaluate the
7When several scrambled versions were possible for a given sentence, we randomly selected one of them.   
8http://nlp.ist.i.kyoto-u.ac.jp/EN/ index.php?JUMAN   
9http://nlp.ist.i.kyoto-u.ac.jp/EN/ index.php?KNP   
10https://crowdsourcing.yahoo.co.jp/   
11Crowdworkers did not know which sentence was the original sentence.   
12We manually created check questions considering the Japanese speakers' preference in trial experiments in advance.
(a) Each verb's ACC-DAT rate.
(b) Relationship between each verb's  $R_{\mathrm{DAT - only}}^v$  and the ACC-DAT rate.
(c) Relationship between the degree of co-occurrence of verb and arguments, and the ACC-DAT rate in each example. For the results of LMs, the ACC-DAT rate of each example is regarded as 1 if LMs prefer ACC-DAT order, otherwise we regard the example as 0.
Figure 3: Overlap of the results of Sasano and Okumura (2016) and that of LMs. In figures (a) and (b), each plot corresponds to each verb. In figure (c), each plot corresponds to each example. The legend of figure (a) and (b) is the same as in figure (c). "S&O 2016" refers to Sasano and Okumura (2016).
claims Sasano and Okumura (2016) focused on with the data they collected. $^{13}$
Word order for each verb First, we analyzed the trend of the double object order for each verb. We analyzed 620 verbs following Sasano and Okumura (2016). For each set of examples  $S^v$  corresponding to a verb  $v$ , we: (i) created an instance with the swapped order of ACC and DAT for each example, and (ii) compared the generation probabilities of the original and swapped instance.  $\hat{S}^v$  is the set of examples preferred by LMs.  $R_{\mathrm{ACC - DAT}}^v$  is calculated as follows:
$$
R _ {\mathrm {A C C - D A T}} ^ {v} = \frac {N _ {\mathrm {A C C - D A T}} ^ {v}}{N _ {\mathrm {A C C - D A T}} ^ {v} + N _ {\mathrm {D A T - A C C}} ^ {v}},
$$
where  $N_{\mathrm{ACC - DAT}}^v /N_{\mathrm{DAT - ACC}}^v$  is the number of examples with the ACC-DAT/DAT-ACC order in  $\hat{S}^v$
Figure 3-(a) shows the relationship between  $R_{\mathrm{ACC - DAT}}^v$  determined by LMs and one reported in a
previous count-based study (Sasano and Okumura, 2016). These results strongly correlate with the Pearson correlation coefficient of 0.91 and 0.88, in CLM and SLM, respectively. In addition, "canonical word order is DAT-ACC" (Hoji, 1985) is unlikely to be valid because there are verbs where  $R_{\mathrm{ACC - DAT}}^v$  is very high (details in Appendix B.1). This conclusion is consistent with Sasano and Okumura (2016).
Word order and verb types In Japanese, there are show-type and pass-type verbs (details in Appendix B.2). Matsuoka (2003) claimed that the order of double objects differs depending on these verb types. Following Sasano and Okumura (2016), we analyzed this trends.
We applied the Wilcoxon rank-sum test between the distributions of  $R_{\mathrm{ACC - DAT}}^v$  determined by LMs in the two groups (show-type and pass-type verbs). The results show no significant difference between the two groups (p-value is 0.17 and 0.12 in the experiments using CLM and SLM, respectively). These results are consistent with the count-based (Sasano and Okumura, 2016) and the human-based (Miyamoto, 2002; Koizumi and Tamaoka, 2004) methods.
Word order and argument omission Sasano and Okumura (2016) claimed that the frequently omitted case is placed near the verb. First, we calculated  $R_{\mathrm{DAT-only}}^v$  for each verb  $v$  as follows:
$$
R _ {\mathrm {D A T - o n l y}} ^ {v} = \frac {N _ {\mathrm {D A T - o n l y}} ^ {v}}{N _ {\mathrm {D A T - o n l y}} ^ {v} + N _ {\mathrm {A C C - o n l y}} ^ {v}},
$$
where  $N_{\mathrm{DAT - only}}^v / N_{\mathrm{ACC - only}}^v$  denotes the number of examples in which the DAT/ACC case appears, and the other case does not in  $S^v$ . A large  $R_{\mathrm{DAT - only}}^v$  score indicates that the DAT argument is less frequently omitted than the ACC argument in  $S^v$ . We analyzed the relationship between  $R_{\mathrm{DAT - only}}^v$  and  $R_{\mathrm{ACC - DAT}}^v$  for each verb.