SlowGuess/ACL_2020_OCR huggingface dataset:
What Determines the Order of Adjectives in English? Comparing Efficiency-Based Theories Using Dependency Treebanks
Richard Futrell
William Dyer
Gregory Scontras
rfutrell@uci.edu william.dyer@oracle.comg.scontras@uci.edu
We take up the scientific question of what determines the preferred order of adjectives in English, in phrases such as big blue box where multiple adjectives modify a following noun. We implement and test four quantitative theories, all of which are theoretically motivated in terms of efficiency in human language production and comprehension. The four theories we test are subjectivity (Scontras et al., 2017), information locality (Futrell, 2019), integration cost (Dyer, 2017), and information gain, which we introduce. We evaluate theories based on their ability to predict orders of unseen adjectives in hand-parsed and automatically-parsed dependency treebanks. We find that subjectivity, information locality, and information gain are all strong predictors, with some evidence for a two-factor account, where subjectivity and information gain reflect a factor involving semantics, and information locality reflects collocational preferences.
Across languages, there exist strong and stable constraints on the order of adjectives when more than one is used to modify a noun (Dixon, 1982; Sproat and Shih, 1991). For example, in English, big blue box sounds natural and appears relatively frequently in corpora, while blue big box sounds less natural and occurs less frequently (Scontras et al., 2017). In this paper, we take up the scientific question of what explains these constraints in natural language. To do so, we implement quantitative models that have been proposed in previous literature as explanations for these constraints, and compare their accuracy in predicting adjective ordering data in parsed corpora of English<sup>1</sup>.
In the last few years, adjective order has become a crucial testing ground for quantitative theories
of syntax. These theories provide mathematical models that can describe the distribution of words in sentences and the way those words combine to yield the meaning of a sentence, in a way that captures the fine-grained quantitative patterns observable in large text datasets (Manning, 2003; Bresnan et al., 2007; Chen and Ferrer-i-Cancho, 2019).
Quantitative syntactic theories are often efficiency-based, meaning that they model word distributions as the result of a process that tries to maximize information transfer while minimizing some measure of cognitive cost; as a result, they often use the mathematical language of information theory. Such theories promise not only to describe distributions of words, but also to explain why they take the shape they do, by viewing human language as an efficient code subject to appropriate constraints. This work informs NLP by providing a theory of language structure that integrates with data-driven, optimization-based machine learning models.
Adjective order is a fruitful empirical target for quantitative theories of syntax because it is an area where the traditional discrete and symbolic theories become highly complex, and a quantitative approach becomes more attractive. For example, in the formal syntax literature, a standard explanation for adjective order constraints is that each adjective belongs to a certain semantic class (e.g., COLOR or SIZE) and that there exists a universal total order on these semantic classes (e.g., COLOR  $<$  SIZE) shared among all languages, which determines the order of adjectives in any given instance (Cinque, 1994; Scott, 2002). Such discrete theories of adjective order become complex rapidly as the number of semantic classes to be posited becomes large (upwards of twelve in Scontras et al. 2017) and more fine-grained (see Bar-Sever et al. 2018 for discussion of the learning problem posed by such classifications).
1All code and data are available at https://github. com/langprocgroup/adjorder.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2003-2012 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In contrast, quantitative syntax theories typically identify a single construct that grounds out in real-valued numerical scores given to adjectives, which determine their ordering preferences. These scores can be estimated based on large-scale corpus data or based on human ratings. In what follows, we test the predictions of four such theories: the subjectivity hypothesis (Scontras et al., 2017; Simonic, 2018; Hahn et al., 2018; Franke et al., 2019; Scontras et al., 2019), the information locality hypothesis (Futrell and Levy, 2017; Futrell et al., 2017; Hahn et al., 2018; Futrell, 2019), the integration cost hypothesis (Dyer, 2017), and the information gain hypothesis, which we introduce.
We begin with a presentation of the details of each theory, then implement the theories and test their predictions against large-scale naturalistic data from English. In addition to comparing the predictors in terms of accuracy, we also perform a number of analyses to determine the important similarities and differences among their predictions. The paper concludes with a discussion of what our results tell us about adjective order and related issues, and a look towards future work.
2 Theories of adjective order
2.1 Subjectivity
Scontras et al. (2017) show that adjective order is strongly predicted by adjectives' subjectivity scores: an average rating obtained by asking human participants to rate adjectives on a numerical scale for how subjective they are. Adjectives that are rated as more subjective typically appear farther from the noun than adjectives rated as less subjective, and the strength of ordering preferences tracks the subjectivity differential between two adjectives. For example, in big blue box, the adjective big has a subjectivity rating of 0.64 (out of 1), and the adjective blue has a subjectivity rating of 0.30. If adjectives are placed in order of decreasing subjectivity, then big must appear before blue, corresponding to the preferred order. The notion of subjectivity as a predictor of adjective order was previously introduced by Hetzron (1978).
Subsequent work has attempted to explain the role of subjectivity in adjective ordering by appealing to the communicative benefit afforded by ordering adjectives with respect to decreasing subjectivity. For example, Franke et al. (2019) use simulated reference games to demonstrate that, given a set of independently-motivated assump
tions concerning the composition of meaning in multi-adjective strings, subjectivity-based orderings lead to a greater probability of successful reference resolution; the authors thus offer an evolutionary explanation for the role of subjectivity in adjective ordering (see also Simonic, 2018; Hahn et al., 2018; Scontras et al., 2019).
2.2 Information locality
The theory of information locality holds that words that have high mutual information are under pressure to be close to each other in linear order (Futrell and Levy, 2017; Futrell et al., 2017). Information locality is a generalization of the well-supported principle of dependency length minimization (Liu et al., 2017; Temperley and Gildea, 2018). In the case of adjective ordering, the prediction is simply that adjectives that have high pointwise mutual information (PMI) with their head noun will tend to be closer to that noun. The PMI of an adjective  $a$  and a noun  $n$  is (Fano, 1961; Church and Hanks, 1990):
$$
\operatorname {P M I} (a: n) \equiv \log \frac {p (a , n)}{p (a) p (n)}. \tag {1}
$$
In this paper, we take the relevant joint distribution  $p(a,n)$  to be the distribution of adjectives and nouns in a dependency relationship, with the marginals calculated as  $p(a) = \sum_{n}p(a,n)$  and  $p(n) = \sum_{a}p(a,n)$ .
Information locality is motivated as a consequence of a more general theory of efficiency in human language. In this theory, languages should maximize information transfer while minimizing cognitive information-processing costs associated with language production and comprehension. Information locality emerges from these theories when we assume that the relevant measure of information-processing cost is the surprisal of words given lossy memory representations (Hale, 2001; Levy, 2008; Smith and Levy, 2013; Futrell and Levy, 2017; Futrell, 2019).
2.3 Integration Cost
The theory of integration cost is also based in the idea of efficiency with regard to information-processing costs. It differs from information locality in that it assumes that the correct metric of processing difficulty for a word  $w$  is the entropy
over the possible heads of  $w$ :
$$
\begin{array}{l} \operatorname {C o s t} (w) \propto H [ T | w ] \\ = \sum_ {t} - p _ {T} (t | w) \log p _ {T} (t | w), \tag {2} \\ \end{array}
$$
where  $T$  is a random variable indicating the head  $t$  of the word  $w$  (Dyer, 2017). This notion of cost captures the amount of uncertainty that has to be resolved about the proper role of the word  $w$  with respect to the rest of the words in the sentence. Like information locality, the theory of integration cost recovers dependency length minimization as a special case. For the case of predicting adjective order, the prediction is that an adjective  $a$  will be closer to a noun when it has lower integration cost:
$$
\operatorname {I C} (a) = H [ N | a ], \tag {3}
$$
where  $N$  is a random variable ranging over nouns.
Integration cost corresponds to an intuitive idea previously articulated in the adjective ordering literature. The idea is that adjectives that can modify a smaller set of nouns appear closer to the noun: for example, an order such as big wooden spoon is preferred over wooden big spoon because the word big can modify nearly any noun, while wooden can only plausibly modify a small set of nouns (Ziff, 1960). The connection between integration cost and set size comes from the information-theoretic notion of the typical set (Cover and Thomas, 2006, pp. 57-71); the entropy of a random variable can be interpreted as the (log) cardinality of the typical set of samples from that variable. When we order adjectives by integration cost, this is equivalent to ordering them such that adjectives that can modify a larger typical set of nouns appear farther from the noun. The result is that each adjective gradually reduces the entropy of the possible nouns to follow, thus avoiding information-processing costs that may be associated with entropy reduction (Hale, 2006, 2016; Dye et al., 2018).
2.4 Information gain
We propose a new efficiency-based predictor of adjective order: information gain. The idea is to view the noun phrase, consisting of prenominal adjectives followed by the noun, as a decision tree for identifying a referent, where each word partitions the space of possible referents. Each partitioning is associated with some information gain, indicating how much the set of possible referents
shrinks. In line with the logic for integration cost, we propose that the word with smaller information gain will be placed earlier, so that the set of referents is gradually narrowed by each word.
As generally implemented in decision trees, information gain refers to the reduction of entropy obtained from partitioning a set on a feature (Quinlan, 1986). In our case, the distribution of nouns  $N$  is partitioned on a given adjective  $a$ , creating two partitions:  $N_{a}$  and its complement  $N_{a}^{c}$ . The difference between the starting entropy  $H[N]$  and the sum of the entropy of each partition, conditioned on the size of that partition, is the information gain of  $a$ :
$$
\begin{array}{l} \operatorname {I G} (a) = H [ N ] \\ - \left[ \frac {\left| N _ {a} \right|}{\left| N \right|} H \left[ N _ {a} \right] + \frac {\left| N _ {a} ^ {c} \right|}{\left| N \right|} H \left[ N _ {a} ^ {c} \right] \right]. \tag {4} \\ \end{array}
$$
Information gain is therefore comprised of both positive and negative evidence. That is, specifying an adjective such as big partitions the probability distribution of nouns into  $N_{big}$ , the subset of  $N$  which takes big as a dependent, and  $N_{big}^{C}$ , the subset of  $N$  which does not.
Crucially,  $H[N_{a}]$  is not  $H[N|a]$  in general.  $H[N|a]$  is the conditional entropy of nouns given a specific adjective, while  $H[N_{a}]$  is the entropy of a distribution over nouns whose support is limited to noun types that have been observed to occur with an adjective  $a$ . Combined with  $H[N_{a}^{c}]$ , information gain tells us how much the entropy of  $N$  is reduced by partitioning on  $a$ . This means that information gain and integration cost, while conceptually similar, are not mathematically equivalent.
To our knowledge, information gain has not been previously suggested as a predictor of adjective ordering, although Danks and Glucksberg (1971) expressed a similar intuition in proposing that adjectives are ordered according to their 'discriminative potential'. Although decision-tree algorithms such as ID3 choose the highest-IG feature first, we predict that the lower-information-gain adjective will precede the higher one.
Previous corpus studies of adjective order include Malouf (2000), who examined methods for ordering adjectives in a natural language generation context, and Wulff (2003), who examined effects of phonological length, syntactic category ambiguity, semantic closeness, adjective frequency, and
a measure similar to PMI called noun specificity. Our work differs from this previous work by focusing on recently-introduced predictors that have theoretical motivations grounded in efficiency and information theory.
The theories we test here (except information gain) have been tested in previous corpus studies, but never compared against each other. Scontras et al. (2017) validate that subjectivity is a good predictor of adjective order in corpora, and Hahn et al. (2018) and Futrell et al. (2019) evaluate both information locality and subjectivity. Dyer (2018) uses integration cost to model the order of same-side sibling dependents cross-linguistically and across all syntactic categories.
Our task is to find predictors of adjective order based solely on data about individual adjectives and nouns. More formally, the goal is to find a scoring function  $S(A,N)$  applying to an adjective  $A$  and a noun  $N$ , such that the order of two adjectives modifying a noun  $A_{1}A_{2}N$  can be predicted accurately by comparing  $S(A_{1},N)$  and  $S(A_{2},N)$ . Furthermore, the scoring function  $S$  should not include information about relative order in observed sequences of the form  $A_{1}A_{2}N$  the scoring function should be based only on corpus data about co-occurrences of  $A$  and  $N$ , or on human ratings about  $A$  and/or  $N$ . We apply this restriction because our goal is to evaluate scientific theories of why adjectives are ordered the way they are, rather than to achieve maximal raw accuracy.
Corpus-based predictors We estimate information-theoretic quantities for adjectives using a large automatically-parsed subsection of the English Common Crawl corpus (Buck et al., 2014; Futrell et al., 2019). The use of a parsed corpus is necessary to identify adjectives that are dependents of nouns in order to calculate PMI and IC. As described in Futrell et al. (2019), this corpus was produced by heuristically filtering Common Crawl to contain only full sentences and to remove web boilerplate text, and then parsing the resulting text using SyntaxNet (Andor et al., 2016), obtaining a total of  $\sim 1$  billion tokens of automatically parsed web text. In this work, we use a subset of this corpus, described below.
From this corpus, we extract two forms of data.
First, we extract adjective-noun (AN) pairs: a set of pairs  $\langle A,N\rangle$  where  $A$  is an adjective and  $N$  is a noun and  $N$  is the head of  $A$  with dependency type amod. As in Futrell (2019), we define  $A$  as an adjective iff its part-of-speech is  $\mathbb{J}\mathbb{J}$  and its wordform is listed as an adjective in the English CELEX database (Baayen et al., 1995). We define  $N$  as a noun iff its part-of-speech is NN or NNS and its wordform is listed as a noun in CELEX. These AN pairs are used to estimate the information-theoretic predictors that we are interested in. We extracted 33,210,207 adjective-noun pairs from the parsed Common Crawl corpus.
Second, we extract adjective-adjective-noun (AAN) triples: a set of triples  $\langle A_1, A_2, N \rangle$  where  $A_1$  and  $A_2$  are adjectives as defined above, and  $A_1$  and  $A_2$  are both adjective dependents with relation type amod of a single noun head  $N$ . Furthermore,  $A_1$  and  $A_2$  must not have any further dependents, and they must appear in the order  $A_1A_2N$  in the corpus with no intervening words. We extracted a total of 842,714 AAN triples from the parsed Common Crawl corpus.
The values of all corpus-based predictors are estimated using the AN pairs. The AAN triples are used only for fitting regressions from the predictors to adjective orders, and for evaluation.
Ratings-based predictors We gathered subjectivity ratings for all 398 adjectives present in AAN triples in the English UD corpus. These subjectivity ratings were collected over Amazon.com's Mechanical Turk, using the methodology of Scontras et al. (2017). 264 English-speaking participants indicated the subjectivity of 30 random adjectives by adjusting a slider between endpoints labeled "completely objective" (coded as 0) and "completely subjective" (coded as 1). Each adjective received an average of 20 ratings.
Test set As a held-out test set for our predictors, we use the English Web Treebank (EWT), a hand-parsed corpus, as contained in Universal Dependencies (UD) v2.4 (Silveira et al., 2014; Nivre, 2015). Following our criteria, we extract 155 AAN triples having scores for all our predictors. Because this test set is very small, we also evaluate against a held-out portion of the parsed Common crawl data. In the Common crawl test set, after including only AAN triples that have scores for all of our predictors, we have 41,822 AAN triples.
4.2 Estimation of predictors
Our information-theoretic predictors require estimates of probability distributions over adjectives and nouns. To estimate these probability distributions, we first use maximum likelihood estimation as applied to counts of wordforms in AN pairs. We call these estimates wordform estimates.
Although maximum likelihood estimation is sufficient to give an estimate of the general entropy of words (Bentz et al., 2017), it is not yet clear that it gives a good measure for conditional entropy or mutual information, due to data sparsity, even with millions of tokens of text (Futrell et al., 2019).
Therefore, as a second method that alleviates the data sparsity issue, we also calculate our probability distributions not over raw wordforms but over clusterings of words in an embedding space, a method which showed promise in Futrell et al. (2019). To derive word clusters, we use sklearn.cluster.KMeans applied to a pretrained set of 1.9 million 300-dimension GloVe vectors $^{2}$  generated from the Common Crawl corpus (Pennington et al., 2014). We classify adjectives into  $k_{A} = 300$  clusters and nouns into  $k_{N} = 1000$  clusters. These numbers  $k$  were found by choosing the largest  $k$  multiple of 100 that did not result in any singleton clusters. We then estimated probabilities  $p(a,n)$  by maximum likelihood estimation after replacing wordforms  $a$  and  $n$  with their cluster indices.
This clustering method alleviates data sparsity by reducing the size of the support of the distributions over adjectives and nouns, to  $k_{A}$  and  $k_{N}$  respectively, and by effectively spreading probability mass among words with similar semantics. The clusters might also end up recapitulating the semantic categories that have played a role in more traditional syntactic theories of adjective order (Dixon, 1982; Cinque, 1994; Scott, 2002). We call these estimates cluster estimates.
Fitting predictors to data Most of our individual predictors come along with theories that say what their effect on adjective order should be. Adjectives with low PMI should be farther from the noun, adjectives with high IC should be farther from the noun, and adjectives with high subjectivity should be farther from the noun. Therefore,
strictly speaking, it is not necessary to fit these predictors to any training data: we can evaluate our theories based on their a priori predictions simply by asking how accurately we can predict the order of adjectives in AAN triples based on the rules above.
However, we can get a deeper picture of the performance of our predictors by using them in classifiers for adjective order. By fitting classifiers using our predictors, we can easily extend our models to ones with multiple predictors, in order to determine if a combined set of the predictors gives increased accuracy over any one.
Logistic regression method We fit logistic regressions to predict adjective order in AAN triples using our predictors. Our goal is to predict the order of the triple from the unordered set of the two adjectives  $\{A_1, A_2\}$  and the noun  $N$ . To do so, we consider the adjectives in lexicographic order: Given an AAN triple, let  $A^1$  denote the lexicographically-first adjective, and  $A^2$  the second. Then any given AAN triple is either of the form  $\langle A^1, A^2, N \rangle$  or  $\langle A^2, A^1, N \rangle$ . We fit a logistic regression to predict this order given the difference in the values of the predictors for the two adjectives. That is, we fit a logistic regression of the form in Figure 1. This method of fitting a classifier to predict order data was used previously in Morgan and Levy (2016). Based on theoretical considerations and previous empirical results, we expect that the fitted values of  $\beta_1$  will be negative for PMI and positive for IC and subjectivity. The regression in Figure 1 can easily be extended to include multiple predictors, with a separate  $\beta$  for each.
Evaluation metrics We evaluate our models using raw accuracy in predicting the order of held-out AAN triples. We also calculate  $95\%$  confidence intervals on these accuracies, indicating our uncertainty about how the accuracy would change in repeated experiments. Following standard experimental practice, if we find that two predictors achieve different accuracies, but their confidence intervals overlap, then we conclude that we do not have evidence that their accuracies are reliably different. We say a difference in accuracy between predictors is significant if the  $95\%$  confidence intervals do not overlap.
Evaluation on held-out hand-parsed data It is crucial that we not evaluate solely on automatically-parsed data. The reason is that both
$^{2}$ http://nlp.stanford.edu/data/glove.42B.300d.zip
$$
\log \frac {p \left(\left\langle A ^ {1} , A ^ {2} , N \right\rangle\right)}{p \left(\left\langle A ^ {2} , A ^ {1} , N \right\rangle\right)} = \beta_ {0} + \beta_ {1} \left(S \left(A ^ {1}, N\right) - S \left(A ^ {2}, N\right)\right) + \epsilon
$$
Figure 1: Logistic regression for adjective order. The function  $S(A, N)$  is the predictor to be evaluated,  $\beta_0$  and  $\beta_1$  are the free parameters to be fit, and  $\epsilon$  is an error term to be minimized.
PMI and IC, as measures of the strength of statistical association between nouns and adjectives, could conceivably double as predictors of parsing accuracy for automatic dependency parsers. If that is the case, then we might observe that AAN triples with low PMI or high IC are rare in automatically parsed data. However, this would not be a consequence of any interesting theory of cognitive cost, but rather simply an artifact of the automatic parser used. To avoid this confound, we include an evaluation based on held-out hand-parsed data in the form of the English Web Treebank.