SlowGuess/ACL_2020_OCR huggingface dataset:
Table 1a shows the accuracies of our predictors in predicting held-out adjective orders in the Common Crawl test set, visualized in Figure 2a. We find that the pattern of results depends on whether predictors are estimated based on wordforms or based on distributional clusters. When estimating based on wordforms, we find that subjectivity and PMI have the best accuracy. When estimating based on clusters, the accuracy of PMI drops, and the best predictor is subjectivity, with IG close behind. We find a negative logistic regression weight for information gain, indicating that the adjective with lower information gain is placed first.
This basic pattern of results is confirmed in the hand-parsed EWT data. Accuracies of predictors on the EWT test set are shown in Table 1b and visualized in Figure 2b. When estimating based on wordforms, the best predictors are subjectivity and PMI, although the confidence intervals of all predictors are overlapping. When estimating based on clusters, IG has the best performance, and PMI again drops in accuracy. For this case, IG, IC, and subjectivity all have overlapping confidence intervals, so we conclude that there is no evidence that one is better than the other. However, we do have evidence that IG and IC are more accurate than PMI when estimated based on clusters.
5.1 Multivariate analysis
Adjective order may be determined by multiple separate factors operating in parallel. In order to
investigate whether our predictors might be making independent contributions to explaining adjective order, we fit logistic regressions containing multiple predictors. If the best accuracy comes from a model with two or more predictors, then this would be evidence that these two predictors are picking up on separate sources of information relevant for predicting adjective order.
We conducted logistic regressions using all sets of two of our predictors. The top 5 such models, in terms of Common Crawl test set accuracy, are shown in Table 2. The best two are cluster/wordform subjectivity and wordform PMI, followed by cluster subjectivity and cluster information gain. No set of three predictors achieves significantly higher accuracy than the best predictors shown in Table 2.
5.2 Qualitative analysis
We manually examined cases where each model made correct and incorrect predictions in the hand-parsed EWT data. Table 3a shows example AAN triples that were ordered correctly by PMI, but not by subjectivity. These are typically cases where a certain adjective-noun pair forms a common collocation whose meaning is in some cases even noncompositional; for example, "bad behaviors" is a common collocation when describing training animals, and "ulterior motives" and "logical fallacy" are likewise common English collocations. In contrast, when subjectivity makes the right prediction and PMI makes the wrong prediction, these are often cases where a word pair which normally would form a collocation is broken up by another adjective, such as "dear sick friend", where "dear friend" is a common collocation.
We also performed a manual qualitative analysis to determine the contribution of information gain beyond subjectivity and PMI. Table 3b shows examples of such cases from the EWT. Many of these seem to be cases with weak preferences, where both the attested order and the flipped order are acceptable (e.g., "tiny little kitten").
Predictor	Accuracy	Conf. Interval
Subj. (cluster)	.661	[.657, .666]
PMI (wordform)	.659	[.654, .664]
Subj. (wordform)	.659	[.654, .664]
IG (cluster)	.650	[.645, .654]
IC (wordform)	.642	[.634, .646]
IG (wordform)	.640	[.635, .645]
IC (cluster)	.613	[.608, .618]
PMI (cluster)	.606	[.601, .610]
(a) Common Crawl  $(N = 41822)$ .
Predictor	Accuracy	Conf. Interval
IG (cluster)	.737	[.668, .806]
Subj. (wordform)	.724	[.654, .795]
IC (cluster)	.705	[.633, .777]
Subj. (cluster)	.692	[.620, .765]
PMI (wordform)	.667	[.592, .741]
IC (wordform)	.641	[.566, .717]
IG (wordform)	.603	[.526, .680]
PMI (cluster)	.590	[.512, .667]
(b) Hand-parsed EWT  $(N = 155)$ . All confidence intervals overlap, other than cluster-based PMI and IG.
(a) Common Crawl  $(N = 41822)$ .
(b) Hand-parsed EWT  $(N = 155)$
Figure 2: Accuracies of predictors on AAN triples in the held-out test data, with  $95\%$  confidence intervals shown.
Table 1: Accuracies of the predictors on AAN triples in the held-out test data.
Predictor	Accuracy	Conf. Interval
Subj. (cluster) + PMI (wordform)	.723	[.719, .727]
Subj. (wordform) + PMI (wordform)	.713	[.708, .717]
Subj. (cluster) + IG (cluster)	.699	[.695, .703]
Subj. (cluster) + IC (cluster)	.690	[.686, .695]
IG (cluster) + IC (cluster)	.684	[.680, .689]
Table 2: Common crawl test set accuracy of the top 5 models combining two predictors.
A1	A2	N
major	bad	behaviors
large	outstanding	debtss
classical	logical	fallacy
dark	ulterior	motives
minor	fine	tuning
(a) Ordered correctly by wordform PMI, but not by word- (b) Ordered correctly by cluster-based information gain, form subjectivity. but not by cluster-based subjectivity nor PMI.
A1	A2	N
tiny	little	kitten
correct	legal	name
chronic	intractable	pain
radical	religious	politics
lonely	eerie	place
Table 3: Selected examples of AAN triples ordered incorrectly by our models, from the EWT test set.
5.3 Interpretation
Our results broadly support the following interpretation. Adjective ordering preferences are largely determined by a semantic factor that can be quantified variously using wordform subjectivity or distributional-cluster-based estimates of information gain. In addition to this factor, another factor is in play: when an adjective-noun pair forms a collocation with a possibly non-compositional meaning, then the adjective in this pair will tend to be placed next to the noun. This latter factor is measured by PMI. This interpretation matches that of Hahn et al. (2018), who found separate contributions from PMI and a model-based operationalization of subjectivity.
Our interpretation is supported by the following points from the analysis above. First, among predictors based solely on wordforms, the best accuracy is obtained by a combination of subjectivity and PMI. Second, when we turn to estimates based on clusters, two things happen: the accuracy of PMI drops, and the accuracy of information gain increases while the accuracy of subjectivity stays about the same. This pattern of results suggests that PMI is measuring a factor that has more to do with specific wordforms, while IG and subjectivity are measuring a factor that has more to do with semantic uncertainty about the noun or about the relationship between the adjective and the noun.
We examined a number of theoretically-motivated predictors of adjective order in dependency treebank corpora of English. We found that the predictors have comparable accuracy, but that it is possible to identify two broad factors: a semantic factor variously captured by subjectivity scores and information gain based on word clusters, and a wordform-based factor captured by PMI.
This study provides a framework for evaluating further theories of adjective order, and for evaluating the theories given here against new data from dependency treebanks. Generalizing to larger datasets of English is straightforward. More excitingly, we now have the opportunity to bring new languages into the fold. The vast majority of research on adjective ordering, and all the corpus work to our knowledge, has been done on English, where adjectives almost always come before the noun. Studying other typologically-distinct languages provides an opportunity to disentangle the theories that we studied here in a way that cannot be done in English.
The available behavioral evidence suggests that mirror-image preferences (e.g., "box blue big") may be the norm in post-nominal adjective languages (Martin, 1969; Scontras et al., 2020). Information locality, subjectivity, and integration cost make precisely that prediction, though none addresses mixed-type languages in which adjectives can precede or follow nouns. It is an open question how to implement IG for these post- or mixed-placement adjectives; one possibility is to measure the information gained when the set of adjectives associated to a noun  $A_{n}$  is partitioned by an adjective  $a$ . In that case, the predictions about post-nominal order could differ substantially from the predictions about pre-nominal order.
Our dependency-treebank-based methods can be applied to any other corpus of any language, provided it has enough data in the form of adjective-noun pairs to get reliable estimates of the information-theoretic predictors. Such studies will be crucial to achieve a complete computational understanding of natural language syntax.
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2442-2452, Berlin, Germany. Association for Computational Linguistics.
R. Harald Baayen, Richard Piepenbrock, and Leon Gulikers. 1995. The CELEX Lexical Database. Release 2 (CD-ROM). Linguistic Data Consortium, University of Pennsylvania.
Galia Bar-Sever, Rachael Lee, Gregory Scontras, and Lisa S. Pearl. 2018. Little lexical learners: Quantitatively assessing the development of adjective ordering preferences. In 42nd annual Boston University Conference on Language Development, pages 58-71.
Christian Bentz, Dimitrios Alikaniotis, Michael Cysouw, and Ramon Ferrer-i-Cancho. 2017. The entropy of words—Learnability and expressivity across more than 1000 languages. Entropy, 19:275-307.
Joan Bresnan, Anna Cueni, Tatiana Nikitina, and Harald Baayen. 2007. Predicting the dative alternation. In Cognitive Foundations of Interpration, pages 69-94. Royal Netherlands Academy of Science, Amsterdam.
Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. N-gram counts and language models from the common crawl. In LREC, volume 2, page 4. Cite-seer.
Xinying Chen and Ramon Ferrer-i-Cancho, editors. 2019. Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019). Association for Computational Linguistics, Paris, France.
Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29.
Guglielmo Cinque. 1994. On the evidence for partial N-movement in the Romance DP. In R S Kayne, G Cinque, J Koster, J.-Y. Pollock, Luigi Rizzi, and R Zanuttini, editors, Paths Towards Universal Grammar. Studies in Honor of Richard S. Kayne, pages 85-110. Georgetown University Press, Washington DC.
Thomas M. Cover and J. A. Thomas. 2006. Elements of Information Theory. John Wiley & Sons, Hoboken, NJ.
J. H. Danks and S. Glucksberg. 1971. Psychological scaling of adjective orders. Journal of Verbal Learning and Verbal Behavior, 10(1):63-67.
Robert M. W. Dixon. 1982. Where have all the adjectives gone? And other essays in semantics and syntax. Mouton, Berlin, Germany.
Melody Dye, Petar Milin, Richard Futrell, and Michael Ramscar. 2018. Alternative solutions to a language design problem: The role of adjectives and gender marking in efficient communication. *Topics in cognitive science*, 10(1):209–224.
William Dyer. 2018. Integration complexity and the order of cosisters. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 55-65, Brussels, Belgium. Association for Computational Linguistics.
William E. Dyer. 2017. Minimizing integration cost: A general theory of constituent order. Ph.D. thesis, University of California, Davis, Davis, CA.
Robert M. Fano. 1961. Transmission of Information: A Statistical Theory of Communication. MIT Press, Cambridge, MA.
Michael Franke, Gregory Scontras, and Mihael Simonic. 2019. Subjectivity-based adjective ordering maximizes communicative success. In Proceedings of the 41st Annual Meeting of the Cognitive Science Society, pages 344-350.
Richard Futrell. 2019. Information-theoretic locality properties of natural language. In Proceedings of the First International Conference on Quantitative Syntax, pages 2-15, Paris.
Richard Futrell and Roger Levy. 2017. Noisycontext surprisal as a human sentence processing cost model. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 688-698, Valencia, Spain.
Richard Futrell, Roger Levy, and Edward Gibson. 2017. Generalizing dependency distance: Comment on "dependency distance: A new perspective on syntactic patterns in natural languages" by Haitao liu et al. Physics of Life Reviews, 21:197-199.
Richard Futrell, Peng Qian, Edward Gibson, Evelina Fedorenko, and Idan Blank. 2019. Syntactic dependencies correspond to word pairs with high mutual information. In Proceedings of the Fifth International Conference on Dependency Linguistics (DepLing 2019), Paris.
Michael Hahn, Judith Degen, Noah Goodman, Daniel Jurafsky, and Richard Futrell. 2018. An information-theoretic explanation of adjective ordering preferences. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society (CogSci).
John Hale. 2006. Uncertainty about the rest of the sentence. Cognitive science, 30(4):643-672.
John T. Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics and Language Technologies, pages 1-8.
John T. Hale. 2016. Information-theoretical complexity metrics. Language and Linguistics Compass, 10(9):397-412.
R. Hetzron. 1978. On the relative order of adjectives. In H. Seller, editor, Language Universals. Narr, Tübingen, Germany.
Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126-1177.
Haitao Liu, Chunshan Xu, and Junying Liang. 2017. Dependency distance: A new perspective on syntactic patterns in natural languages. Physics of Life Reviews, 21:171-193.
Robert Malouf. 2000. The order of prenominal adjectives in natural language generation. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, ACL '00, pages 85-92, Stroudsburg, PA, USA. Association for Computational Linguistics.
Christopher D. Manning. 2003. Probabilistic syntax. In Probabilistic Linguistics, pages 289-341. MIT Press.
J E Martin. 1969. Some competence-process relationships in noun phrases with prenominal and postnominal adjectives. Journal of Verbal Learning and Verbal Behavior, 8:471-480.
Emily Morgan and Roger Levy. 2016. Abstract knowledge versus direct experience in processing of binomial expressions. Cognition, 157:382-402.
Joakim Nivre. 2015. Towards a universal grammar for natural language processing. In Computational Linguistics and Intelligent Text Processing, pages 3-16. Springer.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
J. R. Quinlan. 1986. Induction of decision trees. Machine Learning, 1(1):81-106.
Gregory Scontras, Galia Bar-Sever, Zeinab Kachakeche, Cesar Manuel Rosales Jr., and Suttera Samonte. 2020. Incremental semantic restriction and subjectivity-based adjective ordering. In Proceedings of Sinn und Bedeutung 24.
Gregory Scontras, Judith Degen, and Noah D. Goodman. 2017. Subjectivity predicts adjective ordering preferences. Open Mind: Discoveries in Cognitive Science, 1(1):53-65.
Gregory Scontras, Judith Degen, and Noah D. Goodman. 2019. On the grammatical source of adjective ordering preferences. Semantics and Pragmatics.
G.-J. Scott. 2002. Stacked adjectival modification and the structure of nominal phrases. In G Cinque, editor, The cartography of syntactic structures, Volume 1: Functional structure in the DP and IP, pages 91-120. Oxford University Press, Oxford.
Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffé, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014).
Mihael Simonic. 2018. Functional explanation of adjective ordering preferences using probabilistic programming. Master's thesis, University of Tübingen.
Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302-319.
R. Sproat and C. Shih. 1991. The cross-linguistic distribution of adjective ordering restrictions. In C. Georgopoulos and R. Ishihara, editors, Interdisciplinary approaches to language: Essays in honor of S.-Y. Kuroda, pages 565-593. Kluwer Academic, Dordrecht, Netherlands.
David Temperley and Daniel Gildea. 2018. Minimizing syntactic dependency lengths: Typological/cognitive universal? Annual Review of Linguistics, 4:1-15.
Stefanie Wulff. 2003. A multifactorial corpus analysis of adjective order in english. International Journal of Corpus Linguistics, 8(2):245-282.
P. Ziff. 1960. Semantic analysis. Cornell University Press, Ithaca, NY.