Measuring Gradience in Speakers’ Grammaticality Judgements - eScholarship (2014)

Measuring Gradience in Speakers’ Grammaticality Judgements Jey Han Lau, Alexander Clark, and Shalom Lappin jeyhan.lau@gmail.com, alexander.clark@kcl.ac.uk, shalom.lappin@kcl.ac.uk Department of Philosophy, King’s College London Abstract dark). To sustain a categorical view of grammaticality it is necessary to show that when gradience does arise in speak- ers’ judgements, it is entirely the result of extra-grammatical influences, such as processing factors, semantic acceptabil- ity, or real world knowledge. To the best of our knowledge, the advocates of the categorical view have not managed to demonstrate this in a convincing way over large amounts of grammatically varied experimental data. Grammaticality is a theoretical concept, while acceptabil- ity can be experimentally tested. To maintain the categori- cal view of grammaticality in the face of pervasive gradience in acceptability judgements across a wide range of syntactic structures, one must provide an independent, empirically vi- able criterion for identifying grammaticality, which does not assume the view that is at issue. 1 In the absence of such a cri- terion, solid experimental evidence for gradience in accept- ability for a large number of speakers, across a wide range of data provides strong prima facie support for the hypothesis that grammaticality is a gradient property. Second, both sides of the debate have generally relied solely on linguistic judgements in order to motivate their con- clusions. In fact the discussion would be advanced if indepen- dent non-linguistic paradigms of binary classifiers and gradi- ent properties were identified and used as benchmarks with which to compare the judgement patterns that speakers ex- hibit with respect to a wide range of grammaticality data. We used Google statistical machine translation to map sen- tences randomly selected from the BNC into a number of languages and then back into English. The errors that were generated across these target languages ranged from mild in- felicities of lexical choice and awkward ordering of modi- fiers, through missing arguments, deleted prepositions, and misplaced subordinate clauses, to word salads. We then tested these sentences on speakers through crowd sourcing with Amazon Mechanical Turk (AMT). Each HIT (Human Intelligence Task) contained an original English sen- tence from the BNC and four translated sentences randomly selected from each of the four target languages. We used three presentation modes for these experiments: a binary classifica- tion task, a four point acceptability scale, and a sliding scale with 100 underlying discrete points. We observed a very high Pearson correlation among the mean judgements across all The question of whether grammaticality is a binary categorical or a gradient property has been the subject of ongoing debate in linguistics and psychology for many years. Linguists have tended to use constructed examples to test speakers’ judge- ments on specific sorts of constraint violation. We applied ma- chine translation to randomly selected subsets of the British National Corpus (BNC) to generate a large test set which con- tains well-formed English source sentences, and sentences that exhibit a wide variety of grammatical infelicities. We tested a large number of speakers through (filtered) crowd sourc- ing, with three distinct modes of classification, one binary and two ordered scales. We found a high degree of correlation in mean judgements for sentences across the three classification tasks. We also did two visual image classification tasks to ob- tain benchmarks for binary and gradient judgement patterns, respectively. Finally, we did a second crowd source experi- ment on 100 randomly selected linguistic textbook example sentences. The sentence judgement distributions for individ- ual speakers strongly resemble the gradience benchmark pat- tern. This evidence suggests that speakers represent grammat- ical well-formedness as a gradient property. Keywords: grammaticality, acceptability, gradient classifiers, binary categories, speakers’ judgements Introduction The question of whether grammaticality is a binary categor- ical or a gradient property has been the subject of a long- standing debate in linguistics and psychology (Keller, 2000; Manning, 2003; Crocker & Keller, 2005; Sorace & Keller, 2005; Fanselow, F´ery, Schlesewsky, & Vogel, 2006; Sprouse, 2007; Ambridge, Pine, & Rowland, 2012). While it has been recognized since the very beginning of modern linguistics (Chomsky, 1965) that there are degrees of grammaticality, in practice grammaticality is standardly taken in theoretical lin- guistics to be dichotomous: a binary division between gram- matical and ungrammatical sentences. Most grammar for- malisms are specified so that the grammars that they allow generate sets of sentences, with a binary set membership cri- terion corresponding to a categorical notion of grammatical- ity. Advocates of a categorical view of grammaticality have tended to limit themselves to experimental results involving a small number of constructed examples. These examples ap- pear to show the inviolability of specific kinds of syntactic constraints (such as wh-island conditions). While this work is interesting and important, it suffers from at least two prob- lems. First, it is difficult to see how the existence of a number of cases in which speakers’ judgements are robustly binary in itself entails the categorical nature of grammaticality, even when these cases exhibit clearly identifiable syntactic errors that are well described by a particular theory of syntax. Gra- dient judgments will inevitably appear to be sharp for clear paradigm cases (very tall vs. very short, very light vs very 1 For example, Sprouse (2007) presents some evidence that cer- tain types of syntactic island violations fail to show syntactic prim- ing effects under experimental conditions. He takes this to motivate a categorical view of grammaticality on the grounds that priming is only possible for grammatical sentences. He does not demon- strate that all types of ungrammaticality fail to exhibit priming ef- fects. Also, his experimental results suggest that speakers assign different levels of acceptability to distinct types of island violations.