Towards a Statistical Model of Grammaticality

Towards a Statistical Model of Grammaticality Alexander Clark, Gianluca Giorgolo, and Shalom Lappin firstname.lastname@kcl.ac. uk Department of Philosophy, King’s College London Abstract The question of whether it is possible to characterise grammat- ical knowledge in probabilistic terms is central to determin- ing the relationship of linguistic representation to other cog- nitive domains. We present a statistical model of grammati- cality which maps the probabilities of a statistical model for sentences in parts of the British National Corpus (BNC) into grammaticality scores, using various functions of the parame- ters of the model. We test this approach with a classifier on test sets containing different levels of syntactic infelicity. With ap- propriate tuning, the classifiers achieve encouraging levels of accuracy. These experiments suggest that it may be possible to characterise grammaticality judgements in probabilistic terms using an enriched language model. Keywords: enriched language models, probability distribu- tion, grammaticality judgements, probabilistic syntax Introduction The past two decades have seen a lively debate over whether linguistic knowledge is probabilistic or categorically rule- based in nature (see the papers in Bod, Hay, and Jannedy (2003) for some of this discussion). Given the success of probabilistic accounts of learning, representation, and infer- ence across a wide range of cognitive domains, this debate has considerable importance for the way in which knowledge of language is integrated into our general view of human cog- nition. On the classical view of syntax developed within linguistic theory over the past sixty years, the grammaticality and the probability of a sentence are entirely distinct properties with no direct relationship. Chomsky (1957) presents the origi- nal argument for the irrelevance of probability in determining grammaticality. 1 This argument depends on the inability of a simple word n-gram model to predict a distinction in prob- ability between a syntactically well-formed but unlikely (se- mantically anomalous) sentence like Colorless green ideas sleep furiously, and a word salad like Furiously sleep ideas green colorless. Pereira (2000) shows that a smoothed class- based n-gram model trained on a newspaper corpus predicts a significant distinction in probability between the two sen- tences. While it is certainly the case that grammaticality cannot be directly reduced to probability, the question of whether there is a significant correlation between the two remains open and interesting. Our general approach is as follows. We train a smoothed class-based trigram model on a filtered subclass of the BNC. We test this model on two corpora. One is di- vided into original sentences of part of the BNC and their re- versed counterparts. The second consists of a subset of orig- 1 See Fong, Malioutov, Yankama, and Berwick (2013) for a re- cent discussion of some of the issues involved in identifying gram- maticality with probability of occurrence. inal BNC sentences and their permuted variants in which a word in each sentence is randomly exchanged with another word three positions away from it. These distortions consti- tute syntactic infelicities. The first case involves gross struc- tural ill-formedness similar to the word salad example, while the second introduces subtler, more local mistakes. We score the test corpora using three alternative conditions. Our binary classifiers predict that a string is well-formed (original) or dis- torted (either reversed or permuted) on the basis of a score de- rived through normalising its log probability (logprob) value in various different ways. We also test different standard de- viations from the distributional norm in setting cut off points for our binary classifiers In our best cases we obtain an accu- racy rate of 98.9% for the original-reversal test set, and 79.1% for the original-permutted test set. These results suggest that by looking at the internal com- ponents of a probability distribution and the stages through which it is computed we can identify additional information that may be used to specify significant correlations between probability and grammaticality. This opens up an interesting set of research questions on the relationship between speak- ers’ knowledge of the probability distribution for a language and their grammaticality judgements. Probability and Grammaticality As has often been noted, it is not possible to reduce gram- maticality directly to probability. First, short ungrammati- cal sentences generally receive higher probability values than long, complex grammatical sentences containing words with low frequencies. Second, if one specifies a probability value (or even a range of such values) as the minimal threshold for grammaticality, then one is committed to the existence of a fi- nite number of grammatical sentences. The sum of the prob- abilities of the possible strings of words in a language sum to 1, and so at most 1/e sentences can have a probability of at least e. On the other hand, probabilistic inference does appear to be pervasive throughout all domains of cognition (Chater, Tenenbaum, and Yuille (2006)). Moreover, language mod- els do seem to play a crucial role in speech recognition and sentence processing. Without them we would not be able to identify speech sounds, and meaningful syntactic and seman- tic structures in noisy environments. Finally, grammatical- ity appears to track speakers’ acceptability judgements, and these are, in many cases, graded. Probability provides a nat- ural basis for generating such a gradient (Crocker and Keller Our starting point is a language model: a statistical model that defines a probability distribution over sentences.