Unsupervised Lexical Learning with Categorial Grammars

In this paper we report on an unsupervised approach to learning Categorial Grammar (CG) lexicons. The learner is provided with a set of possible lexical CG categories, the forward and backward application rules of CG and unmarked positive only corpora. Using the categories and rules, the sentences from the corpus are probabilistically parsed. The parses and the history of previously parsed sentences are used to build a lexicon and annotate the corpus. We report the results from experiments on a number of small generated corpora, that contain examples from subsets of the English language. These show that the system is able to generate reasonable lexicons and provide accurately parsed corpora in the process. We also discuss ways in which the approach can be scaled up to deal with larger and more diverse corpora. 1 I n t r o d u c t i o n In this paper we discuss a potential solution to two problems in Natural Language Processing (NLP), using a combination of statistical and symbolic machine learning techniques. The first problem is learning the syntactic roles, or categories, of words o f a language i.e. learning a lexicon. Secondly, we discuss a method of annotating a corpus with parses. The aim is to learn Categorial Grammar (CG) lexicons, starting from a set of lexical categories, the functional application rules of CG and an unannotated corpus of positive examples. The CG formalism (discussed in Section 2) is chosen because it assigns distinct categories to words of different types, and the categories describe the exact syntactic role each word can play in a sentence. This problem is similar to the unsupervised part of speech tagging work of, for example, Brill (Brill, 1997) and Kupiec (Kupiec, 1992). In Brill's work a lexicon containing the parts of speech available to each word is provided and a simple tagger attaches a complex tag to each word in the corpus, which represents all the possible tags that word can have. Transformation rules are then learned which use the context of a word to determine which simple tag it should be assigned. The results are good, generally achieving around 95% accuracy on large corpora such as the Penn Treebank. Kupiec (Kupiec, 1992) uses an unsupervised version of the Baum-Welch algorithm, which is a way of using examples to iteratively estimate the probabilities of a Hidden Markov Model for part of speech tagging. Instead of supplying a lexicon, he places the words in equivalence classes. Words in the same equivalence class must take one of a specific set of parts of speech. This improves the accuracy of this algorithm to about the same level as Brill's approach. In both cases, the learner is provided with a large amount of background knowledge either a complete lexicon or set of equivalence classes. In the approach presented here, the most that is provided is a small partial lexicon. In fact the system learns the lexicon. The second problem annotating the corpus is solved because of the approach we use to learn the lexicon. The system uses parsing to determine which are the correct lexical entries for a word, thus annotating the corpus with the parse derivations (also providing less probable parses if desired). An example of another approach to doing this is the Fidditch parser of Hindle (Hindle, 1983) (based on the deterministic parser of Marcus (Marcus, 1980)), which was used to annotate the Penn Treebank (Marcus et al., 1993). However, instead of learning the lexicon, a complete grammar and lexicon