Bridging the Gap Between Lexicon and Corpus: Convergence of Formalisms

I first consider the spectrum of lexical information from the semantic to the textual. A range of lexicons are classified according towhere they sit on this scale. Lexicographic tools and WSD programs are included in the classification, and this is justified. There is currently a lacuna between the most text-oriented of the lexicographic approaches, and the most sophisticated of the data-driven ones. Lexical tuning requires that the lacuna be filled, so corpus data can flow into the lexicon. Following an analysis of similarities and differences between lexicographic tools and data-driven approaches, a strategy for bridging the gap is proposed. 1 A Spectrum of Lexicons Lexicons look in two directions: towards the text, and towards semantics. When we look up a word or phrase encountered in a text to find the meaning, the textual orientation provides the input, the semantic one, the output. In a language generation system, the roles are reversed. In principle lexicons could be very highly developed in relation to both the textual and the semantic orientation. In practice, for computational lexicons in particular, the emphasis tends to be on one or the other. (The MicroKosmos project is interesting in this regard. It has two teams, one working on the text-oriented lexicon, the other on the ontology (Viegas and Nirenburg, 1995). ) The work was undertaken under EPSRC grant GR/K/18931 Thus lexicons can be placed on a spectrum according to where the emphasis lies: how ‘surfacey’ they are, as in Fig. 1. At the semantic end lie AI ontologies. Traditional nativespeaker dictionaries such as the OED are somewhat closer to the text, and learner dictionaries, with their emphasis on grammatical and textual patterning, more so. (Since the advent of corpus lexicography in the early 1980s, the entire UK dictionary-publishing community has been steadily creeping in a text-ward direction, with the learners’ dictionaries leading the way.) NLP lexicons which have been developed with parsing inmind, such asCOMLEX or the ANLT lexicon, are further along a textual direction. Further still down this road are ‘lexicons’ of patterns as used in Information Extraction. Then there is a gap; and then we move to data-driven artifacts like the decision lists Yarowsky uses for word sense disambiguation, Schütze’s sense clusters, and Grefenstette’s thesauri (Yarowsky, 1995; Schütze, 1998; Grefenstette, 1994). At the end point of the scale are sets of corpus citations for the word. Underlying the analysis is the thesis that the lexical entry for a word is an abstraction from the occurrences of the word in the language. A higher level of abstraction takes us further away from the linguistic data, towards the semantic end of the spectrum.1 2 Correlated distinctions There are several related but distinct dimensions along which lexicons can be analysed: 1Investigators who view lexicons as primarily approximations to mental lexicons might find this perspective odd: here lurks an issue regarding the primacy of linguistic as against psychological data in the study of the lexicon. For discussion see (Kilgarriff, 1992, section 1.4.1). AI Knowledge rep KL-ONE and similar COMLEX, ANLT (lexicons for parsing) CYC, MicroKosmos ontology MT & MicroKosmos lexicons (learners’, bilingual) PENMAN Upper Model Wilks’s Preference semantics