seventh BENELEARN conference , 109-118 , 1997 When small disjuncts abound , try lazy learning : A case study

Machine learning is becoming recognised as a source of generic and powerful tools for tasks studied and implemented in language technology. Lazy learning with information-theoretic similarity matching has appeared a salient approach, demonstrated to be superior over other machine-learning approaches in various comparative studies. It is asserted both in theoretical machine learning and in reports on applications of machine learning to natural language that the success of lazy learning may be due to the fact that language data contains small disjuncts, i.e., small clusters of identically-classified instances. We propose three measures to discover small disjuncts in our data: (i) we count and analyse indexed clusters of instances in induced decision trees; (ii) we count clusters of friendly (identically-classified) instances immediately surrounding instances by using similarity metrics from lazy learning; (iii) we compare average sizes of friendly-instance clusters using different similarity metrics. The measures are illustrated by a sample language task, viz. word pronunciation. Two conclusions are arrived at: (i) our data indeed contains large amounts of small disjuncts of about three to a hundred instances, and (ii) there are important differences in feature relevance in the data, exploited appropriately when lazy learning is augmented with information-theoretic similarity matching. We claim that the measures introduced in this paper are useful for predicting the suitedness of lazy learning in general. 1 Lazy learning of language tasks In language technology, machine learning is becoming recognised as a source of powerful generic tools for learning complex language tasks (Daelemans, Van den Bosch, and Weijters, 1997b). Lazy learning (Aha, Kibler, and Albert, 1991; Aha 1997) has been demonstrated to be an especially salient approach to learning various language tasks such as grapheme-phoneme conversion, stress assignment, morphological segmentation, and part-of-speech tagging; for an overview, we refer to Daelemans et al. (1997). In many comparative studies, lazy learning is shown to outperform traditional linguistics-based models and other machine-learning approaches such as decisiontree learning and connectionist learning, when applied to various language tasks (Daelemans et al., 1997; Van den Bosch, 1997). In these comparative studies, lazy learning augmented with an information-theoretic weighted similarity function, IB1-IG consistently offers the best generalisation performances (Daelemans and Van den Bosch, 1992; Daelemans, Van den Bosch, and Weijters, 1997a). Lazy (or instance based) learning is the common term for a class of learning algorithms descending from the k-nearest neighbour (k-NN) algorithm (Cover and Hart, 1967; Devijverand Kittler, 1982; Aha et al., 1991). Lazy learning is based on the hypothesis that performance in cognitive tasks (e.g., language tasks) is based on computing the similarity between new instances of the task and stored representations of instances encountered earlier, rather than on the application of mental rules abstracted from earlier experiences (Aha et al., 1991; Aha and Goldstone, 1992; Daelemans, 1995). Instance-based learning is commonly referred to as lazy due to the minimal effort put in the learning process. This paper offers concrete estimates relating to two assumptions on the suitability of applying lazy learning to language tasks: 1. When classes are spread disjunctively in an instance space over small clusters of instances (small disjuncts) in a data set, lazy learning is generally assumed to be the preferred learning approach yielding the best performance (Holte, Acker, and Porter, 1989). 2. Two inherent data characteristics of language data are assumed to cause (a) the high performance of lazy learning, viz. that instances occur in pockets of exceptions (Daelemans, 1995) (which may be equivalent to the small disjuncts mentioned in (1)), and (b) the consistent superiority of information-theory-weighted lazy learning in particular, viz. that instances of language tasks display substantial differences in feature relevance (Daelemans, 1995; Van den Bosch, 1997). We provide characterisations of the two assumptions in section 2. The remainder of the paper is devoted to proposing and testing concrete measures for testing the assumption. The tests of the proposed measures are performed on data representing a sample language task, viz. learning word pronunciation. Section 3 briefly introduces the sample task and provides the empirical results of a comparative study with learning algorithms applied to the task. In section 4 we investigate the following: 1. We count and analyse the clusters of instances represented at end nodes in the decision trees generated by IGTREE, applied to the word-pronunciation task. 2. We measure the clusteredness of instances with IB1 applied to the word-pronunciation task. 3. We measure the clusteredness of instances with IB1-IG, and compare it with those measured with IB1 applied to the word-pronunciation task, and two distorted variations on the task. In section 5, we formulate the conclusions to be drawn from the present study. 2 Small disjuncts and feature relevance All language tasks can be defined as classification tasks (Daelemans, 1995). Given a corpus of examples of the task, data bases of these examples can be built to which machine-learning algorithms can readily be applied. In such data bases, pockets of exceptions tend to occur; “Exceptions tend to come in ‘families’ ” (Daelemans, 1996b, p. 5). Keeping members of such families in memory, enables accurate classification of members of the same family occurring in new, unseen data. It can therefore be expected that lazy learning, which keeps all training material in memory, will be able to deal better with reoccurring family members of stored families of exceptions in test material, than decision-tree learning algorithms or supervised connectionist algorithms, which show the tendency to ignore small groups of exceptions considered to be noise. The observation of ‘families of exceptions’ relates to an observation in machine-learning research that “IB1 performs well for highly disjunctive target concepts” (Aha, 1992, p. 6). A target concept, i.e., a class, is highly disjunctive when the instances of that class are very dissimilar globally, and similar locally. In data sets containing highly disjunctive target concepts, similar instances of the same class are only found in small clusters (Daelemans’ (1996) families). The expectation is that IB1 will perform generally better than eager-learning algorithms on such data since it retains all information concerning disjuncts, no matter how small, while decision trees, notably those implementing pruning (Quinlan, 1993), tend to overgeneralise and miss out on disambiguating small disjuncts (Holte et al., 1989; Ali. 1996). Our working assumption here is, by abduction, that language data contains many small disjuncts. Apart from ‘families of exceptions’, many language tasks display outspoken differences in feature relevance; language data is for a considerable part redundant (Zipf, 1935), and contains fairly localised hot spots of information. Therefore, using information-theoretic functions for similarity matching in lazy learning, e.g., as in IB1-IG (Daelemans, Van den Bosch, and Weijters, 1997a), may be advantageous in learning language tasks. With information-theoretic weighted similarity matching, the multi-dimensional feature-value space is rescaled in such a way that instances matching on important features are regarded as more similar to each other than instances matching on an irrelevant feature. It is unclear, however, what the exact influence is on the functioningof lazy learning with information-theoretic weighted similarity matching, as compared to flat weighting in IB1. In sum, our working assumption is that with IB1-IG, clustering of families of instances in the instance base improves. 3 Word pronunciation: the GS task Converting written words to stressed phonemic transcription, i.e., word pronunciation, is a wellknown benchmark task in machine learning (Stanfill and Waltz, 1986; Sejnowski and Rosenberg, 1987; Dietterich, Hild, and Bakiri, 1990). Here, we provide an overview of experiments performed on the word-pronunciation task, i.e., the conversion of fixed-sized instances representing parts of words (windows), to a class representing the phoneme and the stress marker of the instance’s middle letter. The task, henceforth referred to as GS (Grapheme-phoneme conversion and Stress assignment) is similar to the NETTALK task presented by Sejnowski and Rosenberg (1986), but is performed on a larger corpus of 77,565 English word-pronunciation pairs (Van den Bosch, 1997). Converted into fixed-sized instance, the full instance base representing the GS task contains 675,745 instances. The task features 159 classes. When neither of these classes would be disjunctively 0.0 2.5 5.0 7.5 10.0 12.5 15.0 BP IB1 IGTREE IB1-IG ge ne ra lis at io n er ro r (% ) 12.14