Comparative Experiments on Disambiguating Word Senses: An Illustration of the Role of Bias in Machine Learning

This paper describes an experimental comparison of seven different learning algorithms on the problem of learning to disambiguate the meaning of a word from context. The algorithms tested include statistical, neural-network, decision-tree, rule-based, and case-based classification techniques. The specific problem tested involves disambiguating six senses of the word "line" using the words in the current and proceeding sentence as context. The statistical and neural-network methods perform the best on this particular problem and we discuss a potential reason for this observed difference. We also discuss the role of bias in machine learning and its importance in explaining performance differences observed on specific problems. I n t r o d u c t i o n Recent research in empirical (corpus-based) natural language processing has explored a number of different methods for learning from data. Three general approaches are statistical, neural-network, and symbolic machine learning and numerous specific methods have been developed under each of these paradigms (Wermter, Riloff, & Scheler, 1996; Charniak, 1993; Reilly & Sharkey, 1992). An important question is whether some methods perform significantly better than others on particular types of problems. Unfortunately, there have been very few direct comparisons of alternative methods on identical test data. A somewhat indirect comparison of applying stochastic context-free grammars (Periera & Shabes, 1992), a transformation-based method (Brill, 1993), and inductive logic programming (Zelle & Mooney, 1994) to parsing the ATIS (Airline Travel Information Service) corpus from the Penn Treebank (Marcus, Santorini, & Marcinkiewicz, 1993) indicates fairly similar performance for these three very different methods. Also, comparisons of Bayesian, informationretrieval, neural-network, and case-based methods on word-sense disambiguation have also demonstrated similar performance (Leacock, Towell, & Voorhees, 1993b; Lehman, 1994). However, in a comparison of neural-network and decision-tree methods on learning to generate the past tense of an English verb, decision trees performed significantly better (Ling & Marinov, 1993; Ling, 1994). Subsequent experiments on this problem have demonstrated that an inductive logic programming method produces even better results than decision trees (Mooney & Califf, 1995). In this paper, we present direct comparisons of a fairly wide range of general learning algorithms on the problem of discriminating six senses of the word "line" from context, using data assembled by Leacock et al. (1993b). We compare a naive Bayesian classifier (Duda & Hart, 1973), a perceptron (Rosenblatt, 1962), a decision-tree learner (Quinlan, 1993), a k nearest-neighbor classifier (Cover & Hart, 1967), logic-based DNF (disjunctive normal form) and CNF (conjunctive normal form) learners (Mooney, 1995) and a decisionlist learner (Rivest, 1987). Tests on all methods used identical training and test sets, and ten separate random trials were run in order to measure average performance and allow statistical testing of the significance of any observed differences. On this particular task, we found that the Bayesian and perceptron methods perform significantly better than the remaining methods and discuss a potential reason for this observed difference. We also discuss the role of bias in machine learning and its importance in explaining the observed differences in the performance of alternative methods on specific problems. B a c k g r o u n d o n M a c h i n e L e a r n i n g