An Analysis of Bayesian Classiiers Probabilistic Approaches to Induction

In this paper we present an average case analysis of the Bayesian classi er a simple induction algo rithm that fares remarkably well on many learning tasks Our analysis assumes a monotone conjunc tive target concept and independent noise free Boolean attributes We calculate the probability that the algorithm will induce an arbitrary pair of concept descriptions and then use this to compute the probability of correct classi cation over the in stance space The analysis takes into account the number of training instances the number of at tributes the distribution of these attributes and the level of class noise We also explore the be havioral implications of the analysis by presenting predicted learning curves for arti cial domains and give experimental results on these domains as a check on our reasoning Probabilistic Approaches to Induction One goal of research in machine learning is to discover principles that relate algorithms and domain character istics to behavior To this end many researchers have carried out systematic experimentation with natural and arti cial domains in search of empirical regularities e g Kibler Langley Others have focused on theoretical analyses often within the paradigm of probably approximately correct learning e g Haus sler However most experimental studies are based only on informal analyses of the learning task whereas most formal analyses address the worst case and thus bear little relation to empirical results A third approach proposed by Cohen and Howe involves the formulation of average case mod els for speci c algorithms and testing them through experimentation Pazzani and Sarrett s study of conjunctive learning provides an excellent example of this technique as does Hirschberg and Pazzani s work on inducing k CNF concepts By assum ing information about the target concept the num Also a liated with RECOM Technologies Also a liated with Sterling Software ber of attributes and the class and attribute frequen cies they obtain predictions about the behavior of induction algorithms and used experiments to check their analyses However their research does not fo cus on algorithms typically used by the experimental and practical sides of machine learning and it is im portant that average case analyses be extended to such methods Recently there has been growing interest in proba bilistic approaches to inductive learning For example Fisher has described Cobweb an incremental algorithm for conceptual clustering that draws heavily on Bayesian ideas and the literature reports a number of systems that build on this work e g Allen Lang ley Iba Gennari Thompson Langley Cheeseman et al have outlined Auto Class a nonincremental system that uses Bayesian methods to cluster instances into groups and other researchers have focused on the induction of Bayesian inference networks e g Cooper Kerskovits These recent Bayesian learning algorithms are com plex and not easily amenable to analysis but they share a common ancestor that is simpler and more tractable This supervised algorithm which we re fer to simply as a Bayesian classi er comes originally from work in pattern recognition Duda Hart The method stores a probabilistic summary for each class this summary contains the conditional probabil ity of each attribute value given the class as well as the probability or base rate of the class This data structure approximates the representational power of a perceptron it describes a single decision boundary through the instance space When the algorithm en counters a new instance it updates the probabilities stored with the speci ed class Neither the order of training instances nor the occurrence of classi cation errors have any e ect on this process When given a test instance the classi er uses an evaluation function which we describe in detail later to rank the alter A related approach involves deriving the optimal learn ing algorithm under certain assumptions and then imple menting an approximation of that algorithm e g Opper Haussler Analysis of Bayesian Classifiers Domain Bayes IND C Freq