Bayesian Filtering : Beyond Binary Classification

Bayesian Filtering is used in FogBugz 4.0 to remove spam from incoming email. But it’s also used very successfully to classify email into various categories, for example, to separate technical questions from salesrelated questions. This paper describes a tournament algorithm used to classifying incoming email into multiple bins. Bayesian spam filters are rapidly approaching a level of speed and accuracy that must have even Alan Ralsky shaking in his boots. It is only a matter of time until these tools are widely adopted and their ability to remove over 99% of incoming spam puts a chokehold on the kings of junk mail. The technique is so promising when sorting e-mail into spam and not-spam categories that it would be a waste to ignore the possibilities of expansion. This paper summarizes an attempt at using Bayesian filtering to automatically sort documents into a variable number of categories. This sorting capacity was specifically developed as a feature for Fog Creek’s FogBugz software. Much of the sorter’s success is attributed to Paul Graham’s seminal work on Bayesian filtering, Gary Robinson’s subsequent variation, and projects such as SpamBayes that have developed effective spam filters on the shoulders of both Graham and Robinson. By extending the use of Robinson’s (or more accurately, Fischer’s) chisquared combining rule beyond binary classification, the project led to the development of a ‘tournament’ style text classification algorithm which will be described below. FogBugz is a tracking tool used by companies and development teams to keep track of project bugs and customer inquiries. Importantly, it has the ability to receive incoming e-mail from an ordinary POP3 mailbox and automatically create new bugs, or cases, to be tracked. The purpose of the Bayesian filter is to enable automatic categorization of these incoming cases into categories, or areas. The categories are defined by the user. For example, a company might use FogBugz to automatically classify their incoming mail into Spam, Technical Questions, and Customer Service Questions. By default, the separation of spam e-mail into a distinct area is included in this organization process. Anybody who sifts through 300 messages about male enhancement, mortgage payment, and Nigerian treasure every morning to find 4 e-mails from family or coworkers is well aware of the need for this filtering. At the time of writing, Bayesian spam filters are fairly common, and if you’re not using one then I highly suggest you join the club. However, I believe that there are only a few other pieces of software which use Bayesian filtering to extend beyond binary spam/ham classification; POPFile is probably the most widely available. Key differences: POPFile uses the Naïve Bayes technique of combining probabilities 1 “Sick of spam? Then blame Alan Ralsky. He emails a billion of them a day. ” http://observer.guardian. co.uk/uk_news/story/0,6903,968089,00.html 2 Paul Graham. http://www.paulgraham. com/spam.html 3 Gary Robinson. http://www.linuxjournal. com/article.php? sid=6467 4 The SpamBayes Team. http://spambayes.sourceforge. net/ 5 Gary Robinson. http://garyrob.blogs. com/whychi90.pdf 6 The POPFile Team. http://popfile.sourceforge. net/ instead of Fischer’s chi-squared rule, and the FogBugz sorter’s Tournament algorithm strays from the POPFile approach. Significantly, one installation of FogBugz has the ability to sort multiple mailboxes into multiple sets of areas with entirely independent sorting procedures. Let’s say you work for Pfizer Inc. and you have two mailboxes. Your personal mailbox gets slammed with more spam than you can shake a stick at. Your inbox at work gets a few spams per day intermixed with at least 10 legit messages, most of which contain the words ‘Viagra’ and ‘prescription’. For you and your Pfizer coworkers, ‘Viagra’ has a very different meaning depending on the originating mailbox. The FogBugz sorter can treat these two cases separately, and your personal mailbox will filter out the overload of undesired ‘Viagra’ messages while your inbox at work will understand that an email containing ‘Viagra’ is probably an important message. It should be noted that as a byproduct of this project, the FogBugz sorter is not limited to e-mail classification; for instance, it currently serves as an automatic discussion board moderator for Fog Creek Software. Essential Background In order to understand the development of the classification process, some background information in Bayesian filtering is necessary. Reading the entirety of Graham and Robinson’s articles will leave you feeling like an expert, but if you choose to press on (which you probably will, despite my advice), the following should be sufficient preparation for the rest of the paper. In “A Statistical Approach to the Spam Problem”, Robinson explains: For each word that appears in the corpus, we calculate: • b(w) = (the number of spam e-mails containing the word w) / (the total number of spam e-mails). • g(w) = (the number of ham e-mails containing the word w) / (the total number of ham e-mails). • p(w) = b(w) / (b(w) + g(w)) p(w) can be roughly interpreted as the probability that a randomly chosen e-mail containing word w will be a spam. Spam-filtering programs can compute p(w) for every word in an e-mail and use that information as the basis for further calculations to determine whether the e-mail is ham or spam. There are many techniques available for using the collection of p(w) values to obtain a prediction regarding the likelihood that a document is associated with a specific category. Of these, Robinson’s technique, which is borrowed from R.A. Fischer’s combination of probabilities into a chi-squared distribution, has been extensively tested and is used by the most successful filters, including SpamBayes. Robinson provides ample theoretical justification for this improvement in practical accuracy over the original filters conceived by Graham [1]. The resultant value from Robinson’s technique, which he labels ‘I’, is used to create a prediction: I is an indicator that is near 1 when the preponderance of the evidence is in favor of the conclusion that the e-mail is a spam and near 0 when the evidence points to the conclusion that it’s ham.it is a useful characteristic of I that it is near . 5 in such cases [where there is strong evidence in favor of both possible conclusions], just as it is near . 5 when there is no particular evidence in one direction or the other. When there is significant evidence in favor of both conclusions, I takes the cautious approach. [t]he SpamBayes project.takes advantages of this by marking e-mails with I near . 5 as uncertain. When attempting to categorize into an arbitrary number of categories, the logic surrounding ‘I’ and uncertain predictions are among the problems encountered. However, SpamBayes is only one of many successful projects that have reported excellent statistics from Robinson’s indicator. It has proven to be an extremely promising route. I first tried to use the simplest and purest technique of expanding this method beyond binary classification.