 anthony baxter i ve got a test set here that s the last NUMBER and a bit years email to info URL and info URL it s a really ugly set of NUMBER NUMBER messages currently broken into NUMBER NUMBER spam NUMBER NUMBER ham NUMBER NUMBER currently unclassified these addresses are all over the NUMBER some different ekit ekno isiconnect websites so they get a lot of spam as well as the usual spam it also has customers complaining about credit card charges it has people interested in the service and asking questions about long distance rates c c c lots and lots of commercial speech in other words stuff that sa gets pretty badly wrong can this corpus be shared i suppose not i m currently mangling it by feeding all parts text html whatever else into the filters as well as both a selected number of headers to from content type x mailer and also a list of header count_of_header this is showing up some nice stuff e g the x uidl that stoopid spammers blindly copy into their messages if we ever wink have a shared corpus an easy refactoring of timtest should allow to plug in different tokenizers i ve only made three changes to graham s algorithm so far well i ve made dozens only three survived testing as proven winners all the rest has been refining the tokenization to provide better clues i did have received in there but it s out for the moment as it causes rates to drop that s ambiguous accuracy rates or error rates ham or spam rates i m also stripping out html tags except for href and src there s so so much goodness in them note that i m only keeping the contents of the attributes mining embedded http https ftp thingies cut the false negative rate in half in my tests not keying off href just scanning for anything that looked like one that was the single biggest f n improvement i ve seen it didn t change the false positive rate so you know whether src added additional power or did you do both at once 