 tim i d prefer to strip html tags from everything but last time i tried that it still had bad effects on the error rates in my corpora guido your corpora are biased in this respect though newsgroups have a strong social taboo on posting html but in many people s personal inboxes it is quite abundant we re in violent agreement there the comments in URL say that as strongly as possible and i ve repeated it endlessly here too but so long as i was the only one doing serious testing it was a dubious idea to make the code maximally clumsy for me to use on the URL task wink getting a good ham corpus may prove to be a bigger hurdle than i though my own saved mail doesn t reflect what i receive since i save and throw away selectively much more so than in the past yup the system picks up on everything in the tokens graham s proposed delete as ham and delete as spam keys would probably work very well for motivated geeks but paul svensson has pointed out here that they probably wouldn t work nearly so well for real people ah that explains why the html tags didn t get stripped i d again offer to add an optional argument to tokenize so that they d get stripped here too but if it gets glossed over a third time that would feel too much like a loss wink i ll bite sounds like a good idea to strip the html in this case i d like to see how this improves the f p rate on this corpus i ll soon check in this change def tokenize_body self msg retain_pure_html_tags false generate a stream of tokens from an email message if a multipart alternative section has both text plain and text html sections the text html section is ignored this may not be a good idea e g the sections may have different content html tags are always stripped from text plain sections by default html tags are also stripped from text html sections however doing so hurts the false negative rate on tim s comp lang python tests where html only messages are almost never legitimate traffic if optional argument retain_pure_html_tags is specified and true html tags are retained in text html sections you should do a cvs up and establish a new baseline first as i checked in a pure win change in the wee hours that cut the fp and fn rates in my tests 