 guido i meant to say that they were NUMBER NUMBER clues cancelled out by NUMBER NUMBER clues but that s wrong too it looks i haven t grokked this part of your code yet this one has way more than NUMBER clues and it seems the classifier basically ended up counting way more NUMBER NUMBER than NUMBER NUMBER clues and no others made it into the list i thought it was looking for clues with values in between apparently it found none that weren t exactly NUMBER NUMBER there s a brief discussion of this before the definition of max_discriminators all clues with prob min_spamprob and max_spamprob are saved in min and max lists and all other clues are fed into the nbest heap then the shorter of the min and max lists cancels out the same number of clues in the longer list whatever remains of the longer list if anything is then fed into the nbest heap too but no more than max_discriminators of them in no case do more than max_discriminators clues enter into the final probability calculation but all of the min and max lists go into the list of clues else you d have no clue that massive cancellation was occurring and massive cancellation may yet turn out to be a hook to signal that manual review is needed in your specific case the excess of clues in the longer max_spamprob list pushed everything else out of the nbest heap and that s why you didn t see anything other than NUMBER NUMBER and NUMBER NUMBER before adding these special lists the outcome when faced with many NUMBER NUMBER and NUMBER NUMBER clues was too often a coin toss whichever flavor just happened to appear max_discriminators NUMBER NUMBER times first determined the final outcome that sure sets the record for longest list of cancelling extreme clues this happened to be the longest one but there were quite a few similar ones i just beat it wink a tokenization scheme that folds case and ignores punctuation and strips a trailing s from words and saves both word bigrams and word unigrams turned up a low probability very long spam with a list of NUMBER NUMBER NUMBER clues and NUMBER NUMBER NUMBER clues yikes i wonder if there s anything we can learn from looking at the clues and the html it was heavily marked up html with ads in the sidebar but the body text was a serious discussion of oo and soft coding with lots of highly technical words as clues including zope and zeo no matter how often it says zope it gets only one NUMBER NUMBER clue from doing so ditto for zeo in contrast html markup has many unique words that get NUMBER NUMBER btw this is a clear case where the assumption of conditionally independent word probabilities is utterly bogus e g the probability that body appears in a message is highly correlated with the prob of br appearing by treating them as independent naive bayes grossly misjudges the probability that both appear and the only thing you get in return is something that can actually be computed wink read the what about html section in URL from the very start i ve been investigating what would work best for the mailing lists hosted at URL and html decorations have so far been too strong a clue to justify ignoring it in that specific context i haven t done anything geared toward personal email including the case of non mailing list email that happens to go through URL i d prefer to strip html tags from everything but last time i tried that it still had bad effects on the error rates in my corpora the full test results with and without html tag stripping is included in the what about html comment block but as the comment block also says xxx so if another way is found to slash the f n rate the decision here xxx not to strip html from html only msgs should be revisited and we ve since done several things that gave significant f n rate reductions i should test that again now are there any minable but unmined header lines in your corpus left almost all of them apart from mime decorations that appear in both headers and bodies like content type the only header lines the base tokenizer looks at now are subject from x mailer and organization or do we have to start with a different corpus before we can make progress there i would need different data yes my ham is too polluted with mailman header decorations which i may or may not be able to clean out but fudging the data is a mortal sin and i haven t changed a byte so far and my spam too polluted with header clues about the fellow who collected it in particular i have to skip to and received headers now and i suspect they re going to be very valuable in real life for example i don t even catch undisclosed recipients in the to header now no sorry these were all of the following structure multipart mixed text plain brief text plus url s text html long html copied from website ah that explains why the html tags didn t get stripped i d again offer to add an optional argument to tokenize so that they d get stripped here too but if it gets glossed over a third time that would feel too much like a loss wink this seems confused jeremy didn t use my trained classifier pickle he trained his own classifier from scratch on his own corpora i think it s still corpus size i reported on tests i ran with random samples of NUMBER spams and NUMBER hams from my corpus that means training on sets of those sizes as well as predicting on sets of those sizes and while that did harm the error rates the error rates i saw were still much better than jeremy reported when using NUMBER of each ah a full test run just finished on the tokenization scheme that folds case and ignores punctuation and strips a trailing s from words and saves both word bigrams and word unigrams this is the code tokenize everything in the body lastw for w in word_re findall text n len w make sure this range matches in tokenize_word if NUMBER n NUMBER if w NUMBER s w w NUMBER yield w if lastw yield lastw w lastw w elif n NUMBER lastw for t in tokenize_word w yield t where word_re re compile r w xNUMBER xff this at least doubled the process size over what s done now it helped the f n rate significantly but probably hurt the f p rate the f p rate is too low with only NUMBER hams per run to be confident about changes of such small absolute magnitude NUMBER NUMBER is a single message in the f p table false positive percentages NUMBER NUMBER NUMBER NUMBER tied NUMBER NUMBER NUMBER NUMBER lost was NUMBER NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER lost was NUMBER NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER tied NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER tied NUMBER NUMBER NUMBER NUMBER lost was NUMBER NUMBER NUMBER NUMBER NUMBER lost was NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER tied NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER won NUMBER times tied NUMBER times lost NUMBER times total unique fp went from NUMBER to NUMBER false negative percentages NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER tied NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER tied NUMBER NUMBER NUMBER NUMBER won NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER lost NUMBER NUMBER won NUMBER times tied NUMBER times lost NUMBER times total unique fn went from NUMBER to NUMBER 