 but it also identified as spam everything in my inbox that had any mime structure or html parts and several messages in my saved zope geeks list that happened to be using mime and or html do you know why the strangest implied claim there is that it hates mime independent of html for example the spamprob of content type text plain in that pickle is under NUMBER NUMBER content type multipart alternative gets NUMBER NUMBER but that s not a killer clue and one bit of good content will more than cancel it out i reran the experiment with the new spamhamNUMBER pik but it doesn t seem to make a difference here are the clues for the two spams in my inbox in URL s output format which sorts the clues by probability the first two numbers are the message number and overall probability then line folded NUMBER NUMBER NUMBER s facility NUMBER NUMBER speaker NUMBER NUMBER stretch NUMBER NUMBER thursday NUMBER NUMBER young NUMBER NUMBER mistakes NUMBER NUMBER growth NUMBER NUMBER content type text plain NUMBER NUMBER please NUMBER NUMBER capital NUMBER NUMBER series NUMBER NUMBER subject don NUMBER NUMBER companies NUMBER NUMBER content type text html NUMBER NUMBER fee NUMBER NUMBER money NUMBER NUMBER NUMBER NUMBERam NUMBER NUMBER NUMBER NUMBERam NUMBER NUMBER content type image gif NUMBER NUMBER content type multipart alternative NUMBER NUMBER attend NUMBER NUMBER companies NUMBER NUMBER content type type multipart alternative NUMBER NUMBER content type multipart related NUMBER NUMBER economy NUMBER NUMBER economy NUMBER NUMBER this has NUMBER content types as spam clues only one of which is related to html despite there being an html alternative and NUMBER other spam clues vs only NUMBER ham clues this was an announcement of a public event by our building owners with a text part that was the same as the html afaict its language may be spammish but the content type clues didn t help btw it makes me wonder about the wisdom of keeping punctuation economy and economy to me don t seem to deserve two be counted as clues NUMBER NUMBER NUMBER s near NUMBER NUMBER alexandria NUMBER NUMBER conn NUMBER NUMBER from adam NUMBER NUMBER from email addr panix NUMBER NUMBER poked NUMBER NUMBER thorugh NUMBER NUMBER though NUMBER NUMBER i m NUMBER NUMBER reflect NUMBER NUMBER i ve NUMBER NUMBER wednesday NUMBER NUMBER content disposition inline NUMBER NUMBER contacting NUMBER NUMBER sold NUMBER NUMBER financially NUMBER NUMBER prices NUMBER NUMBER rates NUMBER NUMBER discount NUMBER NUMBER hotel NUMBER NUMBER hotels NUMBER NUMBER hotels NUMBER NUMBER nights NUMBER NUMBER plaza NUMBER NUMBER rates NUMBER NUMBER rates NUMBER NUMBER rooms NUMBER NUMBER season NUMBER NUMBER stations NUMBER NUMBER subject hotel NUMBER NUMBER here is the full message received headers stripped with apologies to ziggy and david date fri NUMBER sep NUMBER NUMBER NUMBER NUMBER NUMBER from adam turoff ziggy panix com subject hotel information to guido URL davida URL message id NUMBER gkNUMBER panix com mime version NUMBER NUMBER content type text plain charset us ascii content disposition inline user agent mutt NUMBER NUMBERi i ve been looking into hotels i poked around expedia for availability from march NUMBER to NUMBER NUMBER nights wednesday thorugh saturday i ve also started contacting hotels for group rates some of the group rates are no better than the regular rates and they require signing a contract with a minimum number of rooms sold with someone financially responsible for unbooked rooms most hotels are less than responsive radission barcelo hotel dupont circle NUMBER night NUMBER weekend state plaza hotel foggy bottom near gwu NUMBER night NUMBER weekend hilton silver spring near metro in suburban md NUMBER hight NUMBER weekend windsor park hotel conn ave between dupont circle woodley park metro stations NUMBER night needs a car econo lodge alexandria near metro in suburban va NUMBER night this is a hand picked list i ignored anything over NUMBER night even though there are some really well situated hotels nearby at higher rates also i m not sure how much these prices reflect an expedia only discount i can t vouch for any of these hotels either i also found out that the down season for dc hotels are mid june through mid september and mid november through mid january z this one has no mime structure nor html it even has a content disposition which is counted as a non spam clue it got f p ed because of the many hospitality related and money related terms i m surprised NUMBER night and similar aren t clues too and again several spam clues are duplicated with different variations hotel hotels hotels subject hotel rates rates wrt hating html possibilities include NUMBER it really had to do with something other than mime html NUMBER these are pure html not multipart alternative with a text plain part so that the tags aren t getting stripped the pickled classifier despises all hints of html due to its URL heritage NUMBER these are multipart alternative with a text plain part but the latter doesn t contain the same text as the text html part for example as anthony reported perhaps the text plain part just says something like this is an hmtl message if it s NUMBER it would be easy to add an optional bool argument to tokenize meaning even if it is pure html strip the tags anyway in fact i d like to do that and default it to true the extreme hatred of html on tech lists strikes me as umm extreme wink i also looked in more detail at some f p s in my geeks traffic the first one s a doozie that s the term right it has lots of html clues that are apparently ignored it was a multipart mixed with two parts a brief text plain part containing one or two sentences a mondo weird url URL and some employer generated spammish boilerplate the second part was the html taken directly from the above url clues NUMBER NUMBER NUMBER s main NUMBER NUMBER later NUMBER NUMBER lots NUMBER NUMBER paul NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER adopt NUMBER NUMBER apps NUMBER NUMBER commands NUMBER NUMBER URL NUMBER NUMBER dejanews NUMBER NUMBER discipline NUMBER NUMBER duct NUMBER NUMBER email addr digicool NUMBER NUMBER email name paul NUMBER NUMBER everitt NUMBER NUMBER exist NUMBER NUMBER forwards NUMBER NUMBER framework NUMBER NUMBER from email addr digicool NUMBER NUMBER from email name paul NUMBER NUMBER from paul NUMBER NUMBER height NUMBER NUMBER hodge podge NUMBER NUMBER httpNUMBER deja NUMBER NUMBER httpNUMBER zope NUMBER NUMBER httpNUMBER st_rn NUMBER NUMBER httpNUMBER comp NUMBER NUMBER httpNUMBER getdoc NUMBER NUMBER httpNUMBER ps NUMBER NUMBER http NUMBER NUMBER NUMBER NUMBER http NUMBER NUMBER NUMBER NUMBER http NUMBER NUMBER NUMBER NUMBER http NUMBER an NUMBER NUMBER http NUMBER author NUMBER NUMBER http NUMBER fmt NUMBER NUMBER http NUMBER getdoc NUMBER NUMBER http NUMBER pr NUMBER NUMBER http NUMBER products NUMBER NUMBER http NUMBER query NUMBER NUMBER http NUMBER search NUMBER NUMBER http NUMBER viewthread NUMBER NUMBER http NUMBER xp NUMBER NUMBER http NUMBER zope NUMBER NUMBER inventing NUMBER NUMBER jsp NUMBER NUMBER jsp NUMBER NUMBER logic NUMBER NUMBER maps NUMBER NUMBER neo NUMBER NUMBER newsgroup NUMBER NUMBER object NUMBER NUMBER popup NUMBER NUMBER probable NUMBER NUMBER query NUMBER NUMBER query NUMBER NUMBER resizes NUMBER NUMBER servlet NUMBER NUMBER skip NUMBER NUMBER NUMBER stems NUMBER NUMBER subject jsp NUMBER NUMBER sucks NUMBER NUMBER templating NUMBER NUMBER tempted NUMBER NUMBER url NUMBER NUMBER usenet NUMBER NUMBER usenet NUMBER NUMBER wrote NUMBER NUMBER x mailer mozilla NUMBER NUMBER en windows nt NUMBER NUMBER u NUMBER NUMBER zope NUMBER NUMBER NUMBER NUMBER NUMBER ccNUMBER NUMBER NUMBER ffNUMBER NUMBER NUMBER ffNUMBER NUMBER NUMBER ffffff NUMBER NUMBER copy NUMBER NUMBER gt NUMBER NUMBER nbsp nbsp NUMBER NUMBER quot no NUMBER NUMBER med NUMBER NUMBER small NUMBER NUMBER NUMBERpt NUMBER NUMBER NUMBERpx NUMBER NUMBER NUMBERpx NUMBER NUMBER NUMBERpt NUMBER NUMBER NUMBERpx NUMBER NUMBER NUMBERpt NUMBER NUMBER NUMBERpx NUMBER NUMBER NUMBERpt NUMBER NUMBER NUMBERpx NUMBER NUMBER NUMBER NUMBER NUMBER NUMBERpt NUMBER NUMBER NUMBER NUMBER b NUMBER NUMBER body NUMBER NUMBER head NUMBER NUMBER html NUMBER NUMBER script NUMBER NUMBER select NUMBER NUMBER span NUMBER NUMBER style NUMBER NUMBER table NUMBER NUMBER td NUMBER NUMBER td tr NUMBER NUMBER tr NUMBER NUMBER tr tr NUMBER NUMBER b a NUMBER NUMBER base NUMBER NUMBER body NUMBER NUMBER br NUMBER NUMBER br nbsp NUMBER NUMBER br a NUMBER NUMBER br span NUMBER NUMBER font NUMBER NUMBER form NUMBER NUMBER head NUMBER NUMBER html NUMBER NUMBER img NUMBER NUMBER input NUMBER NUMBER meta NUMBER NUMBER option NUMBER NUMBER p NUMBER NUMBER p a NUMBER NUMBER script NUMBER NUMBER select NUMBER NUMBER span NUMBER NUMBER style NUMBER NUMBER table NUMBER NUMBER td NUMBER NUMBER td NUMBER NUMBER td td NUMBER NUMBER td img NUMBER NUMBER tr NUMBER NUMBER tr NUMBER NUMBER tr td NUMBER NUMBER tr td img NUMBER NUMBER absolute NUMBER NUMBER align left NUMBER NUMBER align center NUMBER NUMBER align left NUMBER NUMBER align middle NUMBER NUMBER align right NUMBER NUMBER align right NUMBER NUMBER alt NUMBER NUMBER bold NUMBER NUMBER border NUMBER NUMBER NUMBER border NUMBER NUMBER NUMBER color NUMBER NUMBER colspan NUMBER NUMBER NUMBER colspan NUMBER NUMBER NUMBER colspan NUMBER NUMBER NUMBER face arial NUMBER NUMBER font family NUMBER NUMBER font size NUMBER NUMBER font weight NUMBER NUMBER footer NUMBER NUMBER for br NUMBER NUMBER fucking br NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER height NUMBER NUMBER NUMBER hspace NUMBER NUMBER NUMBER httpNUMBER g NUMBER NUMBER httpNUMBER webNUMBER NUMBER NUMBER httpNUMBER NUMBER NUMBER NUMBER httpNUMBER ads NUMBER NUMBER httpNUMBER d NUMBER NUMBER httpNUMBER page NUMBER NUMBER httpNUMBER site NUMBER NUMBER http NUMBER article NUMBER NUMBER http NUMBER back NUMBER NUMBER http NUMBER com NUMBER NUMBER http NUMBER d NUMBER NUMBER http NUMBER gif NUMBER NUMBER http NUMBER go NUMBER NUMBER http NUMBER group NUMBER NUMBER http NUMBER http NUMBER NUMBER http NUMBER post NUMBER NUMBER http NUMBER ps NUMBER NUMBER http NUMBER site NUMBER NUMBER http NUMBER st NUMBER NUMBER http NUMBER title NUMBER NUMBER http NUMBER yahoo NUMBER NUMBER inc a NUMBER NUMBER jobs NUMBER NUMBER normal NUMBER NUMBER nowrap NUMBER NUMBER nowrap NUMBER NUMBER nowrap font NUMBER NUMBER padding NUMBER NUMBER rowspan NUMBER NUMBER NUMBER rowspan NUMBER NUMBER NUMBER servlets NUMBER NUMBER size NUMBER NUMBER NUMBER size NUMBER NUMBER NUMBER skip NUMBER NUMBER NUMBER skip b NUMBER NUMBER NUMBER skip h NUMBER NUMBER NUMBER skip h NUMBER NUMBER NUMBER skip h NUMBER NUMBER NUMBER skip h NUMBER NUMBER NUMBER skip h NUMBER NUMBER NUMBER skip h NUMBER NUMBER NUMBER skip v NUMBER NUMBER NUMBER solid NUMBER NUMBER text NUMBER NUMBER NUMBER to br NUMBER NUMBER type image NUMBER NUMBER type text NUMBER NUMBER type hidden NUMBER NUMBER type image NUMBER NUMBER type radio NUMBER NUMBER type submit NUMBER NUMBER type text NUMBER NUMBER valign top NUMBER NUMBER valign top NUMBER NUMBER value NUMBER NUMBER visibility NUMBER NUMBER width NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER width NUMBER NUMBER NUMBER works br NUMBER NUMBER the second f p had the same structure and sender the third f p had the same structure and a different sender ditto the fifth sixth not posting clues for brevity the fourth was different plaintext with one very short sentence and a url clues NUMBER NUMBER NUMBER s from email addr digicool NUMBER NUMBER httpNUMBER news NUMBER NUMBER from email addr com NUMBER NUMBER from tres NUMBER NUMBER http NUMBER NUMBERdigi NUMBER NUMBER proto http NUMBER NUMBER subject geeks NUMBER NUMBER x mailer mozilla NUMBER NUMBER en xNUMBER u linux NUMBER NUMBER NUMBER NUMBER NUMBERsmp iNUMBER NUMBER NUMBER take NUMBER NUMBER bool noorg NUMBER NUMBER httpNUMBER com NUMBER NUMBER skip h NUMBER NUMBER NUMBER http NUMBER htm NUMBER NUMBER subject software NUMBER NUMBER http NUMBER business NUMBER NUMBER http NUMBER local NUMBER NUMBER subject firm NUMBER NUMBER us NUMBER NUMBER the seventh was similar i scanned a bunch more until i got bored and most of them were either of the first form brief text with url followed by quoted html from website or the second brief text with one or more urls it s up to you to decide what to call this but i think these are none of your NUMBER NUMBER or NUMBER they re close to NUMBER but all are multipart mixed rather than multipart alternative so i guess i ll have to retrain it yes you told me so that would be a different experiment i m certainly curious to see whether jeremy s much worse than mine error rates are typical or aberrant it s possible that the corpus you ve trained on is more homogeneous than you thought guido van rossum home page URL 