 i wrote a script some time ago to try an minimize the duplicates i see by calculating a loose checksum but i still have some duplicates should i delete the duplicates before training or not tim people just can t stop thinking wink the classifier should work tim best when trained on a wholly random spattering of real life if tim real life contains duplicates then that s what the classifier tim should see a bit more detail i get destined for many addresses skip URL skip URL concerts URL webmaster URL etc i originally wrote a slightly different version of the URL script i m about to check in to avoid manually scanning all those presumed spams which are really identical once a message was identified as spam what i refer to as a loose checksum was computed to try and avoid saving the same spam multiple times for later review would people be interested in the script i d be happy to extricate it from my local modules and check it into cvs tim sure i think it s relevant but maybe for another purpose paul tim svensson is thinking harder about real people wink than the rest tim of us and he may be able to get use out of approaches that tim identify closely related spam for example some amount of spam is tim going to end up in the ham training data in real life use and any tim sort of similarity score to a piece of known spam may be an aid in tim finding and purging it i ll check it in let me know if you find it useful skip 