 skip montanaro because i get mail through several different email addresses i frequently get duplicates or triplicates or more plicates of various spam messages in saving spam for later analysis i haven t always been careful to avoid saving such duplicates i wrote a script some time ago to try an minimize the duplicates i see by calculating a loose checksum but i still have some duplicates should i delete the duplicates before training or not people just can t stop thinking wink the classifier should work best when trained on a wholly random spattering of real life if real life contains duplicates then that s what the classifier should see would people be interested in the script i d be happy to extricate it from my local modules and check it into cvs sure i think it s relevant but maybe for another purpose paul svensson is thinking harder about real people wink than the rest of us and he may be able to get use out of approaches that identify closely related spam for example some amount of spam is going to end up in the ham training data in real life use and any sort of similarity score to a piece of known spam may be an aid in finding and purging it 