 jeremy hylton i think one step towards deployment is creating a re usable tokenizer for mail messages the current codebase doesn t expose an easy to use or easy to customize tokenizer tokenize couldn t be easier to use it takes a string argument and produces a stream of tokens whether via explicit list or generator or tuple or doesn t matter all the tokenize functions in URL and URL are freely interchangeable this way note that we have no evidence to support that a customizable tokenizer would do any good or if it would in which ways customization could be helpful that s a research issue on which no work has been done the timtest module seems to contain an enormous body of practical knowledge about how to parse mail messages but the module wasn t designed for re use that s partly a failure of imagination wink splitting out all knowledge of tokenization is just a large block cut and paste there it s done change the from timtoken import tokenize at the top to use any other tokenizer now if you want to make it easier still feel free to check in something better i d like to see a module that can take a single message or a collection of messages and tokenize each one the msg and msgstream classes in URL are a start at that but it s hard to do anything truly useful here when people use all sorts of different physical representations for email msgs mboxes in various formats one file per folder one file per msg skip s gzipped gimmick if you re a python coder wink you should find it very easy to change the guts of msg and msgstream to handle your peculiar scheme defining interfaces for these guys should be done i d like to see the tokenize by customizable too tim had to exclude some headers from his test data because there were particular biases in the test data if other people have test data without those biases they ought to be able to customize the tokenizer to include them or exclude others this sounds like a bottomless pit to me and there s no easier way to customize than to edit the code as readme txt still says though massive refactoring would help hop to it 