 neale pickett if you can spare the memory you might get better performance in this case using the pickle store since it only has to go to disk once but boy does it ever go to disk i can t think of anything obvious to speed things up once it s all loaded into memory though on my box the current system scores about NUMBER msgs per second starting in memory of course while that can be a drag while waiting for one of my full test runs to complete one of those scores a message more than NUMBER NUMBER times and trains more than NUMBER NUMBER times i ve got no urge to do any speed optimizations if i were using this for my own email i d never notice the drag guido will bitch like hell about waiting an extra second for his NUMBER msg batches to score but he s the boss so he bitches about everything wink that s profiler territory and profiling is exactly the kind of optimization i just said i wasn t going to do i haven t profiled yet but suspect there aren t any egregious hot spots NUMBER gram ing of long words with high bit characters is likely overly expensive when it happens but it doesn t happen that often and as an approach to non english languages it sucks anyway i e there s no point speeding something that ought to be replaced entirely 