so then tim peters tim one comcast net is all like guido i don t know how big that pickle would be maybe loading it each time is fine or maybe marshalling my tests train on about NUMBER NUMBER msgs and a binary pickle of the database is approaching NUMBER million bytes my paltry NUMBER message training set makes a NUMBER NUMBERmb where NUMBERmb NUMBEReNUMBER bytes pickle URL which i just checked in will optionally let you write stuff out to a dbm file with that same message base the dbm file weighs in at a hefty NUMBER NUMBERmb it also takes longer to write using a database real NUMBERmNUMBER NUMBERs user NUMBERmNUMBER NUMBERs sys NUMBERmNUMBER NUMBERs using a pickle real NUMBERmNUMBER NUMBERs user NUMBERmNUMBER NUMBERs sys NUMBERmNUMBER NUMBERs this is on a piii at NUMBER NUMBERmhz i don t know what it s supposed to be NUMBER NUMBER is what proc cpuinfo says for comparison spamoracle currently the gold standard in my mind at least for speed on the same data blazes along real NUMBERmNUMBER NUMBERs user NUMBERmNUMBER NUMBERs sys NUMBERmNUMBER NUMBERs its data file which appears to be a marshalled hash is NUMBERkb however it s compiled o caml and it uses a much simpler tokenizing algorithm written with a lexical analyzer ocamllex so we ll never be able to outperform it it s something to keep in mind though i don t have statistics yet for scanning unknown messages actually i do and the database blows the pickle out of the water but it scores every word with NUMBER NUMBER so i m not sure that s a fair test in any case NUMBERmb per user is probably too large and NUMBERmb is questionable on the other hand my pickle compressed very well with gzip shrinking down to NUMBER NUMBERmb neale 