fixed the tokenizer so that it reads the whole file into a buffer,
somehow the program is still pretty slow, but not nearly as bad as last time,...