added some error checking to tokenizer.