corrected tokenizer so it does not delete '