fixed escaping issue for tokenization.

This is a bit of a HACK as it uses information about the tokenizer.