Better Tokenizer Diagnostics

Added lines for diagnostic outputs.