Add comment detection and erasure to tokenizer.