set options for end of line and comments in tokenizer
make tokenizer output more similar to input with added token types