Added characters to ignorechars.
Merged tokenizer.