Tokenizer now uses tokens to define ignored input