constructing tokenizer from grammar and variableTypes