added tokenizer selection option, support for different lm types