tokenizer: added methods for tokenizing option