adding logic functions to tmachine tokenizer