added tokenizer operations.
working version 1.0