added function support to tokenizer
restructured code with inner classes