Implemented most of the tree construction in the tokenizer