Updated tokenizer implementation, tests still pending