continued working on editor functionality; integrated tokenizer