did work on tokenizer, removed a bit of reduntant code