Reimplemented tokenizing code.