refactor how tokens and scanner interact