re-roll of 178

This is a re-roll of all of the lexer refactoring done previously in PR #178, this time based off of 9.5 so there are no merge issues.