refactored lexical analysis routines