Refactored handling of lookahead states and

pending tokens