test the comment eater
state machine for tokenizing a rule's RHS