change the loop so that when a token is got while reading something else, it can be tested without loosing it. A break should be used if it is sure that no token is left over (most cases).