improper handling of "NEEDSOMETHING" state, since that requires two
processing steps (first to remove the NEEDSOMETHING, and second to
process the actual token itself)