Disallow not properly delimited tokens.

Also simplifies the logic.