fix EOF handling, including a nasty infinite loop

Once we reach the "raw" EOF token, we know exactly what we're going to
get and we don't need to do any more processing on it. In fact, we
_shouldn't_ do more processing on it, since we now add tokens to the
buffer no matter what when we see an EOF. That was causing an infinite
loop, as the EOF would cause things to get buffered, which would then
empty until we hit the EOF, which would re-buffer it, etc.