I just tried to remove the if-block and indeed some tests fail. I'm not exactly sure why it happens, apparently the generated parser code polls (in some cases) another time for more tokens from the lexer when it sees an EOF. I don't think it's something to be worried about.