Recognizes comments and EOF tokens.