Don't delete lines following lex'd `ignore:end`. Fixes gh-1879

Previously, lines would be cleared (set to the empty string) even after `ignore:end`
is found, if the comment was not processed first.

This change performs some minor comment processing in the lexer itself in order to
solve this problem.

This is a hack, ideas are welcome.

Closes #1879
`commentToken()` isn't bound to the scanner's context, so it's just grabbing it so it can be used.