fix 'trailing whitespace' induced erroneous behaviour which had me invoke the ethereal dieties, with feeling

Added fix for when grammar file leaves some (invisible) whitespace on the empty lines between lexer rules: previously, the lexer would, depending on the exact input, barf an inexplicable hairball or (worse!) fail silently. Unit tests to verify the success of the fix have been included.

> Plus a few pedantic whitespace/semicolon edits: I can be an a^H donkey like that: old style guy likes his semicolons to be consistent and on a 'should-be' rather than 'must-be' basis.