Enhanced lexer to accept (and ignore) non-code

Also tidied things up a bit