Replaced sitedef lexer flags with function pointer.

Removed in_comment and in_path_expression mode flags and added the
alloc_next_token function pointer field.  When the scanning mode changes,
the sitedef lexer changes the scanning function in that field.