Added debugging info to RubyLexer#advance when yylex returns nil.
Removed RubyLexer#expr_beg_push.
Removed RubyLexer#fix_arg_lex_state.
Added RubyLexer#result, refactoring a common pattern.
Added RubyLexer#expr_result (a morph of expr_beg_push).
Added RubyLexer#arg_state (a non-mutative morph of fix_arg_lex_state).
Replaced every section I could with calls to result.
RubyLexer#yylex went from 639 lines to 502! Oddly... flog went up by 64.6. :(

[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8870]
Added RubyLexer::SIMPLE_SSTRING_RE to clean up the huge case stmt.
Added RubyLexer::in_arg_state? to refactor a bunch of patterns.
Added RubyLexer::space_vs_beginning to refactor ** vs *.
Cleaned up more code with result.

[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8871]
- No longer assigning ivar/cvars to env. Only locals should be in env.
Removed yylex's last_state. Use lex_state directly and stop modifying it everywhere.
+ Added RubyLexer#process_token_keyword.
Split process_token out to only deal with lvars. gvar/cvar/ivar all handled directly.
Merged a lot of state checks into cleaner case stmts. Fewer returns.

[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8881]
minor cleanup quelling 1.9 warnings
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8889]
why didn't that go through?
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8890]
Added some tests
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8891]
fixed comment placement
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8904]
Entirely to TestRubyLexer:
Added lex_state ivar, defaulting to :expr_beg. Now you can set start state before lexing.
Renamed util_* to assert_*
util_lex_token became assert_lex & assert_lex3, now I test lex_state everywhere and optionally test nesting
util_bad_token became refute_lex
Added assert_parse, split from setup_lexer.
moved all assertions up top
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8905]
additional comments and todos for later
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8917]
Added extra quoted symbol tests
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8922]
cleanup
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8923]
cleanup
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8924]
+ zero byte is allowed in symbols for 1.9 / 2.0.
+ Added optimizations for simple quoted symbols.

[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8925]
Added test_yylex_colon2_argh to test edge case
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8929]
Sorted methods 'n stuff to make comparisons easier.
+ Added RubyLexer#scan, #matched, #beginning_of_line? and others to decouple from internals.
Finally figured out some of the string logic that has eluded me (but worked!) for years.

[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8930]
simple rearrangement
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8936]
+ Removed unused RubyLexer#warnings.
+ Added RubyLexer#beginning_of_line?, check(re), end_of_stream?
lots of code cleanup with those additions.

[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8937]
+ Aliased Lexer#src to ss (since that is what it is).
Moved everything to ss.

[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8938]
Missed some in last round
[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8939]
+ Renamed *_RE consts to just * (IDENT_CHAR, ESC, etc).
- Refactored initialize and reset to more properly re-initialize as needed.

[git-p4: depot-paths = "//src/ruby_parser/dev/": change = 8952]