Small refactoring of lexer state functions

Instead of having a single function that is able to lex an entire ruby
file, it would be more convenient to have functions that can parse any
particular ast node, and functions that compose that in interesting ways
(e.g.: reading an entire file is just lexing *something* until you reach EOF).

rubyspec requires loading fileutils, which incidentally ends up needing to
parse heredocs better. In this case, I'll want to be able to parse until the
end of the current line, and then switch to parsing the heredoc. This refactor
should make that a bit easier.