Fix scanner to emit a token if it is at the end of the string

* Begin adding integration tests for the scanner and parser
  to make sure there are no regressions.
Fix scanner to handle multibyte encodings

* This uses the ragel-driven buffer to make sure bytes are aligned
  by character boundaries.