Factored out lexing into subfunctions

Also some refactoring of characters, booleans,
bytevectors, and #!r6rs comments.
Cleaned up code a bit:
* generalized simple tokens cases
* removed function calls outrage
* inlined single character case
* moved space & comment ignoring to a separate function
Implemented proper sentinel characters

  * Negative, clearly non-Unicode and easily testable
  * Distinguished end-of-stream and end-of-buffer

Also fixed tests, comments, Cutter assertions.
Quick, dirty, and single-buffer implementation of hex characters
Optimized buffer refilling
Comment and Makefile tweaks
Changed pointers in context for indices. Disabled lexer_context tests.
Replaced old lookahead interface
Removed old implementation
Added modern (partial) implementation
Implemented the crucial rest of the buffering functions.

Also quickly-dirty adapted the code so it does not crash.
Shift-reduce whitespace
Merge branch 'two-part-buffer' into long-tokens

Merging partially implemented branch before total refactoring
Merge branch 'long-tokens'

I'm going to do a total refactoring of this stuff to adopt UTF-8, so
current code state is merged into master to eliminate then-obsolete
branches.