More deterministic tokenization

We now run only one tokeizer, which produces exactly the symbol we want. We can
still have different rules request different tokens at the same position,
though, so not much power is lost (if any).

Signed-off-by: Casey Dahlin <casey.dahlin@gmail.com>
Pass previous token to tokenizers

This will let us inform about errors and also propagate file info like line and
column.

Signed-off-by: Casey Dahlin <casey.dahlin@gmail.com>