Refactor in preparation for error handling

Signed-off-by: Casey Dahlin <casey.dahlin@gmail.com>
Document tokensToSymbols

Signed-off-by: Casey Dahlin <casey.dahlin@gmail.com>
Create StateQueue class

We use a custom type of queue for states that automatically maintains the map
of states onto values and prioritizes states so that those of minimal
Levenshtein distance get processed first.

Signed-off-by: Casey Dahlin <casey.dahlin@gmail.com>
Add Levenshtein distance propagation logic to EPState

We can now propagate "with errors," causing new states to be produced with
invented tokens added, or with tokens skipped over. States are tagged with
their pseudo-Levenshtein distance from the original stream (pseudo because
we're abstracting for our potentially non-deterministic token stream).

Signed-off-by: Casey Dahlin <casey.dahlin@gmail.com>
Introduce error-recovery system

The error recovery system is based on constructing an alternate tokenization
path with a minimal pseudo-Levenshtein distance from the original. When we
arrive at a complete state where no final terminal has been emitted, we sort
all existing states by how suitable they are for continuation (preferring later
states, states that need symbols that are likely to be terminals, etc.). We
then choose the best of those and insert several derived states into the
processing queue: versions with the next token removed, versions with the next
token replaced with the expected token, and versions with the expected token
added before the next token. We mark these states as having a Levenshtein
distance of 1. That distance aggregates to parent states, so if the process
repeats for more errors the total Levenshtein distance stored in the produced
final terminal will be the number of tokens skipped, inserted, or replaced.

When choosing states to continue, we prefer states that have low Levenshtein
distances, so we tend to continue paths that have as few errors as possible,
and thus tend to complete the least erroneous path.

Signed-off-by: Casey Dahlin <casey.dahlin@gmail.com>