extended lexer code to return instances of token datatype along with matches

Instances of the token datatype defined in the lexical spec are
associated with final states of the intermediary NFAs and DFAs and are
used during lexing to return pairs of tokens and strings.

This is done by replacing the set of final states with a map from
state to the token class that was matched. I don't know if there is
any way (let alone a typesafe way) of actually calling the token
constructors with string arguments, but returning a tuple of the token
and any extra information obtained about the match (like line number
and character offset) is equivalent.