Refine token detection

I want the ability to use the same token registry to both tokenize an
input string and detect tokens. I did some regex magic to make this
possible. I fixed and expanded what was considered a valid identifier.
This code appears to be functioning. I want to do some testing before
releasing the next version to NuGet.