single tokens table + support for "new" tokens

replaced all_tokens and extra_tokens with single table call tokens,
which simply has a state column to indicate whether extra or new; tons
of changes to support new tokens and handle duplicate tokens
Merge branch 'master' of https://github.com/kwv/Sandbox