Improve logic behind consuming tokens.

Fixes https://github.com/allenai/taggers/issues/37
Fixes https://github.com/allenai/taggers/issues/43
Fixes https://github.com/allenai/taggers/issues/46
Partially addresses https://github.com/allenai/taggers/issues/48