LUCENE-5608 fix assertions given that tokens are re-used

git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1594391 13f79535-47bb-0310-9956-ffa450edef68
LUCENE-5608 better/more comments

git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1594394 13f79535-47bb-0310-9956-ffa450edef68