LUCENE-5053: Expose PagedGrowableWriter memory usage.


git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1492114 13f79535-47bb-0310-9956-ffa450edef68
LUCENE-5042: Fix the n-gram tokenizers and filters.

This commit fixes n-gram tokenizers and filters so that they handle
supplementary characters correctly and adds the ability to pre-tokenize the
stream in tokenizers.


git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1492185 13f79535-47bb-0310-9956-ffa450edef68
LUCENE-5042: Reset the CharBuffer in TokenStream.reset().


git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1492257 13f79535-47bb-0310-9956-ffa450edef68
LUCENE-5042: Maven configuration: add chars.txt to forbiddenapis config

git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/trunk@1492420 13f79535-47bb-0310-9956-ffa450edef68