My original intension for allowing `posInc == 0` in the query analyzer was to allow SolrTextTagger to work on fields that are configured for normal text search. However it also turned out that this is required for Japanese language support using Kuromoji as this analyzer does insert tokens with `posInc == 0` (see also the example given by [Lucene's TokenStreams are actually graphs!](http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html))

In nearly all cases having tokens with `posInc == 0` is not an issue. There are some cases with the WordDelimiterFilter such as a field define keepOriginal=0 at index time and keepOriginal=1 at query time: With this an Entity with an label modified at index time would not be tagged in a text containing exactly the indexed label. This is because the original token will be emitted first at query time but was not included in the FST. However I do not consider this as a problem as also normal Solr queries are affected by bad FIeld configuration.

Regarding `offsetAtt.startOffset() < lastEndOffset`: Not sure why I added this, but having this is problematic! E.g. the WordDelimiterFilter does not modify offsets if the lengths of the term does not correspond to the length defined by the offset attribute. So an analyzer that first contains a filter that removes `'s` and after that splits `Wi-Fi` to `Wi`and `Fi` would not work as this would results on only looking up the `Wi` token and ignoring `Fi` - as both `Wi` and `Fi` will have overlapping offsets.

to conclude: Having support for `posInc == 0` is necessary to support Kuromoji and we will need to remove the overlapping offset check. I will look into the later. Its a pity that I con no longer remember why I added this in the beginning :(