OpenSextant/SolrTextTagger — pull request #11 by westei (2013-09-18 13:24:12+00:00)
Title: posIncAttr and offsetAttr based support for alternate tokens and Synonyms

This pull request provides support for features as discussed by [Issue-10](https://github.com/OpenSextant/SolrTextTagger/issues/10).

Because current Solr versions do not provide correct `posInc` and `posLen`information this implementation does ignore `posLen` but uses `offset` information instead.

The main addition is the `PhraseBuilder`. This class analyses Tokens emitted by the TokenStream and creates (potentially) multiple phrases. Phrases are represented as integer arrays to be added to the FST.

The algorithm used by the PhraseBuilder is complex. I hope that the extensive comments in the code will help to understand it. The included UnitTests do also contain a lot of examples (especially of corner cases - e.g. alternate tokens removed by StopwordFilters). Running those in a debugger should also help to understand the workflow.

NOTE: that it is not possible to build phrases in all cases. Some Analyzer configurations tend to produce streams of tokens that can not be correctly processed. In such cases the PhraseBuilder will throw an UnsupportedTokenException together with some ERROR level loggings pointing the user to potential problems. Those issues could only be solved when Solr correctly provides posInc AND posLen values. This would also dramatically reduce the complexity of the PhraseBuilder class.