OpenSextant/SolrTextTagger — pull request #7 by westei (2013-07-05 09:29:45+00:00)
Title: Adds a LookupAttribute used by the Tagger to lookup only selected tokens in the FST Corpus

This adds the "LookupAttribute" used by the Tagger to decide if a Token consumed via TokenStream#incrementToken() needs to be looked up in the FST corpus. Analyzer chains may define a TokenFilter component that does set this attribute to TRUE or FALSE. Typically this will be done based on NLP (Natural Language Processing) results.

The Tagger uses the LookupAttribute as follows:

* Tags are only created for Tokens where the LookupAttribute is TRUE
* Advancing tags are continued regardless of the LookupAttribute value

### Tagger API change

The Tagger no longer takes an Analyzer and a Reader but directly the TokenStream. This was necessary to allow to modify (extend) the TokenStream of the `indexField` before parsing it to the Tokenizer.

Analyzer analyzer = req.getSchema().getField(indexedField).getType().getAnalyzer();
//get the TokenStream from the Analyzer
TokenStream baseStream = analyzer.tokenStream("", reader);
//add a FilterStream that sets the LookupAttribute to the end
TokenStream filterStream = new WordLengthLookupFilter(baseStream);
//create the Tagger using the modified analyzer chain.
new Tagger(corpus, filterStream, tagClusterReducer) { /* implement */ }.process();

See also the WordLengthLookupFilter.java and the full example in the class level java doc.

### Further considerations / open issues

Currently the usage requires to modify the Analyzer as defined by the `indexedField`. This is completely fine for my use case, as I do not run this within the SolrServer as request handler. I do have direct access to the SolrCore and can therefore directly inject a TokenFilter that sets the LookupAttribute based on NLP results.

When running within a SolrServer users would need to configure a Analyzer chain that includes this NLP functionality (e.g. [Solr OpenNLP integration](http://wiki.apache.org/solr/OpenNLP). NOTE: that using the OpenNLP analyzers would also require to implement a TokenFilter that sets the LookupAttribute accordingly.

As NLP processing is not useful for short label texts using the 'indexField' Analyzer also for the Tagger seams not the best option in such use cases.

Because of that I think it would be best to add an optional configuration to the SolrTextTagger that allows users to specify a seperate fieldType (defined in the schema.xml) that is used by the Tagger. If defined this would be used instead of the Analyzer `indexField`.

The downside of this option is that users need to be aware that the Analyzer chain used by the Tagger needs to be compatible with the one of the `indexField`. Especially lowercase filters and stemmer used by the `indexField` need also to be present in the Analyzer used by the Tagger. So the [example analyzer chain](http://wiki.apache.org/solr/OpenNLP#Full_Example) would not work with the Tagger, but would need to be adapted to be similar to the chain of the `indexField`