OpenSextant/SolrTextTagger — pull request #9 by westei (2013-08-21 12:11:48+00:00)
Title: Support for multi-valued fields; Handling of missing termId's

Sorry for changing two things in one pull request, but after vacation I was no longer aware of the termId change and so those two things got mixed together in a single commit :(

### (A) Support for multi-valued fields:

While building a FST over all entities in Freebase I noticed that I was not able to find the Freebase Entity for ["BBC"](https://www.freebase.com/m/0g5lhl7) even that I have indexed the data in a way that I had both "BBC" and "British Broadcasting Corporation" as labels for this entity. After some debugging I recognized that TaggerFstCorpus only retrieves a single value for the configured `storedFieldName`. So in case the corpus does use multiple valued fields (like in my case) only a single one will get added into the FST.

This is fixed by this pull request by using `IndexableField[] storedFields = document.getFields(storedFieldName);`and iterating over possible multiple stored values.

### (B) Dealing with missing Term IDs in the inverted index

While building FST models for several languages of Freebase I was running into exceptions like`Couldn't lookup term TEXT=... TERM=...`

One example I investigated further was

TEXT=Einladung zur ACID-HOUSE-IH-WI?T-WO-Party TERM=wi?t

This specific case is related to the German label of https://www.freebase.com/m/03q6lyt.
However note that the term 'WIẞT' in this label does not use the normal 'ß' as typically used in the German language, but an upper case version that was only very recently added to Unicode (see [Wikipedia: Capital_ẞ](http://en.wikipedia.org/wiki/Capital_%E1%BA%9E) for details).

I checked also a some other examples and for all cases it was related to very uncommon Letters in Terms.

As throwing a IllegalStateException for such cases would prevent to create a FST for the whole corpus I decided to change this to a WARN level logging and just to skip missing termIds when building the FST.