elasticsearch/elasticsearch — pull request #7170 by areek (2014-08-06 02:46:18+00:00)
Title: Completion & Context Suggester: Support real-time deleted document filtering for suggestions

Idea:
- encode lucene docids to corresponding FST outputs
- make sure no surface form is lost even though there may be exact duplicates
- use encoded docids to perform real-time deleted doc filtering

Implementation & Considerations:
- Currently the FST BytesRef Output is in the format:
`surface_form` + `PAYLOAD_SEP` + `payload` + `PAYLOAD_SEP` + `docID`
- Duplicate surface forms are stored uniquely with dedup bytes. (uses the `END_BYTE` before the dedup bytes, maybe we can support `exact_first` in the future)
- `maxAnalyzedPathsForOneInput` is now dynamically set by taking into account the # of surfaces per analyzed form (to deepen the `TopNSearcher` queue)
- This implementation modifies the `XAnalyzingSuggester` further, I think it is a good idea to make lucene's `AnalyzingSuggester` more pluggable instead in the future.

closes #7133