elasticsearch/elasticsearch — pull request #7650 by areek (2014-09-09 02:53:44+00:00)
Title: NRTSuggester: Support returning StoredFields with suggestions & Optional output duplication

The ability to return field values for given storedField(s), along with returned suggestions makes the suggesters more flexible, allowing to return associated field values for every suggestion without bloating up the in-memory FST (using `payload` increases the FST size) or having to specify the fields to return at index-time.

**Implementation details:**
- Support returning associated StoredFields of suggestions
- Lucene docid is extracted from suggestion and used to get named StoredField value(s) at query-time
- User can specify arbitrary number of StoredFields to be returned
- Returned `StoredField(s)` can be accessed through the new `XLookupResult`
- Currently supports StoredField values of type: `String`, `BytesRef` and `Number`
- Optional deduplication (query time option)
- Remove exactFirst option - simplify lookup code
- Minor changes to `TopNSearcher` & how suggestions are collected
- Introduce `XLookup` interface
- Benchmarks

**Benchmark Results:**
- **data set**: LineFileDocs
- `AnalyzingSuggester` : vanilla Lucene suggester built from `InputIterator`, uses `payload` to store field value.
- `XAnalyzingSuggester` : current ES suggester built with custom postings format, uses `payload` to store field value
- `XNRTSuggester` : new NRT suggester built with custom postings format, retrieves field value from provided stored field name.
- **Summary**
- number of results returned is inversely proportional to QPS
- longer prefix lengths yields to higher QPS
- Retrieving `storedField` is slower than using `payload`, at the cost of more flexibility
```
....
-- Stored Field Retrieval Performance
-- prefixes: 2-4, num: 2
AnalyzingSuggester queries: 3472, time[ms]: 83 [+- 8.64], ~kQPS: 42
XAnalyzingSuggester queries: 3472, time[ms]: 62 [+- 1.48], ~kQPS: 56
XNRTSuggester queries: 3472, time[ms]: 174 [+- 5.04], ~kQPS: 20
-- prefixes: 2-4, num: 4
AnalyzingSuggester queries: 3472, time[ms]: 122 [+- 3.34], ~kQPS: 29
XAnalyzingSuggester queries: 3472, time[ms]: 96 [+- 2.35], ~kQPS: 36
XNRTSuggester queries: 3472, time[ms]: 307 [+- 6.29], ~kQPS: 11
-- prefixes: 2-4, num: 6
AnalyzingSuggester queries: 3472, time[ms]: 172 [+- 5.38], ~kQPS: 20
XAnalyzingSuggester queries: 3472, time[ms]: 135 [+- 4.58], ~kQPS: 26
XNRTSuggester queries: 3472, time[ms]: 425 [+- 6.98], ~kQPS: 8
-- prefixes: 3-6, num: 2
AnalyzingSuggester queries: 3472, time[ms]: 72 [+- 2.84], ~kQPS: 48
XAnalyzingSuggester queries: 3472, time[ms]: 60 [+- 3.02], ~kQPS: 58
XNRTSuggester queries: 3472, time[ms]: 169 [+- 4.06], ~kQPS: 20
-- prefixes: 3-6, num: 4
AnalyzingSuggester queries: 3472, time[ms]: 107 [+- 4.29], ~kQPS: 32
XAnalyzingSuggester queries: 3472, time[ms]: 85 [+- 3.98], ~kQPS: 41
XNRTSuggester queries: 3472, time[ms]: 266 [+- 5.58], ~kQPS: 13
-- prefixes: 3-6, num: 6
AnalyzingSuggester queries: 3472, time[ms]: 139 [+- 2.76], ~kQPS: 25
XAnalyzingSuggester queries: 3472, time[ms]: 112 [+- 3.71], ~kQPS: 31
XNRTSuggester queries: 3472, time[ms]: 350 [+- 5.82], ~kQPS: 10
-- prefixes: 100-200, num: 2
AnalyzingSuggester queries: 3472, time[ms]: 134 [+- 3.11], ~kQPS: 26
XAnalyzingSuggester queries: 3472, time[ms]: 122 [+- 3.85], ~kQPS: 28
XNRTSuggester queries: 3472, time[ms]: 191 [+- 7.18], ~kQPS: 18
-- prefixes: 100-200, num: 4
AnalyzingSuggester queries: 3472, time[ms]: 136 [+- 2.72], ~kQPS: 26
XAnalyzingSuggester queries: 3472, time[ms]: 123 [+- 3.85], ~kQPS: 28
XNRTSuggester queries: 3472, time[ms]: 191 [+- 4.50], ~kQPS: 18
-- prefixes: 100-200, num: 6
AnalyzingSuggester queries: 3472, time[ms]: 136 [+- 4.07], ~kQPS: 26
XAnalyzingSuggester queries: 3472, time[ms]: 122 [+- 4.15], ~kQPS: 28
XNRTSuggester queries: 3472, time[ms]: 191 [+- 3.81], ~kQPS: 18

- Storage benchmark
AnalyzingSuggester (with payload) size[B]: 1,198,856
XAnalyzingSuggester (with payload) size[B]: 762,592
XNRTSuggester size[B]: 774,504
...
```