Issue 1:
Overview

The main features of the new Completion Suggester are:

Near real-time - ensures no deleted docs are ever suggested (addresses #7761), allows returning arbitrary fields as payload

Flexible scoring - exposed through the new geo scorer (addresses #6444 (comment), #4759)

Flexible filtering with categories - suggestions can be associated with arbitrary categories, AND and OR operators on categories can be used at query-time. (addreses #6444)

Apart from the features above, the new suggester will be able to return multiple outputs matching an input upon configuration (duplicate_output) (addresses #8129, #7641, #4255)

The new Completion Suggester will be using NRTSuggester instead of the previous XAnalyzingSuggester.

Sample use-cases:

Filtering with categories - get completion for some prefix for all or any provided categories

Flexible scoring (geo) - get completion for some prefix for all or any provided categories sorted by the distance (between provided score_context and completion weight).

Usage

NOTE: The API presented is experimental and may change.

Mapping

curl -X PUT localhost:9200/<index>/<type>/_mapping -d '{
"<type>": {
"properties": {
"name": {
"type": "string"
},
"suggest": {
"type": "completion",
"index_analyzer": "..",
"search_analyzer": “..”,
"preserve_separators": true | false,
"preserve_position_increments": true | false,
"duplicate_output" : true | false, (default: false)
"has_categories": true | false, (default: false)
"scorer": "default" | "geo" (default: "default")
}
}
}
}'

Indexing

Indexing with categories and default scorer

curl -X PUT 'localhost:9200/music/song/1?refresh=true' -d '{
"name": "completion_with_default_scorer",
"suggest": {
"categories" : ["rock", "grunge"]
"input": [ "Nevermind", "Nirvana" ],
"output": "Nirvana - Nevermind",
"weight": 34
}
}'

Indexing with categories and geo scorer

curl -X PUT 'localhost:9200/places/food/1?refresh=true' -d '{
"name": "completion_with_geo_scorer",
"suggest": {
"categories" : ["cafe", "snacks"]
"input": [ "timmis", "timhorton" ],
"output": "Tim Hortons",
"weight": {
"lat": ...,
"lon": ...
}
}
}'

Indexing with no categories and default scorer

same as the current Completion Suggester

Querying

Note: payload_template can have references to one or more fields that will be retrieved at query-time.

Querying on suggesters with categories and default scorer

Query all completions associated with category "rock" with payload

curl -X POST 'localhost:9200/music/_suggest?pretty' -d '{
"song-suggest" : {
"text" : "nir",
"categories" : ["rock"],
"payload_template" : "attr1: {{field1Name}}, attr2: {{field2Name}}"
"completion" : {
"field" : "suggest"
}
}
}'
{
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"song-suggest" : [ {
"text" : "nir",
"offset" : 0,
"length" : 4,
"options" : [ {
"text" : "Nirvana - Nevermind",
"score" : 34.0,
"categories" : ["rock"],
"payload" : "attr1: field1Value, attr2: field2Value"
} ]
} ]
}

Query all completions which are either associated with rock category or metal category.

Note: category_operation defaults to "AND"

curl -X POST 'localhost:9200/music/_suggest?pretty' -d '{
"song-suggest" : {
"text" : "nir",
"categories" : ["rock", "metal"],
"category_operation": "OR",
"completion" : {
"field" : "suggest"
}
}
}'

Querying on suggesters with categories and geo scorer

Query all completions which are associated with category "cafe" with some score_context (e.g. current location). The result will be sorted by the distance of the geo score for the completion w.r.t. provided score_context

curl -X POST 'localhost:9200/places/_suggest?pretty' -d '{
"place-suggest" : {
"text" : "tim",
"categories" : ["cafe"],
"score_context" : { "lat": ..., "lon": ... }
"payload_template" : "attr1: {{field1Name}}, attr2: {{field2Name}}"
"completion" : {
"field" : "suggest"
}
}
}'
{
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"place-suggest" : [ {
"text" : "tim",
"offset" : 0,
"length" : 4,
"options" : [ {
"text" : "Tim Hortons",
"score" : <Distance between `score_context` and `completion score`>,
"categories" : ["cafe"],
"payload" : "attr1: field1Value, attr2: field2Value"
} ]
} ]
}

Querying on suggesters with no categories and default scorer

same as the current Completion Suggester

NRTSuggester

The new Completion Suggester will be using a new NRTSuggester. Currently the NRTSuggester is under development and is planned to support at least the functionalities described so far.

In the second phase, the following will be considered:

enable 'fuzziness' (edit distance) to influence suggestion scoring (addresses #4441)

[TENTATIVE] experiment with enabling filter support (addresses #8096)

Migration plan

New Completion Suggester will use a new completion index format under the hood, so some work needs to be done to upgrade from existing completion suggester to the new one. The migration strategy still needs to be decided.

Issue 2:
org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException

at org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:99)

at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)

Caused by: java.lang.NullPointerException

at org.elasticsearch.search.suggest.completion.AnalyzingCompletionLookupProvider$1$1.finish(AnalyzingCompletionLookupProvider.java:134)

at org.elasticsearch.search.suggest.completion.Completion090PostingsFormat$SuggestFieldsConsumer$1.finish(Completion090PostingsFormat.java:152)

at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:204)

at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)

at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)

at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)

at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)

at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)

at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)

at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)

at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

As always, let me know if there's anything more I can provide to help debug.