Issue 1:
The context suggester is returning only one document when two documents have the same input (the higher score, I guess). Also the payload is wrong.

1- create an index called "services" with suggest_field mapped as "completion" with payloads

POST services
{
"mappings": {
"service": {
"properties": {
"name": {
"type": "string"
},
"suggest_field": {
"type": "completion",
"payloads": true,
"context": {
"color": {
"type": "category"
}
}
}
}
}
}
}

2- index a document with id = 1. The input of the completion is "same_input" with weight = 100 and payload {"url": "first_document"}

PUT services/service/1
{
"name": "same_input",
"suggest_field": {
"input": "same_input",
"weight": 100,
"payload": {
"url": "first_document"
},
"context": {
"color": ["red", "yellow"]
}
}
}

3- index a document with id = 2. The input of the completion is "same_input" too with weight = 200 and payload {"url": "second_document"}

PUT services/service/2
{
"name": "same_input",
"suggest_field": {
"input": "same_input",
"weight": 200,
"payload": {
"url": "second_document"
},
"context": {
"color": ["red", "green"]
}
}
}

4- query for suggest using "same_input" as text

POST services/_suggest?pretty'
{
"suggest" : {
"text" : "same_input",
"completion" : {
"field" : "suggest_field",
"size": 10,
"context": {
"color": ["red"]
}
}
}
}

5 - the response includes only one document with the score 200, which is the weigth of the document id = 2 but with the payload of the document of id = 1 when using context "red".

{
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"suggest": [
{
"text": "same_input",
"offset": 0,
"length": 10,
"options": [
{
"text": "same_input",
"score": 200,
"payload": {
"url": "first_document"
}
}
]
}
]
}

Issue 2:
When I create multiple documents which have the same output value in the completion field, a suggest completion request only retrieves one object.

I guess it is a feature, however, once we may set different payloads to those documents, it would make sense to retrieve multiple suggestions with the same output.

My environment settings:

ElasticSearch version number: 1.0.0.Beta1

Lucene version: 4.5.1

In the following example, I create an index with two documents. Each document has a different value in the payload, but the same output and input.

Performing a suggestion completion request, only one document is retrieved.

Scripts:

Create and populate index

curl -XDELETE 'localhost:9200/notebookindex'

curl -XPUT localhost:9200/notebookindex

curl -XPUT localhost:9200/notebookindex/friend/_mapping -d '{
"friend" : {
"properties" : {
"name" : { "type" : "string" },
"suggestField" : { "type" : "completion", "payloads" : true }
}
}
}'

curl -XPUT 'localhost:9200/notebookindex/friend/1' -d ' {
"name": "james smith",
"suggestField": {
"input": ["james", "smith", "james smith"],
"output": "james smith",
"payload": {"id": "1", "phone": "555-55555"}
}
}'

curl -XPUT 'localhost:9200/notebookindex/friend/2' -d '{
"name": "james smith",
"suggestField": {
"input": ["james", "smith", "james smith"],
"output": "james smith",
"payload": {"id": "2", "phone": "444-44444"}
}
}'

Search 1: look for friends starting with j

curl -XPOST 'localhost:9200/notebookindex/_suggest?pretty' -d '{
"my-friends-suggest": {
"text": "j",
"completion": {
"field": "suggestField"
}
}
}'

Only one James is found, even though there are two documents matching the suggestion.

{
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"my-friends-suggest" : [ {
"text" : "j",
"offset" : 0,
"length" : 1,
"options" : [ {
"text" : "james smith",
"score" : 1.0, "payload" : {"id":"2","phone":"444-44444"}
} ]
} ]
}

Removal and Search 2: remove the James (id 2) previously found and look for friends starting with j

curl -XDELETE 'localhost:9200/notebookindex/friend/2'

curl -XPOST 'localhost:9200/notebookindex/_suggest?pretty' -d '{
"my-friends-suggest": {
"text": "j",
"completion": {
"field": "suggestField"
}
}
}'

The other James is found.

{
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"my-friends-suggest" : [ {
"text" : "j",
"offset" : 0,
"length" : 1,
"options" : [ {
"text" : "james smith",
"score" : 1.0, "payload" : {"id":"1","phone":"555-55555"}
} ]
} ]
}