Issue 1:
When searching trough a few documents (1206 in that case) in an index (updated with deletes, inserts, updates from time to time), I got some duplicates or not depending on the sorting I supply.

Elasticsearch version:

2.1.0

JVM version:

openjdk version "1.8.0_66-internal"

OpenJDK Runtime Environment (build 1.8.0_66-internal-b17)

OpenJDK 64-Bit Server VM (build 25.66-b17, mixed mode)

OS version:

Linux CBISES02 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

I have an index with the following mapping:

{
"state":"open",
"settings":{
"index":{
"creation_date":"1466088694234",
"number_of_shards":"10",
"number_of_replicas":"1",
"uuid":"4tiKdB_6RWe-x2LoxDGOFg",
"version":{
"created":"2010099"
}
}
},
"mappings":{
"product":{
"_routing":{
"required":true
},
"dynamic":"false",
"_all":{
"enabled":false
},
"properties":{
"id":{
"index":"not_analyzed",
"type":"string"
},
"entityId":{
"index":"not_analyzed",
"type":"string"
},
"categories":{
"type":"integer"
},
"name":{
"type":"string",
"fields":{
"raw":{
"index":"not_analyzed",
"type":"string"
}
}
},
"routingKey":{
"index":"not_analyzed",
"type":"string"
}
(...)
}
}
},
"aliases":[
(.. some aliases)
}

Retrieving the documents using the following query returns 1206 documents, with duplicates and therefor some documents are missing. Note that re-indexing changes the number of duplicates.

{
"from": page * 3, // page from 0 to no more documents
"size": 3,
"sort": [
{
"name.raw": {
"order": "asc"
}
}
]
}

Retrieving the documents using the following query returns 1206 documents, without duplicates.

{
"from": page * 3, //page from 0 to no more documents
"size": 3,
"sort": [
{
"entityId": {
"order": "asc"
}
}
]
}

As there is a routing key, only one shard is hit at any time for both searches so the sorting should be consistent.

Expected result:

No duplicates, regardless of the sorting supplied.

Issue 2:
Elasticsearch version: 5.0.0 beta1

Plugins installed: []

JVM version:Version 8 Update 101

Steps to reproduce:

Definition of a mapping where I define 2 "testString" blocks.

{
"settings": {
"analysis": {
"analyzer": {
"default": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"asciifolding"
]
},
"lowercasedRaw": {
"type": "custom",
"tokenizer": "keyword",
"filter": [
"lowercase"
]
}
}
}
},
"mappings": {
"_default_": {
"dynamic_templates": [
/* all "test" fields have an analyzed subfield added */
{
"testString": {
"match_mapping_type": "string",
"match" : "test",
"mapping": {
"type": "string",
"analyzer": "lowercasedRaw",
"index": "analyzed",
"fields": {
"analyzed": {
"type": "string",
"index": "analyzed",
"analyzer": "default"
}
}
}
}
},
/* string fields are by default analyzed with the lowercasedRaw analyzer */
/* NOTICE the duplicate "testString" name */
{
"testString": {
"match_mapping_type": "string",
"mapping": {
"analyzer": "lowercasedRaw",
"index": "analyzed",
"type": "string"
}
}
}
]
}
}
}

First case

Add the following records (in bulk):

{"index":{"_type" : "log", "_id" : "1" } }
{"test":"testing"}
{"index":{"_type" : "log", "_id" : "2" } }
{"group":"U2"}

Get the mapping now (I removed the "default" part, because that stays the same):

{
"testing": {
"mappings": {
"log": {
"dynamic_templates": [
{
"testString": {
"match": "test",
"match_mapping_type": "string",
"mapping": {
"analyzer": "lowercasedRaw",
"fields": {
"analyzed": {
"analyzer": "default",
"index": "analyzed",
"type": "string"
}
},
"index": "analyzed",
"type": "string"
}
}
},
{
"testString": {
"match_mapping_type": "string",
"mapping": {
"analyzer": "lowercasedRaw",
"index": "analyzed",
"type": "string"
}
}
}
],
"properties": {
"group": {
"type": "text",
"analyzer": "lowercasedRaw"
},
"test": {
"type": "text",
"fields": {
"analyzed": {
"type": "text"
}
},
"analyzer": "lowercasedRaw"
}
}
}
}
}
}

Second case

Add the following records (in bulk): (these are the same records as above, but only the order is changed:

{"index":{"_type" : "log", "_id" : "2" } }
{"group":"U2"}
{"index":{"_type" : "log", "_id" : "1" } }
{"test":"testing"}

Get the mapping now (I removed the "default" part, because that stays the same):

{
"testing": {
"mappings": {
"log": {
"dynamic_templates": [
{
"testString": {
"match": "test",
"match_mapping_type": "string",
"mapping": {
"analyzer": "lowercasedRaw",
"fields": {
"analyzed": {
"analyzer": "default",
"index": "analyzed",
"type": "string"
}
},
"index": "analyzed",
"type": "string"
}
}
}
],
"properties": {
"group": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"test": {
"type": "text",
"fields": {
"analyzed": {
"type": "text"
}
},
"analyzer": "lowercasedRaw"
}
}
}
}
}
}

If you compare these 2 mappings you'll see the differences.

Notice that if you correct the original mapping and give the 2 items a different name, the mapping stays consistent, regardless the order of the records.