Issue 1:
I decided to reindex my data to take advantage of doc_values, but one of 30 indices (~120m docs in each) got less documents after reindexing. I reindexed again and docs disappeared again.

Then I bisected the problem to specific docs and found that some docs in source index has duplicate ids.

curl -s "http://web245:9200/statistics-20141110/_search?pretty&q=_id:1jC2LxTjTMS1KHCn0Prf1w"

{
"took" : 1156,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "statistics-20141110",
"_type" : "events",
"_id" : "1jC2LxTjTMS1KHCn0Prf1w",
"_score" : 1.0,
"_source":{"@timestamp":"2014-11-10T14:30:00+0300","@key":"client_belarussia_msg_sended_from_mutual__22_1","@value":"149"}
}, {
"_index" : "statistics-20141110",
"_type" : "events",
"_id" : "1jC2LxTjTMS1KHCn0Prf1w",
"_score" : 1.0,
"_source":{"@timestamp":"2014-11-10T14:30:00+0300","@key":"client_belarussia_msg_sended_from_mutual__22_1","@value":"149"}
} ]
}
}

Here are two indices, source and destination:

health status index pri rep docs.count docs.deleted store.size pri.store.size
green open statistics-20141110 5 0 116217042 0 12.3gb 12.3gb
green open statistics-20141110-dv 5 1 116216507 0 32.3gb 16.1gb

Segments of problematic index:

index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound
statistics-20141110 0 p 192.168.0.190 _gga 21322 14939669 0 1.6gb 4943008 true true 4.9.0 false
statistics-20141110 0 p 192.168.0.190 _isc 24348 10913518 0 1.1gb 4101712 true true 4.9.0 false
statistics-20141110 1 p 192.168.0.245 _7i7 9727 7023269 0 766mb 2264472 true true 4.9.0 false
statistics-20141110 1 p 192.168.0.245 _i01 23329 14689581 0 1.5gb 4788872 true true 4.9.0 false
statistics-20141110 2 p 192.168.1.212 _9wx 12849 8995444 0 987.7mb 3326288 true true 4.9.0 false
statistics-20141110 2 p 192.168.1.212 _il1 24085 13205585 0 1.4gb 4343736 true true 4.9.0 false
statistics-20141110 3 p 192.168.1.212 _8pc 11280 10046395 0 1gb 4003824 true true 4.9.0 false
statistics-20141110 3 p 192.168.1.212 _hwt 23213 13226096 0 1.3gb 4287544 true true 4.9.0 false
statistics-20141110 4 p 192.168.2.88 _91i 11718 8328558 0 909.2mb 2822712 true true 4.9.0 false
statistics-20141110 4 p 192.168.2.88 _hms 22852 14848927 0 1.5gb 4777472 true true 4.9.0 false

The only thing that happened with index besides indexing is optimizing to 2 segments per shard.

Issue 2:
Elasticsearch version: 1.7.5

JVM version:

OS version: CentOS 7.2.1511

Description of the problem including expected versus actual behavior: I have been working on this for a few days now, and found similar issues, but nothing that matched what I am seeing. Every time I restart elasticsearch the shards begin to initialize until this point via _cluster/health

{

"cluster_name": "elasticsearch",

"status": "red",

"timed_out": false,

"number_of_nodes": 1,

"number_of_data_nodes": 1,

"active_primary_shards": 5458,

"active_shards": 5458,

"relocating_shards": 0,

"initializing_shards": 0,

"unassigned_shards": 5574,

"delayed_unassigned_shards": 0,

"number_of_pending_tasks": 0,

"number_of_in_flight_fetch": 0

}

I then attempt to manually reroute shards, which fails. I have run reroute?explain and receive the following:

"explanations": [

{

"command": "allocate",

"parameters": {

"index": "logstash-2016.03.11-mr",

"shard": 0,

"node": "loggy",

"allow_primary": true

},

"decisions": [

{

"decider": "same_shard",

"decision": "NO",

"explanation": "shard cannot be allocated on same node [3OedVZN9QVKU9Hw7tn2h2w] it already exists on"

},

{

"decider": "filter",

"decision": "YES",

"explanation": "node passes include/exclude/require filters"

},

{

"decider": "replica_after_primary_active",

"decision": "YES",

"explanation": "primary is already active"

},

{

"decider": "throttling",

"decision": "YES",

"explanation": "below shard recovery limit of [4]"

},

{

"decider": "enable",

"decision": "YES",

"explanation": "allocation disabling is ignored"

},

{

"decider": "disable",

"decision": "YES",

"explanation": "allocation disabling is ignored"

},

{

"decider": "awareness",

"decision": "YES",

"explanation": "no allocation awareness enabled"

},

{

"decider": "shards_limit",

"decision": "YES",

"explanation": "total shard limit disabled: [-1] <= 0"

},

{

"decider": "node_version",

"decision": "YES",

"explanation": "target node version [1.7.5] is same or newer than source node version [1.7.5]"

},

{

"decider": "disk_threshold",

"decision": "YES",

"explanation": "only a single node is present"

},

{

"decider": "snapshot_in_progress",

"decision": "YES",

"explanation": "shard not primary or relocation disabled"

}

]

}

]

I have a single node cluster and also attempted to adjust the number of replicas to 0 per another post I found, which did not resolve the issue. Physically inspecting the file system also doesn't appear to have duplicates. This started about a week ago, and as far as I know nothing major changed on the system. I do think we hit a limit as I also had to adjust the fs filemax, but this issue is still not correct after multiple attempts to restart and re-index.

I have also enabled routing allocation. My current _cluster/settings are:

{

"acknowledged": true,

"persistent": {},

"transient": {

"cluster": {

"routing": {

"allocation": {

"enable": "true"

}

}

}

}

}

Running _cat/nodes?v only returns a single node as I would expect.

_cat/allocation?v returns 20% free space:

shards disk.used disk.avail disk.total disk.percent host ip node

5458 315.2gb 77.3gb 392.6gb 80 vcosdevlog01.[SNIPPED] [X.X.X.X] loggy

5574 UNASSIGNED

Any help would be greatly appreciated!