Issue 1:
Elasticsearch version: 2.2.0

JVM version: 1.8.0_65

OS version: Ubuntu 15.04

Description of the problem including expected versus actual behavior: When hitting any Cat APIs that deal with the list of nodes, the ordering of the nodes is non-deterministic, so depending on which node handles the request, the ordering will be different.

The following APIs are affected:

_cat/nodes

_cat/allocation

_cat/nodeattrs

I'm inclined to raise this as a bug, since the ordering used to be deterministic in 1.7.

Steps to reproduce:

Setup a cluster with at least two nodes

Hit _cat/nodes and note the ordering of entries

Hit _cat/nodes on a different node in the cluster and note the ordering is different

Issue 2:
Opening a new issue as requested by @brwe in the comments to #8788.

We recently saw an unexpectedly high count on an index we're feeding. It's the second time we've seen this; the first time was only off by 1, but this time it was off by over 16,000 in an index that should contain less than 300k docs. Since the dataset is so small I pulled down everything (_search?pretty=true&fields=&size=400000), scraped out the ids and ran them through uniq, which revealed a ton of duplicates.

Notes/observations:

We're using 1.3.4, doing bulk indexing with the Java API's TransportClient and BulkProcessor.

We're not using autogenerated ids, so it doesn't look as if the fix for #8788 is likely to help us.

No id appeared more than twice.

The index has not been migrated from an earlier version. In fact, the index is deleted and recreated irregularly but fairly frequently, from every few days to every couple of weeks.

Around the time the duplicates appeared, we saw problems in other (non-Elastic) parts of the system. I can't see any way that they could directly cause the duplication, but it's possible that network issues were the common cause of both.

Running

curl -s 'http://HOST:9200/std_physicalasset_alpha_a/_search?pretty&q=_id:77309419847&explain&fields=_source,_routing' -d '{"version":true}'

produced

{
"took" : 247,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_shard" : 3,
"_node" : "HRLe1R1iTzqzZ44ocSb4Sw",
"_index" : "std_physicalasset_alpha_a",
"_type" : "other",
"_id" : "77309419847",
"_version" : 2,
"_score" : 1.0,
"_source":{"DEBUGRUN":"2015-04-03T20:48:55.327Z"},
"_explanation" : {
"value" : 1.0,
"description" : "ConstantScore(_uid:_default_#77309419847 _uid:generic#77309419847 _uid:other#77309419847 _uid:plant#77309419847 _uid:vessel#77309419847), product of:",
"details" : [ {
"value" : 1.0,
"description" : "boost"
}, {
"value" : 1.0,
"description" : "queryNorm"
} ]
}
}, {
"_shard" : 3,
"_node" : "HRLe1R1iTzqzZ44ocSb4Sw",
"_index" : "std_physicalasset_alpha_a",
"_type" : "vessel",
"_id" : "77309419847",
"_version" : 456,
"_score" : 1.0,
"_source":{"DEBUGRUN":"2015-04-08T16:12:52.903Z"},
"_explanation" : {
"value" : 1.0,
"description" : "ConstantScore(_uid:_default_#77309419847 _uid:generic#77309419847 _uid:other#77309419847 _uid:plant#77309419847 _uid:vessel#77309419847), product of:",
"details" : [ {
"value" : 1.0,
"description" : "boost"
}, {
"value" : 1.0,
"description" : "queryNorm"
} ]
}
} ]
}
}

I've removed most of _source as verbose and not relevant, but DEBUGBATCH is interesting; it's a manually-set human-readable equivalent of _timestamp, and shows that the two documents with this id were loaded almost 5 days apart. The second has been updated again since I captured that, although the feeder process has now been stopped.

curl -s 'http://HOST:9200/std_physicalasset_alpha_a/_segments?pretty'

produces around 100k of JSON, which seems a bit much to post in full. (That may be telling in itself; the full response contained a suspiciously-round 255 segment objects.) The following is just the portion for the shard and (non-primary) node containing the sample duplicate id above:

{
"_shards" : {
"total" : 10,
"successful" : 10,
"failed" : 0
},
"indices" : {
"std_physicalasset_alpha_a" : {
"shards" : {
====================
# Shards 0-2 omitted
====================
"3" : [ {
"routing" : {
"state" : "STARTED",
"primary" : false,
"node" : "HRLe1R1iTzqzZ44ocSb4Sw"
},
"num_committed_segments" : 26,
"num_search_segments" : 16,
"segments" : {
"_2a5z" : {
"generation" : 106487,
"num_docs" : 43857,
"deleted_docs" : 15215,
"size_in_bytes" : 341737818,
"memory_in_bytes" : 2035824,
"committed" : true,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2a76" : {
"generation" : 106530,
"num_docs" : 1197,
"deleted_docs" : 31,
"size_in_bytes" : 9592426,
"memory_in_bytes" : 418896,
"committed" : true,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2a7t" : {
"generation" : 106553,
"num_docs" : 1382,
"deleted_docs" : 35,
"size_in_bytes" : 11067533,
"memory_in_bytes" : 434208,
"committed" : true,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2a82" : {
"generation" : 106562,
"num_docs" : 1629,
"deleted_docs" : 48,
"size_in_bytes" : 12533722,
"memory_in_bytes" : 443952,
"committed" : true,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2a8n" : {
"generation" : 106583,
"num_docs" : 2070,
"deleted_docs" : 61,
"size_in_bytes" : 15764916,
"memory_in_bytes" : 447088,
"committed" : true,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2a9e" : {
"generation" : 106610,
"num_docs" : 1480,
"deleted_docs" : 28,
"size_in_bytes" : 11471539,
"memory_in_bytes" : 435432,
"committed" : true,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2a9m" : {
"generation" : 106618,
"num_docs" : 151,
"deleted_docs" : 3,
"size_in_bytes" : 1529856,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2a9n" : {
"generation" : 106619,
"num_docs" : 6528,
"deleted_docs" : 145,
"size_in_bytes" : 45294050,
"memory_in_bytes" : 632040,
"committed" : true,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2a9q" : {
"generation" : 106622,
"num_docs" : 186,
"deleted_docs" : 4,
"size_in_bytes" : 1872124,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2a9t" : {
"generation" : 106625,
"num_docs" : 220,
"deleted_docs" : 9,
"size_in_bytes" : 2276799,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2a9w" : {
"generation" : 106628,
"num_docs" : 654,
"deleted_docs" : 18,
"size_in_bytes" : 5614691,
"memory_in_bytes" : 368520,
"committed" : true,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2aaf" : {
"generation" : 106647,
"num_docs" : 101,
"deleted_docs" : 80,
"size_in_bytes" : 1858120,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2aaj" : {
"generation" : 106651,
"num_docs" : 499,
"deleted_docs" : 269,
"size_in_bytes" : 6309352,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : false
},
"_2aaq" : {
"generation" : 106658,
"num_docs" : 609,
"deleted_docs" : 207,
"size_in_bytes" : 6785525,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : false
},
"_2aas" : {
"generation" : 106660,
"num_docs" : 663,
"deleted_docs" : 225,
"size_in_bytes" : 7485334,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : false
},
"_2aat" : {
"generation" : 106661,
"num_docs" : 106,
"deleted_docs" : 0,
"size_in_bytes" : 1942039,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2aau" : {
"generation" : 106662,
"num_docs" : 123,
"deleted_docs" : 0,
"size_in_bytes" : 2226032,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2aaw" : {
"generation" : 106664,
"num_docs" : 75,
"deleted_docs" : 0,
"size_in_bytes" : 1489187,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2aaz" : {
"generation" : 106667,
"num_docs" : 93,
"deleted_docs" : 0,
"size_in_bytes" : 1683929,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2ab0" : {
"generation" : 106668,
"num_docs" : 74,
"deleted_docs" : 0,
"size_in_bytes" : 1368605,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2ab5" : {
"generation" : 106673,
"num_docs" : 68,
"deleted_docs" : 0,
"size_in_bytes" : 1346587,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2ab8" : {
"generation" : 106676,
"num_docs" : 31,
"deleted_docs" : 0,
"size_in_bytes" : 690748,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2ab9" : {
"generation" : 106677,
"num_docs" : 54,
"deleted_docs" : 0,
"size_in_bytes" : 1086051,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2aba" : {
"generation" : 106678,
"num_docs" : 121,
"deleted_docs" : 0,
"size_in_bytes" : 2119135,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2abb" : {
"generation" : 106679,
"num_docs" : 473,
"deleted_docs" : 0,
"size_in_bytes" : 6822800,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : false
},
"_2abc" : {
"generation" : 106680,
"num_docs" : 714,
"deleted_docs" : 388,
"size_in_bytes" : 12332118,
"memory_in_bytes" : 417680,
"committed" : false,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2abd" : {
"generation" : 106681,
"num_docs" : 1,
"deleted_docs" : 0,
"size_in_bytes" : 49613,
"memory_in_bytes" : 0,
"committed" : true,
"search" : false,
"version" : "4.9",
"compound" : true
},
"_2acb" : {
"generation" : 106715,
"num_docs" : 17,
"deleted_docs" : 0,
"size_in_bytes" : 398023,
"memory_in_bytes" : 195496,
"committed" : false,
"search" : true,
"version" : "4.9",
"compound" : true
},
"_2acc" : {
"generation" : 106716,
"num_docs" : 36,
"deleted_docs" : 0,
"size_in_bytes" : 756378,
"memory_in_bytes" : 228552,
"committed" : false,
"search" : true,
"version" : "4.9",
"compound" : true
},
"_2acd" : {
"generation" : 106717,
"num_docs" : 13,
"deleted_docs" : 0,
"size_in_bytes" : 316552,
"memory_in_bytes" : 196488,
"committed" : false,
"search" : true,
"version" : "4.9",
"compound" : true
},
"_2ace" : {
"generation" : 106718,
"num_docs" : 17,
"deleted_docs" : 0,
"size_in_bytes" : 354373,
"memory_in_bytes" : 189920,
"committed" : false,
"search" : true,
"version" : "4.9",
"compound" : true
},
"_2acf" : {
"generation" : 106719,
"num_docs" : 2118,
"deleted_docs" : 96,
"size_in_bytes" : 23788950,
"memory_in_bytes" : 490448,
"committed" : false,
"search" : true,
"version" : "4.9",
"compound" : false
},
"_2acg" : {
"generation" : 106720,
"num_docs" : 24,
"deleted_docs" : 0,
"size_in_bytes" : 516210,
"memory_in_bytes" : 219592,
"committed" : false,
"search" : true,
"version" : "4.9",
"compound" : true
},
"_2ach" : {
"generation" : 106721,
"num_docs" : 67,
"deleted_docs" : 0,
"size_in_bytes" : 1265169,
"memory_in_bytes" : 236688,
"committed" : false,
"search" : true,
"version" : "4.9",
"compound" : true
}
}
}, {
"routing" : {
"state" : "STARTED",
"primary" : true,
"node" : "oyFoYqV1SHyLoJmOzRQrDQ"
},
"num_committed_segments" : 14,
"num_search_segments" : 13,
"segments" : {
==========================
20 segment objects omitted
==========================
} ],
===============
Shard 4 omitted
===============
}
}
}
}

It's highly unlikely that I'll be able to make the ES server logs available in full, but I can grep them if there's anything specific that might help. I can't find any mention of the std_physicalasset_alpha_a index in them except for a bunch of entries of the form

[2015-04-05 22:20:58,598][INFO ][index.engine.internal ] [c115knppaes01-01] [std_physicalasset_alpha_a][2] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2015-04-05 22:20:59,216][INFO ][index.engine.internal ] [c115knppaes01-01] [std_physicalasset_alpha_a][2] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5