Issue 1:
I have a an index with 30M documents, spread across 4 nodes, 30 shards, with 2 replicas each. Each document has a nested mapping type "a", representing a transaction log entry. Each transaction log entry has a positive or negative dollar value and a timestamp. When I attempt to bucket the nested objects using the date range and sum aggregations, the sum aggregation appears to break on transactions with both positive and negative numbers.

For example, consider the following query. I include a reverse nested agg to show that the nested objects are different than the output of the sum agg.

{
"size": 0,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"range": {
"timestamp": {
"gte": "2011-10-01"
}
}
},
{
"term": {
"bar.name": "ASpecialName"
}
}
]
}
}
}
},
"aggs": {
"BAR": {
"terms": {
"field": "bar.name",
"size": 2,
"order": {
"NESTED>TOTAL": "asc"
}
},
"aggs": {
"NESTED": {
"nested": {
"path": "a"
},
"aggs": {
"TOTAL": {
"sum": {
"field": "a.obligatedamount"
}
},
"DATES": {
"date_range": {
"field": "a.signeddate",
"keyed": true,
"ranges": [
{
"key": "FY2012",
"from": "2011-10-01",
"to": "2012-09-30"
},
{
"key": "FY2013",
"from": "2012-10-01",
"to": "2013-09-30"
},
{
"key": "FY2014",
"from": "2013-10-01",
"to": "2014-09-30"
},
{
"key": "FY2015",
"from": "2014-10-01",
"to": "2015-09-30"
}
]
},
"aggs": {
"DATEBUCKET_SUBTOTAL": {
"sum": {
"field": "a.obligatedamount"
}
},
"HITS_REVERSE": {
"reverse_nested": {},
"aggs": {
"HITS": {
"top_hits": {
"_source": {
"include": [
"a.obligatedamount",
"a.signeddate"
]
}
}
}
}
}
}
}
}
}
}
}
}
}

The following result is returned. Note the odd number in the DATEBUCKET_SUBTOTAL in FY2012 for "ASpecialPlace". This appears to only be an issue when the transaction log contains both negative and positive numbers. (Response slightly truncated...)

{
"aggregations": {
"BAR": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "ASpecialPlace",
"doc_count": 2,
"NESTED": {
"doc_count": 3,
"DATES": {
"buckets": {
"FY2012": {
"from_as_string": "2011-10-01T00:00:00.000Z",
"to_as_string": "2012-09-30T00:00:00.000Z",
"doc_count": 1,
"HITS_REVERSE": {
"doc_count": 1,
"HITS": {
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "myIndex",
"_type": "myType",
"_id": "AG9J61P110025",
"_score": 1,
"_source": {
"a": [
{
"signeddate": "2011-02-01T00:00:00+0000",
"obligatedamount": 4000
},
{
"signeddate": "2012-07-11T00:00:00+0000",
"obligatedamount": -1694
}
]
}
}
]
}
}
},
"DATEBUCKET_SUBTOTAL": {
"value": -8.365e-321
}
},
"FY2013": {
"from_as_string": "2012-10-01T00:00:00.000Z",
"to": 1380499200000,
"doc_count": 0,
"HITS_REVERSE": {
"doc_count": 0,
"HITS": {
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
},
"DATEBUCKET_SUBTOTAL": {
"value": 0
}
},
"FY2014": {
"from_as_string": "2013-10-01T00:00:00.000Z",
"to_as_string": "2014-09-30T00:00:00.000Z",
"doc_count": 1,
"HITS_REVERSE": {
"doc_count": 1,
"HITS": {
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "myIndex",
"_type": "myType",
"_id": "AG04GGP140011",
"_score": 1,
"_source": {
"a": [
{
"signeddate": "2013-12-19T00:00:00+0000",
"obligatedamount": 3449
}
]
}
}
]
}
}
},
"DATEBUCKET_SUBTOTAL": {
"value": 3449
}
},
"FY2015": {
"from_as_string": "2014-10-01T00:00:00.000Z",
"to_as_string": "2015-09-30T00:00:00.000Z",
"doc_count": 0,
"HITS_REVERSE": {
"doc_count": 0,
"HITS": {
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
},
"DATEBUCKET_SUBTOTAL": {
"value": 0
}
}
}
},
"TOTAL": {
"value": 3449
}
}
}
]
}
}
}

If I set a filter at the nested level to remove any nested "a" objects with a negative value, everything works fine. Note, this behavior is also apparent if using a stats agg instead of just a sum.

Issue 2:
Today, a new field can be added to the local mapping of two shards simultaneously. If the detected field type is different, then each shard can end up with a different field type. These shards then send their new mapping to the master, but only of the mappings will win.

This can result in incorrect results and even data loss, eg: one shard thinks that the field is a string, and the other shard (and the master) thinks that it is a number. In this case, sorting and aggregations will be wrong (#8485). Then, when the string-shard is allocated to a new node, that node will receive the "number" mapping from the master. Replaying the transaction log can cause shard failures (eg #8684). Or new numeric doc values are written but, when we try to merge segments, it fails with a doc values exception (#8009).

The only way to ensure that dynamic fields are the same on all shards is to wait for the master to acknowledge the mapping changes before indexing the document, so:

parse the document

update local mapping

if changed, send to master and wait for ack

index document

This will potentially slow down indexing when many dynamic fields are being added, but it is the only way to automatically protect against data loss due to mapping conflicts.

Should a user wish to disable waiting for the master (and they are certain that their dynamic mapping rules are good enough to prevent these problems) then we should allow them to opt-out by setting "dynamic": "unsafe"