Issue 1:
To recreate, start with a freshly started cluster (1.5.2) with nothing stored in fielddata.

GET /_cat/fielddata?v

id host ip node total
oXbVJr3_QNqRhTSkKB1WSw Olivers-MacBook-Pro.local 10.4.1.106 Crown 0b

Now create a document that contains fields with the same name as the 'meta' fields that appear in the output to fielddata.

POST /lul/wat/1
{
"id": "hah",
"host": "yeah",
"ip": 1,
"node": "lulwat",
"total": 5
}

Then search with aggregations against those fields, to load them into fielddata.

POST /lul/wat/_search
{
"query": {
"match_all": {}
},
"aggs": {
"one": {
"terms": {
"field": "ip"
}
},
"two": {
"terms": {
"field": "node"
}
}
}
}

Now check the _cat/fielddata endpoint again.

GET /_cat/fielddata?v

id host ip node total node ip
Bt6uoLNcS0mlVSdY1xGgSA Olivers-MacBook-Pro.local 10.4.1.106 Crown 8.9kb Crown 10.4.1.106

Note that node and ip are both repeated, but we can't actually see the usage for those fields - instead we get the metadata again. For the sake of completeness...

POST /lul/wat/_search
{
"query": {
"match_all": {}
},
"aggs": {
"one": {
"terms": {
"field": "ip"
}
},
"two": {
"terms": {
"field": "node"
}
},
"three": {
"terms": {
"field": "host"
}
},
"four": {
"terms": {
"field": "id"
}
},
"five": {
"terms": {
"field": "total"
}
}
}
}

GET /_cat/fielddata?v

id host ip node total node total ip host id
oXbVJr3_QNqRhTSkKB1WSw Olivers-MacBook-Pro.local 10.4.1.106 Crown 18.4kb Crown 18.4kb 10.4.1.106 Olivers-MacBook-Pro.local oXbVJr3_QNqRhTSkKB1WSw

Note that locally I can recreate the field duplication, but in our production environment (1.4.0) we see even stranger things:

id host ip node total content.source name.raw delivery.id timestamp id _type sections _id created delivery.channel content.id campaign published content.sections content.topics ip referrer topics content.created source title.raw p property.type _parent property.id
NcxVMomeTC6TOTEVH9lt3Q [REDACTED] [REDACTED] prd-elastic-c 2.7gb 16mb 0b 805.9kb 172.3mb NcxVMomeTC6TOTEVH9lt3Q 32.8mb 9.9mb 52.2mb 34.4mb 491.3kb 116.7mb 82.5mb 3mb 113.5mb 1.5gb [REDACTED] 110.8mb 29.9mb 31mb 10.6mb 111kb 400.7mb 26.8mb 741.5mb 1.3mb
152.6kb [REDACTED] 0b prd-elastic-a 1.5gb 6.4mb 0b 645.1kb 86.7mb 152.6kb 17.8mb 4.7mb 36.1mb 24mb 458.4kb 79.1mb 48.3mb 1.9mb 75.1mb 859.9mb 0b 84.3mb 12.7mb 0b 4.4mb 836.9kb 197.5mb 11mb 493.9mb 861.5kb
CqC9hO3tT2O_j2gMPrc2Uw [REDACTED] [REDACTED] prd-elastic-f 3.1gb 23.1mb 0b 865.4kb 160.3mb CqC9hO3tT2O_j2gMPrc2Uw 18.1mb 8.5mb 49.3mb 35.2mb 559.1kb 135.2mb 80.2mb 3mb 128.7mb 1.9gb [REDACTED] 124.1mb 22.8mb 35.3mb 8.1mb 577.3kb 368.8mb 22.8mb 671.8mb 1.2mb
5.6kb [REDACTED] 0b prd-elastic-d 3gb 18.7mb 0b 1.2mb 159.9mb 5.6kb 12.5mb 9.2mb 51.3mb 38mb 769.9kb 134.1mb 82mb 3.6mb 132.4mb 1.7gb 0b 141mb 25.7mb 58.8mb 9.5mb 399.7kb 408.7mb 23mb 709.8mb 1.3mb
WkXcjwj8T6yQ1X2Pb_M2ow [REDACTED] [REDACTED] prd-elastic-b 1.9gb 15.8mb 0b 1.1mb 87.3mb WkXcjwj8T6yQ1X2Pb_M2ow 0b 5.1mb 34.5mb 35.9mb 593.9kb 86.5mb 60.7mb 1.9mb 79.1mb 1.1gb [REDACTED] 87.7mb 19.2mb 31mb 7.2mb 110.6kb 202.9mb 20.3mb 505.8mb 1.2mb
116.6kb [REDACTED] 38.2mb prd-elastic-e 3.5gb 25.9mb 21.6kb 1mb 204.5mb 116.6kb 0b 11.1mb 57.2mb 34mb 724.2kb 161.6mb 90.7mb 3.6mb 152.8mb 2.1gb 38.2mb 150mb 31.5mb 29.9mb 10.2mb 615.9kb 368.8mb 32.8mb 764.5mb 1.7mb

ip and id are both repeated, but in the output from some nodes the id and ip fields display the fielddata size for the fields, and some of the nodes display the metadata as I guess they should. I haven't been able to reproduce this locally (it might have been fixed since 1.4.0?), nor have I haven't yet been able to find any pattern to which nodes display fielddata sizes and which display the metadata.

Thanks!

Issue 2:
We should explore the idea of disabling the _all field by default. For

background, the _all field contains the contents of each of a document's

field. During indexing, these contents are copied to the _all field, analyzed

with a specific analyzer, and then indexed. At query time, some queries such as

the query_string and simple_query_string queries search the _all field if

no fields are specified.

There are a number of issues with the _all field:

It uses a fair amount of disk space, due to duplicating the values from all

other fields to an additional field

_all has its own analyzer, which is confusing when expecting to query

text analyzed a certain way (or with synonyms, for example) and discovering

that it does not match due to a analysis difference

Additional indexing overhead caused by data duplication

Since the _all field is not retrievable or part of _source, its contents

cannot easily be inspected for debugging purposes, some users do not know it

even exists, which causes confusion at query time

Better alternatives exist, such as the copy_to value on mappings which can

be used to create custom _all fields

If we were to change the _all field to be disabled by default, some queries

would have to be handled differently. The query_string and

simple_query_string queries would have to know a default field or fields to

query. Perhaps we would be able to change these queries to send a "fields": ["*"] parameter if no fields are specified in the JSON query, so that all field

values can still be queried.