Issue 1:
Elasticsearch version: 2.3.5

Plugins installed: [head, kopf, cloud_aws, marvel, license]

JVM version: 1.8.0_51

OS version: CentOS 6.6

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Cluster has 21 nodes. Previous version (1.7.2) worked with no issues for more than a year.

What we see is huge spike in write IOPS per node (500 writes/s before 2500 writes/s now). Around noon we have a script that reads a lot of data from this cluster and our read IOPS go high as well. We are running on AWS and the volume has only 3,000 max. With all those IOPS we hit the limit of 3,000 and at this point Elastisearch goes down.

When we look at all ES nodes and get the status of Elasticsearch service we get this: elasticsearch dead but subsys locked. When we try to start ES backup we get this:

Starting elasticsearch: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000429990000, 15408234496, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 15408234496 bytes for committing reserved memory.
An error report file with more information is saved as:
/tmp/hs_err_pid8710.log
[FAILED]

So apparently init script does not correctly report Elasticsearch status - ES is still running but it is in a bad state. So we have to kill ES manually with kill -9 (which is not fun at all on 21 servers) and only then we can start ES back online.

So with that multiple questions:

Why does new ES has so much more write IOPS? How do we prevent this?

One possibility is that auto-throttling is in place? We used to have this setting in our previous cluster:

indices.store.throttle.max_bytes_per_sec = "100mb"

So 2.3.5 apparently has auto-throttling enabled by default. Since it is impossible to delete our previous setting - does new ES just ignores it? Or how do we delete it? How can we disable auto-throttling if it causes the issue?

What is wrong with init script? We use official rpm and it seems there is a bug with init script - it does not correctly detect hung process. How can we fix that?

Thanks a lot!

Issue 2:
Hi,

looks like DocumentMapper is creating its own list of root field mappers:

...
// UID first so it will be the first stored field to load (so will benefit from "fields: []" early termination
this.rootMappers.put(UidFieldMapper.class, new UidFieldMapper());
this.rootMappers.put(IdFieldMapper.class, idFieldMapper);
this.rootMappers.put(RoutingFieldMapper.class, new RoutingFieldMapper());
// add default mappers, order is important (for example analyzer should come before the rest to set context.analyzer)
this.rootMappers.put(SizeFieldMapper.class, new SizeFieldMapper());
this.rootMappers.put(IndexFieldMapper.class, new IndexFieldMapper());
this.rootMappers.put(SourceFieldMapper.class, new SourceFieldMapper());
this.rootMappers.put(TypeFieldMapper.class, new TypeFieldMapper());
this.rootMappers.put(AnalyzerMapper.class, new AnalyzerMapper());
this.rootMappers.put(AllFieldMapper.class, new AllFieldMapper());
this.rootMappers.put(BoostFieldMapper.class, new BoostFieldMapper());
this.rootMappers.put(TimestampFieldMapper.class, new TimestampFieldMapper());
this.rootMappers.put(TTLFieldMapper.class, new TTLFieldMapper());
this.rootMappers.put(ParentFieldMapper.class, new ParentFieldMapper());
...

and DocumentMapperParser is creating its own:

...
rootTypeParsers = new MapBuilder<String, Mapper.TypeParser>()
.put(SizeFieldMapper.NAME, new SizeFieldMapper.TypeParser())
.put(IndexFieldMapper.NAME, new IndexFieldMapper.TypeParser())
.put(SourceFieldMapper.NAME, new SourceFieldMapper.TypeParser())
.put(TypeFieldMapper.NAME, new TypeFieldMapper.TypeParser())
.put(AllFieldMapper.NAME, new AllFieldMapper.TypeParser())
.put(AnalyzerMapper.NAME, new AnalyzerMapper.TypeParser())
.put(BoostFieldMapper.NAME, new BoostFieldMapper.TypeParser())
.put(ParentFieldMapper.NAME, new ParentFieldMapper.TypeParser())
.put(RoutingFieldMapper.NAME, new RoutingFieldMapper.TypeParser())
.put(TimestampFieldMapper.NAME, new TimestampFieldMapper.TypeParser())
.put(TTLFieldMapper.NAME, new TTLFieldMapper.TypeParser())
.put(UidFieldMapper.NAME, new UidFieldMapper.TypeParser())
.put(IdFieldMapper.NAME, new IdFieldMapper.TypeParser())
.immutableMap();
...

From external plugin it is possible to modify only root type parsers from DocumentMapperParser, but it is the list from DocumentMapper that is used when creating ParsedDocument instance.

BTW,

im trying to create plugin that will add support for computed fields (based on script), i wanted to borrow some 'inspiration' from how '_all' field is implemented (as it is in a way computed field). Is implementing root field mapper the right way to do it ? I tried to use field mappers (not root), but looks like they are not used unless field value is present in indexed document (which is kinda problem for computed fields, as they are never present in source document)