Issue 1:
Our cluster reached OOM and one node is recovering very slowly after restart.

It has a lot of data locally, but doesn't seem to care about it to restore locally:

web592 ~ # du -ms /var/lib/elasticsearch/statistics/nodes/*
1292882 /var/lib/elasticsearch/statistics/nodes/0

iostat shows that disks are mostly idle. Cluster is on 1.4.1 with aws plugin for s3 backups. Other 4 nodes recovered their state much more quickly and now cluster is rebalancing.

[2014-12-01 19:04:45,186][INFO ][node ] [statistics02] version[1.4.1], pid[87174], build[89d3241/2014-11-26T15:49:29Z]
[2014-12-01 19:04:45,186][INFO ][node ] [statistics02] initializing ...
[2014-12-01 19:04:45,197][INFO ][plugins ] [statistics02] loaded [cloud-aws], sites []
[2014-12-01 19:04:46,729][DEBUG][discovery.zen.elect ] [statistics02] using minimum_master_nodes [3]
[2014-12-01 19:04:46,730][DEBUG][discovery.zen.ping.multicast] [statistics02] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2014-12-01 19:04:46,733][DEBUG][discovery.zen.ping.unicast] [statistics02] using initial hosts [statistics01.es.pretender.local, statistics02.es.pretender.local, statistics03.es.pretender.local, statistics04.es.pretender.local, statistics05.es.pretender.local], with concurrent_connects [10]
[2014-12-01 19:04:46,743][DEBUG][discovery.zen ] [statistics02] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false]
[2014-12-01 19:04:46,744][DEBUG][discovery.zen.fd ] [statistics02] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2014-12-01 19:04:46,746][DEBUG][discovery.zen.fd ] [statistics02] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2014-12-01 19:04:47,024][DEBUG][gateway.local ] [statistics02] using initial_shards [quorum], list_timeout [30s]
[2014-12-01 19:04:47,026][DEBUG][indices.recovery ] [statistics02] using max_bytes_per_sec[200mb], concurrent_streams [5], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true]
[2014-12-01 19:04:47,088][DEBUG][gateway.local.state.meta ] [statistics02] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
[2014-12-01 19:04:47,399][DEBUG][gateway.local.state.meta ] [statistics02] took 310ms to load state
[2014-12-01 19:04:49,314][DEBUG][gateway.local.state.shards] [statistics02] took 1.9s to load started shards state
[2014-12-01 19:04:49,324][INFO ][node ] [statistics02] initialized
[2014-12-01 19:04:49,324][INFO ][node ] [statistics02] starting ...
[2014-12-01 19:04:49,431][INFO ][transport ] [statistics02] bound_address {inet[/192.168.2.82:9300]}, publish_address {inet[/192.168.2.82:9300]}
[2014-12-01 19:04:49,442][INFO ][discovery ] [statistics02] statistics/putJb40cQsSRK8HLRaXcBQ
[2014-12-01 19:04:52,459][DEBUG][discovery.zen ] [statistics02] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], id[60], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
--> ping_response{node [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], id[69], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
--> ping_response{node [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], id[140], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
--> ping_response{node [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], id[12], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
[2014-12-01 19:05:02,432][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24539
[2014-12-01 19:05:02,434][DEBUG][discovery.zen.fd ] [statistics02] [master] restarting fault detection against master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], reason [new cluster state received and we are monitoring the wrong master [null]]
[2014-12-01 19:05:02,435][DEBUG][discovery.zen ] [statistics02] got first state from fresh master [4OG7n-hnTCGUs-81MDxTwQ]
[2014-12-01 19:05:02,435][INFO ][cluster.service ] [statistics02] detected_master [statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]], added {[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]],[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]],[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]],[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]],}, reason: zen-disco-receive(from master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]])
[2014-12-01 19:05:02,461][INFO ][cluster.routing.allocation.decider] [statistics02] updating [cluster.routing.allocation.disk.watermark.low] to [90%]
[2014-12-01 19:05:02,461][INFO ][cluster.routing.allocation.decider] [statistics02] updating [cluster.routing.allocation.disk.watermark.high] to [95%]
[2014-12-01 19:05:03,298][INFO ][http ] [statistics02] bound_address {inet[/192.168.2.82:9200]}, publish_address {inet[/192.168.2.82:9200]}
[2014-12-01 19:05:03,298][INFO ][node ] [statistics02] started
[2014-12-01 19:05:12,238][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24540
[2014-12-01 19:05:12,608][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][4] starting recovery from local ...
[2014-12-01 19:05:12,618][DEBUG][index.gateway ] [statistics02] [statistics-20131231][2] starting recovery from local ...
[2014-12-01 19:05:12,632][DEBUG][index.gateway ] [statistics02] [statistics-20140505][2] starting recovery from local ...
[2014-12-01 19:05:12,646][DEBUG][index.gateway ] [statistics02] [statistics-20140726][4] starting recovery from local ...
[2014-12-01 19:05:12,651][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][4] recovery completed from [local], took [43ms]
[2014-12-01 19:05:12,740][DEBUG][index.gateway ] [statistics02] [statistics-20140505][2] recovery completed from [local], took [108ms]
[2014-12-01 19:05:12,740][DEBUG][index.gateway ] [statistics02] [statistics-20131231][2] recovery completed from [local], took [122ms]
[2014-12-01 19:05:12,767][DEBUG][index.gateway ] [statistics02] [statistics-20140726][4] recovery completed from [local], took [121ms]
[2014-12-01 19:05:18,378][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24541
[2014-12-01 19:05:18,421][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][0] starting recovery from local ...
[2014-12-01 19:05:18,424][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][0] recovery completed from [local], took [3ms]
[2014-12-01 19:05:18,430][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][3] starting recovery from local ...
[2014-12-01 19:05:18,433][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][3] recovery completed from [local], took [3ms]
[2014-12-01 19:05:18,441][DEBUG][index.gateway ] [statistics02] [statistics-20140816][2] starting recovery from local ...
[2014-12-01 19:05:18,505][DEBUG][index.gateway ] [statistics02] [statistics-20140816][2] recovery completed from [local], took [64ms]
[2014-12-01 19:05:24,440][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24542
[2014-12-01 19:05:24,472][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][2] starting recovery from local ...
[2014-12-01 19:05:24,475][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][2] recovery completed from [local], took [3ms]
[2014-12-01 19:05:24,481][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][1] starting recovery from local ...
[2014-12-01 19:05:24,484][DEBUG][index.gateway ] [statistics02] [statistics-20141101-dv][1] recovery completed from [local], took [3ms]
[2014-12-01 19:05:24,490][DEBUG][index.gateway ] [statistics02] [statistics-20140108][2] starting recovery from local ...
[2014-12-01 19:05:24,528][DEBUG][index.gateway ] [statistics02] [statistics-20140108][2] recovery completed from [local], took [38ms]
[2014-12-01 19:05:28,287][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24543
[2014-12-01 19:05:28,769][DEBUG][indices.recovery ] [statistics02] [statistics-20140226][1] recovery completed from [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], took [394ms]
[2014-12-01 19:05:32,681][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24544
[2014-12-01 19:05:38,058][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24545
[2014-12-01 19:05:42,821][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24546
[2014-12-01 19:05:52,443][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24547
[2014-12-01 19:06:02,245][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24548
[2014-12-01 19:06:15,962][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24549
[2014-12-01 19:06:23,949][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24550
[2014-12-01 19:06:24,519][DEBUG][indices.recovery ] [statistics02] [statistics-20140527][4] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [55.6s]
[2014-12-01 19:06:24,520][DEBUG][indices.recovery ] [statistics02] [statistics-20140413][4] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [45.9s]
[2014-12-01 19:06:28,723][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24551
[2014-12-01 19:06:34,943][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24552
[2014-12-01 19:06:50,748][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24553
[2014-12-01 19:06:58,276][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24554
[2014-12-01 19:06:58,745][DEBUG][indices.recovery ] [statistics02] [statistics-20140512][3] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [22.7s]
[2014-12-01 19:07:03,330][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24555
[2014-12-01 19:07:08,646][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24556
[2014-12-01 19:07:08,742][DEBUG][indices.recovery ] [statistics02] [statistics-20140823][3] recovery completed from [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], took [56ms]
[2014-12-01 19:07:13,350][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24557
[2014-12-01 19:07:19,349][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24558
[2014-12-01 19:07:20,403][DEBUG][indices.recovery ] [statistics02] [statistics-20131121][4] recovery completed from [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], took [7s]
[2014-12-01 19:07:27,139][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24559
[2014-12-01 19:07:27,587][DEBUG][indices.recovery ] [statistics02] [statistics-20140723][0] recovery completed from [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], took [404ms]
[2014-12-01 19:07:30,786][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24560
[2014-12-01 19:07:35,493][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24561
[2014-12-01 19:07:36,207][DEBUG][indices.recovery ] [statistics02] [statistics-20140326][1] recovery completed from [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], took [685ms]
[2014-12-01 19:07:39,615][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24562
[2014-12-01 19:07:40,124][DEBUG][indices.recovery ] [statistics02] [statistics-20140703][2] recovery completed from [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], took [475ms]
[2014-12-01 19:07:44,138][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24563
[2014-12-01 19:07:47,642][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24564
[2014-12-01 19:07:52,212][DEBUG][discovery.zen.publish ] [statistics02] received cluster state version 24565```

Issue 2:
When restarting a cluster from green state, each shard appears to undergo some form of checksum to verify it before bringing it online.

Is there a way to journal writes so that recovery is much much faster, in the way that the xfs filesystem does it.

Only review the data that was being written to at the time of the outage or shutdown so that only the in-progress write data needs to be checked.

For a clean shutdown, maybe a complete cluster restart command could tell all nodes to shutdown in a clean state then turn off, allowing a near instantaneous recovery on startup. Like stop allocation then flush all translogs, then shut down,etc.

Would just take lots of the pain out of cluster restarts.

Just an idea