Issue 1:
Using

Elasticsearch: 2.3.1

JVM version: 1.8.0_60-b27

OS version: Centos 7

With

.
.
.
node:
data: true
master: false
name: esd02-data4
tier: hot
os:
available_processors: 8
path:
data:
- /storage/sdb1
- /storage/sdx1
.
.
.

in elasticsearch.yml we have observed twice now where one of the disks (say, /storage/sdx1) will fill up to 100% while the other /storage/sdb1 still has space. The error logs fill up with errors like:

[logstash-syslog-2016.07.21][[logstash-syslog-2016.07.21][1]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -> /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device];
Caused by: [logstash-syslog-2016.07.21][[logstash-syslog-2016.07.21][1]] EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -> /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device];
Caused by: java.nio.file.FileSystemException: /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -> /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device
java.nio.file.FileSystemException: /storage/sdb1/stage/nodes/0/indices/logstash-syslog-2016.07.21/_state/state-13787.st -> /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/_state/state-13787.st.tmp: No space left on device

It's pretty obvious what's happening... storage/sdx1 owns a shard of logstash-syslog-2016.07.21 and ES is refusing to "break" the shard across multiple disk paths....

but it's also failing to recover from this on its own, or to stop sending data to the shard whose disk is full. I would expect either (A) in the event of a full disk, ES would nominate the other shards to accept 100% of the writes to the index or (B) ES would loosen its requirement that shards are not allowed to span multiple write paths.

Issue 2:
#18467 was intended to limit how many times shard allocation was attempted before giving up, instead of constantly retrying. eg if an analyzer requires a local synonyms file, which is missing, then we should stop trying until the situation is resolved.

Unfortunately it doesn't work for unassigned primary shards because of the way allocation works today. For instance, a primary shard might be sitting on a disk which is over the high watermark, or it may have allocation filtering which prevents it from being allocated to the node where it already exists. Today, we just go ahead and try to assign the primary regardless of the allocation deciders (which means that we also don't limit retries to 5).

Instead, we could add extra logic to the appropriate deciders to say "always return YES if the shard in question is a primary which already exists on this node". The decision returned would be YES, but the explanation provided by the reroute or cluster allocation explain could include the reason this decider was ignored.

/cc @ywelsch @dakrone @s1monw

Related to #18321