[RECOVERY] Throw IndexShardClosedException if shard is closed

Today we throw a generic ElasticsearchException when a recovery is cancled. This
causes verbose logging and send shard failures and additional unnecessary cluster state
events. We can just throw IndexShardClosedException which prevents the send shard failures.

we see lots of these exception in the logs which are misleading - this was introduced lately and never released:

```
1&gt; [2014-11-25 11:07:40,742][DEBUG][index.service            ] [node_0] [test2] [3] closed (reason: [recovery failure [RecoveryFailedException[[test2][3]: Recovery failed from [node_1][9pORZ5dyT6C0Y8oZxZ9q-w][ip-10-255-15-175][local[3]]{mode=local} into [node_0][_ZbAkLDCSCa1GB_dEv7nUQ][ip-10-255-15-175][local[2]]{mode=local}]; nested: RemoteTransportException[[node_0][local[2]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test2][3] Phase[3] Execution failed]; nested: ElasticsearchException[recovery was canceled reason [shard is closed]]; ]])
  1&gt; [2014-11-25 11:07:40,742][WARN ][indices.cluster          ] [node_0] [test2][3] sending failed shard after recovery failure
  1&gt; org.elasticsearch.indices.recovery.RecoveryFailedException: [test2][3]: Recovery failed from [node_1][9pORZ5dyT6C0Y8oZxZ9q-w][ip-10-255-15-175][local[3]]{mode=local} into [node_0][_ZbAkLDCSCa1GB_dEv7nUQ][ip-10-255-15-175][local[2]]{mode=local}
  1&gt;         at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:245)
  1&gt;         at org.elasticsearch.indices.recovery.RecoveryTarget.access$500(RecoveryTarget.java:64)
  1&gt;         at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:485)
  1&gt;         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
  1&gt;         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  1&gt;         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  1&gt;         at java.lang.Thread.run(Thread.java:744)
  1&gt; Caused by: org.elasticsearch.transport.RemoteTransportException: [node_0][local[2]][internal:index/shard/recovery/start_recovery]
  1&gt; Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [test2][3] Phase[3] Execution failed
  1&gt;         at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1182)
  1&gt;         at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:675)
  1&gt;         at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:127)
  1&gt;         at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:51)
  1&gt;         at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:148)
  1&gt;         at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:134)
  1&gt;         at org.elasticsearch.transport.local.LocalTransport$2.doRun(LocalTransport.java:267)
  1&gt;         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
  1&gt;         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  1&gt;         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  1&gt;         at java.lang.Thread.run(Thread.java:744)
  1&gt; Caused by: org.elasticsearch.ElasticsearchException: recovery was canceled reason [shard is closed]
  1&gt;         at org.elasticsearch.indices.recovery.ShardRecoveryHandler$CancelableThreads.failIfCanceled(ShardRecoveryHandler.java:640)
  1&gt;         at org.elasticsearch.indices.recovery.ShardRecoveryHandler$CancelableThreads.remove(ShardRecoveryHandler.java:661)
  1&gt;         at org.elasticsearch.indices.recovery.ShardRecoveryHandler$CancelableThreads.run(ShardRecoveryHandler.java:655)
  1&gt;         at org.elasticsearch.indices.recovery.ShardRecoveryHandler.phase3(ShardRecoveryHandler.java:430)
  1&gt;         at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1178)
  1&gt;         ... 10 more
  1&gt; [2014-11-25 11:07:40,743][WARN ][cluster.action.shard     ] [node_0] [test2][3] sending failed shard for [test2][3], node[_ZbAkLDCSCa1GB_dEv7nUQ], [R], s[INITIALIZING], indexUUID [sUuEvRUORCeBRQgKKtg4UQ], reason [Failed to start shard, message [RecoveryFailedException[[test2][3]: Recovery failed from [node_1][9pORZ5dyT6C0Y8oZxZ9q-w][ip-10-255-15-175][local[3]]{mode=local} into [node_0][_ZbAkLDCSCa1GB_dEv7nUQ][ip-10-255-15-175][local[2]]{mode=local}]; nested: RemoteTransportException[[node_0][local[2]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test2][3] Phase[3] Execution failed]; nested: ElasticsearchException[recovery was canceled reason [shard is closed]]; ]]
  1&gt; [2014-11-25 11:07:40,743][WARN ][cluster.action.shard     ] [node_0] [test2][3] received shard failed for [test2][3], node[_ZbAkLDCSCa1GB_dEv7nUQ], [R], s[INITIALIZING], indexUUID [sUuEvRUORCeBRQgKKtg4UQ], reason [Failed to start shard, message [RecoveryFailedException[[test2][3]: Recovery failed from [node_1][9pORZ5dyT6C0Y8oZxZ9q-w][ip-10-255-15-175][local[3]]{mode=local} into [node_0][_ZbAkLDCSCa1GB_dEv7nUQ][ip-10-255-15-175][local[2]]{mode=local}]; nested: RemoteTransportException[[node_0][local[2]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test2][3] Phase[3] Execution failed]; nested: ElasticsearchException[recovery was canceled reason [shard is closed]]; ]]
```
I wanted to factor this out into a utils class and use it on the other side of the recovery too