elasticsearch/elasticsearch — pull request #8779 by s1monw (2014-12-04 13:06:02+00:00)
Title: [Store] Only fail recovery if files are inconsistent

the recovery diff can return file in the `different` category
since it's conservative if it can't tell if the files are the same.
Yet, the cleanup code only needs to ensure both ends of the recovery
are consistent. If we have a very old segments_N file no checksum is present
but for the delete files they might be such that the segments file passes
the consistency check but the .del file doesn't since it's in-fact the same
but this check was missing in the last commit.

this caused BWC test failures like this:

```
Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t6][inet[/144.76.166.238:9707]][internal:index/shard/recovery/clean_files]
Caused by: org.elasticsearch.indices.recovery.RecoveryFailedException: [test0][1]: Recovery failed from [node_t2][yIx4DzQeSXqzx0k5Eg-GOg][Ubuntu-1404-trusty-64-minimal][inet[/144.76.166.238:9708]]{mode=network} into [node_t6][kZHdqPMlQhOznc3Pp5IFFA][Ubuntu-1404-trusty-64-minimal][inet[/144.76.166.238:9707]]{mode=network} (failed to clean after recovery)
at org.elasticsearch.indices.recovery.RecoveryTarget$CleanFilesRequestHandler.messageReceived(RecoveryTarget.java:402)
at org.elasticsearch.indices.recovery.RecoveryTarget$CleanFilesRequestHandler.messageReceived(RecoveryTarget.java:362)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:274)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: local version: name [_0_1.del], length [49], checksum [htw9r2], writtenBy [null] is different from remote version after recovery: name [_0_1.del], length [49], checksum [htw9r2], writtenBy [null]
at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:602)
at org.elasticsearch.indices.recovery.RecoveryTarget$CleanFilesRequestHandler.messageReceived(RecoveryTarget.java:400)
... 6 more
[2014-12-04 10:00:59,730][WARN ][cluster.action.shard ] [node_t6]
```