elasticsearch/elasticsearch — pull request #7351 by s1monw (2014-08-20 14:38:48+00:00)
Title: Improve recovery / snapshot restoring file identity handling

This commit changes the way how files are selected for retransmission
on recovery / restore. Today this happens on a per-file basis where the
rather weak checksum and the file length in bytes is compared to check if
a file is identical. This is prone to fail in the case of a checksum collision
which can happen under certain circumstances.
The changes in this commit move the identity comparison to a per-commit / per-segment
level where files are only treated as identical iff all the other files in the
commit / segment are the same. This `all or nothing` strategy is reducing the chance for
a collision dramatically since we also use a strong hash to identify commits / segments
based on the content of the `.si` / `segments.N` file.