Repository: borgbackup/borg

Issue:
Option to detect hash collision
Hash collisions are unlikely when using SHA256, but as the number of files/chunks increases so does the probability, most de-duplication system do some sort to sanity check; file based - check file size and possibly type, block based - either generate fingerprint in addition to hash or read the block already in archive storage (ie: paranoid mode option in Symantec PureDisk) to verify the archive data really matches new data before incrementing the link counter and discarding the new data.  Easy to test functionality if you are able to change the hash function to md4.



Discussion:
AFAIK, hash collisions start becoming a real problem if the number of hashes is about 2 ^ (N/2) with N being the bit length of the hash. So for sha256 that is 2 ^ 128 hashes. Each chunk is about 2^16 bytes (default, or likely bigger if you backup a lot of data and adapt --chunker-params). So this means that we talk about 2^144 Bytes of data in your backup. In decimal, that is a 1 followed by 43 zeros.

I guess you'll have a lot of other problems before that happens, right?
Like Borg eating all your RAM, all your disks failing at once, the ceiling falling on your head, the sun exploding, ... :-D

So the question here is really how much overhead one wants to add just to deal with such a improbable case.

Currently we can not easily change the id_hash, but I have plans to make that more flexible (together with more flexible crypto).

One check I could imagine (not sure if it is done already), is if we detect we already have some specific hash to additionally compare the length of the current input data to the length of the stored data that originally generated that hash. This would add a few bits (8..10?) of safety margin and is likely to be implementable rather cheaply (and also without significant runtime overhead).

Relevant code: cache.Cache.seen_chunk and also .add_chunk. It does not compare the size yet.

BTW, if chunk length matches, this also means either this was a relatively small file and the file size matches (whole file = 1 chunk) or that the rolling hash used for the content defined chunking triggered at the beginning and/or end of the chunk. This seems to add some more bits of safety than the mere length check.

http://stackoverflow.com/questions/4014090/is-it-safe-to-ignore-the-possibility-of-sha-collisions-in-practice < haha


Resolving patch:
diff --git a/borg/cache.py b/borg/cache.py
--- a/borg/cache.py
+++ b/borg/cache.py
@@ -347,9 +347,9 @@ def create_master_idx(chunk_idx, tf_in, tmp_dir):
     def add_chunk(self, id, data, stats):
         if not self.txn_active:
             self.begin_txn()
-        if self.seen_chunk(id):
-            return self.chunk_incref(id, stats)
         size = len(data)
+        if self.seen_chunk(id, size):
+            return self.chunk_incref(id, stats)
         data = self.key.encrypt(data)
         csize = len(data)
         self.repository.put(id, data, wait=False)
@@ -357,8 +357,14 @@ def add_chunk(self, id, data, stats):
         stats.update(size, csize, True)
         return id, size, csize
 
-    def seen_chunk(self, id):
-        return self.chunks.get(id, (0, 0, 0))[0]
+    def seen_chunk(self, id, size=None):
+        refcount, stored_size, _ = self.chunks.get(id, (0, None, None))
+        if size is not None and stored_size is not None and size != stored_size:
+            # we already have a chunk with that id, but different size.
+            # this is either a hash collision (unlikely) or corruption or a bug.
+            raise Exception("chunk has same id [%r], but different size (stored: %d new: %d)!" % (
+                            id, stored_size, size))
+        return refcount
 
     def chunk_incref(self, id, stats):
         if not self.txn_active:


Test patch: