Repository: borgbackup/borg

Issue:
borg upgrade also upgrades the backup repo
It uses hardlinks to create a backup repo, but then updates the files in place,
which doesn't break the hardlinks.

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/28861170-borg-upgrade-also-upgrades-the-backup-repo)



Discussion:
are you sure? did you conclude this from reading the source or from running the upgrade directly?

there's a unit test that verifies that we operate on a copy of the hardlink here:

https://github.com/borgbackup/borg/blob/master/borg/testsuite/upgrader.py#L188

I tested this with the released version of borg, and the hardlinks weren't broken.

On 2015-12-04 14:40:48, Dan Christensen wrote:

> I tested this with the released version of borg, and the hardlinks weren't broken.

Can you provide a more obvious way to reproduce? Did you run with
--inplace?

Thanks

```
$ attic init test2.attic
Initializing repository at "test2.attic"
Encryption NOT enabled.
Use the "--encryption=passphrase|keyfile" to enable encryption.
Initializing cache...

$ attic create test2.attic::1 foo

$ ./borg-linux64 upgrade test2.attic
making a hardlink copy in /tmp/jdc/test2.attic.upgrade-2015-12-04-15:31:33
opening attic repository with borg and converting
no key file found for repository
converting index index b'/tmp/jdc/test2.attic/index.2'
copying attic cache file from /home/jdc/.cache/attic/1b8ffdb91a5cefae0af6b7ec4a3a9ebcd0ff22aa0e497a0705796a679ee2f95d/config to /home/jdc/.cache/borg/1b8ffdb91a5cefae0af6b7ec4a3a9ebcd0ff22aa0e497a0705796a679ee2f95d/config
copying attic cache file from /home/jdc/.cache/attic/1b8ffdb91a5cefae0af6b7ec4a3a9ebcd0ff22aa0e497a0705796a679ee2f95d/files to /home/jdc/.cache/borg/1b8ffdb91a5cefae0af6b7ec4a3a9ebcd0ff22aa0e497a0705796a679ee2f95d/files
copying attic cache file from /home/jdc/.cache/attic/1b8ffdb91a5cefae0af6b7ec4a3a9ebcd0ff22aa0e497a0705796a679ee2f95d/chunks to /home/jdc/.cache/borg/1b8ffdb91a5cefae0af6b7ec4a3a9ebcd0ff22aa0e497a0705796a679ee2f95d/chunks
converting cache /home/jdc/.cache/borg/1b8ffdb91a5cefae0af6b7ec4a3a9ebcd0ff22aa0e497a0705796a679ee2f95d/chunks
converting 1 segments...
converting segment 1/1, 100.00% done (/tmp/jdc/test2.attic/data/0/2)

$ ls -l test2.attic*/index.2
-rw-rw---- 2 jdc jdc 40978 Dec  4 15:31 test2.attic/index.2
-rw-rw---- 2 jdc jdc 40978 Dec  4 15:31 test2.attic.upgrade-2015-12-04-15:31:33/index.2

$ ls -i test2.attic*/index.2
6700175 test2.attic/index.2
6700175 test2.attic.upgrade-2015-12-04-15:31:33/index.2
```

The two index.2 files have the same inode number.

Ah, but now I see that the data files are different, so in that case the hardlinks are broken.

Here's an approach that would be safer, and still allow progress reporting.

When --inplace is given, do things the way they are currently done.

Without --inplace, require the user to specify the name of the new borg repo to be created.
Then, go through the attic repo one file at a time and write the converted file to the new borg repo.
Then there's no chance that the attic repo will get corrupted.  Moreover, it will work across 
filesystems, which might be a common situation for a large repo.

On 2015-12-04 15:37:42, Dan Christensen wrote:

> The two index.2 files have the same inode number.

Okay, so only the index files are affected. That is a bug.

> Ah, but now I see that the data files are different, so in that case the hardlinks are broken.

Phew! At least those work. :)

On 2015-12-04 16:08:10, Dan Christensen wrote:

> Here's an approach that would be safer, and still allow progress reporting.
> 
> When --inplace is given, do things the way they are currently done.
> 
> Without --inplace, require the user to specify the name of the new borg repo to be created.
> Then, go through the attic repo one file at a time and write the converted file to the new borg repo.
> Then there's no chance that the attic repo will get corrupted.  Moreover, it will work across 
> filesystems, which might be a common situation for a large repo.

That  could work. I guess I got carried away with the hardlink
approach... I think the hardlink approach is still faster than copying
the whole file, especially with COW filesystems like ZFS or BTRFS,
because the file _won't_ need to be completely copied from scratch...

But yeah, this would probably be safer and work across filesystems...

I upgraded a attic repo to borg, using `borg upgrade repo` (not in-place):

```
tw@tux:~/w/borg$ ls -li repo*
repo:
2267577 -rw-rw-r-- 2 tw tw   148 Dec  7 18:49 config
2330626 drwxrwxr-x 3 tw tw  4096 Dec  7 18:49 data
2267586 -rw-rw-r-- 2 tw tw    32 Dec  7 18:51 hints.2
2267587 -rw-rw-r-- 2 tw tw 40978 Dec  7 18:52 index.2
2267582 -rw------- 2 tw tw    31 Dec  7 18:52 lock.roster
2267576 -rw-rw-r-- 2 tw tw    28 Dec  7 18:49 README

repo.upgrade-2015-12-07-18:52:06:
2267577 -rw-rw-r-- 2 tw tw   148 Dec  7 18:49 config
2821173 drwxrwxr-x 3 tw tw  4096 Dec  7 18:49 data
2267586 -rw-rw-r-- 2 tw tw    32 Dec  7 18:51 hints.2
2267587 -rw-rw-r-- 2 tw tw 40978 Dec  7 18:52 index.2
2267582 -rw------- 2 tw tw    31 Dec  7 18:52 lock.roster
2267576 -rw-rw-r-- 2 tw tw    28 Dec  7 18:49 README
```

As one sees at the filenames and the inode numbers at the left, there are the following issues:
- a lock.roster file is created in the attic repo backup that was not there before (attic does not use that file). that's a minor issue, but a little unclean.
- the repo/README is still the same and says `This is an Attic repository` which is not true after the conversion.
- config is same inode (no problem)
- hints.2 is same inode (no problem)
- index.2 are same inode. this is a bug, as it modified the file and the modification is in the attic backup also.
- the segment files are (as it should be) different:

```
2267581 -rw------- 1 tw tw 1911341 Dec  7 18:52 repo/data/0/2
2267585 -rw-rw-r-- 1 tw tw 1911341 Dec  7 18:51 repo.upgrade-2015-12-07-18:52:06/data/0/2
```


Resolving patch:
diff --git a/borg/upgrader.py b/borg/upgrader.py
--- a/borg/upgrader.py
+++ b/borg/upgrader.py
@@ -16,6 +16,10 @@
 
 
 class AtticRepositoryUpgrader(Repository):
+    def __init__(self, *args, **kw):
+        kw['lock'] = False  # do not create borg lock files (now) in attic repo
+        super().__init__(*args, **kw)
+
     def upgrade(self, dryrun=True, inplace=False):
         """convert an attic repository to a borg repository
 
@@ -33,9 +37,19 @@ def upgrade(self, dryrun=True, inplace=False):
             logger.info('making a hardlink copy in %s', backup)
             if not dryrun:
                 shutil.copytree(self.path, backup, copy_function=os.link)
+                # we need to create a real copy (not hardlink copy) of index.* and hints.*
+                # otherwise the backup copy will get modified.
+                transaction_id = self.get_index_transaction_id()
+                for name in ['index', 'hints']:
+                    fname = "%s.%d" % (name, transaction_id)
+                    fname_orig = os.path.join(self.path, fname)
+                    fname_backup = os.path.join(backup, fname)
+                    os.remove(fname_backup)
+                    shutil.copy(fname_orig, fname_backup)
+
         logger.info("opening attic repository with borg and converting")
-        # we need to open the repo to load configuration, keyfiles and segments
-        self.open(self.path, exclusive=False)
+        # now lock the repo, after we have made the copy
+        self.lock = UpgradableLock(os.path.join(self.path, 'lock'), exclusive=True, timeout=1.0).acquire()
         segments = [filename for i, filename in self.io.segment_iterator()]
         try:
             keyfile = self.find_attic_keyfile()
@@ -48,13 +62,21 @@ def upgrade(self, dryrun=True, inplace=False):
         self.lock = UpgradableLock(os.path.join(self.path, 'lock'),
                                    exclusive=True).acquire()
         try:
+            self.convert_repo_index(dryrun)
             self.convert_cache(dryrun)
             self.convert_segments(segments, dryrun=dryrun, inplace=inplace)
+            self.borg_readme()
         finally:
             self.lock.release()
             self.lock = None
         return backup
 
+    def borg_readme(self):
+        readme = os.path.join(self.path, 'README')
+        os.remove(readme)
+        with open(readme, 'w') as fd:
+            fd.write('This is a Borg repository\n')
+
     @staticmethod
     def convert_segments(segments, dryrun=True, inplace=False):
         """convert repository segments from attic to borg
@@ -139,8 +161,8 @@ def convert_keyfiles(keyfile, dryrun):
             with open(keyfile, 'w') as f:
                 f.write(data)
 
-    def convert_cache(self, dryrun):
-        """convert caches from attic to borg
+    def convert_repo_index(self, dryrun):
+        """convert some repo files
 
         those are all hash indexes, so we need to
         `s/ATTICIDX/BORG_IDX/` in a few locations:
@@ -150,22 +172,28 @@ def convert_cache(self, dryrun):
           should probably update, with a lock, see
           `Repository.open()`, which i'm not sure we should use
           because it may write data on `Repository.close()`...
-
-        * the `files` and `chunks` cache (in `$ATTIC_CACHE_DIR` or
-          `$HOME/.cache/attic/<repoid>/`), which we could just drop,
-          but if we'd want to convert, we could open it with the
-          `Cache.open()`, edit in place and then `Cache.close()` to
-          make sure we have locking right
         """
         transaction_id = self.get_index_transaction_id()
         if transaction_id is None:
             logger.warning('no index file found for repository %s' % self.path)
         else:
             index = os.path.join(self.path, 'index.%d' % transaction_id).encode('utf-8')
-            logger.info("converting index index %s" % index)
+            logger.info("converting repo index %s" % index)
             if not dryrun:
                 AtticRepositoryUpgrader.header_replace(index, b'ATTICIDX', b'BORG_IDX')
 
+    def convert_cache(self, dryrun):
+        """convert caches from attic to borg
+
+        those are all hash indexes, so we need to
+        `s/ATTICIDX/BORG_IDX/` in a few locations:
+
+        * the `files` and `chunks` cache (in `$ATTIC_CACHE_DIR` or
+          `$HOME/.cache/attic/<repoid>/`), which we could just drop,
+          but if we'd want to convert, we could open it with the
+          `Cache.open()`, edit in place and then `Cache.close()` to
+          make sure we have locking right
+        """
         # copy of attic's get_cache_dir()
         attic_cache_dir = os.environ.get('ATTIC_CACHE_DIR',
                                          os.path.join(os.path.expanduser('~'),


Test patch: