Repository: borgbackup/borg

Issue:
borg create --read-device repo /dev/vg0/*-snapshot
this feature would read all given (device)files and save their content (not the device file node).



Discussion:
Not sure where you're going with this feature.  Can you expand on the use case?  Maybe it's better to deal with LVM issues in a wrapper like borgmatic, then to try and expand borg to grok LVM?  Maybe keep borg more focused on "here's an already mounted file system, back it up".

The only change intended here is to read from a device file (not save the file as device file).
the archive could then be named repo::somehost-date and contain:
/dev/vg0/root-snapshot
/dev/vg0/home-snapshot
...

This is similar to using `dd if=/dev/vg0/xxx-snapshot | borg create repo:xxx -` - except that we can do multiple snapshots at once and the snapshot device names are reflected in the repo.

I don't intend to add functionality that creates or releases snapshots for lvm or btrfs or ...

+1
This is the last thing that blocks us from using Borg - no nice support for lvm images. We use xen and kvm, this would simplify our solution a lot.

 I will send a bag of Polish candy (krówka) for the one who will develop this :) 

@neutrinus we are also on https://www.bountysource.com/. :)

reading a device is already possible using dd and stdin "-" mode, but this ticket is meant to really have the lv names in the archive.

I know, I can use stdin, but doesn't look clean for us :) 

Bounty created, no candies.

thanks for creating a bounty. guess I need to make bountysource a bit more visible here also.

I just found that this maybe makes sense to be more generic than --read-device (in the sense of a block device). How about just opening and reading (backing up) any given filename as if it were a regular file? Block devices, char devices(?), symlinks, FIFOs, sockets(?). Directories would be the only exception (and just recursed, as usual).


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -145,7 +145,8 @@ def do_create(self, args):
                     continue
             else:
                 restrict_dev = None
-            self._process(archive, cache, args.excludes, args.exclude_caches, skip_inodes, path, restrict_dev)
+            self._process(archive, cache, args.excludes, args.exclude_caches, skip_inodes, path, restrict_dev,
+                          read_special=args.read_special)
         archive.save(timestamp=args.timestamp)
         if args.progress:
             archive.stats.show_progress(final=True)
@@ -163,7 +164,8 @@ def do_create(self, args):
             print('-' * 78)
         return self.exit_code
 
-    def _process(self, archive, cache, excludes, exclude_caches, skip_inodes, path, restrict_dev):
+    def _process(self, archive, cache, excludes, exclude_caches, skip_inodes, path, restrict_dev,
+                 read_special=False):
         if exclude_path(path, excludes):
             return
         try:
@@ -180,7 +182,8 @@ def _process(self, archive, cache, excludes, exclude_caches, skip_inodes, path,
         # Ignore if nodump flag is set
         if has_lchflags and (st.st_flags & stat.UF_NODUMP):
             return
-        if stat.S_ISREG(st.st_mode):
+        if (stat.S_ISREG(st.st_mode) or
+            read_special and not stat.S_ISDIR(st.st_mode)):
             try:
                 status = archive.process_file(path, st, cache)
             except IOError as e:
@@ -197,7 +200,7 @@ def _process(self, archive, cache, excludes, exclude_caches, skip_inodes, path,
                 for filename in sorted(entries):
                     entry_path = os.path.normpath(os.path.join(path, filename))
                     self._process(archive, cache, excludes, exclude_caches, skip_inodes,
-                                  entry_path, restrict_dev)
+                                  entry_path, restrict_dev, read_special=read_special)
         elif stat.S_ISLNK(st.st_mode):
             status = archive.process_symlink(path, st)
         elif stat.S_ISFIFO(st.st_mode):
@@ -687,6 +690,9 @@ def run(self, args=None):
                                     'zlib,0 .. zlib,9 == zlib (with level 0..9), '
                                     'lzma == lzma (default level 6), '
                                     'lzma,0 .. lzma,9 == lzma (with level 0..9).')
+        subparser.add_argument('--read-special', dest='read_special',
+                               action='store_true', default=False,
+                               help='open and read special files as if they were regular files')
         subparser.add_argument('archive', metavar='ARCHIVE',
                                type=location_validator(archive=True),
                                help='archive to create')
diff --git a/borg/cache.py b/borg/cache.py
--- a/borg/cache.py
+++ b/borg/cache.py
@@ -3,6 +3,7 @@
 import errno
 import msgpack
 import os
+import stat
 import sys
 from binascii import hexlify
 import shutil
@@ -381,7 +382,7 @@ def chunk_decref(self, id, stats):
             stats.update(-size, -csize, False)
 
     def file_known_and_unchanged(self, path_hash, st):
-        if not self.do_files:
+        if not (self.do_files and stat.S_ISREG(st.st_mode)):
             return None
         if self.files is None:
             self._read_files()
@@ -398,7 +399,7 @@ def file_known_and_unchanged(self, path_hash, st):
             return None
 
     def memorize_file(self, path_hash, st, ids):
-        if not self.do_files:
+        if not (self.do_files and stat.S_ISREG(st.st_mode)):
             return
         # Entry: Age, inode, size, mtime, chunk ids
         mtime_ns = st_mtime_ns(st)


Test patch: