Repository: borgbackup/borg

Issue:
How to back up only certain files (e.g. *.jpg)
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Question

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

borg 1.1.13

#### Operating system (distribution) and version.

Ubuntu 18.04.4

#### Hardware / network configuration, and filesystems used.

Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
16 GB RAM
ZFS 5 x 3-way mirrors, 22 TiB usable

#### How much data is handled by borg?

15 TiB

## Question

Is this possible? I can't find a reference to a command-line flag that will do this, or related FAQ, doc, or issue.

The reason I'm doing this, is to back up my most important files first, which I can get most optimally "close enough" with wildcard matching.

I understand I can generate a separate "includes" file but if there's a way to keep it native, that would be better for my use case.

Many thanks.

Update: I realize there actually is no `--files-from` flag or similar functionality (and `--patterns-from` is marked experiments - and seems a little dicey to include millions of files).


Discussion:
borg help patterns
Sadly, as mentioned, I'm not comfortable with the "experimental" status particularly on production data, but thanks for the suggestion.

Edit: Also, it appears that includes in `--patterns-from` would take priority over a lengthy list of regexes I've specified in `--exclude_from`, which would mean that I would need to refactor those regexes into `--patterns-from`, and get the order of include/exclude correct. Which, without knowing why it's marked "experimental", seems like it might further push the boundaries of complexity and increase the odds of something going wrong.

And finally, the documentation on `--patterns-from` is on the dicier side, understandably with experimental status. For example, the sum total of help on the syntax specific to `--patterns-from`, above and beyond `--pattern`, apparently anywhere that Google appears to be able to see, consists of this:

```
The patterns.lst file could look like that:

# "sh:" pattern style is the default, so the following line is not needed:
P sh
R /
# can be rebuild
- /home/*/.cache
```

Which leaves to inference, what `P` and `R` mean. It would seem that:

- `P` is some kind of preprocessor command that means "treat everything that follows as if prepended by "sh:", for shell-style matching. But this implies other questions that aren't answered, such as:
  - Presumably the other styles would also be valid?
  - Can you use this `P` keyword on any line, to change the matching style for all subsequent lines until another one is encountered?
  - Instead of `P`, can you specify the matching style for every line? If so, how would that syntax look like in the context of, and relative to, leading `-` and `+`?
- `R` seems to mean "Root", or "Relative". (Or either synonymously.) Meaning, everything sepcified below is relative to whatever `R` says. But similarly, unanswered questions:
  - Why then is "/home/*/.cache" specified with a leading "/"? Does it override `P` (for example if `P` was set to, say, `/etc`)? Is that only because `R` just means where to start scanning from?

Etc. Sure, I could test these theories and answer each question through experimental trial and error, but it would be much more assuring if there were canonical documentation on it.

(And yes I know this is floss and I could improve the docs myself. Unfortunately I invest time on other projects and it would be impossible for anyone to contribute to every open source project they use. For those who contribute, we still have to be OK being an average consumer for 99.99% of floss we rely on. Ditto with borg contributors. And so we evaluate the pros and cons of competing projects for a given solution, through that lens.)
Maybe really read borg help patterns before assuming things?

I'ld like to fix anything unclear there, but would expect the discussion starting from what is already documented.
@ThomasWaldmann Seems like we're miscommunicating. Maybe my previous reply was too long.

I wrote at length, the shortcomings, as I see them, *specifically* of the output of `borg help patterns`. Literally, "starting from what is already documented". I'm not sure why you are having heartburn over that.

Rather than saying I'm "assuming things", maybe try to understand the shortcomings that I saw as an average user, and why - which I went to some effort to think through clearly, and tried to present logically - specifically from the output of `borg help patterns`. And in fact, that output does not seem to be elaborated on in any other documentation that I've been able to find, e.g. [Borg Documentation](https://borgbackup.readthedocs.io/en/stable/).

You may understand perfectly well the specific syntax and grammar expected by `--patterns-from`, as very lightly hinted at in the tiny example file as output in `borg help patterns`. But without that syntax and grammar being documented for users anywhere that I've been able to find, that output is opaque and not very useful.

Just because I don't understand the syntax & grammar expected in the file read by `--patterns-from`, and tried my best to infer/reverse-engineer the syntax & grammar, doesn't mean I'm "assuming things". It means I'm obviously struggling to understand how to use the unclear and incomplete output of `borg help patterns`. I'd much rather not have to resort to what you seem to view as "assuming things".

I understand the feature is experimental. That's all good. I don't expect it (or anything) to be perfect, bulletproof, and well-documented. My only suggestion is to think twice about repeatedly blaming the user, for doing exactly what you suggested (as I already had when you seemed to reflexively suggest it without actually understanding my concerns or that I obviously already did), when s/he finds the output of said suggestion insufficient. My biased feedback which you can consider or ignore.
It clearly says "full path" at quite some places and also explains what that means.

It nowhere says that the R recursion root specification makes patterns defined below it relative. Or that R means relative.
But I agree, there are some shortcomings in that part of the docs:
- usage of leading / in patterns - bug in docs or ignored by the code?
- ability of using a pattern type prefix directly in a pattern line
- scope of a P definition line
- explicitly saying that R means recursion root (starting point of recursion)
- extend the example pattern file to cover more of this

borg 1.2 can also read paths from stdin (e.g. like `find ... | borg ...`).
@ThomasWaldmann I could try a complete overhaul of the help patterns section. Assign this to me if you like.
Yeah, why not. If you could actually test the stuff while writing the examples, that would be great, thanks!

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2485,40 +2485,35 @@ def do_break_lock(self, args, repository):
 
     helptext = collections.OrderedDict()
     helptext['patterns'] = textwrap.dedent('''
-        The path/filenames used as input for the pattern matching start from the
-        currently active recursion root. You usually give the recursion root(s)
-        when invoking borg and these can be either relative or absolute paths.
-
-        If you give `/absolute/` as root, the paths going into the matcher will
-        look relative like `absolute/.../file.ext`, because file paths in Borg
-        archives are always stored normalized and relative. This means that e.g.
-        ``borg create /path/to/repo ../some/path`` will store all files as
-        `some/path/.../file.ext` and ``borg create /path/to/repo /home/user``
-        will store all files as `home/user/.../file.ext`.
-
-        A directory exclusion pattern can end either with or without a slash ('/').
-        If it ends with a slash, such as `some/path/`, the directory will be
-        included but not its content. If it does not end with a slash, such as
-        `some/path`, both the directory and content will be excluded.
-
-        File patterns support these styles: fnmatch, shell, regular expressions,
-        path prefixes and path full-matches. By default, fnmatch is used for
-        ``--exclude`` patterns and shell-style is used for the ``--pattern``
-        option. For commands that support patterns in their ``PATH`` argument
-        like (``borg list``), the default pattern is path prefix.
-
-        Starting with Borg 1.2, discovered fs paths are normalised, have leading
-        slashes removed and then are matched against your patterns.
-        Note: You need to review your include / exclude patterns and make
-        sure they do not expect leading slashes. Borg can only deal with this
-        for some very simple patterns by removing leading slashes there also.
-
-        If followed by a colon (':') the first two characters of a pattern are
-        used as a style selector. Explicit style selection is necessary when a
-        non-default style is desired or when the desired pattern starts with
-        two alphanumeric characters followed by a colon (i.e. `aa:something/*`).
-
-        `Fnmatch <https://docs.python.org/3/library/fnmatch.html>`_, selector `fm:`
+        When specifying one or more file paths in a Borg command that supports
+        patterns for the respective option or argument, you can apply the
+        patterns described here to include only desired files and/or exclude
+        unwanted ones. Patterns can be used
+
+        - for ``--exclude`` option,
+        - in the file given with ``--exclude-from`` option,
+        - for ``--pattern`` option,
+        - in the file given with ``--patterns-from`` option and
+        - for ``PATH`` arguments that explicitly support them.
+
+        Borg always stores all file paths normalized and relative to the
+        current recursion root. The recursion root is also named ``PATH`` in
+        Borg commands like `borg create` that do a file discovery, so do not
+        confuse the root with the ``PATH`` argument of e.g. `borg extract`.
+
+        Starting with Borg 1.2, paths that are matched against patterns always
+        appear relative. If you give ``/absolute/`` as root, the paths going
+        into the matcher will look relative like ``absolute/.../file.ext``.
+        If you give ``../some/path`` as root, the paths will look like
+        ``some/path/.../file.ext``.
+
+        File patterns support five different styles. If followed by a colon ':',
+        the first two characters of a pattern are used as a style selector.
+        Explicit style selection is necessary if a non-default style is desired
+        or when the desired pattern starts with two alphanumeric characters
+        followed by a colon (i.e. ``aa:something/*``).
+
+        `Fnmatch <https://docs.python.org/3/library/fnmatch.html>`_, selector ``fm:``
             This is the default style for ``--exclude`` and ``--exclude-from``.
             These patterns use a variant of shell pattern syntax, with '\\*' matching
             any number of characters, '?' matching any single character, '[...]'
@@ -2526,7 +2521,7 @@ def do_break_lock(self, args, repository):
             matching any character not specified. For the purpose of these patterns,
             the path separator (backslash for Windows and '/' on other systems) is not
             treated specially. Wrap meta-characters in brackets for a literal
-            match (i.e. `[?]` to match the literal character `?`). For a path
+            match (i.e. ``[?]`` to match the literal character '?'). For a path
             to match a pattern, the full path must match, or it must match
             from the start of the full path to just before a path separator. Except
             for the root path, paths will never end in the path separator when
@@ -2534,33 +2529,31 @@ def do_break_lock(self, args, repository):
             separator, a '\\*' is appended before matching is attempted. A leading
             path separator is always removed.
 
-        Shell-style patterns, selector `sh:`
+        Shell-style patterns, selector ``sh:``
             This is the default style for ``--pattern`` and ``--patterns-from``.
             Like fnmatch patterns these are similar to shell patterns. The difference
-            is that the pattern may include `**/` for matching zero or more directory
-            levels, `*` for matching zero or more arbitrary characters with the
+            is that the pattern may include ``**/`` for matching zero or more directory
+            levels, ``*`` for matching zero or more arbitrary characters with the
             exception of any path separator. A leading path separator is always removed.
 
-        Regular expressions, selector `re:`
-            Regular expressions similar to those found in Perl are supported. Unlike
-            shell patterns regular expressions are not required to match the full
+        `Regular expressions <https://docs.python.org/3/library/re.html>`_, selector ``re:``
+            Unlike shell patterns, regular expressions are not required to match the full
             path and any substring match is sufficient. It is strongly recommended to
             anchor patterns to the start ('^'), to the end ('$') or both. Path
             separators (backslash for Windows and '/' on other systems) in paths are
-            always normalized to a forward slash ('/') before applying a pattern. The
-            regular expression syntax is described in the `Python documentation for
-            the re module <https://docs.python.org/3/library/re.html>`_.
+            always normalized to a forward slash '/' before applying a pattern.
 
-        Path prefix, selector `pp:`
+        Path prefix, selector ``pp:``
             This pattern style is useful to match whole sub-directories. The pattern
-            `pp:root/somedir` matches `root/somedir` and everything therein. A leading
-            path separator is always removed.
+            ``pp:root/somedir`` matches ``root/somedir`` and everything therein.
+            A leading path separator is always removed.
 
-        Path full-match, selector `pf:`
+        Path full-match, selector ``pf:``
             This pattern style is (only) useful to match full paths.
             This is kind of a pseudo pattern as it can not have any variable or
-            unspecified parts - the full path must be given. `pf:root/file.ext` matches
-            `root/file.ext` only. A leading path separator is always removed.
+            unspecified parts - the full path must be given. ``pf:root/file.ext``
+            matches ``root/file.ext`` only. A leading path separator is always
+            removed.
 
             Implementation note: this is implemented via very time-efficient O(1)
             hashtable lookups (this means you can have huge amounts of such patterns
@@ -2573,20 +2566,20 @@ def do_break_lock(self, args, repository):
 
         .. note::
 
-            `re:`, `sh:` and `fm:` patterns are all implemented on top of the Python SRE
-            engine. It is very easy to formulate patterns for each of these types which
-            requires an inordinate amount of time to match paths. If untrusted users
-            are able to supply patterns, ensure they cannot supply `re:` patterns.
-            Further, ensure that `sh:` and `fm:` patterns only contain a handful of
-            wildcards at most.
+            ``re:``, ``sh:`` and ``fm:`` patterns are all implemented on top of
+            the Python SRE engine. It is very easy to formulate patterns for each
+            of these types which requires an inordinate amount of time to match
+            paths. If untrusted users are able to supply patterns, ensure they
+            cannot supply ``re:`` patterns. Further, ensure that ``sh:`` and
+            ``fm:`` patterns only contain a handful of wildcards at most.
 
         Exclusions can be passed via the command line option ``--exclude``. When used
         from within a shell, the patterns should be quoted to protect them from
         expansion.
 
         The ``--exclude-from`` option permits loading exclusion patterns from a text
-        file with one pattern per line. Lines empty or starting with the number sign
-        ('#') after removing whitespace on both ends are ignored. The optional style
+        file with one pattern per line. Lines empty or starting with the hash sign
+        '#' after removing whitespace on both ends are ignored. The optional style
         selector prefix is also supported for patterns loaded from a file. Due to
         whitespace removal, paths with whitespace at the beginning or end can only be
         excluded using regular expressions.
@@ -2597,21 +2590,21 @@ def do_break_lock(self, args, repository):
         Examples::
 
             # Exclude '/home/user/file.o' but not '/home/user/file.odt':
-            $ borg create -e '*.o' backup /
+            $ borg create -e '*.o' /path/to/repo::archive /
 
             # Exclude '/home/user/junk' and '/home/user/subdir/junk' but
             # not '/home/user/importantjunk' or '/etc/junk':
-            $ borg create -e 'home/*/junk' backup /
+            $ borg create -e 'home/*/junk' /path/to/repo::archive /
 
             # Exclude the contents of '/home/user/cache' but not the directory itself:
-            $ borg create -e home/user/cache/ backup /
+            $ borg create -e home/user/cache/ /path/to/repo::archive /
 
             # The file '/home/user/cache/important' is *not* backed up:
-            $ borg create -e home/user/cache/ backup / /home/user/cache/important
+            $ borg create -e home/user/cache/ /path/to/repo::archive / /home/user/cache/important
 
             # The contents of directories in '/home' are not backed up when their name
             # ends in '.tmp'
-            $ borg create --exclude 're:^home/[^/]+\\.tmp/' backup /
+            $ borg create --exclude 're:^home/[^/]+\\.tmp/' /path/to/repo::archive /
 
             # Load exclusions from file
             $ cat >exclude.txt <<EOF
@@ -2624,36 +2617,56 @@ def do_break_lock(self, args, repository):
             # Example with spaces, no need to escape as it is processed by borg
             some file with spaces.txt
             EOF
-            $ borg create --exclude-from exclude.txt backup /
+            $ borg create --exclude-from exclude.txt /path/to/repo::archive /
 
-        A more general and easier to use way to define filename matching patterns exists
-        with the ``--pattern`` and ``--patterns-from`` options. Using these, you may
-        specify the backup roots (starting points) and patterns for inclusion/exclusion.
-        A root path starts with the prefix `R`, followed by a path (a plain path, not a
-        file pattern). An include rule starts with the prefix +, an exclude rule starts
-        with the prefix -, an exclude-norecurse rule starts with !, all followed by a pattern.
+        A more general and easier to use way to define filename matching patterns
+        exists with the ``--pattern`` and ``--patterns-from`` options. Using
+        these, you may specify the backup roots, default pattern styles and
+        patterns for inclusion and exclusion.
 
-        .. note::
+        Root path prefix ``R``
+            A recursion root path starts with the prefix ``R``, followed by a path
+            (a plain path, not a file pattern). Use this prefix to have the root
+            paths in the patterns file rather than as command line arguments.
 
-            Via ``--pattern`` or ``--patterns-from`` you can define BOTH inclusion and exclusion
-            of files using pattern prefixes ``+`` and ``-``. With ``--exclude`` and
-            ``--exclude-from`` ONLY excludes are defined.
+        Pattern style prefix ``P``
+            To change the default pattern style, use the ``P`` prefix, followed by
+            the pattern style abbreviation (``fm``, ``pf``, ``pp``, ``re``, ``sh``).
+            All patterns following this line will use this style until another style
+            is specified.
 
-        Inclusion patterns are useful to include paths that are contained in an excluded
-        path. The first matching pattern is used so if an include pattern matches before
-        an exclude pattern, the file is backed up. If an exclude-norecurse pattern matches
-        a directory, it won't recurse into it and won't discover any potential matches for
-        include rules below that directory.
+        Exclude pattern prefix ``-``
+            Use the prefix ``-``, followed by a pattern, to define an exclusion.
+            This has the same effect as the ``--exclude`` option.
 
-        .. note::
+        Exclude no-recurse pattern prefix ``!``
+            Use the prefix ``!``, followed by a pattern, to define an exclusion
+            that does not recurse into subdirectories. This saves time, but
+            prevents include patterns to match any files in subdirectories.
+
+        Include pattern prefix ``+``
+            Use the prefix ``+``, followed by a pattern, to define inclusions.
+            This is useful to include paths that are covered in an exclude
+            pattern and would otherwise not be backed up.
+
+        The first matching pattern is used, so if an include pattern matches
+        before an exclude pattern, the file is backed up. Note that a no-recurse
+        exclude stops examination of subdirectories so that potential includes
+        will not match - use normal exludes for such use cases.
+
+        **Tip: You can easily test your patterns with --dry-run and  --list**::
 
-            It's possible that a sub-directory/file is matched while parent directories are not.
-            In that case, parent directories are not backed up thus their user, group, permission,
-            etc. can not be restored.
+            $ borg create --dry-run --list --patterns-from patterns.txt /path/to/repo::archive
 
-        Note that the default pattern style for ``--pattern`` and ``--patterns-from`` is
-        shell style (`sh:`), so those patterns behave similar to rsync include/exclude
-        patterns. The pattern style can be set via the `P` prefix.
+        This will list the considered files one per line, prefixed with a
+        character that indicates the action (e.g. 'x' for excluding, see
+        **Item flags** in `borg create` usage docs).
+
+        .. note::
+
+            It's possible that a sub-directory/file is matched while parent
+            directories are not. In that case, parent directories are not backed
+            up and thus their user, group, permission, etc. cannot be restored.
 
         Patterns (``--pattern``) and excludes (``--exclude``) from the command line are
         considered first (in the order of appearance). Then patterns from ``--patterns-from``
@@ -2663,44 +2676,44 @@ def do_break_lock(self, args, repository):
 
             # backup pics, but not the ones from 2018, except the good ones:
             # note: using = is essential to avoid cmdline argument parsing issues.
-            borg create --pattern=+pics/2018/good --pattern=-pics/2018 repo::arch pics
+            borg create --pattern=+pics/2018/good --pattern=-pics/2018 /path/to/repo::archive pics
 
-            # use a file with patterns:
-            borg create --patterns-from patterns.lst repo::arch
+            # backup only JPG/JPEG files (case insensitive) in all home directories:
+            borg create --pattern '+ re:\\.jpe?g(?i)$' /path/to/repo::archive /home
+
+            # backup homes, but exclude big downloads (like .ISO files) or hidden files:
+            borg create --exclude 're:\\.iso(?i)$' --exclude 'sh:home/**/.*' /path/to/repo::archive /home
+
+            # use a file with patterns (recursion root '/' via command line):
+            borg create --patterns-from patterns.lst /path/to/repo::archive /
 
         The patterns.lst file could look like that::
 
-            # "sh:" pattern style is the default, so the following line is not needed:
-            P sh
-            R /
-            # can be rebuild
+            # "sh:" pattern style is the default
+            # exclude caches
             - home/*/.cache
-            # they're downloads for a reason
-            - home/*/Downloads
-            # susan is a nice person
             # include susans home
             + home/susan
             # also back up this exact file
             + pf:home/bobby/specialfile.txt
             # don't backup the other home directories
             - home/*
-            # don't even look in /proc
-            ! proc
+            # don't even look in /dev, /proc, /run, /sys, /tmp (note: would exclude files like /device, too)
+            ! re:^(dev|proc|run|sys|tmp)
 
         You can specify recursion roots either on the command line or in a patternfile::
 
             # these two commands do the same thing
-            borg create --exclude home/bobby/junk repo::arch /home/bobby /home/susan
-            borg create --patterns-from patternfile.lst repo::arch
+            borg create --exclude home/bobby/junk /path/to/repo::archive /home/bobby /home/susan
+            borg create --patterns-from patternfile.lst /path/to/repo::archive
 
-        The patternfile::
+        patternfile.lst::
 
             # note that excludes use fm: by default and patternfiles use sh: by default.
             # therefore, we need to specify fm: to have the same exact behavior.
             P fm
             R /home/bobby
             R /home/susan
-
             - home/bobby/junk
 
         This allows you to share the same patterns between multiple repositories


Test patch: