Repository: borgbackup/borg

Issue:
borg 1.2 regular expression pattern behavior change
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Seems like a bug as it's not announced behaviour change, and dangerous as it suddenly includes files (or no longer includes files) it has included before.

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

1.2 locally, no idea remote

#### Operating system (distribution) and version.

Ubuntu 22.04

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

`borg create --progress --one-file-system --exclude-caches --info --stats --exclude-from=/home/jak/.config/borg/exclude.conf -C zlib,9 $repostory::$archive /home/jak`


## Describe the problem you're observing.

`re:` patterns no longer apply to the full filename, but to the relative one in the archive. Meaning I suddenly had all my regex excluded files pop up in my backups, such as files in `/home/jak/mail/.notmuch` which are huge.

I had to remove the leading `/` to get the pattern to match again, as can be seen in https://github.com/julian-klode/dotfiles/commit/7a6f1e0437fe96311c59a38cbf2c743043941e31 and in this excerpt:

```diff
-re:/home/jak/.*/\.notmuch
+re:home/jak/.*/\.notmuch
```





Discussion:
Smaller reproducer:

```
jak@jak-t480s:/tmp$ mkdir in out
jak@jak-t480s:/tmp$ borg init -e none out
jak@jak-t480s:/tmp$ touch in/foo
jak@jak-t480s:/tmp$ touch in/fooz
jak@jak-t480s:/tmp$ touch in/bar
jak@jak-t480s:/tmp$ borg create --exclude "re:/tmp/in/foo" out::slash /tmp/in/
jak@jak-t480s:/tmp$ borg create --exclude "re:tmp/in/foo" out::noslash /tmp/in/
jak@jak-t480s:/tmp$ borg list out::slash
drwxrwxr-x jak    jak           0 Sat, 2022-03-05 19:41:53 tmp/in
-rw-rw-r-- jak    jak           0 Sat, 2022-03-05 19:41:51 tmp/in/foo
-rw-rw-r-- jak    jak           0 Sat, 2022-03-05 19:41:52 tmp/in/fooz
-rw-rw-r-- jak    jak           0 Sat, 2022-03-05 19:41:53 tmp/in/bar
jak@jak-t480s:/tmp$ borg list out::noslash
drwxrwxr-x jak    jak           0 Sat, 2022-03-05 19:41:53 tmp/in
-rw-rw-r-- jak    jak           0 Sat, 2022-03-05 19:41:53 tmp/in/bar
```
I confirm this bug with borg 1.2.0. The following pattern no longer works to exclude files (while it worked with the previous 1.1.x version):

    --pattern=-re:"^$SOURCE_DIR/\.s-data/.+\.(log|sav)$"

@misa-ki please expand `$SOURCE_DIR`. 
Here:

    SOURCE_DIR="/home/sd1"
    ...
    ...
    --pattern=-re:"^$SOURCE_DIR/\.s-data/.+\.(log|sav)$"

I've stumbled over this, too, but not with regular expression patterns, but with fnmatch patterns in an `--exclude-from` file.

I had patterns in there like
```
*/proc/*
*/sys/*
```
(In an all-purpose borg excludes file I'm using for all the machines I administrate. On some of them, those special directories are under a different subdir.)

These used to exclude the `/proc` and `/sys` directories, but not anymore. I had to add, in this example, `proc/*` and `sys/*` to make it work.
Any chance to get this fixed soon?
Just upgraded to 1.2 and made my first backup since the upgrade. It seems it works fine with the `--patterns-from` option. That uses shell-style patterns, tho (according to the docs).
This has caused my repos to balloon in size. Shouldn't there be a prominent mention of this in the release notes?
Running `git bisect` now to find the changeset that introduced the behaviour change...

This is the changeset that changes state from good to bad with the reproducer script:

```
[ece47ee9e40f0c069f48533d34be8f2fa5bff5ad] [BUGFIX] Borg commands accept absolute pathes, #4029
```

#4029 #5133

Sidenote: I thought I could use `git bisect run reproducer.sh`, but the multithreading in setup.py / cythonize caused isses at install time (which are meanwhile fixed, but not in older revisions).

@fantasya-pbem can you have a look?

if you remember what precisely #5133 fixed, please leave a comment there (I tried `borg create repo::archive /somepath` and it worked for me **without #5133**, so it is unclear to me what #5133 was about).

I'm not sure if #5133 is the only cause of this issue. See here: https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns

_„Starting with Borg 1.2, for all but regular expression pattern matching styles, all paths are treated as relative, meaning that a leading path separator is removed after normalizing and before matching. This allows you to use absolute or relative patterns arbitrarily.“_


This is what I used for git bisect (and bisect found exactly the linked changeset for the good -> bad transition):
```
## reproduce-6407.sh

cd ~/w/borg
export BORG_OPENSSL_PREFIX=/opt/homebrew/opt/openssl@1.1
export BORG_LZ4_PREFIX=/opt/homebrew
export BORG_ZSTD_PREFIX=/opt/homebrew
python setup.py clean
python setup.py clean2
pip install -e .

cd /tmp
mkdir in out
borg init -e none out
touch in/foo
touch in/fooz
touch in/bar

borg create --exclude "re:/tmp/in/foo" out::slash /tmp/in/
borg list out::slash | grep foo >/dev/null 2>&1
export RETCODE=$?
borg delete --force out
rm -rf /tmp/in
if [ $RETCODE = 1 ]; then
    echo "good"
    exit 0
else
    echo "bad"
    exit 1
fi
```

But of course could be that at another place some leading slash should also get removed.

Still it would be interesting what precisely that changeset fixed, can you tell?
#5133 was just an addition to #5099 where we made the big change (remove leading slashes). No, I cannot tell or remember more details.
@fantasya-pbem do you think you could fix this issue?
Bounty: https://app.bountysource.com/issues/106797173-borg-1-2-regular-expression-pattern-behavior-change

From my point of view there is no issue (in the sense of a bug) in Borg, but in the exclude file / the usage of exclude patterns. As all paths are considered relative, we made these changes to make clear this fact. So that no absolute paths should be used. In addition to the change in patterns.py I also had to adapt the testsuite - a clear sign that using absolute paths is considered wrong. Of course - this may be seen as a breaking change in Borg 1.2 that we maybe have not emphasized enough in the release.
Maybe this sentence in the "borg help patterns" docs is misleading:

_„Starting with Borg 1.2, for all but regular expression pattern matching styles, all paths are treated as relative, meaning that a leading path separator is removed after normalizing and before matching. **This allows you to use absolute or relative patterns arbitrarily.**“_

But...

@ThomasWaldmann You wrote: _„But of course could be that at another place some leading slash should also get removed.“_

Maybe this is the cause of problems - the leading separators in the exclude files' paths may be removed before matching against backup file paths. Is this the basic issue we have here?
>As all paths are considered relative, we made these changes to make clear this fact. So that no absolute paths should be used.

How about we try to solve this from the other end: What did people expect, what on their end broke and why did they get surprised? I can only speak for myself, but I guess others can add their cases later.

For me:

### a) It did work before

Let's be honest: We all mostly copy working stuff from a to b. In my case things like `-e re:^/tmp/` to exclude `/tmp` but not `/var/tmp` where everywhere. When it no longer matched _/tmp_ several of my exclusion broke, resulting in unexpected backup sizes (hello proc pagemaps). I guess `-e /tmp` would be a better way (or would there the leading slash get removed as well?), but as said: I only copied it from _somewhere_. Even the [documentation](https://github.com/borgbackup/borg/blob/299196ebcb0fe89b28abef750873f6745c7646a6/docs/usage/help.rst.inc#L145) shows examples with a leading `/` like `re:^/home/[^/]+\.tmp/` which now probably won't ever match.

### b) Alternative?

While `re:^/tmp/` is somewhat easy to replace, others might be a bit more complex: I want to exclude every folder named _tmp_ contained in a folder named _var_. Up until recently `re:^.*/var/tmp/` did the trick, now it will exclude _/my/chroot/var/tmp/_ but miss _/var/tmp/_. `-e /var/tmp/` would (if I guess right) only match _/var/tmp/_ but not _/my/chroot/var/tmp/_. And `re:^.*var/tmp/` would work, but also catch something like _/home/god/earth/cultivar/tmp/_. So one would need to match the start of a string or a slash. Sure, it's possible, but I don't think `^(.*/)?var/tmp/` is quite intuitive.

So - I hope you see why exactly this broke for me without a warning. What would be the proper way to catch these cases? And how could we avoid others from falling into this trap when just copying stuff from the internet, that at least partly worked with previous versions?
In the last paragraph of my previous post I suggested (without having confirmed if this solves the issue) to accept both patterns with or without leading separator and "fix" them as needed everywhere in the code where they are compared against file paths. This would fix the BC so that nobody had to adapt their patterns.

For me, two things have to be confirmed first:

1. Did I correctly understand the problems here?
2. If so, are there any line of code left where we have to clean up the patterns before applying them?
I've read through all the stuff and here is my view and todo list suggestion:

- this is not a bug, but an intended change that aligns behaviour as good as possible. borg archives stuff without leading slashes, so having the patterns / matcher behave in the same way is a good thing.
- this needs to be pointed out in the changes.rst, in the compatibility notes. sorry that this was overlooked in 1.2.0, so people ran into this without warning.
- regex is not the only issue, but also non-trivial fnmatch patterns
- fix this:
```
Starting with Borg 1.2, discovered fs paths are normalised, have leading
slashes removed and then are matched against your patterns.
Note: you need to review your include / exclude patterns and make
sure they do not expect leading slashes. borg can only deal with this
for some very simple patterns by removing leading slashes there also.
```
- fix the wrong example(s) in the docs, e.g. this one: https://github.com/borgbackup/borg/blob/299196ebcb0fe89b28abef750873f6745c7646a6/docs/usage/help.rst.inc#L145

Anything missing?

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2489,15 +2489,12 @@ def do_break_lock(self, args, repository):
         currently active recursion root. You usually give the recursion root(s)
         when invoking borg and these can be either relative or absolute paths.
 
-        So, when you give `relative/` as root, the paths going into the matcher
-        will look like `relative/.../file.ext`. When you give `/absolute/` as
-        root, they will look like `/absolute/.../file.ext`.
-
-        File paths in Borg archives are always stored normalized and relative.
-        This means that e.g. ``borg create /path/to/repo ../some/path`` will
-        store all files as `some/path/.../file.ext` and ``borg create
-        /path/to/repo /home/user`` will store all files as
-        `home/user/.../file.ext`.
+        If you give `/absolute/` as root, the paths going into the matcher will
+        look relative like `absolute/.../file.ext`, because file paths in Borg
+        archives are always stored normalized and relative. This means that e.g.
+        ``borg create /path/to/repo ../some/path`` will store all files as
+        `some/path/.../file.ext` and ``borg create /path/to/repo /home/user``
+        will store all files as `home/user/.../file.ext`.
 
         A directory exclusion pattern can end either with or without a slash ('/').
         If it ends with a slash, such as `some/path/`, the directory will be
@@ -2510,10 +2507,11 @@ def do_break_lock(self, args, repository):
         option. For commands that support patterns in their ``PATH`` argument
         like (``borg list``), the default pattern is path prefix.
 
-        Starting with Borg 1.2, for all but regular expression pattern matching
-        styles, all paths are treated as relative, meaning that a leading path
-        separator is removed after normalizing and before matching. This allows
-        you to use absolute or relative patterns arbitrarily.
+        Starting with Borg 1.2, discovered fs paths are normalised, have leading
+        slashes removed and then are matched against your patterns.
+        Note: You need to review your include / exclude patterns and make
+        sure they do not expect leading slashes. Borg can only deal with this
+        for some very simple patterns by removing leading slashes there also.
 
         If followed by a colon (':') the first two characters of a pattern are
         used as a style selector. Explicit style selection is necessary when a
@@ -2603,26 +2601,26 @@ def do_break_lock(self, args, repository):
 
             # Exclude '/home/user/junk' and '/home/user/subdir/junk' but
             # not '/home/user/importantjunk' or '/etc/junk':
-            $ borg create -e '/home/*/junk' backup /
+            $ borg create -e 'home/*/junk' backup /
 
             # Exclude the contents of '/home/user/cache' but not the directory itself:
             $ borg create -e home/user/cache/ backup /
 
             # The file '/home/user/cache/important' is *not* backed up:
-            $ borg create -e /home/user/cache/ backup / /home/user/cache/important
+            $ borg create -e home/user/cache/ backup / /home/user/cache/important
 
             # The contents of directories in '/home' are not backed up when their name
             # ends in '.tmp'
-            $ borg create --exclude 're:^/home/[^/]+\\.tmp/' backup /
+            $ borg create --exclude 're:^home/[^/]+\\.tmp/' backup /
 
             # Load exclusions from file
             $ cat >exclude.txt <<EOF
             # Comment line
-            /home/*/junk
+            home/*/junk
             *.tmp
             fm:aa:something/*
-            re:^/home/[^/]+\\.tmp/
-            sh:/home/*/.thumbnails
+            re:^home/[^/]+\\.tmp/
+            sh:home/*/.thumbnails
             # Example with spaces, no need to escape as it is processed by borg
             some file with spaces.txt
             EOF
@@ -2676,23 +2674,23 @@ def do_break_lock(self, args, repository):
             P sh
             R /
             # can be rebuild
-            - /home/*/.cache
+            - home/*/.cache
             # they're downloads for a reason
-            - /home/*/Downloads
+            - home/*/Downloads
             # susan is a nice person
             # include susans home
-            + /home/susan
+            + home/susan
             # also back up this exact file
-            + pf:/home/bobby/specialfile.txt
+            + pf:home/bobby/specialfile.txt
             # don't backup the other home directories
-            - /home/*
+            - home/*
             # don't even look in /proc
-            ! /proc
+            ! proc
 
         You can specify recursion roots either on the command line or in a patternfile::
 
             # these two commands do the same thing
-            borg create --exclude /home/bobby/junk repo::arch /home/bobby /home/susan
+            borg create --exclude home/bobby/junk repo::arch /home/bobby /home/susan
             borg create --patterns-from patternfile.lst repo::arch
 
         The patternfile::
@@ -2700,10 +2698,10 @@ def do_break_lock(self, args, repository):
             # note that excludes use fm: by default and patternfiles use sh: by default.
             # therefore, we need to specify fm: to have the same exact behavior.
             P fm
-            R /home/bobby
-            R /home/susan
+            R home/bobby
+            R home/susan
 
-            - /home/bobby/junk
+            - home/bobby/junk
 
         This allows you to share the same patterns between multiple repositories
         without needing to specify them on the command line.\n\n''')


Test patch: