Repository: borgbackup/borg

Issue:
Dry Run Statistics, please implement or document
Hi,

this is a follow-up to #265.

I don't agree with the with the way it was closed. And I was tricked by the same assumption as shown there, and there is STILL no documentation nor implementation of this functionality.

Seriously, I do expect that "--stats -C none ..." would give me some useful information. That is what I get from rsync when used for similar purpose ("rsync -a --stats --checksum ...") and a such limitation in borg seems to be weird, frankly speaking.
Dry Run Statistics, please implement or document
Hi,

this is a follow-up to #265.

I don't agree with the with the way it was closed. And I was tricked by the same assumption as shown there, and there is STILL no documentation nor implementation of this functionality.

Seriously, I do expect that "--stats -C none ..." would give me some useful information. That is what I get from rsync when used for similar purpose ("rsync -a --stats --checksum ...") and a such limitation in borg seems to be weird, frankly speaking.


Discussion:
The major use case for `borg create --dry-run` was to optimize the exclude options.

Thus, it doesn't do much except recursing over the the input directories, doing pattern matching on the file names. It does that rather fast though, by getting out there quickly before the real processing of a file begins (reading, chunking, hashing, updating the repo). Adding --stats thus gives no useful information.

Adding a useful --dry-run --stats functionality is not easily possible, I think:
- dry-run requires no changes to the repo
- the deduplication processing (and stats) only works if the repo, chunks index, etc. is updated
- compression processing (and stats) only works if stuff is actually compressed (taking quite some time)

So, there is only one, slightly "dirty" option coming to mind:
It could do a normal borg create run and just not commit the result.
That would give correct stats, but also potentially need a lot of resources (time, space, ...), so not sure we want that. If one wants that, ony could also run a normal borg create and delete the created archive again, which would be a lot cleaner (and is already working with current code).

So, we could just add code that outputs that --stats is not available together with --dry-run.
> The major use case for borg create --dry-run was to optimize the exclude options.

Well, this is exactly what I had in mind. Make a dry-run, estimate whether the data would fit on the target filesystem (even in worse case without compression).

Actually, I was starting with an existing borg archive (which contained an old/similar version of the same source data), I would also like to know whether it's worth to keep the old repo or restart from scratch.

If you say that making a "test commit" and dropping it later is totally harmless then I agree with your conclusion. But please document it somehow in a more appropriate way, ideally in the manpage and also in the program output.

PS: and I still don't understand what you tried to say with "The major use case was ... to optimize the exclude options". Was this a use case for regular users or for developers? From user's point of view, this feature basically does NOT work. There is no way for me to make a QUICK check to see whether my exclude file worked or not. I.e. even if I do RTFM and run something like `borg create --stats --exclude-from exclude.stuff.txt -verbose $PWD::$(date +%F) /src` and then it runs for hours (storage medium is slow) and only then I can see whether the pattern in `exclude.stuff.txt` was correct or not. It's just PITA.
If you run `borg create --dry-run --list --exclude ...` you will get a list of all files it would back up.

By just looking through that list, you can usually:
- spot some files you don't want to backup: downloads, ISOs here and there, unimportant VM images, cache directories, ...
- check whether any include/exclude patterns you have used do really work (it's easy to have typos or syntax issues there)

That's what it was made for. Not for estimating dedup or compression efficiency.

The major use case for `borg create --dry-run` was to optimize the exclude options.

Thus, it doesn't do much except recursing over the the input directories, doing pattern matching on the file names. It does that rather fast though, by getting out there quickly before the real processing of a file begins (reading, chunking, hashing, updating the repo). Adding --stats thus gives no useful information.

Adding a useful --dry-run --stats functionality is not easily possible, I think:
- dry-run requires no changes to the repo
- the deduplication processing (and stats) only works if the repo, chunks index, etc. is updated
- compression processing (and stats) only works if stuff is actually compressed (taking quite some time)

So, there is only one, slightly "dirty" option coming to mind:
It could do a normal borg create run and just not commit the result.
That would give correct stats, but also potentially need a lot of resources (time, space, ...), so not sure we want that. If one wants that, ony could also run a normal borg create and delete the created archive again, which would be a lot cleaner (and is already working with current code).

So, we could just add code that outputs that --stats is not available together with --dry-run.
> The major use case for borg create --dry-run was to optimize the exclude options.

Well, this is exactly what I had in mind. Make a dry-run, estimate whether the data would fit on the target filesystem (even in worse case without compression).

Actually, I was starting with an existing borg archive (which contained an old/similar version of the same source data), I would also like to know whether it's worth to keep the old repo or restart from scratch.

If you say that making a "test commit" and dropping it later is totally harmless then I agree with your conclusion. But please document it somehow in a more appropriate way, ideally in the manpage and also in the program output.

PS: and I still don't understand what you tried to say with "The major use case was ... to optimize the exclude options". Was this a use case for regular users or for developers? From user's point of view, this feature basically does NOT work. There is no way for me to make a QUICK check to see whether my exclude file worked or not. I.e. even if I do RTFM and run something like `borg create --stats --exclude-from exclude.stuff.txt -verbose $PWD::$(date +%F) /src` and then it runs for hours (storage medium is slow) and only then I can see whether the pattern in `exclude.stuff.txt` was correct or not. It's just PITA.
If you run `borg create --dry-run --list --exclude ...` you will get a list of all files it would back up.

By just looking through that list, you can usually:
- spot some files you don't want to backup: downloads, ISOs here and there, unimportant VM images, cache directories, ...
- check whether any include/exclude patterns you have used do really work (it's easy to have typos or syntax issues there)

That's what it was made for. Not for estimating dedup or compression efficiency.


Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2669,8 +2669,9 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
 
         When using ``--stats``, you will get some statistics about how much data was
         added - the "This Archive" deduplicated size there is most interesting as that is
-        how much your repository will grow.
-        Please note that the "All archives" stats refer to the state after creation.
+        how much your repository will grow. Please note that the "All archives" stats refer to
+        the state after creation. Also, the ``--stats`` and ``--dry-run`` options are mutually
+        exclusive because the data is not actually compressed and deduplicated during a dry run.
 
         See the output of the "borg help patterns" command for more help on exclude patterns.
         See the output of the "borg help placeholders" command for more help on placeholders.
@@ -2738,10 +2739,12 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
                                           help='create backup')
         subparser.set_defaults(func=self.do_create)
 
-        subparser.add_argument('-n', '--dry-run', dest='dry_run', action='store_true',
+        dryrun_group = subparser.add_mutually_exclusive_group()
+        dryrun_group.add_argument('-n', '--dry-run', dest='dry_run', action='store_true',
                                help='do not create a backup archive')
-        subparser.add_argument('-s', '--stats', dest='stats', action='store_true',
+        dryrun_group.add_argument('-s', '--stats', dest='stats', action='store_true',
                                help='print statistics for the created archive')
+
         subparser.add_argument('--list', dest='output_list', action='store_true',
                                help='output verbose list of items (files, dirs, ...)')
         subparser.add_argument('--filter', metavar='STATUSCHARS', dest='output_filter',


Test patch: