Repository: borgbackup/borg

Issue:
[Question] borg create from stdin and caching
I was not able to found clarification or any description of this scenarion in documentation:

How metadata caching is working in case of creating snapshot from stdin pipe? Example:

`mysqldump ... | borg create $REPOSITORY::mysql-$now -`

Is there any sense to add option `--files-cache=ctime,size` in this case? It's not file, so may be better to add `--no-cache-sync` or `--no-files-cache`? Will deduplication be working in this case?

btw, does `--files-cache=ctime,size` doing the same as `--ignore-inode` ?



Discussion:
as stdin is only a stream of data, there is no such metadata associated with it.

so, the files cache should not be used for stdin at all. maybe we should check whether borg loads the files cache from disk in such a case (or stores it back to disk).

iirc, --ignore-inode is deprecated (do you see a warning?) and just removes `inode` from the `--files-cache` default of `ctime,size,inode`.
> cache should not be used for stdin at all

According to the documentation, "in order for the deduplication used by Borg to work, it needs to keep a local cache containing checksums of all file chunks already stored in the repository".

As I see by stats after `borg create` (and from repo size) - deduplication is working for backups from stdin.

Example for my random MySQL db (`mysqlump ... | borg create ... -`):

```
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               96.43 MB             11.32 MB            605.73 kB
```

So, how it is working then?
In my previous comment, I meant the `files cache`, not the `chunks cache`.
So, chunk cache is used, files cache is not used, it's ok to set `--no-files-cache` (or do not specify at all), but not good idea to set `--no-cache-sync`, am I correct?
Guess it should not do harm to use --no-files-cache, but maybe that is not even needed because borg does not load the files cache anyway when just processing stdin.

Don't use --no-cache-sync without good reason.
I checked the code:

`LocalCache._do_open` first reads the chunks cache and then the files cache (except if it is **disabled**).

If the files cache is loaded, it is also saved back to disk at the end. You can see that by giving --progress.

So, one could save a bit of files cache loading/saving time by disabling the files cache for doing stdin-only archiving.

Thanks! 
Maybe adding this to the docs would be useful, PRs welcome.
I'll do a PR for the docs.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -3238,7 +3238,8 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         directory.
 
         When giving '-' as path, borg will read data from standard input and create a
-        file 'stdin' in the created archive from that data.
+        file 'stdin' in the created archive from that data. See section *Reading from
+        stdin* below for details.
 
         The archive will consume almost no disk space for files or parts of files that
         have already been stored in other archives.
@@ -3357,6 +3358,25 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         - '-' = dry run, item was *not* backed up
         - 'x' = excluded, item was *not* backed up
         - '?' = missing status code (if you see this, please file a bug report!)
+
+        Reading from stdin
+        ++++++++++++++++++
+
+        To read from stdin, specify ``-`` as path and pipe directly to borg::
+
+            backup-vm --id myvm --stdout | borg create REPO::ARCHIVE -
+
+        Note that piping to borg creates an archive even if the command piping
+        to borg exits with a failure. In this case, **one can end up with
+        truncated output being backed up**.
+
+        Reading from stdin yields just a stream of data without file metadata
+        associated with it, and the files cache is not needed at all. So it is
+        safe to disable it via ``--no-files-cache`` and speed up backup
+        creation a bit.
+
+        By default, the content read from stdin is stored in a file called 'stdin'.
+        Use ``--stdin-name`` to change the name.
         """)
 
         subparser = subparsers.add_parser('create', parents=[common_parser], add_help=False,


Test patch: