Repository: borgbackup/borg

Issue:
Order of traversing files in a directory
Moving from #325 in a ticket of its own.

While looking around Archiver::_process I noticed that:

```
            if not dry_run:
                status = archive.process_dir(path, st)
            try:
                entries = os.listdir(path)
            except OSError as e:
                status = 'E'
                self.print_warning('%s: %s', path, e)
            else:
                for filename in sorted(entries):
                    entry_path = os.path.normpath(os.path.join(path, filename))
                    self._process(archive, cache, matcher, exclude_caches, exclude_if_present,
                                  keep_tag_files, skip_inodes, entry_path, restrict_dev,
                                  read_special=read_special, dry_run=dry_run)
```

I imagine the `sort(entries)` just sorts stuff alphabetically? If so, this is a pretty bad idea on a bunch of filesystems for large directories on spinning media. Affected filesystems include reiserfs, ext4 and probably many more that have static inode allocation.
The problem at hand is when you sort this list in an order different than inode numbers, every lstat that you do seeks the disk to a different inode table block somewhere which is really slow.
Not sorting the output is also not an option since then you get some "random" underlying hash sorting that also might not correspond to creation order. The most optimal strategy here for things with fixed inode tables is to sort by inode number (available in readdir output).
For filesystems with no static inode tables (like reiserfs) I imagine that would also lead to desired effect because it still is a good proxy for create time that leads to ondisk location (earlier created files get closer to start of disk in a chunk if they were written together).
More advanced COW filesystems like zfs and bttrfs - I am less sure, those might be all over the place depending on a bunch of stuff, though. I assume sort ordering would not matter there all that much.

A question was also raised if this makes any practical difference. And it certainly does for large directories (with millions of entries) that are not cached.
Here are some links to ext3 discussions about when htree code became popular and discussion arose. You'll see that performance impact is quite high:
https://bugzilla.kernel.org/show_bug.cgi?id=417
https://lkml.org/lkml/2008/2/18/263

There are tons more and the same issue is present in other filesystems with hashed names. I know about this because I am a Linux fs developer on my day job (and we met this when I was still working on reiserfs).

Additionally as we look outside of Linux (which is really optimized in these areas), other systems caches are (in some cases? in many cases) a lot more suboptimal or even non-exstent (e.g. in BSD where every fs driver used to need to implement their own cache).
I had a big confirmation of this yesterday. So I had a ~2M structure of files in on my Linux server, all on rotating media. find / -perm 0 (to cause stat calls) took 15 seconds there.
Now I also have this macbook I am working on, it has ssd and about 2.5M files on it, same find takes 2m50s. Quite a difference thanks to many-many variables I am sure.



Discussion:
If this has no functional reason - and I don't think it has - then we can just drop that sorting; since it's still a DFS traversal the usual directory-before-containing-files assertion holds. Some care should still be taken that we are not missing anything that depends on it.

The thing is, if you drop the sorting altogether, you don't gain anything.
What I am trying to say is that it would be great if you can sort on inode number (at least for some filesystems).

Stupid question: I always thought directory order ~ inode order of the dirents ... so that's probably wrong, or at least impl. defined?

Readdir order is usually the order the names are laid out in the directory itself. (but intermediate layers and some filesystems might reorder stuff - see the library referenced for example).

For simple filesystems readdir order is also the order in which files were created. But more modern systems use "name hash" to help with lookups, so readdir order and file creation orders are quite different.

What this leads to is:
When I do `for i in`seq 1000`; do touch $i ; done`

and then do `ls -U`, I get back my files in random order.
When caches are cold and I try to do stat on them in that readdir order - that makes hdd heads move all around from one inode table to another instead of moving nicely sequentially as I progress the files in the creation order.

Does this change things significantly for OSX? [feature/inorder](/enkore/borg/tree/feature/inorder)

N.b. experimental code, different error handling characteristics, probably eats your cat ;-)

Also, the actual entries are not stat'ed in inode order, because, uh, we seem to need stat to get to the inode of the entries. So, this shouldn't make a difference after all. Something I am missing here? -- ah, scandir will do. brb.

If I read this correctly (And this reminds me some perl confusing constructions now ;) ) it gets the readdir output, and then the "map" thing calls lstat on every element and then you sort based on inode number from struct stat?
This is not going to help because it's everybody's guess what order you do lstat calls in (probably native readdir order?).

Instead, when you do readdir you get a dirent structure (similar to other OSes too):

```
           struct old_linux_dirent {
               long  d_ino;              /* inode number */
               off_t d_off;              /* offset to this old_linux_dirent */
               unsigned short d_reclen;  /* length of this d_name */
               char  d_name[NAME_MAX+1]; /* filename (null-terminated) */
           }
```

So you already have the inode number in the readdir output and that's all you need to do the sorting. - Whenever python shows you this detail - I don't know. But if you plan to do stat beforehand, then might as well not bother sorting.

Also modern systems don't use readdir(2). but getdents64(2) that returns you:

```
           struct linux_dirent {
               unsigned long  d_ino;     /* Inode number */
               unsigned long  d_off;     /* Offset to next linux_dirent */
               unsigned short d_reclen;  /* Length of this linux_dirent */
               char           d_name[];  /* Filename (null-terminated) */
                                 /* length is actually (d_reclen - 2 -
                                    offsetof(struct linux_dirent, d_name)) */
               /*
               char           pad;       // Zero padding byte
               char           d_type;    // File type (only since Linux
                                         // 2.6.4); offset is (d_reclen - 1)
               */
           }
```

Now, this in addition to all other things also brings you the d_type - from which you can know what file type is that name correspond to without doing stat (useful in various workloads, as stat is actually quite heavy in many cases, but borg needs to do stat anyway so probably of no significant importance here, esp. since it's not portable).

Yes, I noticed now as well. I pushed a new version which uses Pythons scandir instead, which from the looks of it is a quite thin wrapper around readdir. This should now _really_ sort-then-stat.

This makes a consistent performance difference for me, about 10 seconds for

```
/bin/time sudo borg create testrepo::1234 --dry-run / -x
```

with master, but only ~4.5 seconds with this. Maybe I put a bug in that skips every other file? :-)

scandir is in stdlib since 3.5, so if we use that before requiring >= 3.5, we would need to add the scandir pypi package as addtl. dependency.

Can you redo your measurement for both versions, for both hot and cold cache?

Now this looks mostly good.

The only concern I have left is - does os.scandir return you fully qualified path name so you no longer need to do `entry_path = os.path.normpath(os.path.join(path, filename))` or was that an omission?

dirent.path is absolute if path in scandir(path) was absolute. So. Not sure. Didn't test if this is the same thing as before.

Thomas: We could just make it a soft-dependency, _if_ this proves to be a worthwhile improvement.

Edit: The difference comes mainly from listdir vs scandir it seems, not sorting scandir makes a bit of a difference (Linux, SSD, hot+cold), but not much, mostly for cold caches, for hot caches it is ~the same. Now, we already concluded that this doesn't affect Linux too much anyway, since it has optimizations for this. Benchmarks on OSX would be more interesting, I think, where verygreen saw these awfully long times for what's basically ls -R. Edit2: vvv Excellent, thanks!

I'll do some measurements a bit later today and post them here 

I did some 1.0.1 Instrumentation so far and pasting it here for the record.

The test case was to create 1M randomly named files in a dir. Here's the code with whch it was created:

```
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <linux/limits.h>

#define NUMBER 1000000

int main(int argc, char **argv)
{
    int i;
    char filename[PATH_MAX];

    if (argc != 2) {
        printf("Usage: Dir to create files in\n");
        return -1;
    }

    if (chdir(argv[1])) {
        perror("chdir");
        return -2;
    }

    for(i = 0 ; i <= NUMBER ; i++) {
        long rand;
        int fd;

        rand = random();

again:
        sprintf(filename, "%li\n", rand);
        fd = open(filename, O_CREAT|O_EXCL|O_WRONLY, 022);
        if ( fd == -1) {
            /* Must be a double hit, increase and try again */
            rand++;
            goto again;
        }

        write(fd, &rand, sizeof(rand) * 2); // some stack garbage too */
        close(fd);
    }

    return 0;
}
```

```
#### Testing on 7200 RPM single HDD, no other activity in the system or on it.
#### The node is i7 2600K, 32G RAM
[root@intelbox2 ~]# lvcreate -n TEST -L 100G slowdisk
  Logical volume "TEST" created.
[root@intelbox2 ~]# mkfs.ext4 /dev/slowdisk/TEST
mke2fs 1.42.13 (17-May-2015)
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: 8c7042cb-fbdc-448e-9e31-227cd2883ab2
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
[root@intelbox2 ~]# mount /dev/slowdisk/TEST /mnt/
[root@intelbox2 ~]# mkdir /mnt/test
[root@intelbox2 ~]# time ./create-a-mil /mnt/test/

real    1m22.594s
user    0m0.422s
sys     0m13.195s
[root@intelbox2 ~]# time sync

real    2m5.392s
user    0m0.000s
sys     0m0.243s

```

The 1.0.1 testing results:

```
### Testing with borg 1.0.1 release
[root@intelbox2 ~]# borg init --encryption=none /tmp/test-backup # tmpfs
[root@intelbox2 ~]# umount /mnt # Clear cache
[root@intelbox2 ~]# mount /dev/slowdisk/TEST /mnt/ 
[root@intelbox2 ~]# time borg create /tmp/test-backup::Initial /mnt/test

real    166m9.992s
user    6m33.515s
sys 1m43.802s
[root@intelbox2 ~]# time borg create /tmp/test-backup::HOT /mnt/test

real    1m25.694s
user    1m18.799s
sys 0m5.124s
[root@intelbox2 ~]# umount /mnt
[root@intelbox2 ~]# mount /dev/slowdisk/TEST /mnt/ 
[root@intelbox2 ~]# time borg create /tmp/test-backup::COLD /mnt/test

real    2m29.803s
user    1m23.781s
sys 0m9.970s
[root@intelbox2 ~]# time borg create /tmp/test-backup::HOT2 /mnt/test

real    1m26.764s
user    1m19.594s
sys 0m4.982s
[root@intelbox2 ~]# umount /mnt
[root@intelbox2 ~]# du -hs /tmp/test-backup 
293M    /tmp/test-backup
```

You can see how disproportionally large time it takes to create the archive. the time is spent thrashing the disk.

I am running borg 1.0.1.dev6+ng95f9286 and it looks like something still incorrectly sorts the dentries not by inode, but by name.
I did an strace and here's what I am seeing:

```
lstat("/mnt/test/1019429875\n", {st_mode=S_IFREG, st_size=16, ...}) = 0
open("/mnt/test/1019429875\n", O_RDONLY|O_NOATIME|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
ioctl(5, TCGETS, 0x7ffc41602cd0)        = -1 ENOTTY (Inappropriate ioctl for device)
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
lseek(5, 0, SEEK_CUR)                   = 0
read(5, "\363C\303<\0\0\0\00010194298", 8388608) = 16
fadvise64(5, 0, 16, POSIX_FADV_DONTNEED) = 0
close(5)                                = 0
llistxattr("/mnt/test/1019429875\n", NULL, 0) = 0
getxattr("/mnt/test/1019429875\n", "system.posix_acl_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("/mnt/test/1019429875\n", "system.posix_acl_default", 0x0, 0) = -1 ENODATA (No data available)
lstat("/mnt/test/1019430414\n", {st_mode=S_IFREG, st_size=16, ...}) = 0
open("/mnt/test/1019430414\n", O_RDONLY|O_NOATIME|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
ioctl(5, TCGETS, 0x7ffc41602cd0)        = -1 ENOTTY (Inappropriate ioctl for device)
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
lseek(5, 0, SEEK_CUR)                   = 0
read(5, "\16F\303<\0\0\0\00010194304", 8388608) = 16
fadvise64(5, 0, 16, POSIX_FADV_DONTNEED) = 0
close(5)                                = 0
llistxattr("/mnt/test/1019430414\n", NULL, 0) = 0
getxattr("/mnt/test/1019430414\n", "system.posix_acl_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("/mnt/test/1019430414\n", "system.posix_acl_default", 0x0, 0) = -1 ENODATA (No data available)
lstat("/mnt/test/1019431109\n", {st_mode=S_IFREG, st_size=16, ...}) = 0
open("/mnt/test/1019431109\n", O_RDONLY|O_NOATIME|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
ioctl(5, TCGETS, 0x7ffc41602cd0)        = -1 ENOTTY (Inappropriate ioctl for device)
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
lseek(5, 0, SEEK_CUR)                   = 0
read(5, "\305H\303<\0\0\0\00010194311", 8388608) = 16
fadvise64(5, 0, 16, POSIX_FADV_DONTNEED) = 0
close(5)                                = 0
llistxattr("/mnt/test/1019431109\n", NULL, 0) = 0
getxattr("/mnt/test/1019431109\n", "system.posix_acl_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("/mnt/test/1019431109\n", "system.posix_acl_default", 0x0, 0) = -1 ENODATA (No data available)
lstat("/mnt/test/1019433040\n", {st_mode=S_IFREG, st_size=16, ...}) = 0
open("/mnt/test/1019433040\n", O_RDONLY|O_NOATIME|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
ioctl(5, TCGETS, 0x7ffc41602cd0)        = -1 ENOTTY (Inappropriate ioctl for device)
fstat(5, {st_mode=S_IFREG, st_size=16, ...}) = 0
lseek(5, 0, SEEK_CUR)                   = 0
read(5, "PP\303<\0\0\0\00010194330", 8388608) = 16
fadvise64(5, 0, 16, POSIX_FADV_DONTNEED) = 0
close(5)                                = 0
llistxattr("/mnt/test/1019433040\n", NULL, 0) = 0
getxattr("/mnt/test/1019433040\n", "system.posix_acl_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("/mnt/test/1019433040\n", "system.posix_acl_default", 0x0, 0) = -1 ENODATA (No data available)
```

the names appear to be sorted by name, not by inode, for example:
[root@intelbox2 borg]# stat /mnt/test/1019430414*

```
  File: Б─≤/mnt/test/1019430414\nБ─≥
  Size: 16          Blocks: 8          IO Block: 4096   regular file
Device: fd0fh/64783d    Inode: 692616      Links: 1
Access: (0000/----------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-04-14 13:40:12.447085161 -0400
Modify: 2016-04-14 13:40:12.447085161 -0400
Change: 2016-04-14 13:40:12.447085161 -0400
 Birth: -
[root@intelbox2 borg]# stat /mnt/test/1019429875*
  File: Б─≤/mnt/test/1019429875\nБ─≥
  Size: 16          Blocks: 8          IO Block: 4096   regular file
Device: fd0fh/64783d    Inode: 1173226     Links: 1
Access: (0000/----------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-04-14 13:40:43.904431890 -0400
Modify: 2016-04-14 13:40:43.904431890 -0400
Change: 2016-04-14 13:40:43.904431890 -0400
 Birth: -
```

Also take a note of all the other unnecessary operations in the strace output:
llistxattr() is performed the last after we queried some posix ACL attrs - in fact you can do it first and if it reutnrs nothing - we know there are no Posix permissions I think?
Additionally, calling llistxattr like that is racy! we have opened the file already to read it anyway, so to ensure 100% mapping of data to attrs, flistxattr/fgetxattr should be used.
Even if you want to not use the file handle - lgetxattr must be used, otherwise symlinks would be processed incorrectly.
fstat is performed at least 3 times per file - huh? and that is AFTER we already did an lstat (ok, I guess to remain safe or races, at least one fstat is fine).
fadvise is performed after the read, huh? it's a "predeclare intents", so it needs to be done before the read so that the data is not cached and cache is not polluted.

whoops, I was testing it against master, not against the proper branch, let me restart this other thing.

AttributeError: 'module' object has no attribute 'scandir'

That's what I get when using the correct branch. I have python-scandir installed:
[root@intelbox2 ~]# pip3 list | grep scandir
You are using pip version 7.1.0, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
scandir (1.1)

You can patch borg/archiver.py

Somewhere at the top put (below the large chunk of imports)

``` py
try:
    from os import scandir, walk
except ImportError:
    from scandir import scandir, walk
```

Replace `os.scandir` with `scandir`. Should work now.

That it doesn't do inode order w/ this change is strange. I'll try debugging that here.

Edit: Can't reproduce this here (Linux). 

Breaking at line 336 (you can just insert a line `import pdb; pdb.set_trace()` to break) in archiver.py:

``` py
            try:
                entries = sorted(os.scandir(path), key=lambda dirent: dirent.inode()) <<<<
            except OSError as e:
```

I get (for a very simple case):

```
(Pdb) p list(map(lambda d: d.inode(), sorted(os.scandir(path), key=lambda dirent: dirent.inode())))
[114706, 114707, 114708, 114709]
```

I'll try with your test program.

> I'll try with your test program.

Also can't reproduce, patched this after the `entries = sorted(...` line:

``` py
                for e in entries:
                    print(e.inode())
```

```
$ ./vg-gen 1m
$ borg init m1repo
$ borg create m1repo/:1m 1m -vp > inodes.borg
# a good while later (also many errno 13, but that's not important here)
$ ls -l inodes.borg 
-rw-r--r-- 1 mabe mabe 11000011 Apr 15 00:09 inodes.borg
$ sort -n inodes.borg > inodes.sorted
$ diff inodes.borg inodes.sorted
$
```

Ok, the results for the patched code are in.
I am running with this patch so that it actually works:

```
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -16,6 +16,11 @@ import sys
 import textwrap
 import traceback

+try:
+    from os import scandir, walk
+except ImportError:
+    from scandir import scandir, walk
+
 from . import __version__
 from .helpers import Error, location_validator, archivename_validator, format_time, format_file_size, \
     parse_pattern, PathPrefixPattern, to_localtime, timestamp, \
@@ -333,7 +338,7 @@ class Archiver:
             if not dry_run:
                 status = archive.process_dir(path, st)
             try:
-                entries = sorted(os.scandir(path), key=lambda dirent: dirent.inode())
+                entries = sorted(scandir(path), key=lambda dirent: dirent.inode())
             except OSError as e:
                 status = 'E'
                 self.print_warning('%s: %s', path, e)
diff --git a/tox.ini b/tox.ini
index 0473cb2..22b01a3 100644
--- a/tox.ini
+++ b/tox.ini
@@ -11,6 +11,7 @@ changedir = {toxworkdir}
 deps =
      -rrequirements.d/development.txt
      attic
+     scandir
 commands = py.test --cov=borg --cov-config=../.coveragerc --benchmark-skip --pyargs {posargs:borg.testsuite}
 # fakeroot -u needs some env vars:
 passenv = *
```

The actual results:

```
# Sorted by ino readdir 1.0.1.dev145+ng2074f00
[root@intelbox2 ~]# ~green/bk/borg/.tox/py34/bin/borg -V
borg 1.0.1.dev145+ng2074f00
[root@intelbox2 ~]# mount /dev/slowdisk/TEST /mnt/
[root@intelbox2 ~]# ~green/bk/borg/.tox/py34/bin/borg init --encryption=none /tmp/test-backup2 # tmpfs
[root@intelbox2 ~]# time ~green/bk/borg/.tox/py34/bin/borg create /tmp/test-backup2::Initial /mnt/test

real    11m9.944s
user    3m43.852s
sys 1m7.596s
[root@intelbox2 ~]# time ~green/bk/borg/.tox/py34/bin/borg create /tmp/test-backup2::HOT /mnt/test

real    1m19.860s
user    1m13.378s
sys 0m4.779s
[root@intelbox2 ~]# umount /mnt
[root@intelbox2 ~]# mount /dev/slowdisk/TEST /mnt/
[root@intelbox2 ~]# time ~green/bk/borg/.tox/py34/bin/borg create /tmp/test-backup2::COLD /mnt/test

real    2m14.050s
user    1m16.178s
sys 0m9.368s
[root@intelbox2 ~]# time ~green/bk/borg/.tox/py34/bin/borg create /tmp/test-backup2::HOT2 /mnt/test
real    1m20.349s
user    1m13.521s
sys 0m4.650s
[root@intelbox2 ~]# umount /mnt
[root@intelbox2 ~]# du -hs /tmp/test-backup2
308M    /tmp/test-backup2
```

So initial repo generation is more than 10x faster. the actual walks are faster, but not as much, both hot and cold.
there's a notable "user time" reduction in the create phase too that I have no good explanation for.

I do not know why the repo size ended up bigger too, some differences accumulated due to the version change too?

> I do not know why the repo size ended up bigger too, some differences accumulated due to the version change too?

It's the same repo from above, or fresh created repo?

> So initial repo generation is more than 10x faster. the actual walks are faster, but not as much, both hot and cold.

That's good to hear! :)

it's a fresh repo - you can see borg init at the start of the output, goes to a different dir too.
I still have both.

Hm, not sure. Since the majority of this repository will be metadata (with a very fine-grained chunking) it _could_ be variance due to different slicing. (Since it's not encrypted the chunker seed is constant, and hence chunking is deterministic between different unencrypted repos for the same bytestring).

Also some more notes.

I guess the lstat in the process is not really necessary? Instead if we pass in d_type, we can use that to find what the file type is and switch based on that, then inside the actual process_\* they can do open/whatever and also fstat/lstat as appropriate. (I suspect it's mostly symlinks where we cannot do open so they would remain racy wrt attrs & content, the rest of file types we should be able to open and then do fstat, flistxattrs, ....).
Good way to catch races would also be to compare inode number from direntry vs what we get from lstat (in process_symlink) or fstat (everywhere else), if it changed - the file type might have changed too and we might need to bail out or instead reevaluate file type and jump to a different process_ function instead.

Do you want me to create a separate ticket about these races?

> Do you want me to create a separate ticket about these races?

That would be nice, thanks.

> I guess the lstat in the process is not really necessary? Instead if we pass in d_type, we can use that to find what the file type is and switch based on that, then inside the actual process_\* they can do open/whatever and also fstat/lstat as appropriate.

Yes, using as much information from scandir as possible is a good idea; we won't get around a stat() in the end, but avoiding these races, or at least being able to detect them should be done.

With the significant speed-up on OSX (/BSD?) alone these changes would be already worth the effort.

My results are not on osx, they are on Linux.

Additional tests on other OSes need to be performed, but I suspect they'd be pretty good since anything that reduces hard drive seeking is going to have huge impact.

> My results are not on osx, they are on Linux.

Oh! Well then that's even better :)

Pretty impressive, I didn't expect a difference that big. OTOH, the testcase with 1M files in 1 dir isn't what we usually find in filesystems, so maybe this could be tested on some real data.

Actually 1M files is not all that uncommon in certain environments.
There's this thing called maildir, for example that stores your emails one per message.
There are /home for large installs with a lot of users, there are picture folders with lot of files and so on.
People have poorly written "directory with file per token 'databases'" too (common with webapps).

People regularly bump into 10MB directory size limit in ext4 as well.
you'd get same problems at 100k and even 10k entries, they are just a bit milder due to less of time compounding.

So in short - the usecases ARE real, I know about them because we got bugreports about it.

directory hashes were implemented because looking up a nonexistent filename in a huge directory was taking terrible amount of time (and you need to do that when creating new files, so your large number of creates turns into O(n*n) exercise without hashes) and so on.

@verygreen ok, so not unreal, just uncommon. If we can easily tune it, we should. :)

Note: see also https://github.com/jborg/attic/issues/91

Probably needs to be changed as part of #908 since it's also a security fix, but wanted to mention that here too since it's also going to optimize things.

Doing full-path lookups on every file is expensive. If you are lucky Linux user, its VFS does a remarkably good job about caching and stuff, but still.
Every time you do open("/dir/dir2/..../dirN/file"), and not just open, any operation really, before getting to file itself, N directories need to be looked up. Yes, they are all cached, but depending on your OS internals and based on the filesystem used those lookups might be more or less expensive.
And they also add up, if you have a couple of million files and an average of 5 levels deep, that's 10 millions of unnecessary intermediate component path lookups.

Would chdir()ing into the currently processed directory also help with the directory being removed while borg is processing it? I'm not sure whether the "open fildes will prevent a file from vanishing entirely" rule applies to the cwd as well.

If you chdir() into a directory, it's pinned (just as if you just open it with open(2)). In Windows it makes the directory undeletable, I believe (not sure about renames).
In Unix directory might disappear from the namespace or renamed into a different place and your iteration over it would still be fine. It's just if it was removed, the rest of entries from your readdir/scandir would disappear (since only empty directory could be removed) and you'd get ENOENT.

Also on e.g. NFS directory might be deleted under you on the server, then next time NFS decides to verify it (as part of some operation involving it), you'd get back ESTALE.


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -25,6 +25,7 @@
     log_multi, PatternMatcher, ItemFormatter
 from .logger import create_logger, setup_logging
 logger = create_logger()
+from . import helpers
 from .compress import Compressor, COMPR_BUFFER
 from .upgrader import AtticRepositoryUpgrader, BorgRepositoryUpgrader
 from .repository import Repository
@@ -247,17 +248,18 @@ def create_inner(archive, cache):
                     self.print_file_status(status, path)
                     continue
                 path = os.path.normpath(path)
+                try:
+                    st = os.lstat(path)
+                except OSError as e:
+                    self.print_warning('%s: %s', path, e)
+                    continue
                 if args.one_file_system:
-                    try:
-                        restrict_dev = os.lstat(path).st_dev
-                    except OSError as e:
-                        self.print_warning('%s: %s', path, e)
-                        continue
+                    restrict_dev = st.st_dev
                 else:
                     restrict_dev = None
                 self._process(archive, cache, matcher, args.exclude_caches, args.exclude_if_present,
                               args.keep_tag_files, skip_inodes, path, restrict_dev,
-                              read_special=args.read_special, dry_run=dry_run)
+                              read_special=args.read_special, dry_run=dry_run, st=st)
             if not dry_run:
                 archive.save(comment=args.comment, timestamp=args.timestamp)
                 if args.progress:
@@ -292,16 +294,16 @@ def create_inner(archive, cache):
 
     def _process(self, archive, cache, matcher, exclude_caches, exclude_if_present,
                  keep_tag_files, skip_inodes, path, restrict_dev,
-                 read_special=False, dry_run=False):
+                 read_special=False, dry_run=False, st=None):
         if not matcher.match(path):
             self.print_file_status('x', path)
             return
-
-        try:
-            st = os.lstat(path)
-        except OSError as e:
-            self.print_warning('%s: %s', path, e)
-            return
+        if st is None:
+            try:
+                st = os.lstat(path)
+            except OSError as e:
+                self.print_warning('%s: %s', path, e)
+                return
         if (st.st_ino, st.st_dev) in skip_inodes:
             return
         # Entering a new filesystem?
@@ -331,15 +333,15 @@ def _process(self, archive, cache, matcher, exclude_caches, exclude_if_present,
             if not dry_run:
                 status = archive.process_dir(path, st)
             try:
-                entries = os.listdir(path)
+                entries = helpers.scandir_inorder(path)
             except OSError as e:
                 status = 'E'
                 self.print_warning('%s: %s', path, e)
             else:
-                for filename in sorted(entries):
-                    entry_path = os.path.normpath(os.path.join(path, filename))
+                for dirent in entries:
+                    normpath = os.path.normpath(dirent.path)
                     self._process(archive, cache, matcher, exclude_caches, exclude_if_present,
-                                  keep_tag_files, skip_inodes, entry_path, restrict_dev,
+                                  keep_tag_files, skip_inodes, normpath, restrict_dev,
                                   read_special=read_special, dry_run=dry_run)
         elif stat.S_ISLNK(st.st_mode):
             if not dry_run:
@@ -461,7 +463,7 @@ def get_mode(item):
                 return [None]
 
         def has_hardlink_master(item, hardlink_masters):
-            return item.get(b'source') in hardlink_masters and get_mode(item)[0] != 'l'
+            return stat.S_ISREG(item[b'mode']) and item.get(b'source') in hardlink_masters
 
         def compare_link(item1, item2):
             # These are the simple link cases. For special cases, e.g. if a
@@ -524,9 +526,6 @@ def compare_items(output, path, item1, item2, hardlink_masters, deleted=False):
             """
             changes = []
 
-            if item1.get(b'hardlink_master') or item2.get(b'hardlink_master'):
-                hardlink_masters[path] = (item1, item2)
-
             if has_hardlink_master(item1, hardlink_masters):
                 item1 = hardlink_masters[item1[b'source']][0]
 
@@ -559,8 +558,26 @@ def print_output(line):
             print("{:<19} {}".format(line[1], line[0]))
 
         def compare_archives(archive1, archive2, matcher):
+            def hardlink_master_seen(item):
+                return b'source' not in item or not stat.S_ISREG(item[b'mode']) or item[b'source'] in hardlink_masters
+
+            def is_hardlink_master(item):
+                return item.get(b'hardlink_master', True) and b'source' not in item
+
+            def update_hardlink_masters(item1, item2):
+                if is_hardlink_master(item1) or is_hardlink_master(item2):
+                    hardlink_masters[item1[b'path']] = (item1, item2)
+
+            def compare_or_defer(item1, item2):
+                update_hardlink_masters(item1, item2)
+                if not hardlink_master_seen(item1) or not hardlink_master_seen(item2):
+                    deferred.append((item1, item2))
+                else:
+                    compare_items(output, item1[b'path'], item1, item2, hardlink_masters)
+
             orphans_archive1 = collections.OrderedDict()
             orphans_archive2 = collections.OrderedDict()
+            deferred = []
             hardlink_masters = {}
             output = []
 
@@ -569,31 +586,40 @@ def compare_archives(archive1, archive2, matcher):
                     archive2.iter_items(lambda item: matcher.match(item[b'path'])),
             ):
                 if item1 and item2 and item1[b'path'] == item2[b'path']:
-                    compare_items(output, item1[b'path'], item1, item2, hardlink_masters)
+                    compare_or_defer(item1, item2)
                     continue
                 if item1:
                     matching_orphan = orphans_archive2.pop(item1[b'path'], None)
                     if matching_orphan:
-                        compare_items(output, item1[b'path'], item1, matching_orphan, hardlink_masters)
+                        compare_or_defer(item1, matching_orphan)
                     else:
                         orphans_archive1[item1[b'path']] = item1
                 if item2:
                     matching_orphan = orphans_archive1.pop(item2[b'path'], None)
                     if matching_orphan:
-                        compare_items(output, item2[b'path'], matching_orphan, item2, hardlink_masters)
+                        compare_or_defer(matching_orphan, item2)
                     else:
                         orphans_archive2[item2[b'path']] = item2
             # At this point orphans_* contain items that had no matching partner in the other archive
+            deleted_item = {
+                b'deleted': True,
+                b'chunks': [],
+                b'mode': 0,
+            }
             for added in orphans_archive2.values():
-                compare_items(output, added[b'path'], {
-                    b'deleted': True,
-                    b'chunks': [],
-                }, added, hardlink_masters, deleted=True)
+                path = added[b'path']
+                deleted_item[b'path'] = path
+                update_hardlink_masters(deleted_item, added)
+                compare_items(output, path, deleted_item, added, hardlink_masters, deleted=True)
             for deleted in orphans_archive1.values():
-                compare_items(output, deleted[b'path'], deleted, {
-                    b'deleted': True,
-                    b'chunks': [],
-                }, hardlink_masters, deleted=True)
+                path = deleted[b'path']
+                deleted_item[b'path'] = path
+                update_hardlink_masters(deleted, deleted_item)
+                compare_items(output, path, deleted, deleted_item, hardlink_masters, deleted=True)
+            for item1, item2 in deferred:
+                assert hardlink_master_seen(item1)
+                assert hardlink_master_seen(item2)
+                compare_items(output, item1[b'path'], item1, item2, hardlink_masters)
 
             for line in sorted(output):
                 print_output(line)
diff --git a/borg/helpers.py b/borg/helpers.py
--- a/borg/helpers.py
+++ b/borg/helpers.py
@@ -6,6 +6,7 @@
 import hashlib
 from itertools import islice
 import os
+import os.path
 import stat
 import textwrap
 import pwd
@@ -1340,3 +1341,68 @@ def consume(iterator, n=None):
     else:
         # advance to the empty slice starting at position n
         next(islice(iterator, n, n), None)
+
+# GenericDirEntry, scandir_generic (c) 2012 Ben Hoyt
+# from the python-scandir package (3-clause BSD license, just like us, so no troubles here)
+# note: simplified version
+
+
+class GenericDirEntry:
+    __slots__ = ('name', '_scandir_path', '_path')
+
+    def __init__(self, scandir_path, name):
+        self._scandir_path = scandir_path
+        self.name = name
+        self._path = None
+
+    @property
+    def path(self):
+        if self._path is None:
+            self._path = os.path.join(self._scandir_path, self.name)
+        return self._path
+
+    def stat(self, follow_symlinks=True):
+        assert not follow_symlinks
+        return os.lstat(self.path)
+
+    def _check_type(self, type):
+        st = self.stat(False)
+        return stat.S_IFMT(st.st_mode) == type
+
+    def is_dir(self, follow_symlinks=True):
+        assert not follow_symlinks
+        return self._check_type(stat.S_IFDIR)
+
+    def is_file(self, follow_symlinks=True):
+        assert not follow_symlinks
+        return self._check_type(stat.S_IFREG)
+
+    def is_symlink(self):
+        return self._check_type(stat.S_IFLNK)
+
+    def inode(self):
+        st = self.stat(False)
+        return st.st_ino
+
+    def __repr__(self):
+        return '<{0}: {1!r}>'.format(self.__class__.__name__, self.path)
+
+
+def scandir_generic(path='.'):
+    """Like os.listdir(), but yield DirEntry objects instead of returning a list of names."""
+    for name in sorted(os.listdir(path)):
+        yield GenericDirEntry(path, name)
+
+try:
+    from os import scandir
+except ImportError:
+    try:
+        # Try python-scandir on Python 3.4
+        from scandir import scandir
+    except ImportError:
+        # If python-scandir is not installed, then use a version that is just as slow as listdir.
+        scandir = scandir_generic
+
+
+def scandir_inorder(path='.'):
+    return sorted(scandir(path), key=lambda dirent: dirent.inode())
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -17,7 +17,7 @@
 
 import pytest
 
-from .. import xattr
+from .. import xattr, helpers
 from ..archive import Archive, ChunkBuffer, ArchiveRecreater
 from ..archiver import Archiver
 from ..cache import Cache
@@ -1314,11 +1314,16 @@ def _test_recreate_interrupt(self, change_args, interrupt_early):
         assert 'dir2/abcdef' in files
         assert 'file1' not in files
 
+    # The _test_create_interrupt requires a deterministic (alphabetic) order of the files to easily check if
+    # resumption works correctly. Patch scandir_inorder to work in alphabetic order.
+
     def test_recreate_interrupt(self):
-        self._test_recreate_interrupt(False, True)
+        with patch.object(helpers, 'scandir_inorder', helpers.scandir_generic):
+            self._test_recreate_interrupt(False, True)
 
     def test_recreate_interrupt2(self):
-        self._test_recreate_interrupt(True, False)
+        with patch.object(helpers, 'scandir_inorder', helpers.scandir_generic):
+            self._test_recreate_interrupt(True, False)
 
     def _test_recreate_chunker_interrupt_patch(self):
         real_add_chunk = Cache.add_chunk


Test patch: