opened
sipsma
username_0
The snapshotter tests create loop devices using [`losetup --find`](http://man7.org/linux/man-pages/man8/losetup.8.html), which finds the first unused loop device or, if all are occupied, creates a new one.

The command appears to work, in part, by asking `/dev/loop-control` for a free loop device and then trying to open the returned loop device. It can get an `EBUSY` when trying to open the actual loop device though (likely due to a race with other parallel tests) in which case it will ask `/dev/loop-control` for another and repeat until it opens one successfully. 

i.e. `/sbin/losetup --find --show /tmp/snapshot-suite-Snapshotter-556085586/root/images/1`
gives repeated strace output like the following for a number of `loopN` devices:
```
40328 open("/dev/loop-control", O_RDWR|O_CLOEXEC) = 3
40328 ioctl(3, LOOP_CTL_GET_FREE)       = 47
...
40328 open("/dev/loop47", O_RDWR|O_CLOEXEC) = 4
40328 ioctl(4, LOOP_SET_FD, 3)          = -1 EBUSY (Device or resource busy)
40328 close(3)                          = 0
```

It seems like a single run of the snapshotter test suites will result in the creation of about 54 loop devices. If invocations of that test suite are run in serial, we should thus only ever see a max of about 54 loop devices under `/dev`. This results in tests taking about 4-5 minutes, which is expected.

However, if tests are run in parallel, 54 devices will be occupied by one suite and thus tests that are part of another suite end up needing to iterate over numerous occupied devices before realizing they need to create a new one, which takes up time and results in over 100 loop devices to appear under `/dev`.

You can see from `ps` output that numerous `losetup --find` commands end up hanging for more than a few seconds when test suites are run in parallel due to the tests stomping on each other (resulting in `EBUSY` and retries during creation of loop devices). This often results in tests taking more than 10 minutes and timing out.
Parallel snapshotter test suites significantly slow down tests
issue
created
sipsma
username_0
If helpful for reproducing, I obtained the strace output by mv'ing `/sbin/losetup` to `/sbin/losetup.real` and then putting the following script at `/sbin/losetup`:
```
#!/bin/sh
OUTPUT_DIR="/tmp/losetup.strace"
mkdir -p ${OUTPUT_DIR}
exec strace -f -D -s 4096 -o ${OUTPUT_DIR}/"$$" /sbin/losetup.real "$@"
```
comment
closed
sipsma
username_0
issue
created
sipsma
username_0
This was addressed in #184 

Further followups can be handled in #190
comment