Repository: borgbackup/borg

Issue:
add automated benchmarks
use pytest-benchmark for it. start with: create, delete, extract.

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/25553185-add-automated-benchmarks)



Discussion:
pytest-benchmark looks nice... i have done [some preliminary benchmarks on attic vs bup](http://anarc.at/blog/2014-11-18-bup-vs-attic-silly-benchmark/) about a year ago. i then found it useful to have [resource usage information](https://docs.python.org/3/library/resource.html) (CPU, RAM, I/O, etc) during the benchmarks as well, to see what was going on more clearly. this doesn't seem to be implemented in pytest-benchmark, so i filed a feature request about this in https://github.com/ionelmc/pytest-benchmark/issues/28.

here's a stub benchmark i used to test the software:

```
from ..helpers import Location
from ..remote import RemoteRepository

def test_ssh_cmd(benchmark, tmpdir):
    repo = RemoteRepository(Location('__testsuite__:' + str(tmpdir)), create=True)
    result = benchmark(repo.ssh_cmd, Location('example.com:foo'))
```

here's the result:

```
===================================================================== test session starts =====================================================================
platform linux -- Python 3.4.2, pytest-2.8.1, py-1.4.30, pluggy-0.3.1
rootdir: /home/anarcat/src/borg, inifile: setup.cfg
plugins: cov-1.8.1, benchmark-2.5.0, pep8-1.0.6
collected 1 items

borg/testsuite/benchmark.py .

 benchmark: 1 tests, min 5 rounds (of min 25.00us), 1.00s max time, timer: time.perf_counter
Name (time in us)         Min      Max     Mean  StdDev  Rounds  Iterations
---------------------------------------------------------------------------
test_ssh_cmd          24.3047  73.0390  24.8964  1.0002    6323           1
---------------------------------------------------------------------------

========================================================= 1 passed, 2 pytest-warnings in 1.39 seconds =========================================================
```

of course, we should disable that in the default unit test suite and only have that run when we explicitely require it, because it would kill the CI server completely.

although the nice thing with this is that we can just wrap existing calls with `benchmark` and get metrics for free. just keep in mind that, by default, every `benchmark` item can be run 5 times, which can have some obvious side-effects in our case. there is some calibration magic going on that can make that run only once (and in fact that is what is happening in my test above, it seems), but still... there's a `benchmark.pendantic` fixture to control better the number of calls per function.

otherwise it looks like we could just start shoving some of that stuff around our test suite, it would be great!

oh and you need to ignore the warning on install, see https://github.com/ionelmc/pytest-benchmark/issues/18

oh and unfortunately, integrating the fixture in our unittest-based class hierarchy would require .. getting rid of the class hierarchy, because it seems [basically impossible to mix fixtures and unittest](https://pytest.org/latest/unittest.html#autouse-fixtures-and-accessing-other-fixtures)... sigh.

so this depends on #28, so later.

I think we would need some extra "tests" (in a testsuite/benchmark module) that are specially made for benchmarking, like generating and processing a bit bigger amount of data, not just the usual 10 small files as the archiver tests do - otherwise we might rather measure the overhead than the core performance and ressource usage would be rather low also.

As a start, we could concentrate on the most frequent operations there: create, extract, list, delete.
The code I posted there might come in handy to produce the bigger amount of data in a reproducable way:

https://github.com/borgbackup/backupdata

If course the benchmark data is only meaningful when running on same machine, under same load conditions (and of course we don't run that on Travis CI).

i agree. i was just saying that we are already running some of those benchmarks, just not measuring them. hooking the benchmark fixture in there may produce basic results that would be part of the CI process, where we could see regressions and so on...

as for https://github.com/borgbackup/backupdata, it seems it would need to actually generate original data, not just modify some existing data... especially if we want to automate it. we could use the tool to alter existing data sets in the future, but that doesn't seem as useful as generating new "standard" datasets.

[bup](https://github.com/bup/bup/) has a [random](https://github.com/bup/bup/blob/master/cmd/random-cmd.py) command to generate a stream of N bytes, based on a [c function](https://github.com/bup/bup/blob/master/lib/bup/_helpers.c#L832). they also have a [memtest](https://github.com/bup/bup/blob/master/cmd/memtest-cmd.py) command that seems to report memory usage.

seivot [uses a genbackupdata command](http://git.liw.fi/cgi-bin/cgit/cgit.cgi/seivot/tree/seivot#n483) that is, like seivot, written by the obnam author: http://liw.fi/genbackupdata/ - the tool is fairly simple and generates randomly filled files... sample usage:

```
$ genbackupdata --create=100M foo
Generating: 95.37 MiB of 95.37 MiB 100 % (107.15 MiB/s)
$ find foo -ls  | head
7872801    4 drwxr-xr-x   3 anarcat  anarcat      4096 oct  6 13:54 foo
7872802    4 drwxr-xr-x   3 anarcat  anarcat      4096 oct  6 13:54 foo/0
7872803    4 drwxr-xr-x  50 anarcat  anarcat      4096 oct  6 13:54 foo/0/0
7877221    4 drwxr-xr-x   2 anarcat  anarcat      4096 oct  6 13:54 foo/0/0/15
7877336   16 -rw-r--r--   1 anarcat  anarcat     16384 oct  6 13:54 foo/0/0/15/114
7877237   16 -rw-r--r--   1 anarcat  anarcat     16384 oct  6 13:54 foo/0/0/15/15
7877293   16 -rw-r--r--   1 anarcat  anarcat     16384 oct  6 13:54 foo/0/0/15/71
7877347   16 -rw-r--r--   1 anarcat  anarcat     16384 oct  6 13:54 foo/0/0/15/125
7877327   16 -rw-r--r--   1 anarcat  anarcat     16384 oct  6 13:54 foo/0/0/15/105
7877249   16 -rw-r--r--   1 anarcat  anarcat     16384 oct  6 13:54 foo/0/0/15/27
```

could be useful, and very fast.

random data is easy to produce and wouldn't require including sample data, but otoh, it is rather unnatural data and thus the benchmark results would be also not very practical. so we could just include some freely licensed real data (not 1GB as I used for my last test, of course, rather few MBs) and feed that into my tool to make more from it (guess we need about 1GB output data to get reasonable measurements). 

one of the things that bup does is use the bup source code as part of the test, so we could use the borg source code as one source of text files. then we'd need images, movies, disk images, compressed files... there can be quite a diversity out there, not sure how to generate that reliably...

oh, and genbackupdata can also alter existing datasets and it is deterministic... http://liw.fi/genbackupdata/genbackupdata.1.txt

another question here is how we can avoid the influence of multitasking, a potentially changing OS and hardware environment on the performance tests? do we need a piece of hardware and OS that does not change over time and that does nothing else than benchmarking borg or is there a more elegant solution?

On 2015-10-09 06:57:05, TW wrote:

> another question here is how we can avoid the influence of multitasking, a potentially changing OS and hardware environment on the performance tests? do we need a piece of hardware and OS that does not change over time and that does nothing else than benchmarking borg or is there a more elegant solution?

i know that pytest's benchmark plugin does a calibration step that
alleviates the influence of the environment.

but unfortunately, different hardware will lead to different benchmarks
- they will only be relevant if ran on the same machine...

a.

## 

We will create a civilization of the Mind in Cyberspace. May it be
more humane and fair than the world your governments have made
before.
                         - John Perry Barlow


Resolving patch:
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -71,6 +71,29 @@ def __exit__(self, *args, **kw):
                 os.environ[k] = v
 
 
+def exec_cmd(*args, archiver=None, fork=False, **kw):
+    if fork:
+        try:
+            borg = (sys.executable, '-m', 'borg.archiver')
+            output = subprocess.check_output(borg + args)
+            ret = 0
+        except subprocess.CalledProcessError as e:
+            output = e.output
+            ret = e.returncode
+        return ret, os.fsdecode(output)
+    else:
+        stdin, stdout, stderr = sys.stdin, sys.stdout, sys.stderr
+        try:
+            sys.stdin = StringIO()
+            sys.stdout = sys.stderr = output = StringIO()
+            if archiver is None:
+                archiver = Archiver()
+            ret = archiver.run(list(args))
+            return ret, output.getvalue()
+        finally:
+            sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
+
+
 class ArchiverTestCaseBase(BaseTestCase):
 
     prefix = ''
@@ -102,34 +125,12 @@ def tearDown(self):
         shutil.rmtree(self.tmpdir)
 
     def cmd(self, *args, **kw):
-        exit_code = kw.get('exit_code', 0)
-        fork = kw.get('fork', False)
-        if fork:
-            try:
-                output = subprocess.check_output((sys.executable, '-m', 'borg.archiver') + args)
-                ret = 0
-            except subprocess.CalledProcessError as e:
-                output = e.output
-                ret = e.returncode
-            output = os.fsdecode(output)
-            if ret != exit_code:
-                print(output)
-            self.assert_equal(exit_code, ret)
-            return output
-        args = list(args)
-        stdin, stdout, stderr = sys.stdin, sys.stdout, sys.stderr
-        try:
-            sys.stdin = StringIO()
-            output = StringIO()
-            sys.stdout = sys.stderr = output
-            ret = self.archiver.run(args)
-            sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
-            if ret != exit_code:
-                print(output.getvalue())
-            self.assert_equal(exit_code, ret)
-            return output.getvalue()
-        finally:
-            sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
+        exit_code = kw.pop('exit_code', 0)
+        ret, output = exec_cmd(*args, archiver=self.archiver, **kw)
+        if ret != exit_code:
+            print(output)
+        self.assert_equal(ret, exit_code)
+        return output
 
     def create_src_archive(self, name):
         self.cmd('create', self.repository_location + '::' + name, src_dir)
diff --git a/borg/testsuite/benchmark.py b/borg/testsuite/benchmark.py
new file mode 100644
--- /dev/null
+++ b/borg/testsuite/benchmark.py
@@ -0,0 +1,106 @@
+"""
+Do benchmarks using pytest-benchmark.
+
+Usage:
+
+    py.test --benchmark-only
+"""
+
+import os
+
+import pytest
+
+from .archiver import changedir, exec_cmd
+
+
+# TODO: use fixture params to test python code and binary
+@pytest.fixture
+def cmd():
+    return exec_cmd
+
+
+@pytest.yield_fixture
+def repo_url(request, tmpdir):
+    os.environ['BORG_PASSPHRASE'] = '123456'
+    os.environ['BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'] = '1'
+    os.environ['BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK'] = '1'
+    os.environ['BORG_KEYS_DIR'] = str(tmpdir.join('keys'))
+    os.environ['BORG_CACHE_DIR'] = str(tmpdir.join('cache'))
+    yield str(tmpdir.join('repository'))
+    tmpdir.remove(rec=1)
+
+
+@pytest.fixture(params=["none", "passphrase"])
+def repo(request, cmd, repo_url):
+    cmd('init', '--encryption', request.param, repo_url)
+    return repo_url
+
+
+@pytest.yield_fixture(scope='session', params=["zeros", "random"])
+def testdata(request, tmpdir_factory):
+    count, size = 10, 1000*1000
+    p = tmpdir_factory.mktemp('data')
+    data_type = request.param
+    if data_type == 'zeros':
+        # do not use a binary zero (\0) to avoid sparse detection
+        data = lambda: b'0' * size
+    if data_type == 'random':
+        rnd = open('/dev/urandom', 'rb')
+        data = lambda: rnd.read(size)
+    for i in range(count):
+        with open(str(p.join(str(i))), "wb") as f:
+            f.write(data())
+    if data_type == 'random':
+        rnd.close()
+    yield str(p)
+    p.remove(rec=1)
+
+
+@pytest.fixture(params=['none', 'lz4'])
+def archive(request, cmd, repo, testdata):
+    archive_url = repo + '::test'
+    cmd('create', '--compression', request.param, archive_url, testdata)
+    return archive_url
+
+
+def test_create_none(benchmark, cmd, repo, testdata):
+    result, out = benchmark.pedantic(cmd, ('create', '--compression', 'none', repo + '::test', testdata))
+    assert result == 0
+
+
+def test_create_lz4(benchmark, cmd, repo, testdata):
+    result, out = benchmark.pedantic(cmd, ('create', '--compression', 'lz4', repo + '::test', testdata))
+    assert result == 0
+
+
+def test_extract(benchmark, cmd, archive, tmpdir):
+    with changedir(str(tmpdir)):
+        result, out = benchmark.pedantic(cmd, ('extract', archive))
+    assert result == 0
+
+
+def test_delete(benchmark, cmd, archive):
+    result, out = benchmark.pedantic(cmd, ('delete', archive))
+    assert result == 0
+
+
+def test_list(benchmark, cmd, archive):
+    result, out = benchmark(cmd, 'list', archive)
+    assert result == 0
+
+
+def test_info(benchmark, cmd, archive):
+    result, out = benchmark(cmd, 'info', archive)
+    assert result == 0
+
+
+def test_check(benchmark, cmd, archive):
+    repo = archive.split('::')[0]
+    result, out = benchmark(cmd, 'check', repo)
+    assert result == 0
+
+
+def test_help(benchmark, cmd):
+    result, out = benchmark(cmd, 'help')
+    assert result == 0
+


Test patch: