Repository: aeye-lab/pymovements
Pull request #1087: ci: create workflow for benchmarking import times
Author: login: dkrako
Labels: name: internal
Created: 2025-04-11T22:53:33Z | Merged: 2025-04-28T15:37:50Z

## Description

This creates a first benchmarking workflow for pymovements.
The most trivial thing to measure is the import time. Currently it is about 3 seconds which should be lowerered.

## Implemented changes

- [x] add pytest-benchmark to test dependencies
- [x] implement tox benchmark workflow
- [x] implement github worker
- [x] implement benchmarking test




Diff:
diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml
new file mode 100644
index 000000000..20c478606
--- /dev/null
+++ b/.github/workflows/benchmark.yml
@@ -0,0 +1,56 @@
+name: benchmark
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
+
+jobs:
+  pytest-benchmark:
+    name: pytest-benchmark
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout pymovements repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Setup python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install tox
+        run: python -m pip install tox
+
+      - name: Setup benchmark environment
+        run: tox -vv --notest -e benchmark
+
+      - name: Run benchmark
+        run: tox --skip-pkg-install -e benchmark
+
+      - name: Download previous benchmark data
+        uses: actions/cache@v4
+        with:
+          path: ./cache
+          key: benchmark-${{ github.ref_name }}-${{ github.run_id }}
+          restore-keys: benchmark-main
+
+      - name: Store benchmark result
+        uses: benchmark-action/github-action-benchmark@v1
+        with:
+          name: pytest-benchmark
+          tool: 'pytest'
+          output-file-path: benchmark-data.json
+          external-data-json-path: ./cache/benchmark-data.json
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          auto-push: false
+          alert-threshold: '150%'
+          comment-on-alert: true
+          fail-on-alert: true
+          summary-always: true
+          alert-comment-cc-users: '@dkrako,@SiQube'
diff --git a/pyproject.toml b/pyproject.toml
index 43cf80f28..333b71a5b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -67,6 +67,7 @@ docs = [
 test = [
   "pybtex",
   "pytest>=6.0.0",
+  "pytest-benchmark>=5.1.0",
   "pytest-cov>=4.0.0",
   "setuptools>=61.0.0",
   "types-tqdm>=4.64.0,<5"
diff --git a/tests/benchmark/import_test.py b/tests/benchmark/import_test.py
new file mode 100644
index 000000000..f94864017
--- /dev/null
+++ b/tests/benchmark/import_test.py
@@ -0,0 +1,34 @@
+# Copyright (c) 2025 The pymovements Project Authors
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""Benchmark import times."""
+import subprocess
+import sys
+
+
+def import_pymovements_subprocess():
+    cmd = [sys.executable, '-c', 'import pymovements']
+    subprocess.run(cmd, check=True)
+
+
+def test_import_pymovements_subprocess(benchmark):
+    benchmark.pedantic(
+        import_pymovements_subprocess,
+        iterations=1, rounds=10,
+    )
diff --git a/tox.ini b/tox.ini
index 32fecc71c..b1f019c47 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,6 +7,7 @@ envlist =
     py311
     py312
     py313
+    benchmark
     build
     docs
     coverage
@@ -21,6 +22,7 @@ setenv =
     COVERAGE_FILE = .coverage.{envname}
 commands =
     pytest \
+        --ignore=tests/benchmark \
         --ignore=tests/integration \
         --tb=long \
         --cov --cov-append \
@@ -30,6 +32,20 @@ commands =
         {posargs:.}
 
 
+[testenv:benchmark]
+deps =
+    .[test]
+commands =
+    pytest \
+        --ignore=src \
+        --ignore=tests/functional \
+        --ignore=tests/integration \
+        --ignore=tests/unit \
+        --benchmark-json \
+        {toxinidir}/benchmark-data.json \
+        {posargs:.}
+
+
 [testenv:build]
 usedevelop = False
 skip_install = True
@@ -78,6 +94,7 @@ deps =
 commands =
     pytest \
         --ignore=src \
+        --ignore=tests/benchmark \
         --ignore=tests/functional \
         --ignore=tests/unit \
         -v \


Reviews:
user: login: dkrako
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-12T11:41:09Z
user: login: SiQube
body: that you for looking into this, I think this will be a cool feature but currently I think this is not yet ready to be merged.

1. see my comments about runtimes
2. I think we should find import leakage and figure out what is going on there instead of merging it 

but your vision is clearer than mine, I think we should rather focus on features etc. instead of refactoring and benchmarking
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T21:15:42Z
user: login: dkrako
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T11:35:04Z
user: login: SiQube
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T15:14:24Z

Review comments:
diff_hunk: @@ -0,0 +1,60 @@
+# Copyright (c) 2025 The pymovements Project Authors
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""Benchmark import times."""
+import subprocess
+import sys
+
+
+def setup_import_pymovements():
+    try:
+        del pymovements  # noqa: F821 # pylint: disable=undefined-variable
+    except BaseException:
+        pass
+
+    for module in list(sys.modules.keys()):
+        if module.startswith('pymovements'):
+            del sys.modules[module]
+
+    assert 'pymovements' not in sys.modules
+    assert not any(module.startswith('pymovements') for module in sys.modules)
+
+
+def import_pymovements():
+    import pymovements  # noqa: F401 # pylint: disable=import-outside-toplevel,unused-import
+
+
+def test_import_pymovements(benchmark):
+    benchmark.pedantic(
+        import_pymovements,
+        setup=setup_import_pymovements,
+        iterations=1, rounds=1,  # only a single round due to leakage to subsequent rounds
path: tests/benchmark/import_test.py
user: login: dkrako
body: this benchmark is only valid for a single round. subsequent rounds are much shorter, so there's some leakage between rounds, that I just couldn't get rid of.

we could probably remove this test and only run the subprocess kind, as the values are very similar, but I think it's nice to have the two different benchmarks.
created_at: 2025-04-12T11:41:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,52 @@
+name: benchmark
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  pytest-benchmark:
+    name: pytest-benchmark
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout pymovements repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Setup python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
path: .github/workflows/benchmark.yml
user: login: SiQube
body: while I think people *should* use the newest version of python, it is very unrealisitic. even bigger libraries have not adopted to 3.13 yet, so maybe benchmarking on <3.13, e.g. 3.10 or so would be more practical.
created_at: 2025-04-22T21:04:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,60 @@
+# Copyright (c) 2025 The pymovements Project Authors
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""Benchmark import times."""
+import subprocess
+import sys
+
+
+def setup_import_pymovements():
+    try:
+        del pymovements  # noqa: F821 # pylint: disable=undefined-variable
+    except BaseException:
+        pass
+
+    for module in list(sys.modules.keys()):
+        if module.startswith('pymovements'):
+            del sys.modules[module]
+
+    assert 'pymovements' not in sys.modules
+    assert not any(module.startswith('pymovements') for module in sys.modules)
+
+
+def import_pymovements():
+    import pymovements  # noqa: F401 # pylint: disable=import-outside-toplevel,unused-import
+
+
+def test_import_pymovements(benchmark):
+    benchmark.pedantic(
+        import_pymovements,
+        setup=setup_import_pymovements,
+        iterations=1, rounds=1,  # only a single round due to leakage to subsequent rounds
path: tests/benchmark/import_test.py
user: login: SiQube
body: for me this is not the case:

when appyling this diff
```diff
(venv) siqube@c-cube:~/lab/aeye-lab/pymovements
$ git diff
diff --git a/tests/benchmark/import_test.py b/tests/benchmark/import_test.py
index 1b6a251..85288d0 100644
--- a/tests/benchmark/import_test.py
+++ b/tests/benchmark/import_test.py
@@ -56,5 +56,5 @@ def import_pymovements_subprocess():
 def test_import_pymovements_subprocess(benchmark):
     benchmark.pedantic(
         import_pymovements_subprocess,
-        iterations=1, rounds=10,
+        iterations=1, rounds=1,
     )

```

I get

```console
(venv) siqube@c-cube:~/lab/aeye-lab/pymovements
$ tox -e benchmark
.pkg: _optional_hooks> python /home/siqube/lab/aeye-lab/pymovements/venv/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /home/siqube/lab/aeye-lab/pymovements/venv/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /home/siqube/lab/aeye-lab/pymovements/venv/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
benchmark: install_package> python -I -m pip install --force-reinstall --no-deps /home/siqube/lab/aeye-lab/pymovements/.tox/.tmp/package/8/pymovements-0.21.0.post46+git.e60c44d7.dirty.tar.gz
benchmark: commands[0]> pytest --ignore=src --ignore=tests/functional --ignore=tests/integration --ignore=tests/unit --benchmark-json /home/siqube/lab/aeye-lab/pymovements/benchmark-data.json .
================================================================================================================= test session starts ==================================================================================================================
platform linux -- Python 3.12.3, pytest-8.3.5, pluggy-1.5.0
cachedir: .tox/benchmark/.pytest_cache
benchmark: 5.1.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/siqube/lab/aeye-lab/pymovements
configfile: pyproject.toml
plugins: cov-6.1.1, benchmark-5.1.0
collected 2 items                                                                                                                                                                                                                                      

tests/benchmark/import_test.py ..                                                                                                                                                                                                                [100%]
Wrote benchmark data in: <_io.BufferedWriter name='/home/siqube/lab/aeye-lab/pymovements/benchmark-data.json'>



----------------------------------------------------------------------------------------------- benchmark: 2 tests ----------------------------------------------------------------------------------------------
Name (time in ms)                             Min                   Max                  Mean            StdDev                Median               IQR            Outliers     OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_import_pymovements                  154.3648 (1.0)        154.3648 (1.0)        154.3648 (1.0)      0.0000 (1.0)        154.3648 (1.0)      0.0000 (1.0)           0;0  6.4782 (1.0)           1           1
test_import_pymovements_subprocess     1,268.2934 (8.22)     1,268.2934 (8.22)     1,268.2934 (8.22)     0.0000 (1.0)      1,268.2934 (8.22)     0.0000 (1.0)           0;0  0.7885 (0.12)          1           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

```

similarly when using 10 runs:

```diff
```

the numbers seem to be kinda consistent

```console
(venv) siqube@c-cube:~/lab/aeye-lab/pymovements
$ git diff
diff --git a/tests/benchmark/import_test.py b/tests/benchmark/import_test.py
index 1b6a251..bc19808 100644
--- a/tests/benchmark/import_test.py
+++ b/tests/benchmark/import_test.py
@@ -44,7 +44,7 @@ def test_import_pymovements(benchmark):
     benchmark.pedantic(
         import_pymovements,
         setup=setup_import_pymovements,
-        iterations=1, rounds=1,  # only a single round due to leakage to subsequent rounds
+        iterations=1, rounds=10,  # only a single round due to leakage to subsequent rounds
     )
```
created_at: 2025-04-22T21:13:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+name: benchmark
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  pytest-benchmark:
+    name: pytest-benchmark
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout pymovements repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Setup python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
path: .github/workflows/benchmark.yml
user: login: dkrako
body: This is the definition of a GitHub workflow. Github workflows are internal to github and neither targeted at the user nor at the developer. They are handled purely by the CI.

we should benchmark on the newest python versions with the newest optimizations.
I don't care much about performance on old python systems.
created_at: 2025-04-28T11:35:04Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-04-11T23:00:12Z
body: ## [Codecov](https://app.codecov.io/gh/aeye-lab/pymovements/pull/1087?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 100.00%. Comparing base ([`8768ff9`](https://app.codecov.io/gh/aeye-lab/pymovements/commit/8768ff97e2d47fa484b32a059b6e6122ae35db67?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab)) to head ([`b369b85`](https://app.codecov.io/gh/aeye-lab/pymovements/commit/b369b85cee63e54c8e4b16d061b3a651e9681909?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab)).
:warning: Report is 54 commits behind head on main.

<details><summary>Additional details and impacted files</summary>



```diff
@@            Coverage Diff            @@
##              main     #1087   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           87        87           
  Lines         3818      3818           
  Branches       679       679           
=========================================
  Hits          3818      3818           
```
</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/aeye-lab/pymovements/pull/1087?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: dkrako
created_at: 2025-04-12T11:33:41Z
body: An example of a summary is here: https://github.com/aeye-lab/pymovements/actions/runs/14413954118?pr=1087

It will always compare against the latest benchmark run on main.

It is missing from the last workflow runs in this PR as no benchmark was run on main yet.
author_association: MEMBER
user: login: dkrako
created_at: 2025-04-22T16:39:00Z
body: @SiQube just a small reminder that this PR is ready for review
author_association: MEMBER
user: login: dkrako
created_at: 2025-04-28T11:40:34Z
body: > but your vision is clearer than mine, I think we should rather focus on features etc. instead of refactoring and benchmarking

I don't care much about new features as long as the current state of pymovements is as bad as it is. Especially all the stuff added last summer requires a major overhaul as it's pretty unusable. Also implementing performance improvements doesn't make any sense if there is no benchmark in place.
author_association: MEMBER
user: login: dkrako
created_at: 2025-04-28T11:46:31Z
body: >     2. I think we should find import leakage and figure out what is going on there instead of merging it
> > the numbers seem to be kinda consistent

You didn't add the results for 10 runs. Your first run already shows that there's import leakage already for the first run.

I removed the leaking brenchmark. I don't see any reason why we should try to hack around `pytest-benchmark` as it's clearly not designed to measure import times. see: https://github.com/ionelmc/pytest-benchmark/issues/177

Importing pymovements in a subprocess is the best way to avoid import leakage.
author_association: MEMBER
user: login: dkrako
created_at: 2025-04-28T15:04:47Z
body: To further clarify: the major contribution of this PR is implementing a benchmarking workflow for github workers.
the benchmarking of import times is minor in that regard.

Nevertheless, import times are very long currently. You're machine must be extremely fast. Mine usually takes 3 to 4 seconds.

The import leakage that you get already in your first run may misguide you: the `test_import_pymovements_subprocess` is much closer to the real import time than your results on `test_import_pymovements`.

You can try this out yourself using 

```python
>>> import timeit
>>> timeit.timeit('import pymovements')
Backend tkagg is interactive backend. Turning interactive mode on.
3.9113128779572435
```
author_association: MEMBER
user: login: SiQube
created_at: 2025-04-28T15:07:13Z
body: ```python
>>> import timeit
>>> timeit.timeit('import pymovements')
1.0151693140001044
```
author_association: CONTRIBUTOR
user: login: dkrako
created_at: 2025-04-28T15:09:51Z
body: > ```python
> >>> import timeit
> >>> timeit.timeit('import pymovements')
> 1.0151693140001044
> ```

Which is nearly ten times as high as your single run of `test_import_pymovements()` and proves my case.
author_association: MEMBER
user: login: SiQube
created_at: 2025-04-28T15:13:45Z
body: yea perfect, you already removed the flaky test, what are we discussing here?
author_association: CONTRIBUTOR
user: login: dkrako
created_at: 2025-04-28T15:18:45Z
body: Thanks!

As an additional note: I won't set this workflow up as a hard merge requirement for now.

We will get notified in case commits or PRs deteriorate performance and can then act accordingly.

Let's see how this works out first.
author_association: MEMBER