Repository: tahoe-lafs/zfec (language: python)

Issue:
Try switching to -O3, see if it speeds things up
-O3 enables auto-vectorization, among other compiler optimizations, so might make things run faster. Blocked on #81 


Discussion / hints:
Some numbers from experiments:

| Compiler | Options              | Speed (MB/sec) |
|----------|----------------------|---------------:|
| gcc 11   | Default              |            360 |
| gcc 11   | -O3                  |            230 |
| clang 14 | Default              |            375 |
| clang 14 | -O3                  |            377 |
| gcc 11   | -march=x86-64-v2     |            478 |
| gcc 11   | -O3 -march=x86-64-v2 |            446 |
| gcc 11   | -O3 -march=x86-64-v3 |            446 |

-O3 was not helpful!

Main speed up for compiler options, as seen above, is telling the compiler you're not on an ancient CPU. Here was RHEL 9's explanation of why they switched to this: https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level

It's hard to get survey data, best I know of is Steam's. It suggests maybe... 0.6% of computers in their survey won't work with x86-64-v2 due to e.g. lack of SSE4.2. https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam


I am now failing to reproduce the above numbers :(
Ah, nevermind, cached build artifacts. Still seems like `-march=x86-64-v2` ~doubles speed of both encoding and decoding.

Fix patch:
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 58f3f88..6e0c136 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -38,7 +38,6 @@ jobs:
           - "ubuntu-22.04"
           - "windows-2022"
         wheel-selector:
-          - "cp38-*"
           - "cp39-*"
           - "cp310-*"
           - "cp311-*"
@@ -57,7 +56,7 @@ jobs:
           fetch-depth: 0
 
       - name: Build wheels
-        uses: pypa/cibuildwheel@v2.20.0
+        uses: pypa/cibuildwheel@v2.21.3
         with:
           output-dir: wheelhouse
         env:
@@ -71,6 +70,11 @@ jobs:
           CIBW_ARCHS_MACOS: "x86_64 universal2 arm64"
           # Skip testing arm64 builds on Intel Macs.
           CIBW_TEST_SKIP: "*-macosx_arm64 *-macosx_universal2:arm64"
+          # Use newer manylinux so we get newer gcc and can take advantage of
+          # speed boost from x86-64-v2, at the cost of not supporting some old
+          # patch releases of Python 3.9 if they haven't updated pip in their
+          # virtualenv.
+          CIBW_MANYLINUX_X86_64_IMAGE: "manylinux_2_28"
       - uses: actions/upload-artifact@v3
         name: Upload artifacts
         with:
diff --git a/setup.py b/setup.py
index 27695b6..8c1789d 100644
--- a/setup.py
+++ b/setup.py
@@ -1,8 +1,8 @@
 from setuptools import setup
 from setuptools.extension import Extension
 
+import platform
 import sys
-import os
 import versioneer
 
 DEBUGMODE = False
@@ -24,6 +24,11 @@
         break
 
 extra_compile_args.append("-std=c99")
+if platform.machine() == "x86_64" and platform.system().lower().startswith("linux"):
+    # Only support CPUs starting from 2008/2009 or so, and run twice as fast as
+    # a result! This requires sufficiently new gcc, 11 maybe?
+    extra_compile_args.append("-march=x86-64-v2")
+
 if DEBUGMODE:
     extra_compile_args.append("-O0")
     extra_compile_args.append("-g")


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 852ee25..2a51799 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -21,13 +21,11 @@ jobs:
           - "ubuntu-22.04"
           - "windows-2022"
         python-version:
-          - "3.8"
           - "3.9"
           - "3.10"
           - "3.11"
           - "3.12"
-          - "3.13-dev"
-          - "pypy-3.8"
+          - "3.13"
           - "pypy-3.9"
           - "pypy-3.10"