Repository: milesgranger/cramjam (language: python)

Issue:
test_bench.py: Snappy tests fail
Something I've stumbled upon packaging this for nixpkgs https://github.com/NixOS/nixpkgs/pull/124862.
Here is the build log https://gist.github.com/veprbl/345d882b01923f31fcfc75c1238d305b
Fastparquet's round trip tests seem to pass just fine. Perhaps they don't use this code path?


Discussion / hints:
Please disregard the previous log, it was build without `python-snappy`. Here is a corrected one: https://gist.github.com/veprbl/288bc21cede2c2aa676362ff0e033e0b
The log is quite hard to read, but are you attempting to reverse `cramjam.snappy.compress` with `decompress_raw_into`? The pairings are
- compress_raw/decompress_raw[_into] 
- compress/decompress[_into] (also known as the "framed" format in some places)

Fix patch:
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
index 56ce049a..0e27b070 100644
--- a/.github/workflows/CI.yml
+++ b/.github/workflows/CI.yml
@@ -118,7 +118,7 @@ jobs:
     - name: Build
       run: cargo build --release
     - name: Tests
-      run: cargo test --no-default-features
+      run: cargo test --no-default-features --release
     - uses: actions/setup-python@v2
       with:
         python-version: ${{ matrix.python-version }}
@@ -139,6 +139,12 @@ jobs:
       run: |
         pip uninstall numpy -y
         python -m pytest tests/test_no_numpy.py
+    - name: Test benchmarks
+      if: matrix.target == 'x86_64'
+      run: |
+        sudo apt-get install -y libsnappy-dev
+        pip install -r benchmark-requirements.txt
+        make test-bench
     - name: Upload wheels
       uses: actions/upload-artifact@v2
       with:
@@ -211,7 +217,7 @@ jobs:
     - name: Build
       run: cargo build --release
     - name: Tests
-      run: cargo test --no-default-features
+      run: cargo test --no-default-features --release
     - uses: actions/setup-python@v2
       with:
         python-version: pypy-3.6
diff --git a/Makefile b/Makefile
index f775a21b..2e060596 100644
--- a/Makefile
+++ b/Makefile
@@ -3,6 +3,9 @@ BASE_BENCH_CMD = python -m pytest -v --benchmark-sort name --benchmark-only benc
 test:
 	python -m pytest tests -v --ignore benchmarks
 
+test-bench:
+	python -m pytest -v --benchmark-disable benchmarks/
+
 bench:
 	python -m pytest -v --benchmark-only --benchmark-sort name benchmarks/
 
diff --git a/benchmark-requirements.txt b/benchmark-requirements.txt
index ff30fc0a..d8331363 100644
--- a/benchmark-requirements.txt
+++ b/benchmark-requirements.txt
@@ -3,3 +3,5 @@ python-snappy==0.5.4
 lz4==3.1.0
 brotlipy==0.7.0
 zstd==1.4.5.1
+numpy
+memory-profiler
diff --git a/dev-requirements.txt b/dev-requirements.txt
index 9591711f..716fce47 100644
--- a/dev-requirements.txt
+++ b/dev-requirements.txt
@@ -1,4 +1,4 @@
-black==19.10b0
+black==21.5b1
 maturin
 numpy
 pytest>=5.3.0


Test patch:
diff --git a/benchmarks/test_bench.py b/benchmarks/test_bench.py
index be376a26..ad72ca76 100644
--- a/benchmarks/test_bench.py
+++ b/benchmarks/test_bench.py
@@ -97,7 +97,7 @@ def test_snappy_framed(benchmark, file, use_cramjam: bool):
         )
 
 
-@pytest.mark.parametrize("op", ("decompress_raw_into", "compress_raw_into"))
+@pytest.mark.parametrize("op", ("decompress_into", "compress_into"))
 @pytest.mark.parametrize("file", FILES, ids=lambda val: val.name)
 def test_cramjam_snappy_de_compress_into(benchmark, op, file):
     """
@@ -110,7 +110,8 @@ def test_cramjam_snappy_de_compress_into(benchmark, op, file):
 
     operation = getattr(snappy, op)
     buffer = np.zeros(
-        len(data) if op == "decompress_into" else len(compressed_data), dtype=np.uint8
+        len(data) if op == "decompress_into" else len(compressed_data),
+        dtype=np.uint8,
     )
 
     benchmark(
@@ -187,7 +188,10 @@ def test_lz4_block(benchmark, file, use_cramjam: bool):
         )
     else:
         benchmark(
-            round_trip, compress=block.compress, decompress=block.decompress, data=data,
+            round_trip,
+            compress=block.compress,
+            decompress=block.decompress,
+            data=data,
         )
 
 
@@ -240,7 +244,10 @@ def test_zstd(benchmark, file, use_cramjam: bool):
         )
     else:
         benchmark(
-            round_trip, compress=zstd.compress, decompress=zstd.decompress, data=data,
+            round_trip,
+            compress=zstd.compress,
+            decompress=zstd.decompress,
+            data=data,
         )