Repository: mwlon/pcodec (language: python)

Issue:
Python bindings?
Thanks for sharing this exciting new compression codec. :pray:

I'm a maintainer of [numcodecs](https://numcodecs.readthedocs.io/en/stable/), a Python library which attempts to wrap all common numerical compression codecs (e.g. blosc, zstd, lz4, etc.) and present them via a standard interface. Numcodecs is used, for example, by the [Zarr](https://zarr.dev/) library as a compression layer.

We'd love to include pcodec in numcodecs. To do so, we'd need some sort of Python wrapper. This should be straightforward with pyo3. We would be happy to make a PR to accomplish this, but I thought I would first open an issue to see whether this was already on the roadmap. Given the amount of data science that happens in Python, Python bindings would help your work reach a much wider audience.

Thanks again for your generous open source contributions. 
Python bindings?
Thanks for sharing this exciting new compression codec. :pray:

I'm a maintainer of [numcodecs](https://numcodecs.readthedocs.io/en/stable/), a Python library which attempts to wrap all common numerical compression codecs (e.g. blosc, zstd, lz4, etc.) and present them via a standard interface. Numcodecs is used, for example, by the [Zarr](https://zarr.dev/) library as a compression layer.

We'd love to include pcodec in numcodecs. To do so, we'd need some sort of Python wrapper. This should be straightforward with pyo3. We would be happy to make a PR to accomplish this, but I thought I would first open an issue to see whether this was already on the roadmap. Given the amount of data science that happens in Python, Python bindings would help your work reach a much wider audience.

Thanks again for your generous open source contributions. 


Discussion / hints:
@rabernat sounds like a great idea! This is something I've wanted to do, but haven't made an explicit plan to do yet. A PR would certainly be welcome.

Are you thinking of just wrapping the auto_compress/decompress functions or lower level functionality as well?
Having taken a closer look at numcodecs: seems like the public functionality is compressing/decompressing an entire buffer, so the standalone format and auto_* would be a good fit. It would compress/decompress to a numpy array (or something like one), and it would take a config that looks like [ChunkConfig](https://docs.rs/pco/latest/pco/struct.ChunkConfig.html).

Does that all match up with your expectations?
Thanks for the quick reply!

> Are you thinking of just wrapping the auto_compress/decompress functions or lower level functionality as well?

I think that's a fine place to start. From the numcodecs pov, the interface we need is simple: we want to turn numpy arrays into compressed `bytes` and vice versa, ideally with as few memory copies as much as possible. `auto_compress` and `simple_compress` should be fine, since they can accept views into an existing numpy array. They need to allocate a new (unknown size) bytes object, and that's fine.

For `auto_decompress`, the one thing missing is the ability to decompress into an existing buffer. For numcodecs, we will always know the size of the array we are decompressing, and many users of the library will provide an `out` argument: https://github.com/zarr-developers/numcodecs/blob/4896680087d3ff1f959401c51cf5aea0fd56554e/numcodecs/abc.py#L59-L70

But it looks like `auto_decompress` always allocates a new vec for its output:
https://github.com/mwlon/pcodec/blob/2314e329f36684b3f6476f4d69150acc6ac3786b/pco/src/standalone/simple.rs#L64

So we would have to copy the results into `out`. This means we would likely want to use the lower-level interface... or add a new high-level function with this capability.
Aha - in that case, probably best I add a new piece of high level functionality for you to use; I might make an unrelated change to the low level standalone API in the next release.

What is numcodec's intended behavior when the `dest` is too small?
sounds like a great idea! This is something I've wanted to do, but haven't made an explicit plan to do yet. A PR would certainly be welcome.

Are you thinking of just wrapping the auto_compress/decompress functions or lower level functionality as well?


Having taken a closer look at numcodecs: seems like the public functionality is compressing/decompressing an entire buffer, so the standalone format and auto_* would be a good fit. It would compress/decompress to a numpy array (or something like one), and it would take a config that looks like [ChunkConfig](https://docs.rs/pco/latest/pco/struct.ChunkConfig.html).

Does that all match up with your expectations?
> What is numcodec's intended behavior when the `dest` is too small?

The way we use numcodecs from zarr, that should never happen. We are usually decompressing ND arrays with an a priori known size. It's the user's responsibility to make the buffer the right size: "this buffer must be exactly the right size to store the decoded data."

So the correct behavior if `dest` is too small would be to raise an error.
I've added `pco::standalone::simple_decompress_into` to main, and I'll probably make a 0.1.0 release including it in a couple days.
I just released 0.1.0, so I think you should be unblocked. Looking forward to your PR!
I have a basic working version up here: https://github.com/rabernat/pcodec-python

The way I have approached this to start was to create a tiny standalone crate with dependencies on pco, pyo3, and numpy. This contains:
- [lib.rs](https://github.com/rabernat/pcodec-python/blob/main/src/lib.rs) - the simple glue code to make the pco functions callable from python
- [test_pcodec.py](https://github.com/rabernat/pcodec-python/blob/main/test/test_pcodec.py) - a simple python test

Before making a PR, I thought I would get your feedback on how you'd want to structure this.
- How do you feel about making a new top-level `pco_python` crate in this package? Or would you rather have this embedded in the `pco` crate itself?
- Are you supportive of running the python tests as part of the CI in this repo?

Thanks for working on this!

> How do you feel about making a new top-level pco_python crate in this package? Or would you rather have this embedded in the pco crate itself?

I'm happy with a top-level `publish = false` crate under the workspace.

> Are you supportive of running the python tests as part of the CI in this repo?

Yes. We'd configure that under .github/workflows somehow

I took a quick look at your code, and it looks like the right direction. Two observations:

- I think we'll actually want to use `pco::standalone::simple_compress` so we can pass in an exact configuration. We can start with just compression level but should make it easy to add more.
- To avoid the repetition you mention, we could either use generic functions (probably best, requires only a small amount of repetition) or macros (more confusing but requires no repetition). I can help you out with this once you have a PR going.
@rabernat sounds like a great idea! This is something I've wanted to do, but haven't made an explicit plan to do yet. A PR would certainly be welcome.

Are you thinking of just wrapping the auto_compress/decompress functions or lower level functionality as well?
Having taken a closer look at numcodecs: seems like the public functionality is compressing/decompressing an entire buffer, so the standalone format and auto_* would be a good fit. It would compress/decompress to a numpy array (or something like one), and it would take a config that looks like [ChunkConfig](https://docs.rs/pco/latest/pco/struct.ChunkConfig.html).

Does that all match up with your expectations?
Thanks for the quick reply!

> Are you thinking of just wrapping the auto_compress/decompress functions or lower level functionality as well?

I think that's a fine place to start. From the numcodecs pov, the interface we need is simple: we want to turn numpy arrays into compressed `bytes` and vice versa, ideally with as few memory copies as much as possible. `auto_compress` and `simple_compress` should be fine, since they can accept views into an existing numpy array. They need to allocate a new (unknown size) bytes object, and that's fine.

For `auto_decompress`, the one thing missing is the ability to decompress into an existing buffer. For numcodecs, we will always know the size of the array we are decompressing, and many users of the library will provide an `out` argument: https://github.com/zarr-developers/numcodecs/blob/4896680087d3ff1f959401c51cf5aea0fd56554e/numcodecs/abc.py#L59-L70

But it looks like `auto_decompress` always allocates a new vec for its output:
https://github.com/mwlon/pcodec/blob/2314e329f36684b3f6476f4d69150acc6ac3786b/pco/src/standalone/simple.rs#L64

So we would have to copy the results into `out`. This means we would likely want to use the lower-level interface... or add a new high-level function with this capability.
Aha - in that case, probably best I add a new piece of high level functionality for you to use; I might make an unrelated change to the low level standalone API in the next release.

What is numcodec's intended behavior when the `dest` is too small?
sounds like a great idea! This is something I've wanted to do, but haven't made an explicit plan to do yet. A PR would certainly be welcome.

Are you thinking of just wrapping the auto_compress/decompress functions or lower level functionality as well?


Having taken a closer look at numcodecs: seems like the public functionality is compressing/decompressing an entire buffer, so the standalone format and auto_* would be a good fit. It would compress/decompress to a numpy array (or something like one), and it would take a config that looks like [ChunkConfig](https://docs.rs/pco/latest/pco/struct.ChunkConfig.html).

Does that all match up with your expectations?
> What is numcodec's intended behavior when the `dest` is too small?

The way we use numcodecs from zarr, that should never happen. We are usually decompressing ND arrays with an a priori known size. It's the user's responsibility to make the buffer the right size: "this buffer must be exactly the right size to store the decoded data."

So the correct behavior if `dest` is too small would be to raise an error.
I've added `pco::standalone::simple_decompress_into` to main, and I'll probably make a 0.1.0 release including it in a couple days.
I just released 0.1.0, so I think you should be unblocked. Looking forward to your PR!
I have a basic working version up here: https://github.com/rabernat/pcodec-python

The way I have approached this to start was to create a tiny standalone crate with dependencies on pco, pyo3, and numpy. This contains:
- [lib.rs](https://github.com/rabernat/pcodec-python/blob/main/src/lib.rs) - the simple glue code to make the pco functions callable from python
- [test_pcodec.py](https://github.com/rabernat/pcodec-python/blob/main/test/test_pcodec.py) - a simple python test

Before making a PR, I thought I would get your feedback on how you'd want to structure this.
- How do you feel about making a new top-level `pco_python` crate in this package? Or would you rather have this embedded in the `pco` crate itself?
- Are you supportive of running the python tests as part of the CI in this repo?

Thanks for working on this!

> How do you feel about making a new top-level pco_python crate in this package? Or would you rather have this embedded in the pco crate itself?

I'm happy with a top-level `publish = false` crate under the workspace.

> Are you supportive of running the python tests as part of the CI in this repo?

Yes. We'd configure that under .github/workflows somehow

I took a quick look at your code, and it looks like the right direction. Two observations:

- I think we'll actually want to use `pco::standalone::simple_compress` so we can pass in an exact configuration. We can start with just compression level but should make it easy to add more.
- To avoid the repetition you mention, we could either use generic functions (probably best, requires only a small amount of repetition) or macros (more confusing but requires no repetition). I can help you out with this once you have a PR going.

Fix patch:
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
new file mode 100644
index 00000000..23316b1e
--- /dev/null
+++ b/.github/workflows/python.yml
@@ -0,0 +1,28 @@
+name: Python
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.11'
+    # TODO: there has to be a cleaner way to do this and also add caching
+    - name: "create venv"
+      run: python -m venv .venv
+    - uses: PyO3/maturin-action@v1
+      with:
+        command: develop
+        args: --manifest-path pco_python/Cargo.toml
+        sscache: true
+    - name: Install test requirements
+      run: source .venv/bin/activate && pip install -r pco_python/test_requirements.txt
+    - name: Run tests
+      run: source .venv/bin/activate && pytest -v
diff --git a/.github/workflows/python_ci.yml b/.github/workflows/python_ci.yml
new file mode 100644
index 00000000..0cb8cb2a
--- /dev/null
+++ b/.github/workflows/python_ci.yml
@@ -0,0 +1,119 @@
+# This file was originally autogenerated by maturin v1.4.0, but has been
+# modified by hand.
+name: Python CI
+
+on:
+  release:
+    types:
+      - published
+
+permissions:
+  contents: read
+
+jobs:
+  linux:
+    runs-on: ubuntu-latest
+    if: contains(github.event.release.name, 'Python')
+    strategy:
+      matrix:
+        target: [x86_64, x86, aarch64, armv7, s390x, ppc64le]
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+      - name: Build wheels
+        uses: PyO3/maturin-action@v1
+        with:
+          target: ${{ matrix.target }}
+          args: --release --out dist --find-interpreter --manifest-path pco_python/Cargo.toml
+          sccache: 'true'
+          manylinux: auto
+      - name: Upload wheels
+        uses: actions/upload-artifact@v3
+        with:
+          name: wheels
+          path: dist
+
+  windows:
+    runs-on: windows-latest
+    if: contains(github.event.release.name, 'Python')
+    strategy:
+      matrix:
+        target: [x64, x86]
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+          architecture: ${{ matrix.target }}
+      - name: Build wheels
+        uses: PyO3/maturin-action@v1
+        with:
+          target: ${{ matrix.target }}
+          args: --release --out dist --find-interpreter --manifest-path pco_python/Cargo.toml
+          sccache: 'true'
+      - name: Upload wheels
+        uses: actions/upload-artifact@v3
+        with:
+          name: wheels
+          path: dist
+
+  macos:
+    runs-on: macos-latest
+    if: contains(github.event.release.name, 'Python')
+    strategy:
+      matrix:
+        target: [x86_64, aarch64]
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+      - name: Build wheels
+        uses: PyO3/maturin-action@v1
+        with:
+          target: ${{ matrix.target }}
+          args: --release --out dist --find-interpreter --manifest-path pco_python/Cargo.toml
+          sccache: 'true'
+      - name: Upload wheels
+        uses: actions/upload-artifact@v3
+        with:
+          name: wheels
+          path: dist
+
+  sdist:
+    runs-on: ubuntu-latest
+    if: contains(github.event.release.name, 'Python')
+    steps:
+      - uses: actions/checkout@v3
+      - name: Build sdist
+        uses: PyO3/maturin-action@v1
+        with:
+          command: sdist
+          args: --out dist --manifest-path pco_python/Cargo.toml
+      - name: Upload sdist
+        uses: actions/upload-artifact@v3
+        with:
+          name: wheels
+          path: dist
+
+  release:
+    name: Release
+    runs-on: ubuntu-latest
+    if: contains(github.event.release.name, 'Python')
+    needs: [linux, windows, macos, sdist]
+    environment:
+      name: pypi
+      url: https://pypi.org/p/pcodec
+    permissions:
+      id-token: write  # IMPORTANT: mandatory for trusted publishing
+    steps:
+      - uses: actions/download-artifact@v3
+        with:
+          name: wheels
+      - name: Publish to PyPI
+        uses: PyO3/maturin-action@v1
+        with:
+          command: upload
+          args: --non-interactive --skip-existing *
diff --git a/Cargo.lock b/Cargo.lock
index 7f0011ba..2510491b 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -364,14 +364,6 @@ version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "a2bd12c1caf447e69cd4528f47f94d203fd2582878ecb9e9465484c4148a8223"
 
-[[package]]
-name = "c_sharp"
-version = "0.0.0"
-dependencies = [
- "libc",
- "q_compress",
-]
-
 [[package]]
 name = "cc"
 version = "1.0.79"
@@ -720,6 +712,12 @@ dependencies = [
  "cc",
 ]
 
+[[package]]
+name = "indoc"
+version = "2.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1e186cfbae8084e513daff4240b4797e342f988cecda4fb6c939150f96315fd8"
+
 [[package]]
 name = "integer-encoding"
 version = "3.0.4"
@@ -868,12 +866,31 @@ dependencies = [
  "cfg-if",
 ]
 
+[[package]]
+name = "matrixmultiply"
+version = "0.3.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7574c1cf36da4798ab73da5b215bbf444f50718207754cb522201d78d1cd0ff2"
+dependencies = [
+ "autocfg",
+ "rawpointer",
+]
+
 [[package]]
 name = "memchr"
 version = "2.4.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "308cc39be01b73d0d18f82a0e7b2a3df85245f84af96fdddc5d202d27e47b86a"
 
+[[package]]
+name = "memoffset"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5a634b1c61a95585bd15607c6ab0c4e5b226e695ff2800ba0cdccddf208c406c"
+dependencies = [
+ "autocfg",
+]
+
 [[package]]
 name = "mio"
 version = "0.8.2"
@@ -897,6 +914,19 @@ dependencies = [
  "winapi",
 ]
 
+[[package]]
+name = "ndarray"
+version = "0.15.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "adb12d4e967ec485a5f71c6311fe28158e9d6f4bc4a447b474184d0f91a8fa32"
+dependencies = [
+ "matrixmultiply",
+ "num-complex",
+ "num-integer",
+ "num-traits",
+ "rawpointer",
+]
+
 [[package]]
 name = "ntapi"
 version = "0.3.7"
@@ -993,6 +1023,21 @@ dependencies = [
  "libc",
 ]
 
+[[package]]
+name = "numpy"
+version = "0.20.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bef41cbb417ea83b30525259e30ccef6af39b31c240bda578889494c5392d331"
+dependencies = [
+ "libc",
+ "ndarray",
+ "num-complex",
+ "num-integer",
+ "num-traits",
+ "pyo3",
+ "rustc-hash",
+]
+
 [[package]]
 name = "once_cell"
 version = "1.18.0"
@@ -1099,6 +1144,15 @@ dependencies = [
  "pco",
 ]
 
+[[package]]
+name = "pco_python"
+version = "0.1.0"
+dependencies = [
+ "numpy",
+ "pco",
+ "pyo3",
+]
+
 [[package]]
 name = "pin-project-lite"
 version = "0.2.9"
@@ -1162,6 +1216,67 @@ dependencies = [
  "unicode-ident",
 ]
 
+[[package]]
+name = "pyo3"
+version = "0.20.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "04e8453b658fe480c3e70c8ed4e3d3ec33eb74988bd186561b0cc66b85c3bc4b"
+dependencies = [
+ "cfg-if",
+ "indoc",
+ "libc",
+ "memoffset",
+ "parking_lot",
+ "pyo3-build-config",
+ "pyo3-ffi",
+ "pyo3-macros",
+ "unindent",
+]
+
+[[package]]
+name = "pyo3-build-config"
+version = "0.20.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a96fe70b176a89cff78f2fa7b3c930081e163d5379b4dcdf993e3ae29ca662e5"
+dependencies = [
+ "once_cell",
+ "target-lexicon",
+]
+
+[[package]]
+name = "pyo3-ffi"
+version = "0.20.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "214929900fd25e6604661ed9cf349727c8920d47deff196c4e28165a6ef2a96b"
+dependencies = [
+ "libc",
+ "pyo3-build-config",
+]
+
+[[package]]
+name = "pyo3-macros"
+version = "0.20.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dac53072f717aa1bfa4db832b39de8c875b7c7af4f4a6fe93cdbf9264cf8383b"
+dependencies = [
+ "proc-macro2",
+ "pyo3-macros-backend",
+ "quote",
+ "syn 2.0.23",
+]
+
+[[package]]
+name = "pyo3-macros-backend"
+version = "0.20.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7774b5a8282bd4f25f803b1f0d945120be959a36c72e08e7cd031c792fdfd424"
+dependencies = [
+ "heck",
+ "proc-macro2",
+ "quote",
+ "syn 2.0.23",
+]
+
 [[package]]
 name = "q_compress"
 version = "0.11.7"
@@ -1171,6 +1286,14 @@ dependencies = [
  "tokio",
 ]
 
+[[package]]
+name = "q_compress_ffi"
+version = "0.0.0"
+dependencies = [
+ "libc",
+ "q_compress",
+]
+
 [[package]]
 name = "quote"
 version = "1.0.29"
@@ -1219,6 +1342,12 @@ dependencies = [
  "rand_core",
 ]
 
+[[package]]
+name = "rawpointer"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3"
+
 [[package]]
 name = "redox_syscall"
 version = "0.2.13"
@@ -1257,6 +1386,12 @@ version = "0.8.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c08c74e62047bb2de4ff487b251e4a92e24f48745648451635cec7d591162d9f"
 
+[[package]]
+name = "rustc-hash"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
+
 [[package]]
 name = "rustc_version"
 version = "0.4.0"
@@ -1404,6 +1539,12 @@ dependencies = [
  "syn 1.0.86",
 ]
 
+[[package]]
+name = "target-lexicon"
+version = "0.12.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "14c39fd04924ca3a864207c66fc2cd7d22d7c016007f9ce846cbb9326331930a"
+
 [[package]]
 name = "thrift"
 version = "0.17.0"
@@ -1483,6 +1624,12 @@ version = "0.2.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3"
 
+[[package]]
+name = "unindent"
+version = "0.2.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c7de7d73e1754487cb58364ee906a499937a0dfabd86bcb980fa99ec8c8fa2ce"
+
 [[package]]
 name = "utf8parse"
 version = "0.2.1"
diff --git a/Cargo.toml b/Cargo.toml
index 14c067cc..d907c5ca 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -6,6 +6,7 @@ members = [
   "better_io",
   "pco",
   "pco_cli",
+  "pco_python",
   "quantile-compression/ffi",
   "quantile-compression/q_compress",
 ]
diff --git a/pco_python/Cargo.toml b/pco_python/Cargo.toml
new file mode 100644
index 00000000..2663c929
--- /dev/null
+++ b/pco_python/Cargo.toml
@@ -0,0 +1,18 @@
+[package]
+name = "pco_python"
+version = "0.0.0"
+edition = "2021"
+publish = false
+
+[lib]
+# The name of the native library. This is the name which will be used in Python to import the
+# library (i.e. `import string_sum`). If you change this, you must also change the name of the
+# `#[pymodule]` in `src/lib.rs`.
+name = "pcodec"
+crate-type = ["cdylib"]
+
+
+[dependencies]
+numpy = "0.20.0"
+pco = {version = "0.1.0", path = "../pco" }
+pyo3 = { version = "0.20.0", features = ["extension-module"] }
diff --git a/pco_python/pyproject.toml b/pco_python/pyproject.toml
new file mode 100644
index 00000000..b56afeb8
--- /dev/null
+++ b/pco_python/pyproject.toml
@@ -0,0 +1,23 @@
+[project]
+authors = [
+  {name = "mwlon", email = "m.w.loncaric@gmail.com"}
+]
+description = "Good compression for numerical sequences"
+dependencies = [
+  "numpy",
+]
+dynamic = ["version"]
+keywords = ["compression", "numerical"]
+license = "Apache-2.0"
+name = "pcodec"
+
+[project.urls]
+Repository = "https://github.com/mwlon/pcodec"
+
+[build-system]
+build-backend = "maturin"
+requires = ["maturin>=1.0,<2.0"]
+
+[tool.maturin]
+# "extension-module" tells pyo3 we want to build an extension module (skips linking against libpython.so)
+features = ["pyo3/extension-module"]
\ No newline at end of file
diff --git a/pco_python/src/lib.rs b/pco_python/src/lib.rs
new file mode 100644
index 00000000..f2a9274d
--- /dev/null
+++ b/pco_python/src/lib.rs
@@ -0,0 +1,82 @@
+use numpy::{Element, PyArrayDyn};
+use pyo3::exceptions::PyRuntimeError;
+use pyo3::prelude::{pymodule, FromPyObject, PyModule, PyObject, PyResult, Python};
+use pyo3::pyclass;
+use pyo3::types::PyBytes;
+
+use pco::data_types::NumberLike;
+use pco::standalone::{auto_compress, simple_decompress_into};
+use pco::DEFAULT_COMPRESSION_LEVEL;
+
+#[pyclass]
+struct PyProgress {
+  #[pyo3(get)]
+  n_processed: usize,
+  #[pyo3(get)]
+  finished: bool,
+}
+
+// The Numpy crate recommends using this type of enum to write functions that accept different Numpy dtypes
+// https://github.com/PyO3/rust-numpy/blob/32740b33ec55ef0b7ebec726288665837722841d/examples/simple/src/lib.rs#L113
+#[derive(FromPyObject)]
+enum ArrayDynFloat<'py> {
+  F32(&'py PyArrayDyn<f32>),
+  F64(&'py PyArrayDyn<f64>),
+  I32(&'py PyArrayDyn<i32>),
+  I64(&'py PyArrayDyn<i64>),
+  U32(&'py PyArrayDyn<u32>),
+  U64(&'py PyArrayDyn<u64>),
+}
+
+fn compress_typed<T: NumberLike + Element>(py: Python, arr: &PyArrayDyn<T>) -> PyResult<PyObject> {
+  let arr_ro = arr.readonly();
+  let src = arr_ro.as_slice()?;
+  let compressed = auto_compress(src, DEFAULT_COMPRESSION_LEVEL);
+  Ok(PyBytes::new(py, &compressed).into())
+}
+
+fn decompress_typed<T: NumberLike + Element>(
+  compressed: &PyBytes,
+  out: &PyArrayDyn<T>,
+) -> PyResult<PyProgress> {
+  let mut out_rw = out.readwrite();
+  let dst = out_rw.as_slice_mut()?;
+  let src = compressed.as_bytes();
+  let progress = simple_decompress_into(src, dst)
+    .map_err(|e| PyRuntimeError::new_err(format!("pco decompression error: {}", e)))?;
+  Ok(PyProgress {
+    n_processed: progress.n_processed,
+    finished: progress.finished,
+  })
+}
+
+#[pymodule]
+fn pcodec(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
+  m.add("__version__", env!("CARGO_PKG_VERSION"))?;
+
+  #[pyfn(m)]
+  fn auto_compress<'py>(py: Python<'py>, x: ArrayDynFloat<'py>) -> PyResult<PyObject> {
+    match x {
+      ArrayDynFloat::F32(py_arr) => compress_typed(py, py_arr),
+      ArrayDynFloat::F64(py_arr) => compress_typed(py, py_arr),
+      ArrayDynFloat::I32(py_arr) => compress_typed(py, py_arr),
+      ArrayDynFloat::I64(py_arr) => compress_typed(py, py_arr),
+      ArrayDynFloat::U32(py_arr) => compress_typed(py, py_arr),
+      ArrayDynFloat::U64(py_arr) => compress_typed(py, py_arr),
+    }
+  }
+
+  #[pyfn(m)]
+  fn simple_decompress_into(compressed: &PyBytes, out: ArrayDynFloat) -> PyResult<PyProgress> {
+    match out {
+      ArrayDynFloat::F32(out) => decompress_typed(compressed, out),
+      ArrayDynFloat::F64(out) => decompress_typed(compressed, out),
+      ArrayDynFloat::I32(out) => decompress_typed(compressed, out),
+      ArrayDynFloat::I64(out) => decompress_typed(compressed, out),
+      ArrayDynFloat::U32(out) => decompress_typed(compressed, out),
+      ArrayDynFloat::U64(out) => decompress_typed(compressed, out),
+    }
+  }
+
+  Ok(())
+}


Test patch:
diff --git a/pco_python/test/test_pcodec.py b/pco_python/test/test_pcodec.py
new file mode 100644
index 00000000..3bab3572
--- /dev/null
+++ b/pco_python/test/test_pcodec.py
@@ -0,0 +1,54 @@
+import numpy as np
+from pcodec import auto_compress, simple_decompress_into
+import pytest
+
+np.random.seed(12345)
+
+@pytest.mark.parametrize(
+    "shape",
+    [
+        (100,),
+        (100, 100),
+        (10, 10, 100),
+        (2, 10, 10, 50),
+    ],
+)
+@pytest.mark.parametrize("dtype", ['f4', 'f8', 'i4', 'i8', 'u4', 'u8'])
+def test_round_trip(shape, dtype):
+    data = np.random.uniform(0, 1000, size=shape).astype(dtype)
+    compressed = auto_compress(data)
+
+    # decompress exactly
+    out = np.empty_like(data)
+    progress = simple_decompress_into(compressed, out)
+    np.testing.assert_array_equal(data, out)
+    assert progress.n_processed == data.size
+    assert progress.finished
+
+def test_inexact_decompression():
+    data = np.random.uniform(size=300)
+    compressed = auto_compress(data)
+
+    # decompress partially
+    out = np.zeros(3)
+    progress = simple_decompress_into(compressed, out)
+    np.testing.assert_array_equal(out, data[:3])
+    assert progress.n_processed == 3
+    assert not progress.finished
+
+    # decompress with room to spare
+    out = np.zeros(600)
+    progress = simple_decompress_into(compressed, out)
+    np.testing.assert_array_equal(out[:300], data)
+    np.testing.assert_array_equal(out[300:], np.zeros(300))
+    assert progress.n_processed == 300
+    assert progress.finished
+
+def test_errors():
+    data = np.random.uniform(size=100).astype(np.float32)
+    compressed = auto_compress(data)
+
+    out = np.zeros(100).astype(np.float64)
+    with pytest.raises(RuntimeError):
+        simple_decompress_into(compressed, out)
+
diff --git a/pco_python/test_requirements.txt b/pco_python/test_requirements.txt
new file mode 100644
index 00000000..55b033e9
--- /dev/null
+++ b/pco_python/test_requirements.txt
@@ -0,0 +1,1 @@
+pytest
\ No newline at end of file