Repository: zarr-developers/zarr-extensions
Pull request #8: add `zfp` codec
Author: login: LDeakin
Created: 2025-04-18T00:40:33Z | Merged: 2025-04-24T15:01:59Z

I intend to release this codec with the next [`zarrs`](https://crates.io/crates/zarrs) release. If this codec configuration and implementation is not controversial, it would be good if it were just named `zfp`?

Previous `zarrs` releases had some differences from this spec:
- `"name": "https://codec.zarrs.dev/array_to_bytes/zfp"`
- Supports a `write_header` parameter that I would prefer to remove

The inclusion of a `write_header` parameter might be worthy of discussion though. If it were present, then this codec is fully compatible with data encoded with `zfpy` in Zarr V2 / `numcodecs.zfpy` in Zarr V3 with just a change of metadata. I support `zfpy` independently in `zarrs`, although it is the same underlying codec implementation.


Diff:
diff --git a/codecs/zarrs.zfp/README.md b/codecs/zarrs.zfp/README.md
new file mode 100644
index 0000000..d5af1b5
--- /dev/null
+++ b/codecs/zarrs.zfp/README.md
@@ -0,0 +1,115 @@
+# ZFP codec
+
+Defines a `array -> bytes` codec that compresses chunks using the [zfp](https://github.com/LLNL/zfp) algorithm.
+
+## Codec name
+
+The value of the `name` member in the codec object MUST be `zfp`.
+
+## Configuration parameters
+
+The configuration of this codec matches the compression modes defined at <https://zfp.readthedocs.io/en/latest/modes.html#expert-mode>.
+Refer to that page for usage information.
+
+The codec has one parameter which is always required:
+- `mode` (string).
+
+The other required parameters are dependent on the `mode`:
+- `"mode": "reversible"`
+- `"mode": "expert"`
+    - `minbits` (unsigned integer)
+    - `maxbits` (unsigned integer)
+    - `maxprec` (unsigned integer)
+    - `minexp` (signed integer)
+- `"mode": "fixed_accuracy"`
+    - `tolerance` (number)
+- `"mode": "fixed_rate"`
+    - `rate` (number)
+- `"fixed_precision"`
+    - `precision` (unsigned integer)
+
+## Example
+
+For example, the array metadata below specifies that the array contains `zfp` compressed chunks using `fixed_accuracy` mode with a tolerance of 0.05:
+
+```json
+{
+    "codecs": [{
+        "name": "zfp",
+        "configuration": {
+            "mode": "fixed_accuracy",
+            "tolerance": 0.05
+        }
+    }],
+}
+```
+
+More examples can be viewed in the [examples](./examples/) subdirectory.
+
+## Supported Chunk Shapes
+
+`zfp` natively only supports 1, 2, 3 and 4 dimensional arrays.
+Chunk shapes are mapped to `zfp` field sizes according to the [`zfp_field_Nd`](https://zfp.readthedocs.io/en/release0.5.5/high-level-api.html#array-metadata) APIs as follows:
+ - 1D: `[nx]`
+ - 2D: `[ny, nx]`
+ - 3D: `[nz, ny, nx]`
+ - 4D: `[nw, nz, ny, nx]`
+
+The chunk of a zero-dimensional Zarr array is interpreted as a 1D `zfp` field with `nx = 1`.
+
+Chunks with more than four dimensions are not supported directly by this codec.
+However, higher-dimensional arrays could be supported by preceeding this codec with a [`np.squeeze`](https://numpy.org/doc/stable/reference/generated/numpy.squeeze.html) inspired array-to-array codec that collapses singleton dimensions (dimensions of size 1), provided the resulting dimensionality is four or fewer.
+For example, a chunk with shape `[4, 1, 3, 1, 2, 1]` would be squeezed to a `zfp` field size of `[nz, ny, nx] = [4, 3, 2]`.
+
+These rules apply to the inner chunk shape if this codec is used as the array-to-bytes codec within the `sharding_indexed` codec.
+
+## Supported Data Types
+
+- `int32`, `uint32`, `int64`, `uint64`, `float32`, `float64`
+
+Implementations may support lower-precision data types (e.g. `float16`, `bfloat16`, `int4`, etc.) through promotion / casting to the above data types.
+
+Implementations may support additional data types that could be interpreted or promoted to the above data types (e.g. `datetime64` -> `int64`).
+
+## Format and algorithm
+
+This format is tightly coupled to the [`zfp` C library](https://zfp.readthedocs.io/en/latest/).
+
+### Compression
+
+1. Lower-precision data types must first be promoted to 32-bit
+   - Floating point data types (e.g. `float16`, `bfloat16`, etc.) can be supported by casting to `float32` in the normal way.
+   - Integer data types must be promoted to `int32` in accordance with the [`zfp_promote_*`](https://zfp.readthedocs.io/en/release0.5.5/low-level-api.html#utility-functions) functions:
+     - `int` with `N` bits: `int32_value = (int32_t)intN_value << (31 - N)`
+     - `uint` with `N` bits: `int32_value = ((int32_t)uintN_value - (1<<(N-1))) << (31 - N)`
+2. The uncompressed data is represented as a contiguous array in a [`zfp_field`](https://zfp.readthedocs.io/en/release0.5.5/high-level-api.html#c.zfp_field) and compressed with `zfp_compress`:
+   - The field sizes are set in accordance with the rules described in [Supported Chunk Shapes](#supported-chunk-shapes).
+
+### Decompression
+
+1. The data is decompressed into a contiguous array with `zfp_decompress`.
+2. Lower-precision data types are restored through demotion:
+   - Floating point data types (e.g. `float16`, `bfloat16`, etc.) can be supported by casting from `float32` in the normal way.
+   - Integer data types must be be demoted from `int32` in accordance with the appropriate [`zfp_demote_*`](https://zfp.readthedocs.io/en/release0.5.5/low-level-api.html#utility-functions) functions:
+     - `int` with `N` bits: `intN_value = (intN_t)clamp(int32_value >> (31 - N), 1<<(N-1), (1<<(N-1)) - 1)`
+     - `uint` with `N` bits: `uintN_value = (uintN_t)clamp((int32_value >> (31 - N)) + (1<<(N-1)), 0, (1<<N) - 1)`
+
+## Differences from `numcodecs.zfpy`
+
+- `mode` is a string rather than an integer.
+- `mode` supports `reversible` and `expert` mode.
+- Lower-precision integer and floating point data types are supported.
+- a header is not written with [`zfp_write_header`](https://zfp.readthedocs.io/en/release0.5.5/high-level-api.html#c.zfp_write_header).
+  - This header is redundant given the information in the codec configuration.
+
+> [!NOTE]
+> An earlier version of the `zfp` codec in the [`zarrs`](https://github.com/LDeakin/zarrs) Rust crate included an optional [`write_header` parameter](https://docs.rs/zarrs_metadata/0.3.7/zarrs_metadata/v3/array/codec/zfp/struct.ZfpCodecConfigurationV1.html) in the codec configuration.
+> This has since been removed in favor of better separating `zfp` and `zfpy`.
+
+## Change log
+
+No changes yet.
+
+## Current maintainers
+
+* [Lachlan Deakin](https://github.com/LDeakin)
diff --git a/codecs/zarrs.zfp/examples/expert.json b/codecs/zarrs.zfp/examples/expert.json
new file mode 100644
index 0000000..4acbb33
--- /dev/null
+++ b/codecs/zarrs.zfp/examples/expert.json
@@ -0,0 +1,10 @@
+{
+  "name": "zfp",
+  "configuration": {
+    "mode": "expert",
+    "minbits": 1,
+    "maxbits": 13,
+    "maxprec": 19,
+    "minexp": -2
+  }
+}
diff --git a/codecs/zarrs.zfp/examples/fixed_accuracy.json b/codecs/zarrs.zfp/examples/fixed_accuracy.json
new file mode 100644
index 0000000..12ffe15
--- /dev/null
+++ b/codecs/zarrs.zfp/examples/fixed_accuracy.json
@@ -0,0 +1,7 @@
+{
+  "name": "zfp",
+  "configuration": {
+    "mode": "fixed_accuracy",
+    "tolerance": 0.05
+  }
+}
diff --git a/codecs/zarrs.zfp/examples/fixed_precision.json b/codecs/zarrs.zfp/examples/fixed_precision.json
new file mode 100644
index 0000000..e17b4f4
--- /dev/null
+++ b/codecs/zarrs.zfp/examples/fixed_precision.json
@@ -0,0 +1,7 @@
+{
+  "name": "zfp",
+  "configuration": {
+    "mode": "fixed_precision",
+    "precision": 19
+  }
+}
diff --git a/codecs/zarrs.zfp/examples/fixed_rate.json b/codecs/zarrs.zfp/examples/fixed_rate.json
new file mode 100644
index 0000000..6d97dc0
--- /dev/null
+++ b/codecs/zarrs.zfp/examples/fixed_rate.json
@@ -0,0 +1,7 @@
+{
+  "name": "zfp",
+  "configuration": {
+    "mode": "fixed_rate",
+    "rate": 10.5
+  }
+}
diff --git a/codecs/zarrs.zfp/examples/reversible.json b/codecs/zarrs.zfp/examples/reversible.json
new file mode 100644
index 0000000..fbf1964
--- /dev/null
+++ b/codecs/zarrs.zfp/examples/reversible.json
@@ -0,0 +1,6 @@
+{
+  "name": "zfp",
+  "configuration": {
+    "mode": "reversible"
+  }
+}
diff --git a/codecs/zarrs.zfp/schema.json b/codecs/zarrs.zfp/schema.json
new file mode 100644
index 0000000..0af1875
--- /dev/null
+++ b/codecs/zarrs.zfp/schema.json
@@ -0,0 +1,94 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "oneOf": [
+    {
+      "type": "object",
+      "properties": {
+        "name": {
+          "const": "zarrs.zfp"
+        },
+        "configuration": {
+          "oneOf": [
+            {
+              "type": "object",
+              "properties": {
+                "mode": {
+                  "const": "reversible"
+                }
+              },
+              "required": ["mode"],
+              "additionalProperties": false
+            },
+            {
+              "type": "object",
+              "properties": {
+                "mode": {
+                  "const": "expert"
+                },
+                "minbits": {
+                  "type": "integer",
+                  "minimum": 0
+                },
+                "maxbits": {
+                  "type": "integer",
+                  "minimum": 0
+                },
+                "maxprec": {
+                  "type": "integer",
+                  "minimum": 0
+                },
+                "minexp": {
+                  "type": "integer"
+                }
+              },
+              "required": ["mode", "minbits", "maxbits", "maxprec", "minexp"],
+              "additionalProperties": false
+            },
+            {
+              "type": "object",
+              "properties": {
+                "mode": {
+                  "const": "fixed_accuracy"
+                },
+                "tolerance": {
+                  "type": "number"
+                }
+              },
+              "required": ["mode", "tolerance"],
+              "additionalProperties": false
+            },
+            {
+              "type": "object",
+              "properties": {
+                "mode": {
+                  "const": "fixed_rate"
+                },
+                "rate": {
+                  "type": "number"
+                }
+              },
+              "required": ["mode", "rate"],
+              "additionalProperties": false
+            },
+            {
+              "type": "object",
+              "properties": {
+                "mode": {
+                  "const": "fixed_precision"
+                },
+                "precision": {
+                  "type": "integer",
+                  "minimum": 0
+                }
+              },
+              "required": ["mode", "precision"],
+              "additionalProperties": false
+            }
+          ]
+        }
+      },
+      "required": ["name", "configuration"],
+      "additionalProperties": false
+    }
+  ]
+}


Reviews:
user: login: jbms
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:27:59Z
user: login: jbms
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:37:19Z
user: login: jbms
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:42:26Z
user: login: jbms
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T17:00:51Z
user: login: LDeakin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T21:21:58Z
user: login: jbms
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T21:55:59Z
user: login: LDeakin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T22:12:51Z
user: login: normanrz
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-24T15:01:53Z

Review comments:
diff_hunk: @@ -0,0 +1,88 @@
+# ZFP codec
+
+Defines a `array -> bytes` codec that compresses chunks using the [zfp](https://github.com/LLNL/zfp) algorithm.
+
+## Codec name
+
+The value of the `name` member in the codec object MUST be `zarrs.zfp`.
+
+## Configuration parameters
+
+The configuration of this codec matches the compression modes defined at <https://zfp.readthedocs.io/en/latest/modes.html#expert-mode>.
+Refer to that page for usage information.
+
+The codec has one parameter which is always required:
+- `mode` (string).
+
+The other required parameters are dependent on the `mode`:
+- `"mode": "reversible"`
+- `"mode": "expert"`
+    - `minbits` (unsigned integer)
+    - `maxbits` (unsigned integer)
+    - `maxprec` (unsigned integer)
+    - `minexp` (signed integer)
+- `"mode": "fixed_accuracy"`
+    - `tolerance` (number)
+- `"mode": "fixed_rate"`
+    - `rate` (number)
+- `"fixed_precision"`
+    - `precision` (unsigned integer)
+
+## Example
+
+For example, the array metadata below specifies that the array contains zfp compressed chunks using `fixed_accuracy` mode with a tolerance of 0.05:
+
+```json
+{
+    "codecs": [{
+        "name": "zarrs.zfp",
+        "configuration": {
+            "mode": "fixed_accuracy",
+            "tolerance": 0.05
+        }
+    }],
+}
+```
+
+More examples can be viewed in the [examples](./examples/) subdirectory.
+
+## Supported Data Types
+
+- `int32`, `uint32`, `int64`, `uint64`, `float32`, `float64`
+- `int8`, `uint8`, `int16`, `uint16` (through internal promotion 32-bit)
+
+Implementations are permitted to support additional data types that could be interpreted as the above data types (e.g. `datetime64` -> `uint64`).
path: codecs/zarrs.zfp/README.md
user: login: jbms
body: I think datetime64 would correspond to int64 not uint64.
created_at: 2025-04-18T16:27:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+# ZFP codec
+
+Defines a `array -> bytes` codec that compresses chunks using the [zfp](https://github.com/LLNL/zfp) algorithm.
+
path: codecs/zarrs.zfp/README.md
user: login: jbms
body: zfp natively only supports 1, 2, 3 and 4 dimensional arrays.  However, for zarr it might be useful to allow arrays with more than 4 dimensions provided that at most 4 dimensions have a chunk size greater than 1.  That would allow zfp to be combined with sharding_indexed to handle arbitrary numbers of dimensions.  You would just need to define exactly how the zfp dimensions are determined from the zarr dimensions. (See also https://github.com/seung-lab/zfpc which is supported by the neuroglancer-precomputed format --- with zarr I think an extra container format is unnecessary because sharding_indexed already serves that purpose quite well)
created_at: 2025-04-18T16:37:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+# ZFP codec
+
+Defines a `array -> bytes` codec that compresses chunks using the [zfp](https://github.com/LLNL/zfp) algorithm.
+
path: codecs/zarrs.zfp/README.md
user: login: jbms
body: My suggestion for defining that would be to say: if the number of dimensions with chunk size > 1 is within the range `[1, 4]`, those are the zfp dimensions.    If there are more than 4 such dimensions, it is an error.  If there are zero such dimensions, i.e. the chunk contains just a single element, treat it as a 1-d array of 1 element (so that users don't have to worry about this as a special case).
created_at: 2025-04-18T16:42:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+# ZFP codec
+
+Defines a `array -> bytes` codec that compresses chunks using the [zfp](https://github.com/LLNL/zfp) algorithm.
+
+## Codec name
+
+The value of the `name` member in the codec object MUST be `zarrs.zfp`.
+
+## Configuration parameters
+
+The configuration of this codec matches the compression modes defined at <https://zfp.readthedocs.io/en/latest/modes.html#expert-mode>.
+Refer to that page for usage information.
+
+The codec has one parameter which is always required:
+- `mode` (string).
+
+The other required parameters are dependent on the `mode`:
+- `"mode": "reversible"`
+- `"mode": "expert"`
+    - `minbits` (unsigned integer)
+    - `maxbits` (unsigned integer)
+    - `maxprec` (unsigned integer)
+    - `minexp` (signed integer)
+- `"mode": "fixed_accuracy"`
+    - `tolerance` (number)
+- `"mode": "fixed_rate"`
+    - `rate` (number)
+- `"fixed_precision"`
+    - `precision` (unsigned integer)
+
+## Example
+
+For example, the array metadata below specifies that the array contains zfp compressed chunks using `fixed_accuracy` mode with a tolerance of 0.05:
+
+```json
+{
+    "codecs": [{
+        "name": "zarrs.zfp",
+        "configuration": {
+            "mode": "fixed_accuracy",
+            "tolerance": 0.05
+        }
+    }],
+}
+```
+
+More examples can be viewed in the [examples](./examples/) subdirectory.
+
+## Supported Data Types
+
+- `int32`, `uint32`, `int64`, `uint64`, `float32`, `float64`
+- `int8`, `uint8`, `int16`, `uint16` (through internal promotion 32-bit)
+
+Implementations are permitted to support additional data types that could be interpreted as the above data types (e.g. `datetime64` -> `uint64`).
path: codecs/zarrs.zfp/README.md
user: login: jbms
body: You could note that lower-precision floating point types, such as float16 and the new float-precision floating point extension types I added, can be supported by first converting to float32 in the normal way.

Lower-precision integer types, e.g. {int,uint}{2,4} can also be supported but would need to be promoted in the same way that {int,uint}{8,16} are promoted:

for intN:

```c++
int32_value = (int32_t)intN_value << (31 - N)
intN_value = (intN_t)clamp(int32_value >> (31 - N), 1<<(N-1), (1<<(N-1)) - 1)
```

for uintN

```c++
int32_value = ((int32_t)uintN_value - (1<<(N-1))) << (31 - N)
uintN_value = (uintN_t)clamp((int32_value >> (31 - N)) + (1<<(N-1)), 0, (1<<N) - 1)
```

created_at: 2025-04-18T17:00:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+# ZFP codec
+
+Defines a `array -> bytes` codec that compresses chunks using the [zfp](https://github.com/LLNL/zfp) algorithm.
+
path: codecs/zarrs.zfp/README.md
user: login: LDeakin
body: Great catch, and a good idea in regards to supporting higher dimensional arrays. I'll add support for that in `zarrs`.

I wasn't aware of https://github.com/seung-lab/zfpc, but it highlights the reason why I didn't include complex numbers for this codec. A Zarr V3 way to do that could be an array-to-bytes codec that deinterleaves complex data types and encodes them independently.
created_at: 2025-04-18T21:21:58Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,88 @@
+# ZFP codec
+
+Defines a `array -> bytes` codec that compresses chunks using the [zfp](https://github.com/LLNL/zfp) algorithm.
+
path: codecs/zarrs.zfp/README.md
user: login: jbms
body: Regarding your "may support" change:  what I described (and what you have) is a breaking change, because previously if you had a chunk size of [64, 64, 1, 64] it would get encoded as a 4-d zfp (effectively padded to [64, 64, 4, 64]) --- I'm not sure exactly what the implications are in terms of runtime performance and compression ratio, but it seems plausible that it may make compression and decompression 4x as slow.  Under the new language, it would instead get encoded as a 3-d zfp with chunk shape `[64, 64, 64]`.  Without the header I don't know that this can be detected automatically.  For zarrs, assuming backwards compatibility is important, perhaps you can support the old behavior under the old identifier that you used, and the new behavior under the plain `zfp` identifier. 
created_at: 2025-04-18T21:55:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+# ZFP codec
+
+Defines a `array -> bytes` codec that compresses chunks using the [zfp](https://github.com/LLNL/zfp) algorithm.
+
path: codecs/zarrs.zfp/README.md
user: login: LDeakin
body: In that case, always maintaining the chunk shape for <=4D data and only collapsing dimensions for 5D+ data might be a good option. I'll trial this in implementation, and come back to this PR afterwards.
created_at: 2025-04-18T22:12:51Z
author_association: MEMBER

PR comments:
user: login: normanrz
created_at: 2025-04-18T11:03:08Z
body: Thanks! This PR looks very good.
I don't see why this codec couldn't be registered as `zfp`. 
author_association: MEMBER
user: login: jbms
created_at: 2025-04-19T05:31:53Z
body: At least based on my understanding, having a chunk size of 1 for one of the zfp dimensions is never actually desirable and you are retaining that solely for backwards compatibility.  Having different behavior for >4 dimensions where chunk size=1 dimensions get dropped may be rather confusing.

An alternative approach would be to delegate the dimension adjustments to a separate "array -> array" codec, perhaps called "squeeze" in line with `np.squeeze`, that allows dimensions of chunk size 1 to be dropped.   Dimension manipulations of various sorts will surely be needed to support various image formats as well, and doing that with common "array -> array" codecs rather than ad-hoc options as part of the "array -> bytes" codec may be better.
author_association: CONTRIBUTOR
user: login: LDeakin
created_at: 2025-04-19T06:26:41Z
body: > At least based on my understanding, having a chunk size of 1 for one of the zfp dimensions is never actually desirable

For sure, having just read through the algorithm.

> An alternative approach would be to delegate the dimension adjustments to a separate "array -> array" codec, perhaps called "squeeze" in line with `np.squeeze`, that allows dimensions of chunk size 1 to be dropped

I like that! If you are onboard with it, I will remove mention of collapsing dimensions in this codec, but keep a note about how higher dimensionality arrays could be supported with a `squeeze`-like codec.
author_association: MEMBER
user: login: LDeakin
created_at: 2025-04-19T10:31:18Z
body: I added a `squeeze` codec to `zarrs` and updated this spec accordingly. I like this approach best for supporting higher-dimensional arrays. Thanks for all your comments @jbms.
author_association: MEMBER
user: login: normanrz
created_at: 2025-04-23T08:52:53Z
body: Thanks for working on this! I'll merge this tomorrow unless I hear from you that you still want to iterate pre-merge.
author_association: MEMBER
user: login: normanrz
created_at: 2025-04-24T15:05:07Z
body: Right after merging, I noticed that the `schema.json` and folder name still referenced `zarrs.zfp`. I changed that in https://github.com/zarr-developers/zarr-extensions/commit/77c6d6ce498754d3acdfdaf9901bd88366421861. I hope that is what you intended.
author_association: MEMBER