Repository: zarr-developers/zarr-python

Issue:
Will V3 compressor codecs require all parameters in array metadata?
### Zarr version

v2.14.3

### Numcodecs version

v0.11.0

### Python Version

3.10

### Operating System

Mac

### Installation

pip

### Description

The current V3 implementation seems to require a more specific set of parameters for codecs/compressors than V2 did. The code sample below demonstrates how all configuration parameters are extracted separately:

https://github.com/zarr-developers/zarr-python/blob/b3cafa96b88c362e7843df8e2a07499588b7fa49/zarr/meta.py#L443-L460

Whereas in V2, the compressor parameters were simply passed on to [`numcodecs.get_config`](https://github.com/zarr-developers/numcodecs/blob/4f194b661faf1f945cb21b1ac1f19c4314ff8f05/numcodecs/registry.py#L51C1-L52C1)

https://github.com/zarr-developers/zarr-python/blob/b3cafa96b88c362e7843df8e2a07499588b7fa49/zarr/meta.py#L169-L171

## Question:
Was this an intentional change? If so, I foresee a number of downstream projects that will need to adjust their expectations (e.g. Kerchunk, Tifffile). 

### Steps to reproduce

I ran into this working on a V2->V3 conversion tool for Kerchunk. A minimal example is not available at this time.

xref: https://github.com/cgohlke/tifffile/issues/211

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/zarr-developers/zarr-python/pull/1447?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zarr-developers) Report
> Merging [#1447](https://app.codecov.io/gh/zarr-developers/zarr-python/pull/1447?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zarr-developers) (0f1c596) into [main](https://app.codecov.io/gh/zarr-developers/zarr-python/commit/327b6942284283d0435d2513514f4712c71376b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zarr-developers) (327b694) will **not change** coverage.
> The diff coverage is `100.00%`.

```diff
@@            Coverage Diff            @@
##              main     #1447   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           37        37           
  Lines        14866     14863    -3     
=========================================
- Hits         14866     14863    -3     
```

| [Impacted Files](https://app.codecov.io/gh/zarr-developers/zarr-python/pull/1447?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zarr-developers) | Coverage Δ | |
|---|---|---|
| [zarr/meta.py](https://app.codecov.io/gh/zarr-developers/zarr-python/pull/1447?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zarr-developers#diff-emFyci9tZXRhLnB5) | `100.00% <100.00%> (ø)` | |

... and [3 files with indirect coverage changes](https://app.codecov.io/gh/zarr-developers/zarr-python/pull/1447/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zarr-developers)

jhamman: Test failures are unrelated and are fixed here https://github.com/zarr-developers/zarr-python/pull/1450
jhamman: @zarr-developers/python-core-devs - looking for some final feedback here. If I don't hear any, I'll plan to merge ~7/12. 
martindurant: LGTM. I suppose if the conf has invalid values, you get an exception when the codec is instantiated -  do we have any expectation/test of how that should go?

For the related issue in TIFFfile, I will comment that many codec options, specifically for compressors, are only actually important at compress time, not when reading. It would be useful to know, though, if the defaults in python implementations are compatible with whatever TIFF assumes - I don't know how to do that rigorously.
cgohlke: > It would be useful to know, though, if the defaults in python implementations are compatible with whatever TIFF assumes - I don't know how to do that rigorously.

The only compression method specified in the TIFF standard that has a codec in the numcodecs package is "deflate". The spec says that "each image segment consists of a single complete zlib data stream.". This is compatible what numcodecs.Zlib handles.

All the other dozens of compression methods found in TIFF are either nowhere specified specifically for TIFF (e.g. LZMA, Zstd) , not supported by numcodecs (PackBits, LZW, CCITT, JPEG), or both (e.g., JPEG2000, WebP, PNG, JetRaw).

The imagecodecs package provides many of those codecs made for TIFF. Few of them might require setting extra arguments, such as JPEG (`jpegtables`), WebP (`hasalpha`), or JetRaw (`shape`). This is handled through tifffile. If not, open an issue there.

rabernat: @jhamman - we just need release notes for this. That's important, since this change could hypothetically impact existing code.
martindurant: @cgohlke , thank you for the thorough reply. It seems we have nothing to worry about here.

Patch:
diff --git a/docs/release.rst b/docs/release.rst
index e8d1c440..46bd1f02 100644
--- a/docs/release.rst
+++ b/docs/release.rst
@@ -18,10 +18,14 @@ Release notes
 Unreleased
 ----------
 
-Bug fixes
-~~~~~~~~~
+Enhancements
+~~~~~~~~~~~~
+
+* Allow for partial codec specification in V3 array metadata.
+  By :user:`Joe Hamman <jhamman>` :issue:`1443`.
 
-* Add ``__contains__`` method to ``KVStore``. By :user:`Christoph Gohlke <cgohlke>` :issue:`1454`.
+* Add ``__contains__`` method to ``KVStore``. 
+  By :user:`Christoph Gohlke <cgohlke>` :issue:`1454`.
 
     * **Block Indexing**: Implemented blockwise (chunk blocks) indexing to ``zarr.Array``.
       By :user:`Altay Sansal <tasansal>` :issue:`1428`
diff --git a/zarr/meta.py b/zarr/meta.py
index 59c56abf..aacffd7f 100644
--- a/zarr/meta.py
+++ b/zarr/meta.py
@@ -441,26 +441,22 @@ class Metadata3(Metadata2):
         uri = 'https://purl.org/zarr/spec/codec/'
         conf = meta['configuration']
         if meta['codec'].startswith(uri + 'gzip/'):
-            codec = numcodecs.GZip(level=conf['level'])
+            conf["id"] = "gzip"
         elif meta['codec'].startswith(uri + 'zlib/'):
-            codec = numcodecs.Zlib(level=conf['level'])
+            conf["id"] = "zlib"
         elif meta['codec'].startswith(uri + 'blosc/'):
-            codec = numcodecs.Blosc(clevel=conf['clevel'],
-                                    shuffle=conf['shuffle'],
-                                    blocksize=conf['blocksize'],
-                                    cname=conf['cname'])
+            conf["id"] = "blosc"
         elif meta['codec'].startswith(uri + 'bz2/'):
-            codec = numcodecs.BZ2(level=conf['level'])
+            conf["id"] = "bz2"
         elif meta['codec'].startswith(uri + 'lz4/'):
-            codec = numcodecs.LZ4(acceleration=conf['acceleration'])
+            conf["id"] = "lz4"
         elif meta['codec'].startswith(uri + 'lzma/'):
-            codec = numcodecs.LZMA(format=conf['format'],
-                                   check=conf['check'],
-                                   preset=conf['preset'],
-                                   filters=conf['filters'])
+            conf["id"] = "lzma"
         else:
             raise NotImplementedError
 
+        codec = numcodecs.get_codec(conf)
+
         return codec
 
     @classmethod


Test patch:
diff --git a/zarr/tests/test_meta.py b/zarr/tests/test_meta.py
index 8acd634a..a7837598 100644
--- a/zarr/tests/test_meta.py
+++ b/zarr/tests/test_meta.py
@@ -314,6 +314,34 @@ def test_encode_decode_array_dtype_shape_v3():
     assert 'filters' not in meta_dec
 
 
+@pytest.mark.parametrize("comp_id", ["gzip", "zlib", "blosc", "bz2", "lz4", "lzma"])
+def test_decode_metadata_implicit_compressor_config_v3(comp_id):
+    meta = {
+        "attributes": {},
+        "chunk_grid": {
+            "chunk_shape": [10],
+            "separator": "/",
+            "type": "regular"
+        },
+        "chunk_memory_layout": "C",
+        "compressor": {
+            "codec": f"https://purl.org/zarr/spec/codec/{comp_id}/1.0",
+            "configuration": {
+                # intentionally left empty
+            }
+        },
+        "data_type": "<f8",
+        "extensions": [],
+        "fill_value": None,
+        "shape": [100, 10, 10]
+    }
+    meta_json = json.dumps(meta)
+
+    # test decoding
+    meta_dec = Metadata3.decode_array_metadata(meta_json)
+    assert meta_dec['compressor'].codec_id == comp_id
+
+
 def test_encode_decode_array_structured():
 
     meta = dict(