Repository: zarr-developers/zarr-python

Issue:
Cannot read attributes that contain non-ASCII characters
### Zarr version

2.13.4.dev68

### Numcodecs version

0.11.0

### Python Version

3.10.6

### Operating System

Linux

### Installation

With pip, using the instructions here https://zarr.readthedocs.io/en/stable/contributing.html

### Description

I expect zarr to be able to read attribute files that contain non-ASCII characters, because JSON files use utf-8 encoding. However, currently, there is a check to error if the JSON file contains any non-ASCII characters.



### Steps to reproduce

```python
import zarr
import tempfile
tempdir = tempfile.mkdtemp()
f = open(tempdir + '/.zgroup','w', encoding='utf-8')
f.write('{"zarr_format":2}')
f.close()
f = open(tempdir + '/.zattrs','w', encoding='utf-8')
f.write('{"foo": "た"}')
f.close()
z = zarr.open(tempdir, mode='r')
z.attrs['foo']
```

```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/github/zarr-python/zarr/attrs.py", line 74, in __getitem__
    return self.asdict()[item]
  File "/root/github/zarr-python/zarr/attrs.py", line 55, in asdict
    d = self._get_nosync()
  File "/root/github/zarr-python/zarr/attrs.py", line 48, in _get_nosync
    d = self.store._metadata_class.parse_metadata(data)
  File "/root/github/zarr-python/zarr/meta.py", line 104, in parse_metadata
    meta = json_loads(s)
  File "/root/github/zarr-python/zarr/util.py", line 56, in json_loads
    return json.loads(ensure_text(s, 'ascii'))
  File "/root/pyenv/zarr-dev/lib/python3.10/site-packages/numcodecs/compat.py", line 181, in ensure_text
    s = codecs.decode(s, encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 9: ordinal not in range(128)
```

### Additional output

_No response_

Discussion:
nhz2: @MSanKeys963 @jakirkham  This PR is ready for review. Also, because I am a first-time contributor, I am not able to trigger Github Actions myself.
MSanKeys963: Thanks for sending the PR @nhz2.
@jakirkham, @joshmoore and @rabernat, please have a look.
joshmoore: Thanks for this, @nhz2! I've triggered the actions and will go through the code ASAP.
joshmoore: Green :tada: in case anyone else wants to now take a closer look. (`release.rst` will need its conflict handled)
nhz2: This PR should have no effect on the JSON files zarr-python writes, they will still be ASCII only with non-ASCII characters escaped. It only affects what kind of JSON files zarr-python can read.
joshmoore: Thanks, @nhz2. Guess I was more concerned whether any calling code (and/or subclasses) would need modification.
joshmoore: Thanks for the analysis, @nhz2! Sounds like it should not impact external consumers.

Any objections to rolling this into 2.14, anyone?
jakirkham: Thanks for the PR Nathan! 🙏

This is a good improvement to include.

Noting some changes in the typing to still allow `str` in functions (since it should still technically be valid). Though agree `bytes` should be included in a few places as well for clarity

Ideally we would capture any `bytes`-like type here (since they are all valid), but that is somewhat complicated to do in practice. Maybe there will be guidance on this in the future ( https://github.com/python/typing/issues/593 )

Patch:
diff --git a/docs/release.rst b/docs/release.rst
index fdff4002..905ccd2e 100644
--- a/docs/release.rst
+++ b/docs/release.rst
@@ -31,6 +31,9 @@ Major changes
 Bug fixes
 ~~~~~~~~~
 
+* Allow reading utf-8 encoded json files
+  By :user:`Nathan Zimmerberg <nhz2>` :issue:`1308`.
+
 * Ensure contiguous data is give to ``FSStore``. Only copying if needed.
   By :user:`Mads R. B. Kristensen <madsbk>` :issue:`1285`.
 * NestedDirectoryStore.listdir now returns chunk keys with the correct '/' dimension_separator.
diff --git a/fixture/utf8attrs/.zattrs b/fixture/utf8attrs/.zattrs
new file mode 100644
index 00000000..7f85af5d
--- /dev/null
+++ b/fixture/utf8attrs/.zattrs
@@ -0,0 +1,1 @@
+{"foo": "た"}
\ No newline at end of file
diff --git a/fixture/utf8attrs/.zgroup b/fixture/utf8attrs/.zgroup
new file mode 100644
index 00000000..3b7daf22
--- /dev/null
+++ b/fixture/utf8attrs/.zgroup
@@ -0,0 +1,3 @@
+{
+    "zarr_format": 2
+}
\ No newline at end of file
diff --git a/zarr/meta.py b/zarr/meta.py
index b493e833..59c56abf 100644
--- a/zarr/meta.py
+++ b/zarr/meta.py
@@ -92,7 +92,7 @@ class Metadata2:
     ZARR_FORMAT = ZARR_FORMAT
 
     @classmethod
-    def parse_metadata(cls, s: Union[MappingType, str]) -> MappingType[str, Any]:
+    def parse_metadata(cls, s: Union[MappingType, bytes, str]) -> MappingType[str, Any]:
 
         # Here we allow that a store may return an already-parsed metadata object,
         # or a string of JSON that we will parse here. We allow for an already-parsed
@@ -110,7 +110,7 @@ class Metadata2:
         return meta
 
     @classmethod
-    def decode_array_metadata(cls, s: Union[MappingType, str]) -> MappingType[str, Any]:
+    def decode_array_metadata(cls, s: Union[MappingType, bytes, str]) -> MappingType[str, Any]:
         meta = cls.parse_metadata(s)
 
         # check metadata format
@@ -198,7 +198,7 @@ class Metadata2:
         return np.dtype(d)
 
     @classmethod
-    def decode_group_metadata(cls, s: Union[MappingType, str]) -> MappingType[str, Any]:
+    def decode_group_metadata(cls, s: Union[MappingType, bytes, str]) -> MappingType[str, Any]:
         meta = cls.parse_metadata(s)
 
         # check metadata format version
@@ -351,7 +351,7 @@ class Metadata3(Metadata2):
             return get_extended_dtype_info(np.dtype(d))
 
     @classmethod
-    def decode_group_metadata(cls, s: Union[MappingType, str]) -> MappingType[str, Any]:
+    def decode_group_metadata(cls, s: Union[MappingType, bytes, str]) -> MappingType[str, Any]:
         meta = cls.parse_metadata(s)
         # 1 / 0
         # # check metadata format version
@@ -390,7 +390,7 @@ class Metadata3(Metadata2):
 
     @classmethod
     def decode_hierarchy_metadata(
-        cls, s: Union[MappingType, str]
+        cls, s: Union[MappingType, bytes, str]
     ) -> MappingType[str, Any]:
         meta = cls.parse_metadata(s)
         # check metadata format
@@ -495,7 +495,7 @@ class Metadata3(Metadata2):
         return StorageTransformerCls.from_config(transformer_type, conf)
 
     @classmethod
-    def decode_array_metadata(cls, s: Union[MappingType, str]) -> MappingType[str, Any]:
+    def decode_array_metadata(cls, s: Union[MappingType, bytes, str]) -> MappingType[str, Any]:
         meta = cls.parse_metadata(s)
 
         # extract array metadata fields
diff --git a/zarr/util.py b/zarr/util.py
index 5b307b7c..be5f174a 100644
--- a/zarr/util.py
+++ b/zarr/util.py
@@ -56,9 +56,9 @@ def json_dumps(o: Any) -> bytes:
                       separators=(',', ': '), cls=NumberEncoder).encode('ascii')
 
 
-def json_loads(s: str) -> Dict[str, Any]:
+def json_loads(s: Union[bytes, str]) -> Dict[str, Any]:
     """Read JSON in a consistent way."""
-    return json.loads(ensure_text(s, 'ascii'))
+    return json.loads(ensure_text(s, 'utf-8'))
 
 
 def normalize_shape(shape) -> Tuple[int]:


Test patch:
diff --git a/zarr/tests/test_attrs.py b/zarr/tests/test_attrs.py
index e4baf182..27ec8fea 100644
--- a/zarr/tests/test_attrs.py
+++ b/zarr/tests/test_attrs.py
@@ -4,9 +4,10 @@ import pytest
 
 from zarr._storage.store import meta_root
 from zarr.attrs import Attributes
-from zarr.storage import KVStore
+from zarr.storage import KVStore, DirectoryStore
 from zarr._storage.v3 import KVStoreV3
 from zarr.tests.util import CountingDict, CountingDictV3
+from zarr.hierarchy import group
 
 
 @pytest.fixture(params=[2, 3])
@@ -42,11 +43,17 @@ class TestAttributes():
         a['baz'] = 42
         assert attrs_key in store
         assert isinstance(store[attrs_key], bytes)
-        d = json.loads(str(store[attrs_key], 'ascii'))
+        d = json.loads(str(store[attrs_key], 'utf-8'))
         if zarr_version == 3:
             d = d['attributes']
         assert dict(foo='bar', baz=42) == d
 
+    def test_utf8_encoding(self, zarr_version):
+
+        # fixture data
+        fixture = group(store=DirectoryStore('fixture'))
+        assert fixture['utf8attrs'].attrs.asdict() == dict(foo='た')
+
     def test_get_set_del_contains(self, zarr_version):
 
         store = _init_store(zarr_version)