Repository: pydata/xarray

Issue:
# Saving to zarr-store using shards changes underlying data

### What happened?

Hey! 
I tried saving a dataset (CMIP6 derived data) with very small chunks (dask array) to a zarr store using shards to limit the number of files used in the store. But, I noticed that computation based on the sharded store returned different results compared to the results from a store with the same data, but no shards.

### What did you expect to happen?

I'm expecting the store using shards to represent the same data as the store not using shards, and any following computations would be identical.

### Minimal Complete Verifiable Example

```Python
import xarray as xr
import dask.array as da
from dask.distributed import Client

client = Client()
rng = da.random.default_rng(seed=42)

# Generate some moderately size data to save to disk.
# Chunksize should be (255, 255, 255)
test_data = rng.integers(0, 2, size=(10000, 300, 300))

# Put in DataArray.
test_data = xr.DataArray(test_data)
test_data.name = "test_data"

#Setup encoding for shards.
encoding = {
    "test_data": {
        "shards": (255*2, 255, 255),
    }
}

# Save to disk.
test_data.to_zarr("test_random.zarr", zarr_format=3, mode="w")
test_data.to_zarr("test_random_shard.zarr", zarr_format=3, encoding=encoding, mode="w")

# Read from disk
test_data = xr.open_zarr("test_random.zarr").test_data
test_data_shard = xr.open_zarr("test_random_shard.zarr").test_data


#Expect this to return true, but it doesn't.
assert test_data.sum().compute().values == test_data_shard.sum().compute().values
```

### Steps to reproduce

Note that setting shards to `(255*1, 255, 255) for the encoding above makes the assert return true.

### MVCE confirmation

- [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- [x] Complete example — the example is self-contained, including all data and the text of any traceback.
- [x] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result.
- [x] New issue — a search of GitHub Issues suggests this is not a duplicate.
- [x] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

### Relevant log output

```Python

```

### Anything else we need to know?

Maybe this a zarr issue. Happy to raise there instead if that is the case.

### Environment

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.12.11 | packaged by conda-forge | (main, Jun  4 2025, 14:45:31) [GCC 13.3.0]
python-bits: 64
OS: Linux
OS-release: 6.17.0-2-default
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.6
libnetcdf: 4.9.3

xarray: 2025.10.1
pandas: 2.3.2
numpy: 2.2.6
scipy: 1.16.1
netCDF4: 1.7.2
pydap: None
h5netcdf: None
h5py: None
zarr: 3.1.3
cftime: 1.6.4
nc_time_axis: 1.4.1
iris: None
bottleneck: 1.5.0
dask: 2025.9.1
distributed: 2025.9.1
matplotlib: 3.10.6
cartopy: 0.24.0
seaborn: None
numbagg: 0.9.2
fsspec: 2025.9.0
cupy: None
pint: None
sparse: 0.17.0
flox: 0.10.6
numpy_groupies: 0.11.3
setuptools: 80.9.0
pip: None
conda: None
pytest: None
mypy: None
IPython: 9.5.0
sphinx: None


</details>


Implementation patch:
diff --git a/xarray/backends/zarr.py b/xarray/backends/zarr.py
index af2aad34d5e..1eadace1d0e 100644
--- a/xarray/backends/zarr.py
+++ b/xarray/backends/zarr.py
@@ -1240,16 +1240,22 @@ def set_variables(
                 zarr_format=3 if is_zarr_v3_format else 2,
             )
 
-            if self._align_chunks and isinstance(encoding["chunks"], tuple):
+            # When shards are specified, dask chunks must align with shard boundaries
+            # (not just zarr chunk boundaries) to avoid data corruption during
+            # parallel writes. See https://github.com/pydata/xarray/issues/10831
+            effective_write_chunks = encoding.get("shards") or encoding["chunks"]
+
+            if self._align_chunks and isinstance(effective_write_chunks, tuple):
                 v = grid_rechunk(
                     v=v,
-                    enc_chunks=encoding["chunks"],
+                    enc_chunks=effective_write_chunks,
                     region=region,
                 )
 
-            if self._safe_chunks and isinstance(encoding["chunks"], tuple):
+            if self._safe_chunks and isinstance(effective_write_chunks, tuple):
                 # the hard case
                 # DESIGN CHOICE: do not allow multiple dask chunks on a single zarr chunk
+                # (or shard, when sharding is enabled)
                 # this avoids the need to get involved in zarr synchronization / locking
                 # From zarr docs:
                 #  "If each worker in a parallel computation is writing to a
@@ -1260,7 +1266,7 @@ def set_variables(
                 shape = zarr_shape or v.shape
                 validate_grid_chunks_alignment(
                     nd_v_chunks=v.chunks,
-                    enc_chunks=encoding["chunks"],
+                    enc_chunks=effective_write_chunks,
                     region=region,
                     allow_partial_chunks=self._mode != "r+",
                     name=name,


Test patch:
diff --git a/xarray/tests/test_backends.py b/xarray/tests/test_backends.py
index 3d694d7791a..d4a39cfcf73 100644
--- a/xarray/tests/test_backends.py
+++ b/xarray/tests/test_backends.py
@@ -2869,6 +2869,41 @@ def test_shard_encoding(self) -> None:
                 with self.roundtrip(data) as actual:
                     pass
 
+    @requires_dask
+    def test_shard_encoding_with_dask(self) -> None:
+        # Test that dask chunks must align with shard boundaries.
+        # See https://github.com/pydata/xarray/issues/10831
+        if not (has_zarr_v3 and zarr.config.config["default_zarr_format"] == 3):
+            pytest.skip("sharding requires zarr v3 format")
+
+        ds = xr.DataArray(np.arange(12), dims="x", name="var1").to_dataset()
+
+        # Case 1: Dask chunks equal to shards should work
+        # (zarr chunk=3, shard=6, dask chunk=6)
+        ds1 = ds.chunk({"x": 6})
+        ds1["var1"].encoding = {"chunks": (3,), "shards": (6,)}
+        with self.roundtrip(ds1) as actual:
+            assert_identical(ds, actual)
+
+        # Case 2: Dask chunks that are multiples of shards should work
+        # (zarr chunk=1, shard=3, dask chunk=6)
+        ds2 = ds.chunk({"x": 6})
+        ds2["var1"].encoding = {"chunks": (1,), "shards": (3,)}
+        with self.roundtrip(ds2) as actual:
+            assert_identical(ds, actual)
+
+        # Case 3: Dask chunks smaller than shards should fail
+        # (zarr chunk=2, shard=4, dask chunk=3) - dask chunk doesn't align with shard
+        ds3 = ds.chunk({"x": 3})
+        ds3["var1"].encoding = {"chunks": (2,), "shards": (4,)}
+        with pytest.raises(ValueError, match=r"would overlap"):
+            with self.roundtrip(ds3) as actual:
+                pass
+
+        # Case 4: Can bypass with safe_chunks=False (but data may be corrupted)
+        with self.roundtrip(ds3, save_kwargs={"safe_chunks": False}) as actual:
+            pass
+
     @requires_dask
     @pytest.mark.skipif(
         ON_WINDOWS,