Repository: dask/dask

Issue:
Pyarrow string bug
```python
# Running in distributed cluster
data = pd.util.testing.makeCustomDataframe(nrows=1000, ncols=10).reset_index()
data[["R0", "C_l0_g0", "C_l0_g1", "C_l0_g2", "C_l0_g3", "C_l0_g4", "C_l0_g5" ,"C_l0_g6", "C_l0_g7", "C_l0_g8", "C_l0_g9"]] = data[["R0", "C_l0_g0", "C_l0_g1", "C_l0_g2", "C_l0_g3", "C_l0_g4", "C_l0_g5" ,"C_l0_g6", "C_l0_g7", "C_l0_g8", "C_l0_g9"]].astype("string[pyarrow]")
data = dd.from_pandas(data, npartitions=128)
data = data.persist()
data.groupby(["R0", "C_l0_g0", "C_l0_g1", "C_l0_g2", "C_l0_g3", "C_l0_g4", "C_l0_g5" ,"C_l0_g6"]).apply(lambda d: pd.DataFrame([sum(d["C_l0_g9"].str.len())], columns=["length"]), meta={"length": "int32"}).compute()
```

**What happened**:
In the example above, an error that seems to relate to pyarrow string happened. It kills the workers, and scheduler, and the whole python process exits.
```
pyarrow.lib.ArrowInvalid: Buffer #0 too small in array of type string and length 3: expected at least 2 byte(s), got 1
```
Notice the `nrows` argument. If you change that to a small number of rows, such as 300, then it is okay. If the nrows is about 1000, it starts to break.

**What you expected to happen**:
The computation should happen successfully. If run without a distributed cluster, this will complete successfully, and the output is
<img width="649" alt="image" src="https://user-images.githubusercontent.com/3459416/172172645-68759936-532d-430d-b159-20c1759dc43d.png">

**Environment**:

- Dask version: 2022.1.1, 2022.5.1, 2022.5.2
- Pandas version: 1.3.5
- Pyarrow version: 7.0.0, 8.0.0
- Python version: 3.8
- Operating System: MacOS, Ubuntu 22.04
- Install method (conda, pip, source): pip in conda environment


<details>
<summary>Cluster Dump State:</summary>
Workers are restarted
</details>


Discussion:
Thanks for reporting @nlhkh -- I can confirm this is a bug. Further, running your example on a few different commits, it looks like https://github.com/dask/dask/pull/9024 is where this serialization error was introduced. For context, here's the full traceback

```python
2022-06-06 14:10:37,614 - distributed.worker - ERROR - Buffer #0 too small in array of type string and length 2: expected at least 2 byte(s), got 1
Traceback (most recent call last):
  File "/Users/james/projects/dask/distributed/distributed/worker.py", line 3331, in gather_dep
    response = await get_data_from_worker(
  File "/Users/james/projects/dask/distributed/distributed/worker.py", line 4661, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 381, in retry_operation
    return await retry(
  File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 366, in retry
    return await coro()
  File "/Users/james/projects/dask/distributed/distributed/worker.py", line 4641, in _get_data
    response = await send_recv(
  File "/Users/james/projects/dask/distributed/distributed/core.py", line 742, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 254, in read
    msg = await from_frames(
  File "/Users/james/projects/dask/distributed/distributed/comm/utils.py", line 98, in from_frames
    res = _from_frames()
  File "/Users/james/projects/dask/distributed/distributed/comm/utils.py", line 81, in _from_frames
    return protocol.loads(
  File "/Users/james/projects/dask/distributed/distributed/protocol/core.py", line 159, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/Users/james/projects/dask/distributed/distributed/protocol/core.py", line 139, in _decode_default
    return merge_and_deserialize(
  File "/Users/james/projects/dask/distributed/distributed/protocol/serialize.py", line 487, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/Users/james/projects/dask/distributed/distributed/protocol/serialize.py", line 416, in deserialize
    return loads(header, frames)
  File "/Users/james/projects/dask/distributed/distributed/protocol/serialize.py", line 95, in pickle_loads
    return pickle.loads(x, buffers=buffers)
  File "/Users/james/projects/dask/distributed/distributed/protocol/pickle.py", line 64, in loads
    return pickle.loads(x, buffers=buffers)
  File "/Users/james/projects/dask/dask/dask/dataframe/_pyarrow_compat.py", line 122, in rebuild_arrowstringarray
    [pyarrow_stringarray_from_parts(*parts) for parts in chunk_parts],
  File "/Users/james/projects/dask/dask/dask/dataframe/_pyarrow_compat.py", line 122, in <listcomp>
    [pyarrow_stringarray_from_parts(*parts) for parts in chunk_parts],
  File "/Users/james/projects/dask/dask/dask/dataframe/_pyarrow_compat.py", line 116, in pyarrow_stringarray_from_parts
    return pa.StringArray.from_buffers(nitems, data_offsets, data, mask, offset=offset)
  File "pyarrow/array.pxi", line 2365, in pyarrow.lib.StringArray.from_buffers
  File "pyarrow/array.pxi", line 1099, in pyarrow.lib.Array.from_buffers
  File "pyarrow/array.pxi", line 1493, in pyarrow.lib.Array.validate
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Buffer #0 too small in array of type string and length 2: expected at least 2 byte(s), got 1
```

cc @jcrist @ian-r-rose @jorisvandenbossche for visibility 
Thanks @jrbourbeau 

Resolving patch:
diff --git a/dask/dataframe/_pyarrow_compat.py b/dask/dataframe/_pyarrow_compat.py
--- a/dask/dataframe/_pyarrow_compat.py
+++ b/dask/dataframe/_pyarrow_compat.py
@@ -86,7 +86,7 @@ def pyarrow_stringarray_to_parts(array):
         # elements so the mask array can always be serialized zero copy.
         npad = array.offset % 8
         mask_start = array.offset // 8
-        mask_stop = mask_start + math.ceil(nitems / 8)
+        mask_stop = math.ceil((array.offset + nitems) / 8)
         mask = mask[mask_start:mask_stop]
 
     # Subtract the offset of the starting element from every used offset in the


Test patch:
diff --git a/dask/dataframe/tests/test_pyarrow_compat.py b/dask/dataframe/tests/test_pyarrow_compat.py
--- a/dask/dataframe/tests/test_pyarrow_compat.py
+++ b/dask/dataframe/tests/test_pyarrow_compat.py
@@ -25,10 +25,20 @@ def randstr(i):
     )
 
 
-@pytest.mark.parametrize("length", [6, 8, 12, 17])
+@pytest.mark.parametrize("length", [6, 8, 12, 20])
 @pytest.mark.parametrize(
     "slc",
-    [slice(None), slice(0, 5), slice(2), slice(2, 5), slice(2, None, 2), slice(0, 0)],
+    [
+        slice(None),
+        slice(0, 5),
+        slice(2),
+        slice(2, 5),
+        slice(2, None, 2),
+        slice(0, 0),
+        slice(7, 10),
+        slice(7, 19),
+        slice(15, 19),
+    ],
 )
 @pytest.mark.parametrize("has_mask", [True, False])
 def test_roundtrip_stringarray(length, slc, has_mask):
@@ -51,7 +61,7 @@ def unpack(nitems, offsets, data, mask=None, offset=0):
     assert bytes(data) == expected_data
 
     if mask is not None:
-        assert len(mask) == math.ceil(nitems / 8)
+        assert len(mask) == math.ceil((nitems + offset) / 8)
         assert x.offset % 8 == offset
 
     # Test rebuilding from components works