Repository: dask/distributed

Issue:
cannot set WRITEABLE flag to True of this array
My colleague is seeing the following error on distirbuted 2.8.0. It looks related to #3180 and this set of lines

https://github.com/dask/distributed/blob/35551998d7350cd5ae6a5c24970d8437fd8d521d/distributed/protocol/numpy.py#L110-L114


```
distributed.worker - INFO - Start worker at: tcp://10.12.205.19:34307

distributed.worker - INFO - Listening to: tcp://10.12.205.19:34307

distributed.worker - INFO - dashboard at: 10.12.205.19:34094

distributed.worker - INFO - Waiting to connect to: tcp://128.117.181.211:37309

distributed.worker - INFO - -------------------------------------------------

distributed.worker - INFO - Threads: 1

distributed.worker - INFO - Memory: 25.00 GB

distributed.worker - INFO - Local Directory: /glade/scratch/deppenme/dask-tmp/worker-k05sxxku

distributed.worker - INFO - -------------------------------------------------

distributed.worker - INFO - Registered to: tcp://128.117.181.211:37309

distributed.worker - INFO - -------------------------------------------------

distributed.worker - ERROR - cannot set WRITEABLE flag to True of this array 

Traceback (most recent call last): File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/worker.py", line 894, in handle_scheduler comm, every_cycle=[self.ensure_communicating, self.ensure_computing] File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/core.py", line 447, in handle_stream msgs = await comm.read() File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/comm/tcp.py", line 208, in read frames, deserialize=self.deserialize, deserializers=deserializers 
File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/comm/utils.py", line 63, in from_frames res = await offload(_from_frames) File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/utils.py", line 1371, in offload return await loop.run_in_executor(_offload_executor, fn, *args, **kwargs) File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) 
File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/comm/utils.py", line 51, in _from_frames frames, deserialize=deserialize, deserializers=deserializers File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/protocol/core.py", line 124, in loads value = _deserialize(head, fs, deserializers=deserializers) 
File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 268, in deserialize return loads(header, frames) 
File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 54, in dask_loads return loads(header, frames) 
File "/glade/work/deppenme/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/protocol/numpy.py",
 line 113, in deserialize_numpy_ndarray x.setflags(write=writeable) ValueError: cannot set WRITEABLE flag to True of this array
```

This is coming out of a complicated analysis pipeline with xarray, zarr and dask so we don't have a minimal example yet. We could work to find one if you have some pointers on what to look for.


Discussion:
Just to clarify, this doesn't happen with distributed 2.7.0?
Yes. i just confirmed that the same code works with 2.7.0
Thanks. cc @jrbourbeau if you have any guesses on this.
Thanks for reporting @dcherian, this is indeed a bug introduce in #3180. Looks like arrays cannot be set to writeable if they don't own their memory and their underlying base object is read-only (to avoid writing to readonly memory).

Here's an example which works fine prior to the changes in #3180, but fails with `ValueError: cannot set WRITEABLE flag to True of this array` after those changes are included:

```python
import numpy as np
from distributed.protocol import serialize, deserialize

x = np.arange(3)
# Create zero-strided array
y = np.broadcast_to(x, (3, 3))

assert x.flags.writeable == True     # arange produces writeable arrays
assert y.flags.writeable == False    # broadcase_to produces read-only arrays

# Make y writeable and it's base object (x) read-only
y.setflags(write=True)
x.setflags(write=False)

# Serialize / deserialize y
z = deserialize(*serialize(y))
```

Any idea if this situation pops up in your workflow @dcherian? I'm looking for a fix now
Hmmm.. i just looked over my colleague's code. There's a lot of broadcasting multiplications and divisions happening so my answer is "quite likely maybe" ;)

Sorry I can't help more.
@jrbourbeau for a fix, when we serialize we might skip this optimization when the `x.base.flags.writable` is False? That check for `.base` may need to be done recursively, if you have a writable view of a writable view of a readonly array...

FWIW, I find it strange that you can do

```python
x.setflags(write=False)
```

when there are writable views on `x` but that's probably a larger issue :)
This might be a good time to use git bisect

On Wed, Nov 20, 2019, 2:16 PM Tom Augspurger <notifications@github.com>
wrote:

> Thanks. cc @jrbourbeau <https://github.com/jrbourbeau> if you have any
> guesses on this.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/3252?email_source=notifications&email_token=AACKZTER3VTXKKP2BKNUH3TQUWSMXA5CNFSM4JPXOOEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEVBYWY#issuecomment-556407899>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTFDHKIOHJJMODQFJJLQUWSMXANCNFSM4JPXOOEA>
> .
>


Resolving patch:
diff --git a/distributed/protocol/numpy.py b/distributed/protocol/numpy.py
--- a/distributed/protocol/numpy.py
+++ b/distributed/protocol/numpy.py
@@ -46,7 +46,7 @@ def serialize_numpy_ndarray(x):
 
     # Only serialize non-broadcasted data for arrays with zero strided axes
     if 0 in x.strides:
-        broadcast_to = (x.shape, x.flags.writeable)
+        broadcast_to = x.shape
         x = x[tuple(slice(None) if s != 0 else slice(1) for s in x.strides)]
     else:
         broadcast_to = None
@@ -103,14 +103,12 @@ def deserialize_numpy_ndarray(header, frames):
         else:
             dt = np.dtype(dt)
 
-        x = np.ndarray(
-            header["shape"], dtype=dt, buffer=frames[0], strides=header["strides"]
-        )
-
         if header.get("broadcast_to"):
-            shape, writeable = header["broadcast_to"]
-            x = np.broadcast_to(x, shape)
-            x.setflags(write=writeable)
+            shape = header["broadcast_to"]
+        else:
+            shape = header["shape"]
+
+        x = np.ndarray(shape, dtype=dt, buffer=frames[0], strides=header["strides"])
 
         return x
 


Test patch:
diff --git a/distributed/protocol/tests/test_numpy.py b/distributed/protocol/tests/test_numpy.py
--- a/distributed/protocol/tests/test_numpy.py
+++ b/distributed/protocol/tests/test_numpy.py
@@ -288,3 +288,22 @@ def test_non_zero_strided_array():
     header, frames = serialize(x)
     assert "broadcast_to" not in header
     assert sum(map(nbytes, frames)) == x.nbytes
+
+
+def test_serialize_writeable_array_readonly_base_object():
+    # Regression test for https://github.com/dask/distributed/issues/3252
+
+    x = np.arange(3)
+    # Create array which doesn't own it's own memory
+    y = np.broadcast_to(x, (3, 3))
+
+    # Make y writeable and it's base object (x) read-only
+    y.setflags(write=True)
+    x.setflags(write=False)
+
+    # Serialize / deserialize y
+    z = deserialize(*serialize(y))
+    np.testing.assert_equal(z, y)
+
+    # Ensure z and y have the same flags (including WRITEABLE)
+    assert z.flags == y.flags