Repository: ray-project/ray

Issue:
Serialization of certain objects not handled correctly after 0.8.1
### What is the problem?

When using Ray 0.8.2 on Ubuntu 18.04, I am serializing data structures from the astropy library (https://pypi.org/project/astropy/ , version 4.0).

Previously, I have configured ray to use cloudpickle for many of these astropy types, as they couldn't be natively handled by ray/pyarrow. In 0.8.2 however, ray seems to be ignoring this configuration, and it converts the data type in question (astropy.constants.constant.Constant) to a numpy ndarray upon ray.put/ray.get, which is incorrect behavior and breaks downstream parts of my code.

Version 0.8.1 seems to be fine. I also reproduced the issue with the test cases below in  ray-0.9.0.dev0 as of today.

I suspect that this is a bug related to classes that have numpy-related base class.

### Reproduction (REQUIRED)

Case 1: ignoring custom serializer

```python
import numpy
import ray

class MyConstant(numpy.ndarray):
    def __init__(self, value):
        super().__init__()
        self.constant = value
    
    def __str__(self):
        print(self.constant)

constant = MyConstant(123)

ray.shutdown()
ray.init()

def explode(x):
    raise RuntimeError()

ray.register_custom_serializer(type(constant), serializer=explode, deserializer=explode)

try:
    ray.put(constant)
    print('Should never get here!')
except (RuntimeError, IndexError):
    print('Correct behavior, proof that customer serializer was used.')
```

Case 2: Incorrect round-trip
```python
import numpy
import ray

class MyConstant(numpy.ndarray):
    def __init__(self, value):
        super().__init__()
        self.constant = value
    
    def __str__(self):
        print(self.constant)

constant = MyConstant(123)

ray.shutdown()
ray.init()

ray.register_custom_serializer(type(constant), use_pickle=True)

repr_orig = repr(constant)
repr_ser =repr(ray.get(ray.put(constant)))

if repr_orig == repr_ser:
    print('Good round trip')
else:
    print('Bad round trip!')

print(repr_orig)
print(repr_ser)
```

If we cannot run your script, we cannot fix your issue.

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://ray.readthedocs.io/en/latest/installation.html).



Discussion:
Thanks, I can reproduce this and it looks like it was introduced in https://github.com/ray-project/ray/pull/6675

@suquark Can you look into this?
@mc-allen it's a great catch! let me fix this

Resolving patch:
diff --git a/python/ray/cloudpickle/cloudpickle_fast.py b/python/ray/cloudpickle/cloudpickle_fast.py
--- a/python/ray/cloudpickle/cloudpickle_fast.py
+++ b/python/ray/cloudpickle/cloudpickle_fast.py
@@ -541,8 +541,8 @@ def reducer_override(self, obj):
         # This is a patch for python3.5
         if isinstance(obj, numpy.ndarray):
             if (self.proto < 5 or
-                    (not obj.flags.c_contiguous and
-                     not obj.flags.f_contiguous) or
+                    (not obj.flags.c_contiguous and not obj.flags.f_contiguous) or
+                    (issubclass(type(obj), numpy.ndarray) and type(obj) is not numpy.ndarray) or
                     obj.dtype == "O" or obj.itemsize == 0):
                 return NotImplemented
             return _numpy_ndarray_reduce(obj)


Test patch:
diff --git a/python/ray/tests/test_basic.py b/python/ray/tests/test_basic.py
--- a/python/ray/tests/test_basic.py
+++ b/python/ray/tests/test_basic.py
@@ -378,6 +378,56 @@ def test_complex_serialization_with_pickle(shutdown_only):
     complex_serialization(use_pickle=True)
 
 
+def test_numpy_serialization(ray_start_regular):
+    array = np.zeros(314)
+    from ray.cloudpickle import dumps
+    buffers = []
+    inband = dumps(array, protocol=5, buffer_callback=buffers.append)
+    assert len(inband) < array.nbytes
+    assert len(buffers) == 1
+
+
+def test_numpy_subclass_serialization(ray_start_regular):
+    class MyNumpyConstant(np.ndarray):
+        def __init__(self, value):
+            super().__init__()
+            self.constant = value
+
+        def __str__(self):
+            print(self.constant)
+
+    constant = MyNumpyConstant(123)
+
+    def explode(x):
+        raise RuntimeError("Expected error.")
+
+    ray.register_custom_serializer(
+        type(constant), serializer=explode, deserializer=explode)
+
+    try:
+        ray.put(constant)
+        assert False, "Should never get here!"
+    except (RuntimeError, IndexError):
+        print("Correct behavior, proof that customer serializer was used.")
+
+
+def test_numpy_subclass_serialization_pickle(ray_start_regular):
+    class MyNumpyConstant(np.ndarray):
+        def __init__(self, value):
+            super().__init__()
+            self.constant = value
+
+        def __str__(self):
+            print(self.constant)
+
+    constant = MyNumpyConstant(123)
+    ray.register_custom_serializer(type(constant), use_pickle=True)
+
+    repr_orig = repr(constant)
+    repr_ser = repr(ray.get(ray.put(constant)))
+    assert repr_orig == repr_ser
+
+
 def test_function_descriptor():
     python_descriptor = ray._raylet.PythonFunctionDescriptor(
         "module_name", "function_name", "class_name", "function_hash")