Repository: mars-project/mars

Issue:
[BUG][mars on ray] Failed to run in ray cluster mode.
**Describe the bug**
Ray cluster will load code from local (not deserialize from bytes), but mars `RemoteMetaStore` is nested in `RayStorage` class. Ray worker can't load the `RemoteMetaStore` from local code.

```
2020-11-17 16:48:29,349	WARNING worker.py:1157 -- Traceback (most recent call last):
  File "/home/admin/.local/lib/python3.6/site-packages/ray/function_manager.py", line 445, in _load_actor_class_from_local
    actor_class = getattr(module, class_name)
AttributeError: module 'mars.ray.core' has no attribute 'RemoteMetaStore'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "python/ray/_raylet.pyx", line 563, in ray._raylet.task_execution_handler
  File "python/ray/_raylet.pyx", line 567, in ray._raylet.task_execution_handler
  File "python/ray/_raylet.pyx", line 364, in ray._raylet.execute_task
  File "/home/admin/.local/lib/python3.6/site-packages/ray/function_manager.py", line 394, in load_actor_class
    job_id, actor_creation_function_descriptor)
  File "/home/admin/.local/lib/python3.6/site-packages/ray/function_manager.py", line 454, in _load_actor_class_from_local
    class_name))
RuntimeError: Actor RemoteMetaStore failed to be imported from local code.
An unexpected internal error occurred while the worker was executing a task.
```

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Python 3.6.5
2. pymars 0.6.0b2
3. ray 1.0.1
4. Minimal reproduce example.
``` python
from mars.session import new_session

ray_session = new_session(backend='ray', _load_code_from_local=True).as_default()
print(ray_session)


def main():
    import mars.dataframe as md
    import mars.tensor as mt
    t = mt.random.rand(100, 4, chunk_size=30)
    df = md.DataFrame(t, columns=list('abcd'))
    print(df.describe().execute())


if __name__ == "__main__":
    main()
```

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.



Discussion:
Any idea about how to address the problem?
> Any idea about how to address the problem?

Just move the `RemoteMetaStore` class outside the `RayStorage` class. Make the `RemoteMetaStore` visible at the module level.
> > Any idea about how to address the problem?
> 
> Just move the `RemoteMetaStore` class outside the `RayStorage` class. Make the `RemoteMetaStore` visible at the module level.

Great, so how about submitting a PR?
> > > Any idea about how to address the problem?
> > 
> > 
> > Just move the `RemoteMetaStore` class outside the `RayStorage` class. Make the `RemoteMetaStore` visible at the module level.
> 
> Great, so how about submitting a PR?

OK. I will create a fix PR.
@fyrestone can you also fix this issue on the Ray side? I guess it's because we lost the parent class information in the function descriptor. 

Resolving patch:
diff --git a/mars/ray/core.py b/mars/ray/core.py
--- a/mars/ray/core.py
+++ b/mars/ray/core.py
@@ -94,37 +94,37 @@ def handle_op(self, *args, **kw):
 ChunkMeta = namedtuple('ChunkMeta', ['shape', 'object_id'])
 
 
-class RayStorage:
-    """
-    `RayStorage` is a dict-like class. When executed in local, Mars executor will store chunk result in a
-    dict(chunk_key -> chunk_result), here uses Ray actor to store them as remote objects.
-    """
+@ray.remote
+class RemoteMetaStore:
+    def __init__(self):
+        self._store = dict()
 
-    @ray.remote
-    class RemoteMetaStore:
-        def __init__(self):
-            self._store = dict()
+    def set_meta(self, chunk_key, meta):
+        self._store[chunk_key] = meta
 
-        def set_meta(self, chunk_key, meta):
-            self._store[chunk_key] = meta
+    def get_meta(self, chunk_key):
+        return self._store[chunk_key]
 
-        def get_meta(self, chunk_key):
-            return self._store[chunk_key]
+    def get_shape(self, chunk_key):
+        return self._store[chunk_key].shape
 
-        def get_shape(self, chunk_key):
-            return self._store[chunk_key].shape
+    def chunk_keys(self):
+        return list(self._store.keys())
 
-        def chunk_keys(self):
-            return list(self._store.keys())
+    def delete_keys(self, keys):
+        if not isinstance(keys, (list, tuple)):
+            keys = [keys]
+        for k in keys:
+            del self._store[k]
 
-        def delete_keys(self, keys):
-            if not isinstance(keys, (list, tuple)):
-                keys = [keys]
-            for k in keys:
-                del self._store[k]
 
+class RayStorage:
+    """
+    `RayStorage` is a dict-like class. When executed in local, Mars executor will store chunk result in a
+    dict(chunk_key -> chunk_result), here uses Ray actor to store them as remote objects.
+    """
     def __init__(self, meta_store=None):
-        self.meta_store = meta_store or RayStorage.RemoteMetaStore.remote()
+        self.meta_store = meta_store or RemoteMetaStore.remote()
 
     def __getitem__(self, item):
         meta: ChunkMeta = ray.get(self.meta_store.get_meta.remote(item))


Test patch:
diff --git a/mars/ray/tests/test_ray_execution.py b/mars/ray/tests/test_ray_execution.py
--- a/mars/ray/tests/test_ray_execution.py
+++ b/mars/ray/tests/test_ray_execution.py
@@ -81,3 +81,10 @@ def testOperandSerialization(self):
 
         for c1, c2 in zip(op.outputs, new_op.outputs):
             self.assertEqual(c1.key, c2.key)
+
+    def testRayClusterMode(self):
+        with new_session(backend='ray', _load_code_from_local=True).as_default():
+            t = mt.random.rand(100, 4, chunk_size=30)
+            df = md.DataFrame(t, columns=list('abcd'))
+            r = df.describe().execute()
+            self.assertEqual(r.shape, (8, 4))