Repository: mars-project/mars

Issue:
[BUG][DISTRIBUTED] client will hang if submit the same graph again
**Describe the bug**
if we resubmit the same tensor by session, the client will hang, at the same time, i found scheduler raise an  error `ActorAlreadyExist`, which i think can be solved by prune existed operand.

**To Reproduce**
I creat a distributed environment with 1 worker.

```
In [1]: import mars.tensor as mt

In [2]: from mars.session import new_session

In [3]: session = new_session('http://0.0.0.0:33335')

In [4]: a = mt.ones((10, 10)) + 1

In [5]: session.run(a)
Out[5]: 
array([[2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]])

In [6]: session.run(a)

```
the client hang when execute line 6, and scheduler log shows:
```
Creating operand actors for graph 93ba7372-e748-40ab-af14-6c250f845ba4
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "mars/actors/pool/gevent_pool.pyx", line 88, in mars.actors.pool.gevent_pool.ActorExecutionContext.fire_run
  File "mars/actors/pool/gevent_pool.pyx", line 91, in mars.actors.pool.gevent_pool.ActorExecutionContext.fire_run
  File "mars/actors/pool/gevent_pool.pyx", line 102, in mars.actors.pool.gevent_pool.ActorExecutionContext.fire_run
  File "mars/actors/pool/gevent_pool.pyx", line 96, in mars.actors.pool.gevent_pool.ActorExecutionContext.fire_run
  File "mars/actors/core.pyx", line 108, in mars.actors.core.FunctionActor.on_receive
  File "mars/actors/core.pyx", line 110, in mars.actors.core.FunctionActor.on_receive
  File "/Users/hekaisheng/Documents/mars/mars/scheduler/graph.py", line 156, in execute_graph
    self.create_operand_actors()
  File "/Users/hekaisheng/Documents/mars/mars/scheduler/graph.py", line 569, in create_operand_actors
    [future.result() for future in itertools.chain(six.itervalues(op_refs), [write_future])]
  File "/Users/hekaisheng/Documents/mars/mars/scheduler/graph.py", line 569, in <listcomp>
    [future.result() for future in itertools.chain(six.itervalues(op_refs), [write_future])]
  File "src/gevent/event.py", line 457, in gevent._event.AsyncResult.result
  File "src/gevent/event.py", line 381, in gevent._event.AsyncResult.get
  File "src/gevent/event.py", line 399, in gevent._event.AsyncResult.get
  File "src/gevent/event.py", line 379, in gevent._event.AsyncResult._raise_exception
  File "/Users/hekaisheng/miniconda3/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise
    raise value.with_traceback(tb)
  File "mars/actors/pool/gevent_pool.pyx", line 758, in mars.actors.pool.gevent_pool.Communicator._create_local_actor
  File "mars/actors/pool/gevent_pool.pyx", line 237, in mars.actors.pool.gevent_pool.LocalActorPool.create_actor
mars.actors.errors.ActorAlreadyExist: Actor s:operator$2724de06-fc4a-11e8-8d24-47a7eeef77d5$c84e3ef10df7fabaf2dc028149609284 already exist, cannot create

```



Discussion:
What's the behavior of **compute on the executed tensor**?

e.g.

```python
a = mt.ones((10, 10)) + 1
session.run(a)

b = a + 1
session.run(b)
```
I guess maybe it's the same error as submitting the same graph.

The issue can be considered from two sides.

1. from the client side, we don't need to submit the entire tensor graph, we must have known which part of the tensor graph could have been executed
2. from the server side, we may need to skip the OperandActor which have been created.

Resolving patch:
diff --git a/mars/deploy/local/session.py b/mars/deploy/local/session.py
--- a/mars/deploy/local/session.py
+++ b/mars/deploy/local/session.py
@@ -81,6 +81,12 @@ def run(self, *tensors, **kw):
 
         return data_list
 
+    def fetch(self, tensor):
+        key = tensor.key
+        graph_key = self._tensor_to_graph[key]
+        resp = self._api.fetch_data(self._session_id, graph_key, key)
+        return dataserializer.loads(resp)
+
     def decref(self, *keys):
         for k in keys:
             if k not in self._tensor_to_graph:
diff --git a/mars/session.py b/mars/session.py
--- a/mars/session.py
+++ b/mars/session.py
@@ -46,6 +46,27 @@ def run(self, *tensors, **kw):
             kw['n_parallel'] = cpu_count()
         return self._executor.execute_tensors(tensors, **kw)
 
+    def fetch(self, tensor):
+        from .tensor.expressions.datasource import TensorFetchChunk
+
+        if self._executor is None:
+            raise RuntimeError('Session has closed')
+
+        if len(tensor.chunks) == 1:
+            return self._executor.chunk_result[tensor.chunks[0].key]
+
+        chunks = []
+        for c in tensor.chunks:
+            op = TensorFetchChunk(dtype=c.dtype, to_fetch_key=c.key)
+            chunk = op.new_chunk(None, c.shape, index=c.index, _key=c.key)
+            chunks.append(chunk)
+
+        new_op = tensor.op.copy()
+        tensor = new_op.new_tensor([None], tensor.shape, chunks=chunks,
+                                   nsplits=tensor.nsplits)
+
+        return self._executor.execute_tensor(tensor, concat=True)[0]
+
     def decref(self, *keys):
         self._executor.decref(*keys)
 
@@ -87,9 +108,22 @@ def run(self, *tensors, **kw):
             ret_list = True
 
         tensors = tuple(mt.tensor(t) for t in tensors)
-        result = self._sess.run(*tensors, **kw)
-        self._executed_keys.update(t.key for t in tensors)
+        run_tensors = []
+        fetch_results = dict()
+
+        # those executed tensors should fetch data directly, submit the others
         for t in tensors:
+            if t.key in self._executed_keys:
+                fetch_results[t.key] = self.fetch(t)
+            else:
+                run_tensors.append(t)
+        if all([t.key in fetch_results for t in tensors]):
+            results = [fetch_results[t.key] for t in tensors]
+            return results if ret_list else results[0]
+
+        result = self._sess.run(*run_tensors, **kw)
+        self._executed_keys.update(t.key for t in run_tensors)
+        for t in run_tensors:
             t._execute_session = self
 
         ret = []
@@ -101,9 +135,22 @@ def run(self, *tensors, **kw):
                 ret.append(np.asscalar(r))
             else:
                 ret.append(r)
+
+        results = []
+        result_iter = iter(ret)
+        for k in [t.key for t in tensors]:
+            if k in fetch_results:
+                results.append(fetch_results[k])
+            else:
+                results.append(next(result_iter))
         if ret_list:
-            return ret
-        return ret[0]
+            return results
+        return results[0]
+
+    def fetch(self, tensor):
+        if tensor.key not in self._executed_keys:
+            raise ValueError('Cannot fetch the unexecuted tensor')
+        return self._sess.fetch(tensor)
 
     @property
     def endpoint(self):
@@ -114,6 +161,7 @@ def endpoint(self, endpoint):
         self._sess.endpoint = endpoint
 
     def decref(self, *keys):
+        self._executed_keys = self._executed_keys.difference(keys)
         if hasattr(self._sess, 'decref'):
             self._sess.decref(*keys)
 
@@ -131,7 +179,8 @@ def __enter__(self):
     def __exit__(self, exc_type, exc_val, exc_tb):
         self._sess.__exit__(exc_type, exc_val, exc_tb)
 
-    close = __exit__
+    def close(self):
+        self.__exit__(None, None, None)
 
     def as_default(self):
         Session._default_session = self
diff --git a/mars/web/session.py b/mars/web/session.py
--- a/mars/web/session.py
+++ b/mars/web/session.py
@@ -147,6 +147,16 @@ def run(self, *tensors, **kw):
             data_list.append(dataserializer.loads(resp.content))
         return data_list
 
+    def fetch(self, tensor):
+        key = tensor.key
+        session_url = self._endpoint + '/api/session/' + self._session_id
+        data_url = session_url + '/graph/%s/data/%s' % (self._tensor_to_graph[key], key)
+        resp = self._req_session.get(data_url)
+        if resp.status_code >= 400:
+            raise ValueError('Failed to fetch data from server. Code: %d, Reason: %s, Content:\n%s' %
+                             (resp.status_code, resp.reason, resp.text))
+        return dataserializer.loads(resp.content)
+
     def decref(self, *keys):
         session_url = self._endpoint + '/api/session/' + self._session_id
         for k in keys:
@@ -199,5 +209,5 @@ def count_workers(self):
     def __enter__(self):
         return self
 
-    def __exit__(self, exc_type, exc_val, exc_tb):
+    def __exit__(self, *_):
         self.close()


Test patch:
diff --git a/mars/deploy/local/tests/test_cluster.py b/mars/deploy/local/tests/test_cluster.py
--- a/mars/deploy/local/tests/test_cluster.py
+++ b/mars/deploy/local/tests/test_cluster.py
@@ -203,6 +203,24 @@ def testExecutableTuple(self):
                 u, s, v = (mt.linalg.svd(a)).execute()
                 np.testing.assert_allclose(u.dot(np.diag(s).dot(v)), np.ones((20, 10)))
 
+    def testRerunTensor(self):
+        with new_cluster(scheduler_n_process=2, worker_n_process=2) as cluster:
+            session = cluster.session
+
+            a = mt.ones((10, 10)) + 1
+            result1 = session.run(a)
+            np.testing.assert_array_equal(result1, np.ones((10, 10)) + 1)
+            result2 = session.run(a)
+            np.testing.assert_array_equal(result1, result2)
+
+            with new_session(cluster.endpoint) as session2:
+                a = mt.random.rand(10, 10)
+                a_result1 = session2.run(a)
+                b = mt.ones((10, 10))
+                a_result2, b_result = session2.run(a, b)
+                np.testing.assert_array_equal(a_result1, a_result2)
+                np.testing.assert_array_equal(b_result, np.ones((10, 10)))
+
     def testGraphFail(self):
         op = SerializeMustFailOperand(f=3)
         tensor = op.new_tensor(None, (3, 3))
diff --git a/mars/tests/test_session.py b/mars/tests/test_session.py
--- a/mars/tests/test_session.py
+++ b/mars/tests/test_session.py
@@ -19,6 +19,7 @@
 import numpy as np
 
 import mars.tensor as mt
+from mars.session import new_session
 
 
 class Test(unittest.TestCase):
@@ -52,3 +53,50 @@ def testSessionExecute(self):
         expected = np.split(data, 3, axis=1)[0]
 
         np.testing.assert_array_equal(result, expected)
+
+        # test run the same tensor
+        arr4 = mt.tensor(data.copy(), chunk_size=3) + 1
+        result1 = arr4.execute()
+        expected = data + 1
+
+        np.testing.assert_array_equal(result1, expected)
+
+        result2 = arr4.execute()
+
+        np.testing.assert_array_equal(result1, result2)
+
+        # test run the same tensor with single chunk
+        arr4 = mt.tensor(data.copy())
+        result1 = arr4.execute()
+        expected = data
+
+        np.testing.assert_array_equal(result1, expected)
+
+        result2 = arr4.execute()
+
+        np.testing.assert_array_equal(result1, result2)
+
+        # test run same key tensor
+        arr5 = mt.ones((10, 10), chunk_size=3)
+        result1 = arr5.execute()
+
+        del arr5
+        arr6 = mt.ones((10, 10), chunk_size=3)
+        result2 = arr6.execute()
+
+        np.testing.assert_array_equal(result1, result2)
+
+    def testClosedSession(self):
+        session = new_session()
+        arr = mt.ones((10, 10))
+
+        result = session.run(arr)
+
+        np.testing.assert_array_equal(result, np.ones((10, 10)))
+
+        session.close()
+        with self.assertRaises(RuntimeError):
+            session.run(arr)
+
+        with self.assertRaises(RuntimeError):
+            session.run(arr + 1)
diff --git a/mars/web/tests/test_api.py b/mars/web/tests/test_api.py
--- a/mars/web/tests/test_api.py
+++ b/mars/web/tests/test_api.py
@@ -158,9 +158,12 @@ def testApi(self):
             value = sess.run(c)
             assert_array_equal(value, np.ones((100, 100)) * 100)
 
+            value2 = sess.run(c)
+            assert_array_equal(value, value2)
+
             # todo this behavior may change when eager mode is introduced
             with self.assertRaises(ExecutionFailed):
-                sess.run(c)
+                sess.run(c + 1)
 
             va = np.random.randint(0, 10000, (100, 100))
             vb = np.random.randint(0, 10000, (100, 100))