Repository: ray-project/ray

Issue:
[dashboard] dashboard.py crashes in example where actors are quickly created and go out of scope.
### What is the problem?
I'm using
- `ray==0.8.2`
- Python 3.7.2
- MacOS

### Reproduction (REQUIRED)

#### Bug 1

This is likely somewhat timing dependent, but when I run the following script

```python
import ray
ray.init()
@ray.remote
class GrandChild:
    def __init__(self):
        pass
@ray.remote
class Child:
    def __init__(self):
        grand_child_handle = GrandChild.remote()
@ray.remote
class Parent:
    def __init__(self):
        children_handles = [Child.remote() for _ in range(2)]
parent_handle = Parent.remote()
```

the `dashboard.py` process crashes with the following error.

```
Traceback (most recent call last):
  File "/Users/rkn/opt/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/Users/rkn/opt/anaconda3/lib/python3.7/site-packages/ray/dashboard/dashboard.py", line 546, in run
    current_actor_table = ray.actors()
  File "/Users/rkn/opt/anaconda3/lib/python3.7/site-packages/ray/state.py", line 1133, in actors
    return state.actor_table(actor_id=actor_id)
  File "/Users/rkn/opt/anaconda3/lib/python3.7/site-packages/ray/state.py", line 369, in actor_table
    ray.ActorID(actor_id_binary))
  File "/Users/rkn/opt/anaconda3/lib/python3.7/site-packages/ray/state.py", line 321, in _actor_table
    assert len(gcs_entries.entries) == 1
AssertionError
```

#### Bug 2

As a consequence, the dashboard doesn't display anything.

There is a second bug here, which is that the error message wasn't pushed to the driver or displayed in the UI in any way.

We have a test testing that tests that an error message is pushed to the driver when the monitor process dies. https://github.com/ray-project/ray/blob/007333b96006e090b14a415a8c5993d8b907f02a/python/ray/tests/test_failure.py#L544

I thought we had a similar test for the dashboard, but I don't see it anywhere. Note that we appear to have the plumbing in place to push error messages to the driver when the dashboard process dies, but I didn't see the relevant error message. See https://github.com/ray-project/ray/blob/f2faf8d26e489e0d879fd691abddeb3140c86bee/python/ray/dashboard/dashboard.py#L928-L929

cc @pcmoritz @simon-mo @mitchellstern 


Discussion:

Resolving patch:
diff --git a/python/ray/state.py b/python/ray/state.py
--- a/python/ray/state.py
+++ b/python/ray/state.py
@@ -318,9 +318,9 @@ def _actor_table(self, actor_id):
             return {}
         gcs_entries = gcs_utils.GcsEntry.FromString(message)
 
-        assert len(gcs_entries.entries) == 1
+        assert len(gcs_entries.entries) > 0
         actor_table_data = gcs_utils.ActorTableData.FromString(
-            gcs_entries.entries[0])
+            gcs_entries.entries[-1])
 
         actor_info = {
             "ActorID": binary_to_hex(actor_table_data.actor_id),


Test patch:
diff --git a/python/ray/tests/test_global_state.py b/python/ray/tests/test_global_state.py
--- a/python/ray/tests/test_global_state.py
+++ b/python/ray/tests/test_global_state.py
@@ -76,6 +76,36 @@ def test_add_remove_cluster_resources(ray_start_cluster_head):
     assert ray.cluster_resources()["CPU"] == 6
 
 
+def test_global_state_actor_table(ray_start_regular):
+    @ray.remote
+    class Actor:
+        def ready(self):
+            pass
+
+    # actor table should be empty at first
+    assert len(ray.actors()) == 0
+
+    # actor table should contain only one entry
+    a = Actor.remote()
+    ray.get(a.ready.remote())
+    assert len(ray.actors()) == 1
+
+    # actor table should contain only this entry
+    # even when the actor goes out of scope
+    del a
+
+    def get_state():
+        return list(ray.actors().values())[0]["State"]
+
+    dead_state = ray.gcs_utils.ActorTableData.DEAD
+    for _ in range(10):
+        if get_state() == dead_state:
+            break
+        else:
+            time.sleep(0.5)
+    assert get_state() == dead_state
+
+
 if __name__ == "__main__":
     import pytest
     import sys