Repository: ray-project/ray

Issue:
Submitting a large number of actor tasks in a loop gives reconstruction error.
I'm seeing this issue on Ubuntu 16.04 with Python 3.6.2 (Anaconda) and the current Ray master (building from source).

I'm running the following.

```python
import ray

ray.worker._init(start_ray_local=True, object_store_memory=100000000)

@ray.remote
class Foo():
    def m(self):
        return 1

actors = [Foo.remote() for _ in range(30)]

i = 0
while True:
    print(i)
    i += 1
    ids = []
    for _ in range(50):
        ids += [a.m.remote() for a in actors]
    ray.get(ids)
```

The relevant output is

```
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
There is not enough space to create this object, so evicting 32259 objects to free up 20000580 bytes.
31
32
33
34
35
There is not enough space to create this object, so evicting 20708 objects to free up 12839980 bytes.
[INFO] (/home/ubuntu/ray/src/local_scheduler/local_scheduler_algorithm.cc:443) A task that has already been executed has been resubmitted, so we are ignoring it. This should only happen during reconstruction.
[INFO] (/home/ubuntu/ray/src/local_scheduler/local_scheduler_algorithm.cc:443) A task that has already been executed has been resubmitted, so we are ignoring it. This should only happen during reconstruction.
[INFO] (/home/ubuntu/ray/src/local_scheduler/local_scheduler_algorithm.cc:443) A task that has already been executed has been resubmitted, so we are ignoring it. This should only happen during reconstruction.
[INFO] (/home/ubuntu/ray/src/local_scheduler/local_scheduler_algorithm.cc:443) A task that has already been executed has been resubmitted, so we are ignoring it. This should only happen during reconstruction.
```

The message repeats a large number of times very quickly and then the script hangs.


Discussion:
The error goes away if I don't save dummy objects on the workers. I'm worried this may be the cause of a memory leak that I'm seeing (e.g., the dummy objects just accumulate forever).

Do we need to pin *all* of the dummy objects?

@stephanie-wang 
For now, we will have to pin all the objects for forked actor handles.

The plan for the future is to remove the dummy objects entirely from the object store. Only the local scheduler hosting the actor process needs to know about them, so we can have the actor process mock the object notifications that would normally come from the object store.

Resolving patch:
diff --git a/python/ray/actor.py b/python/ray/actor.py
--- a/python/ray/actor.py
+++ b/python/ray/actor.py
@@ -6,7 +6,6 @@
 import hashlib
 import inspect
 import json
-import numpy as np
 import traceback
 
 import pyarrow.plasma as plasma
@@ -110,37 +109,6 @@ def get_actor_checkpoint(worker, actor_id):
         return checkpoint_index, checkpoint
 
 
-def put_dummy_object(worker, dummy_object_id):
-    """Put a dummy actor object into the local object store.
-
-    This registers a dummy object ID in the local store with an empty numpy
-    array as the value. The resulting object is pinned to the store by storing
-    it to the worker's state.
-
-    For actors, dummy objects are used to store the stateful dependencies
-    between consecutive method calls. This function should be called for every
-    actor method execution that updates the actor's internal state.
-
-    Args:
-        worker: The worker to use to perform the put.
-        dummy_object_id: The object ID of the dummy object.
-    """
-    # Add the dummy output for actor tasks. TODO(swang): We use
-    # a numpy array as a hack to pin the object in the object
-    # store. Once we allow object pinning in the store, we may
-    # use `None`.
-    dummy_object = np.zeros(1)
-    worker.put_object(dummy_object_id, dummy_object)
-    # Keep the dummy output in scope for the lifetime of the
-    # actor, to prevent eviction from the object store.
-    dummy_object = worker.get_object([dummy_object_id])
-    dummy_object = dummy_object[0]
-    worker.actor_pinned_objects.append(dummy_object)
-    if (len(worker.actor_pinned_objects) >
-            ray._config.actor_max_dummy_objects()):
-        worker.actor_pinned_objects.pop(0)
-
-
 def make_actor_method_executor(worker, method_name, method):
     """Make an executor that wraps a user-defined actor method.
 
@@ -168,11 +136,10 @@ def actor_method_executor(dummy_return_id, task_counter, actor,
         if method_name == "__ray_checkpoint__":
             # Execute the checkpoint task.
             actor_checkpoint_failed, error = method(actor, *args)
-            # If the checkpoint was successfully loaded, put the dummy object
-            # and update the actor's task counter, so that the task following
-            # the checkpoint can run.
+            # If the checkpoint was successfully loaded, update the actor's
+            # task counter and set a flag to notify the local scheduler, so
+            # that the task following the checkpoint can run.
             if not actor_checkpoint_failed:
-                put_dummy_object(worker, dummy_return_id)
                 worker.actor_task_counter = task_counter + 1
                 # Once the actor has resumed from a checkpoint, it counts as
                 # loaded.
@@ -188,7 +155,6 @@ def actor_method_executor(dummy_return_id, task_counter, actor,
         else:
             # Update the worker's internal state before executing the method in
             # case the method throws an exception.
-            put_dummy_object(worker, dummy_return_id)
             worker.actor_task_counter = task_counter + 1
             # Once the actor executes a task, it counts as loaded.
             worker.actor_loaded = True
diff --git a/python/ray/worker.py b/python/ray/worker.py
--- a/python/ray/worker.py
+++ b/python/ray/worker.py
@@ -230,10 +230,6 @@ def __init__(self):
         # task assigned. Workers are not assigned a task on startup, so we
         # initialize to False.
         self.actor_checkpoint_failed = False
-        # TODO(swang): This is a hack to prevent the object store from evicting
-        # dummy objects. Once we allow object pinning in the store, we may
-        # remove this variable.
-        self.actor_pinned_objects = None
         # The number of threads Plasma should use when putting an object in the
         # object store.
         self.memcopy_threads = 12
@@ -1920,9 +1916,6 @@ def connect(info, object_id_seed=None, mode=WORKER_MODE, worker=global_worker,
         actor_key = b"Actor:" + worker.actor_id
         class_id = worker.redis_client.hget(actor_key, "class_id")
         worker.class_id = class_id
-        # Store a list of the dummy outputs produced by actor tasks, to pin the
-        # dummy outputs in the object store.
-        worker.actor_pinned_objects = []
 
     # Initialize the serialization library. This registers some classes, and so
     # it must be run before we export all of the cached remote functions.


Test patch: