Repository: ray-project/ray

Issue:
Garbage collection for actor dummy objects
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Any
- **Ray installed from (source or binary)**: Any
- **Ray version**: 0.5.3
- **Python version**: Any
- **Exact command to reproduce**:

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
1. For long running jobs, actor won't exit and it will submit tasks continuously. Each task will have a dummy object put in `local_objects_` in task_dependency_manager.cc. This unordered map will grow very big. When the size grows to 13-16 million, the rehashing algorithm will take more than 10 second, which cause heartbeat timeout. 
2. For python functions, even if there is no return value, a `None` object will be put into Plasma. However, users don't aware there is a None value put into Plasma and they will not call Free to release it, which will also cause very big Plasma usage or large amount of `local_objects_` elements.
### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
For 2, if you don't want any return values, then you should decorate the actor method with `@ray.method(num_return_vals=0)` as follows.

```python
@ray.remote
class Actor:
    @ray.method(num_return_vals=0)
    def method(self):
        pass
```

For 1, maybe the solution is to release unneeded dummy objects every time we do an actor checkpoint.
Thanks, @robertnishihara for the `num_return_vals=0` method. That is what I want!
For 1, we have implemented in local repo to delete the GCS table as well as the dummy object after checkpointing.
Actually, I think we can delete the previous dummy object on the fly after we do `ExtendFrontier`. Because we only need the most recent dummy object to represent Actor's state. I'll experiment this idea when I have time. cc @stephanie-wang 
@raulchen, I'm not 100% sure, but there may be some issues with that if the actor handles are passed into other tasks because then you may receive tasks that depend on an earlier actor dummy object that has been removed.
@robertnishihara I think that's true for now, because when we pass a handle to another task, the new handle will have the same cursor. But I don't really understand why the new handle needs to depend on the old handle's cursor. I think from the perspective of the actor, these 2 handles can be simply 2 independent clients. Their task execution order shouldn't be related, right?
@raulchen I think in a lot of cases, it is intuitive to expect the execution orders to be related. For example,

```python
a = Actor.remote()  # Create an actor.

a.initialize.remote()  # Do some initialization.

f.remote(a)  # Pass a into f, which uses a.
```

In this example, I think people would expect the `initialize` task (and definitely the actor creation task) to happen before whatever tasks `f` submits to `a`.

https://github.com/ray-project/ray/issues/1149 is a related example in which the lack of dependence caused confusion (though this is with the actor creation task).
@robertnishihara Thanks. 
Regarding the 4 options you mentioned in [this comment](https://github.com/ray-project/ray/issues/1149#issuecomment-345153290), I think it makes sense to have option 2 the default behavior. because in normal cases, users may only want to distribute a handle to multiple workers, and don't care about the order.

Regarding the example you commented above, one can explicitly ensure order by doing the following:
```
ray.get(a.initialize.remote())
f.remote(a)
```
or
```
initialized = a.initialize.remote()
f.remote(a, initialized)
```
As @robertnishihara, the original reasoning for keeping the dummy objects around is for ordering. It's hard to say definitively how often users do or don't care about ordering, so I think we'll just have to wait and see.

As a longer-term solution, you can actually decide when to release dummy objects as long as the frontend provides information about how many times an actor handle has been passed around. So in the example posted above, if you did, this then you would only have to keep around the `initialize` dummy object, and it could be released once `f` calls its first method.
```python
a = Actor.remote()  # Create an actor.
a.initialize.remote()  # Do some initialization.
f.remote(a)  # Pass a into f, which uses a.
a.foo.remote()  # Tell the backend that a was passed into f.
```
And yes, I completely agree that we should release old dummy objects after a checkpoint.
It is possible that an old dummy object ID could become relevant a long time later, right (even after multiple checkpoints)? E.g., if a handle was passed into a task but the task didn't get scheduled for a very long time due to some other object dependency.

In that case, we really just need to be able to compare the dummy Object ID with the current actor frontier, so if we can encode more semantic information in the IDs, e.g., a vector clock essentially, then we wouldn't need to keep around all of the dummy object IDs.

@stephanie-wang Does something like that make sense?
> In that case, we really just need to be able to compare the dummy Object ID with the current actor frontier, so if we can encode more semantic information in the IDs, e.g., a vector clock essentially, then we wouldn't need to keep around all of the dummy object IDs.

@robertnishihara can you elaborate on how that solves the case you just described (late-scheduled tasks)? Why would having that ordering let you release an old dummy object? I was trying to figure out a solution to the same problem in general for lineage GC after checkpointing.
```
a -> b -> c -> d
      \
        -> x
```

@ujvl say we have the above dependency tree, each letter is a dummy object. Right now, we cannot clean up b until x starts executing, even if c and d have already finished. Because otherwise we don't know x's dependencies are satisfied. (Considering x might be re-submitted, we can never clean up b, right? @stephanie-wang )
However, if we dummy objects are comparable, we can clean up b and c, and only keep d. Because when we try schedule x, we know there's a newer dummy object d than b, x's dependency is satisfied.  
I really don't think that it's a good idea to pursue comparable object IDs. The best we can do is probabilistic comparison. You will run out of bits very quickly if you try to do perfect comparison.

The way that I would go about the case that @robertnishihara listed is:
1. The frontend notifies the backend that a handle has been passed to another task with dummy object `x`. The backend records the fact that it is waiting for `x`. It can do this by preemptively adding a new handle to the actor frontier.
2. The task with the new actor handle hasn't been scheduled yet, but a bunch of other tasks get submitted on other handles. All dummy objects for these handles in between `x` and the present task can be released. A checkpoint may be taken at any point, but it doesn't affect which objects get released (the set of dummy objects held is already the minimum).
4. The task gets scheduled and submits new tasks on its actor handle. The backend releases `x`.

Also, @raulchen, I think it's okay to release b as soon as x is scheduled the first time. If x gets resubmitted, then you have to roll back the actor to the latest checkpoint anyway, so b will get recreated.
@stephanie-wang, regarding

> I think it's okay to release b as soon as x is scheduled the first time. If x gets resubmitted, then you have to roll back the actor to the latest checkpoint anyway, so b will get recreated.

This assumes that the checkpoint is before `b`, right? E.g., if the checkpoint was taken at `d` then reloading from the checkpoint won't cause `b` to get recreated, right?

Also, is there a way to make this work in the setting where the actor is using the "restart only" fault tolerance mode, where the actor is recreated, but no methods are replayed? I suppose that for such actors, task ordering probably doesn't matter and so maybe such actors simply shouldn't use dummy object IDs.
If the checkpoint happens before `b`, then `b` will get recreated. If the checkpoint was after `b`, then I think we should make it so that `b` is considered unreconstructable (similar to what we do right now for evicted actor objects), and gets failed immediately.

I think that the task ordering is still important for "restart only" actors, since it would not be very intuitive to users if tasks could suddenly be executed out-of-order for that failure mode. The part that would be different is after recovering from a failure, we would only require that any **later** task could be executed immediately, instead of requiring that the **next consecutive** task be executed.
@stephanie-wang the scenario I'm concerned about is a little different I think.

I'm imagining a setting with no failures.

- Actor methods `a`, `b`, `c`, `d`, and `x` are submitted as above.
- Methods `a`, `b`, `c`, `d` execute.
- A checkpoint is taken (say after `d`), and so dummy object IDs for `a`, `b`, `c`, `d` are flushed.
- Then `x` finally arrives at the raylet (it was delayed for some reason).
- The correct behavior is to just execute `x`. Doing so respects all ordering constraints. However, `x` depends on the dummy object for `b` which has been flushed, so we don't know that we can execute it.

Does that make sense?
Actually, I don't think checkpoints really matter at all. The algorithm I'm proposing is to record the fact that the actor handle was forked from `b` in the task that produces `c`. When `c` gets executed, the raylet will see this and extend the actor's frontier to include the new handle on which `x` will be submitted. Any dummy object on the actor's frontier will be pinned, and everything before the frontier will be released. Therefore, `x`'s dependency will be pinned until it arrives at the raylet.

Resolving patch:
diff --git a/python/ray/actor.py b/python/ray/actor.py
--- a/python/ray/actor.py
+++ b/python/ray/actor.py
@@ -485,6 +485,10 @@ class ActorHandle(object):
         _ray_actor_driver_id: The driver ID of the job that created the actor
             (it is possible that this ActorHandle exists on a driver with a
             different driver ID).
+        _ray_new_actor_handles: The new actor handles that were created from
+            this handle since the last task on this handle was submitted. This
+            is used to garbage-collect dummy objects that are no longer
+            necessary in the backend.
     """
 
     def __init__(self,
@@ -520,6 +524,7 @@ def __init__(self,
             actor_creation_dummy_object_id)
         self._ray_actor_method_cpus = actor_method_cpus
         self._ray_actor_driver_id = actor_driver_id
+        self._ray_new_actor_handles = []
 
     def _actor_method_call(self,
                            method_name,
@@ -585,6 +590,7 @@ def _actor_method_call(self,
             actor_creation_dummy_object_id=(
                 self._ray_actor_creation_dummy_object_id),
             execution_dependencies=execution_dependencies,
+            new_actor_handles=self._ray_new_actor_handles,
             # We add one for the dummy return ID.
             num_return_vals=num_return_vals + 1,
             resources={"CPU": self._ray_actor_method_cpus},
@@ -596,6 +602,9 @@ def _actor_method_call(self,
         # The last object returned is the dummy object that should be
         # passed in to the next actor method. Do not return it to the user.
         self._ray_actor_cursor = object_ids.pop()
+        # We have notified the backend of the new actor handles to expect since
+        # the last task was submitted, so clear the list.
+        self._ray_new_actor_handles = []
 
         if len(object_ids) == 1:
             object_ids = object_ids[0]
@@ -702,6 +711,19 @@ def _serialization_helper(self, ray_forking):
 
         if ray_forking:
             self._ray_actor_forks += 1
+            new_actor_handle_id = actor_handle_id
+        else:
+            # The execution dependency for a pickled actor handle is never safe
+            # to release, since it could be unpickled and submit another
+            # dependent task at any time. Therefore, we notify the backend of a
+            # random handle ID that will never actually be used.
+            new_actor_handle_id = ray.ObjectID(_random_string())
+        # Notify the backend to expect this new actor handle. The backend will
+        # not release the cursor for any new handles until the first task for
+        # each of the new handles is submitted.
+        # NOTE(swang): There is currently no garbage collection for actor
+        # handles until the actor itself is removed.
+        self._ray_new_actor_handles.append(new_actor_handle_id)
 
         return state
 
diff --git a/python/ray/experimental/named_actors.py b/python/ray/experimental/named_actors.py
--- a/python/ray/experimental/named_actors.py
+++ b/python/ray/experimental/named_actors.py
@@ -56,5 +56,8 @@ def register_actor(name, actor_handle):
     # Add the actor to Redis if it does not already exist.
     already_exists = _internal_kv_put(actor_name, pickled_state)
     if already_exists:
+        # If the registration fails, then erase the new actor handle that
+        # was added when pickling the actor handle.
+        actor_handle._ray_new_actor_handles.pop()
         raise ValueError(
             "Error: the actor with name={} already exists".format(name))
diff --git a/python/ray/worker.py b/python/ray/worker.py
--- a/python/ray/worker.py
+++ b/python/ray/worker.py
@@ -524,6 +524,7 @@ def submit_task(self,
                     actor_creation_dummy_object_id=None,
                     max_actor_reconstructions=0,
                     execution_dependencies=None,
+                    new_actor_handles=None,
                     num_return_vals=None,
                     resources=None,
                     placement_resources=None,
@@ -594,6 +595,9 @@ def submit_task(self,
             if execution_dependencies is None:
                 execution_dependencies = []
 
+            if new_actor_handles is None:
+                new_actor_handles = []
+
             if driver_id is None:
                 driver_id = self.task_driver_id
 
@@ -628,8 +632,8 @@ def submit_task(self,
                 num_return_vals, self.current_task_id, task_index,
                 actor_creation_id, actor_creation_dummy_object_id,
                 max_actor_reconstructions, actor_id, actor_handle_id,
-                actor_counter, execution_dependencies, resources,
-                placement_resources)
+                actor_counter, new_actor_handles, execution_dependencies,
+                resources, placement_resources)
             self.raylet_client.submit_task(task)
 
             return task.returns()
@@ -1949,7 +1953,7 @@ def connect(ray_params,
             worker.current_task_id, worker.task_index,
             ray.ObjectID(NIL_ACTOR_ID), ray.ObjectID(NIL_ACTOR_ID), 0,
             ray.ObjectID(NIL_ACTOR_ID), ray.ObjectID(NIL_ACTOR_ID),
-            nil_actor_counter, [], {"CPU": 0}, {})
+            nil_actor_counter, [], [], {"CPU": 0}, {})
 
         # Add the driver task to the task table.
         global_state._execute_command(driver_task.task_id(), "RAY.TABLE_ADD",


Test patch:
diff --git a/src/ray/raylet/worker_pool_test.cc b/src/ray/raylet/worker_pool_test.cc
--- a/src/ray/raylet/worker_pool_test.cc
+++ b/src/ray/raylet/worker_pool_test.cc
@@ -66,9 +66,9 @@ static inline TaskSpecification ExampleTaskSpec(
     const ActorID actor_id = ActorID::nil(),
     const Language &language = Language::PYTHON) {
   std::vector<std::string> function_descriptor(3);
-  return TaskSpecification(UniqueID::nil(), UniqueID::nil(), 0, ActorID::nil(),
-                           ObjectID::nil(), 0, actor_id, ActorHandleID::nil(), 0, {}, 0,
-                           {{}}, {{}}, language, function_descriptor);
+  return TaskSpecification(UniqueID::nil(), TaskID::nil(), 0, ActorID::nil(),
+                           ObjectID::nil(), 0, actor_id, ActorHandleID::nil(), 0, {}, {},
+                           0, {{}}, {{}}, language, function_descriptor);
 }
 
 TEST_F(WorkerPoolTest, HandleWorkerRegistration) {
diff --git a/test/actor_test.py b/test/actor_test.py
--- a/test/actor_test.py
+++ b/test/actor_test.py
@@ -1854,8 +1854,87 @@ def fork(queue, key, num_items):
     # Fork num_iters times.
     num_forks = 10
     num_items_per_fork = 100
-    ray.get(
-        [fork.remote(queue, i, num_items_per_fork) for i in range(num_forks)])
+
+    # Submit some tasks on new actor handles.
+    forks = [
+        fork.remote(queue, i, num_items_per_fork) for i in range(num_forks)
+    ]
+    # Submit some more tasks on the original actor handle.
+    for item in range(num_items_per_fork):
+        local_fork = queue.enqueue.remote(num_forks, item)
+    forks.append(local_fork)
+    # Wait for tasks from all handles to complete.
+    ray.get(forks)
+    # Check that all tasks from all handles have completed.
+    items = ray.get(queue.read.remote())
+    for i in range(num_forks + 1):
+        filtered_items = [item[1] for item in items if item[0] == i]
+        assert filtered_items == list(range(num_items_per_fork))
+
+
+def test_pickled_handle_consistency(setup_queue_actor):
+    queue = setup_queue_actor
+
+    @ray.remote
+    def fork(pickled_queue, key, num_items):
+        queue = ray.worker.pickle.loads(pickled_queue)
+        x = None
+        for item in range(num_items):
+            x = queue.enqueue.remote(key, item)
+        return ray.get(x)
+
+    # Fork num_iters times.
+    num_forks = 10
+    num_items_per_fork = 100
+
+    # Submit some tasks on the pickled actor handle.
+    new_queue = ray.worker.pickle.dumps(queue)
+    forks = [
+        fork.remote(new_queue, i, num_items_per_fork) for i in range(num_forks)
+    ]
+    # Submit some more tasks on the original actor handle.
+    for item in range(num_items_per_fork):
+        local_fork = queue.enqueue.remote(num_forks, item)
+    forks.append(local_fork)
+    # Wait for tasks from all handles to complete.
+    ray.get(forks)
+    # Check that all tasks from all handles have completed.
+    items = ray.get(queue.read.remote())
+    for i in range(num_forks + 1):
+        filtered_items = [item[1] for item in items if item[0] == i]
+        assert filtered_items == list(range(num_items_per_fork))
+
+
+def test_nested_fork(setup_queue_actor):
+    queue = setup_queue_actor
+
+    @ray.remote
+    def fork(queue, key, num_items):
+        x = None
+        for item in range(num_items):
+            x = queue.enqueue.remote(key, item)
+        return ray.get(x)
+
+    @ray.remote
+    def nested_fork(queue, key, num_items):
+        # Pass the actor into a nested task.
+        ray.get(fork.remote(queue, key + 1, num_items))
+        x = None
+        for item in range(num_items):
+            x = queue.enqueue.remote(key, item)
+        return ray.get(x)
+
+    # Fork num_iters times.
+    num_forks = 10
+    num_items_per_fork = 100
+
+    # Submit some tasks on new actor handles.
+    forks = [
+        nested_fork.remote(queue, i, num_items_per_fork)
+        for i in range(0, num_forks, 2)
+    ]
+    ray.get(forks)
+    # Check that all tasks from all handles have completed.
     items = ray.get(queue.read.remote())
     for i in range(num_forks):
         filtered_items = [item[1] for item in items if item[0] == i]