Repository: ray-project/ray

Issue:
Actor handle GC is problematic.
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: all
- **Ray installed from (source or binary)**: source
- **Ray version**: latest master
- **Python version**: all
- **Exact command to reproduce**: n/a

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

Currently, when the original actor handle is garbage collected by Python interpreter, a `__ray_terminate__` message will be sent to also terminate the actor, see https://github.com/ray-project/ray/blob/06f6431765b16da7cdb924cb4716a35acf0fba84/python/ray/actor.py#L662. 

This mechanism is problematic, because 
1) If there're forked handles, the actor shouldn't be terminated.
2) `__del__` is unreliable and not recommended to use, see https://stackoverflow.com/questions/1481488/what-is-the-del-method-how-to-call-it.

The first point is a critical problem and is hard to handle automatically. Even if we count the number of forked handles, and only terminate the actor when all handles are GC'ed. It's still problematic if users serialize a handle and deserialize it after a long time. Ray doesn't know when an actor is not needed any more. Only the user knows. 

Thus, I'm in favor of letting the users manually GC an actor with `actor.__ray_terminate__.remote()`, when they don't need the actor any more. To prevent the actor from wasting resource forever, there's already the mechanism that we'll clean up all workers when a driver exits.



Discussion:
@raulchen is this still an issue?
@robertnishihara Hi Robert. Can confirm, still an issue on `0.8.1`
Yeah, this is still an issue on latest master. BTW, we've removed the `__del__` function in our production code.

Resolving patch:
diff --git a/python/ray/actor.py b/python/ray/actor.py
--- a/python/ray/actor.py
+++ b/python/ray/actor.py
@@ -652,6 +652,14 @@ def __init__(self,
                     decorator=self._ray_method_decorators.get(method_name))
                 setattr(self, method_name, method)
 
+    def __del__(self):
+        # Mark that this actor handle has gone out of scope. Once all actor
+        # handles are out of scope, the actor will exit.
+        worker = ray.worker.get_global_worker()
+        if worker.connected and hasattr(worker, "core_worker"):
+            worker.core_worker.remove_actor_handle_reference(
+                self._ray_actor_id)
+
     def _actor_method_call(self,
                            method_name,
                            args=None,
@@ -752,36 +760,6 @@ def __repr__(self):
             self._ray_actor_creation_function_descriptor.class_name,
             self._actor_id.hex())
 
-    def __del__(self):
-        """Terminate the worker that is running this actor."""
-        # TODO(swang): Also clean up forked actor handles.
-        # Kill the worker if this is the original actor handle, created
-        # with Class.remote(). TODO(rkn): Even without passing handles around,
-        # this is not the right policy. the actor should be alive as long as
-        # there are ANY handles in scope in the process that created the actor,
-        # not just the first one.
-        worker = ray.worker.get_global_worker()
-        exported_in_current_session_and_job = (
-            self._ray_session_and_job == worker.current_session_and_job)
-        if (worker.mode == ray.worker.SCRIPT_MODE
-                and not exported_in_current_session_and_job):
-            # If the worker is a driver and driver id has changed because
-            # Ray was shut down re-initialized, the actor is already cleaned up
-            # and we don't need to send `__ray_terminate__` again.
-            logger.warning(
-                "Actor is garbage collected in the wrong driver." +
-                " Actor id = %s, class name = %s.", self._ray_actor_id,
-                self._ray_actor_creation_function_descriptor.class_name)
-            return
-        if worker.connected and self._ray_original_handle:
-            # Note: in py2 the weakref is destroyed prior to calling __del__
-            # so we need to set the hardref here briefly
-            try:
-                self.__ray_terminate__._actor_hard_ref = self
-                self.__ray_terminate__.remote()
-            finally:
-                self.__ray_terminate__._actor_hard_ref = None
-
     def __ray_kill__(self):
         """Deprecated - use ray.kill() instead."""
         logger.warning("actor.__ray_kill__() is deprecated and will be removed"
@@ -792,13 +770,9 @@ def __ray_kill__(self):
     def _actor_id(self):
         return self._ray_actor_id
 
-    def _serialization_helper(self, ray_forking):
+    def _serialization_helper(self):
         """This is defined in order to make pickling work.
 
-        Args:
-            ray_forking: True if this is being called because Ray is forking
-                the actor handle and false if it is being called by pickling.
-
         Returns:
             A dictionary of the information needed to reconstruct the object.
         """
@@ -807,10 +781,11 @@ def _serialization_helper(self, ray_forking):
 
         if hasattr(worker, "core_worker"):
             # Non-local mode
-            state = worker.core_worker.serialize_actor_handle(self)
+            state = worker.core_worker.serialize_actor_handle(
+                self._ray_actor_id)
         else:
             # Local mode
-            state = {
+            state = ({
                 "actor_language": self._ray_actor_language,
                 "actor_id": self._ray_actor_id,
                 "method_decorators": self._ray_method_decorators,
@@ -819,18 +794,20 @@ def _serialization_helper(self, ray_forking):
                 "actor_method_cpus": self._ray_actor_method_cpus,
                 "actor_creation_function_descriptor": self.
                 _ray_actor_creation_function_descriptor,
-            }
+            }, None)
 
         return state
 
     @classmethod
-    def _deserialization_helper(cls, state, ray_forking):
+    def _deserialization_helper(cls, state, outer_object_id=None):
         """This is defined in order to make pickling work.
 
         Args:
             state: The serialized state of the actor handle.
-            ray_forking: True if this is being called because Ray is forking
-                the actor handle and false if it is being called by pickling.
+            outer_object_id: The ObjectID that the serialized actor handle was
+                contained in, if any. This is used for counting references to
+                the actor handle.
+
         """
         worker = ray.worker.get_global_worker()
         worker.check_connected()
@@ -838,7 +815,7 @@ def _deserialization_helper(cls, state, ray_forking):
         if hasattr(worker, "core_worker"):
             # Non-local mode
             return worker.core_worker.deserialize_and_register_actor_handle(
-                state)
+                state, outer_object_id)
         else:
             # Local mode
             return cls(
@@ -855,8 +832,8 @@ def _deserialization_helper(cls, state, ray_forking):
 
     def __reduce__(self):
         """This code path is used by pickling but not by Ray forking."""
-        state = self._serialization_helper(False)
-        return ActorHandle._deserialization_helper, (state, False)
+        state = self._serialization_helper()
+        return ActorHandle._deserialization_helper, (state)
 
 
 def modify_class(cls):
diff --git a/python/ray/serialization.py b/python/ray/serialization.py
--- a/python/ray/serialization.py
+++ b/python/ray/serialization.py
@@ -135,11 +135,18 @@ def __init__(self, worker):
         self._thread_local = threading.local()
 
         def actor_handle_serializer(obj):
-            return obj._serialization_helper(True)
+            serialized, actor_handle_id = obj._serialization_helper()
+            # Update ref counting for the actor handle
+            self.add_contained_object_id(actor_handle_id)
+            return serialized
 
         def actor_handle_deserializer(serialized_obj):
+            # If this actor handle was stored in another object, then tell the
+            # core worker.
+            context = ray.worker.global_worker.get_serialization_context()
+            outer_id = context.get_outer_object_id()
             return ray.actor.ActorHandle._deserialization_helper(
-                serialized_obj, True)
+                serialized_obj, outer_id)
 
         self._register_cloudpickle_serializer(
             ray.actor.ActorHandle,
@@ -153,15 +160,7 @@ def id_deserializer(serialized_obj):
             return serialized_obj[0](*serialized_obj[1])
 
         def object_id_serializer(obj):
-            if self.is_in_band_serialization():
-                self.add_contained_object_id(obj)
-            else:
-                # If this serialization is out-of-band (e.g., from a call to
-                # cloudpickle directly or captured in a remote function/actor),
-                # then pin the object for the lifetime of this worker by adding
-                # a local reference that won't ever be removed.
-                ray.worker.get_global_worker(
-                ).core_worker.add_object_id_reference(obj)
+            self.add_contained_object_id(obj)
             owner_id = ""
             owner_address = ""
             # TODO(swang): Remove this check. Otherwise, we will not be able to
@@ -243,10 +242,20 @@ def get_and_clear_contained_object_ids(self):
         return object_ids
 
     def add_contained_object_id(self, object_id):
-        if not hasattr(self._thread_local, "object_ids"):
-            self._thread_local.object_ids = set()
-
-        self._thread_local.object_ids.add(object_id)
+        if self.is_in_band_serialization():
+            # This object ID is being stored in an object. Add the ID to the
+            # list of IDs contained in the object so that we keep the inner
+            # object value alive as long as the outer object is in scope.
+            if not hasattr(self._thread_local, "object_ids"):
+                self._thread_local.object_ids = set()
+            self._thread_local.object_ids.add(object_id)
+        else:
+            # If this serialization is out-of-band (e.g., from a call to
+            # cloudpickle directly or captured in a remote function/actor),
+            # then pin the object for the lifetime of this worker by adding
+            # a local reference that won't ever be removed.
+            ray.worker.get_global_worker().core_worker.add_object_id_reference(
+                object_id)
 
     def _deserialize_pickle5_data(self, data):
         if not self.use_pickle:
diff --git a/streaming/python/runtime/graph.py b/streaming/python/runtime/graph.py
--- a/streaming/python/runtime/graph.py
+++ b/streaming/python/runtime/graph.py
@@ -62,7 +62,7 @@ def __init__(self, task_pb):
         self.task_id = task_pb.task_id
         self.task_index = task_pb.task_index
         self.worker_actor = ray.actor.ActorHandle.\
-            _deserialization_helper(task_pb.worker_actor, False)
+            _deserialization_helper(task_pb.worker_actor)
 
 
 class ExecutionGraph:


Test patch:
diff --git a/java/test/src/main/resources/test_cross_language_invocation.py b/java/test/src/main/resources/test_cross_language_invocation.py
--- a/java/test/src/main/resources/test_cross_language_invocation.py
+++ b/java/test/src/main/resources/test_cross_language_invocation.py
@@ -34,7 +34,7 @@ def py_func_call_java_actor(value):
 @ray.remote
 def py_func_call_java_actor_from_handle(value):
     assert isinstance(value, bytes)
-    actor_handle = ray.actor.ActorHandle._deserialization_helper(value, False)
+    actor_handle = ray.actor.ActorHandle._deserialization_helper(value)
     r = actor_handle.concat.remote(b"2")
     return ray.get(r)
 
@@ -42,7 +42,7 @@ def py_func_call_java_actor_from_handle(value):
 @ray.remote
 def py_func_call_python_actor_from_handle(value):
     assert isinstance(value, bytes)
-    actor_handle = ray.actor.ActorHandle._deserialization_helper(value, False)
+    actor_handle = ray.actor.ActorHandle._deserialization_helper(value)
     r = actor_handle.increase.remote(2)
     return ray.get(r)
 
@@ -52,7 +52,7 @@ def py_func_pass_python_actor_handle():
     counter = Counter.remote(2)
     f = ray.java_function("org.ray.api.test.CrossLanguageInvocationTest",
                           "callPythonActorHandle")
-    r = f.remote(counter._serialization_helper(False))
+    r = f.remote(counter._serialization_helper())
     return ray.get(r)
 
 
diff --git a/python/ray/tests/test_actor.py b/python/ray/tests/test_actor.py
--- a/python/ray/tests/test_actor.py
+++ b/python/ray/tests/test_actor.py
@@ -106,6 +106,7 @@ class Actor(object):
 
     # The cache of ActorClassMethodMetadata.
     cache = ray.actor.ActorClassMethodMetadata._cache
+    cache.clear()
 
     # Check cache hit during ActorHandle deserialization.
     A1 = ray.remote(Actor)
@@ -532,6 +533,34 @@ def method(self):
     assert ray.get(Actor.remote().method.remote()) == 1
 
 
+def test_distributed_actor_handle_deletion(ray_start_regular):
+    @ray.remote
+    class Actor:
+        def method(self):
+            return 1
+
+        def getpid(self):
+            return os.getpid()
+
+    @ray.remote
+    def f(actor, signal):
+        ray.get(signal.wait.remote())
+        return ray.get(actor.method.remote())
+
+    signal = ray.test_utils.SignalActor.remote()
+    a = Actor.remote()
+    pid = ray.get(a.getpid.remote())
+    # Pass the handle to another task that cannot run yet.
+    x_id = f.remote(a, signal)
+    # Delete the original handle. The actor should not get killed yet.
+    del a
+
+    # Once the task finishes, the actor process should get killed.
+    ray.get(signal.send.remote())
+    assert ray.get(x_id) == 1
+    ray.test_utils.wait_for_pid_to_exit(pid)
+
+
 def test_multiple_actors(ray_start_regular):
     @ray.remote
     class Counter:
diff --git a/python/ray/tests/test_metrics.py b/python/ray/tests/test_metrics.py
--- a/python/ray/tests/test_metrics.py
+++ b/python/ray/tests/test_metrics.py
@@ -202,7 +202,7 @@ def getpid(self):
             try:
                 assert len(actor_info) == 1
                 _, parent_actor_info = actor_info.popitem()
-                assert parent_actor_info["numObjectIdsInScope"] == 11
+                assert parent_actor_info["numObjectIdsInScope"] == 13
                 assert parent_actor_info["numLocalObjects"] == 10
                 children = parent_actor_info["children"]
                 assert len(children) == 2
diff --git a/src/ray/core_worker/test/core_worker_test.cc b/src/ray/core_worker/test/core_worker_test.cc
--- a/src/ray/core_worker/test/core_worker_test.cc
+++ b/src/ray/core_worker/test/core_worker_test.cc
@@ -618,9 +618,10 @@ TEST_F(ZeroNodeTest, TestTaskSpecPerf) {
                                      /*is_detached*/ false,
                                      /*is_asyncio*/ false};
   const auto job_id = NextJobId();
-  ActorHandle actor_handle(ActorID::Of(job_id, TaskID::ForDriverTask(job_id), 1), job_id,
-                           ObjectID::FromRandom(), function.GetLanguage(), true,
-                           function.GetFunctionDescriptor(), "");
+  ActorHandle actor_handle(ActorID::Of(job_id, TaskID::ForDriverTask(job_id), 1),
+                           TaskID::Nil(), rpc::Address(), job_id, ObjectID::FromRandom(),
+                           function.GetLanguage(), true, function.GetFunctionDescriptor(),
+                           "");
 
   // Manually create `num_tasks` task specs, and for each of them create a
   // `PushTaskRequest`, this is to batch performance of TaskSpec
@@ -734,8 +735,9 @@ TEST_F(ZeroNodeTest, TestWorkerContext) {
 TEST_F(ZeroNodeTest, TestActorHandle) {
   // Test actor handle serialization and deserialization round trip.
   JobID job_id = NextJobId();
-  ActorHandle original(ActorID::Of(job_id, TaskID::ForDriverTask(job_id), 0), job_id,
-                       ObjectID::FromRandom(), Language::PYTHON, /*is_direct_call=*/false,
+  ActorHandle original(ActorID::Of(job_id, TaskID::ForDriverTask(job_id), 0),
+                       TaskID::Nil(), rpc::Address(), job_id, ObjectID::FromRandom(),
+                       Language::PYTHON, /*is_direct_call=*/false,
                        ray::FunctionDescriptorBuilder::BuildPython("", "", "", ""), "");
   std::string output;
   original.Serialize(&output);