Repository: ray-project/ray

Issue:
Ray kill remote tasks
Is there any python interface for terminating remote tasks?
In some situations, one may need to stop some tasks before they finish.
For example, we may create several NNs to test a dataset.

cnn1 = CNN1.remote()
cnn2 = CNN2.remote()
lstm1 = LSTM1.remote()
lstm2 = LSTM2.remote()
...

then we use `ray.wait` to wait for these tasks.
and results (given by some tasks finished earlier) show that CNN is much better than LSTM for this dataset. So we would like to kill most LSTM tasks (we know their object_ids) to make more space for CNNs.

We wonder how to do that with ray. 
@mitar 


Discussion:
Currently there is no interface for terminating tasks, but I agree this could be really useful.

One workaround would be to take your training tasks and divide them into smaller segments (e.g., 10 second chunks). Then whenever one training segment finishes, you can decide to launch a new training segment (picking up where the previous one left off) or to stop training that model.

Does that achieves what you're looking for?

As for canceling tasks, what sort of API do you think would make sense? E.g., `ray.cancel(object_id)` or something like that?
I would suggest provide some mechanism for sending messages and then task can check periodically if there is a message. Because in our case we might want to terminate a task, but also maybe just pause a task. So different messages could be beneficial for this. So API could be:

```python
ray.send_message(object_id, {message})
```

If there is no task running for `object_id`, nothing happens (or maybe a return value could signal that).

On receiving end, one could do something like `ray.get_messages()` to return any current pending messages available for the current task.
I see. You could certainly try that out (e.g., hijacking the existing Redis server for that purpose). I wouldn't want to encourage anyone to actually do this though, but we can see if that provides the functionality that you're looking for.

Something like the following.

```python
def broadcast_message(key, message):
    ray.global_worker.redis_client.set(key, message)

def check_message(key):
    return ray.global_worker.redis_client.get(key)
```

E.g.,

```python
import ray
import time

ray.init()

def broadcast_message(key, message):
    ray.worker.global_worker.redis_client.set(key, message)

def check_message(key):
    return ray.worker.global_worker.redis_client.get(key)

@ray.remote
def f():
    while True:
        if check_message("XXX") == b"stop":
            break
        time.sleep(1)

object_id = f.remote()

broadcast_message("XXX", b"stop")

# This should return
ray.get(object_id)
```
A good way to send messages to tasks in the current API is to use actors. Each training task is a long running actor which has a train() method that does the training for a few steps (which takes say a few seconds); you can then send messages to the actor by invoking methods on it, for example for killing it or for changing some training settings, etc. These can be invoked in between the training tasks.

-- Philipp.
Does a task have access to its ObjectId? Maybe with some context manager?

@pcmoritz: But wouldn't this introduce some latency? After each iteration, method has to return to the caller, then caller decides if to continue? Wouldn't be easier if method just check for any messages which arrived in meantime and then decide locally?
While a task is executing, it does not have access to the object IDs of its return values. What would you use that for?

As for latency, presumably you would train for 10 seconds or a minute and then you would have 1 millisecond of latency and then do another 10 seconds of training and so it probably wouldn't be an issue.
BTW, if we just leave the actor there (stop calling any function of the actor), will the actor try to release all the resources it used? 

For example, when we initialize an actor, it creates a tf session and occupies some resources of GPU. Then we call `step()` of the actor or `keep_training_until_receive_msg()`. After breaking the training process, will it still stay in GPU until we manually close the tensorflow session in the actor?
Once a process using TensorFlow starts to use space on a GPU, it never releases it until the process is killed (see #616), so the actor will continue to occupy space on the GPU until the actor is killed. The actor will naturally be killed when the actor handle goes out of scope in Python.

E.g., if you do

```python
actor = CNN1.remote()

del actor
```

Then the call to `del actor` will send a task to the actor that will cause the actor to kill itself when the task runs.
While we still haven't implemented regular cancelation, now that we have distributed actor handles, it is possible to work around it by using an actor to send messages from the driver to the task. 

Note that this requires the cooperation of the task itself. For example,

```python
import ray
import time

ray.init()

@ray.remote
class TaskCanceler(object):
    def __init__(self):
        self.task_canceled = False

    def cancel_task(self):
        """The driver can call this to cancel the task."""
        self.task_canceled = True

    def is_task_canceled(self):
        """The task calls this to check if it is canceled."""
        return self.task_canceled

@ray.remote
def f(task_canceler):
    i = 0
    while True:
        print(i)
        i += 1
        time.sleep(1)
        # Check if the task should abort.
        if ray.get(task_canceler.is_task_canceled.remote()):
            return

task_canceler = TaskCanceler.remote()

# Create the task.
f.remote(task_canceler)

# Cancel the task.
task_canceler.cancel_task.remote()
```

And of course this can be generalized in a lot of ways (e.g., to use one actor to selectively cancel many different tasks).
This has just again turned up as an issue for our project. I realized that in fact besides cooperative termination also forceful termination would be useful. Sometimes there is simply no small iterations to have (like in traditional ML algorithms) to check some status.

So I would ask for some way to kill a task forcefully, which would probably just terminate whole worker (I assume each worker is running only one task at a time, no). No matter the state or anything.
I also need to kill the worker via API, it would be very handy.

I'd start remote workers, then I'd do .wait with time limit and then I would like to kill the workers that didn't finish.

I tried implementing cooperative stopping in the workers but that's very hard, because the worker is calling compiled 3rd party simulator. 
Hi guys,

Any update on this? Do we have any APIs to kill the remote workers yet? 
I've ended up doing this by running my task in an actor on a background thread, a la

```python
import ray
import threading
import time

@ray.remote
class Task:

    def __init__(self):
        self._thread = threading.Thread(target=self._run, daemon=True)
        self._thread.start()
    
    def _run(self):
        print('Started long-running task')
        time.sleep(10)
        print('You shouldn\'t see this because the process has been killed')

ray.init()
task = Task.remote()
task.__ray_terminate__.remote()   # kills the process with an `exit(0)`
```

This should work as long as your long-running task doesn't hang on `ray.get`/`ray.wait`/the GIL.

`__ray_terminate__` will also be called when the original actor handle goes out of scope, so you don't have to call it explicitly if you don't want to.
You can now kill actors while they're still processing tasks using the `actor.__ray_kill__()`. This is similar to the solution @andyljones posted except it will work even if the actor is blocking on `ray.get()`, `ray.wait()`, or the GIL.
What about non-actors? So a simple Ray task?
(BTW, any pointers how this is implemented for actors?)
Not currently supported for tasks. See the original PR for details:
https://github.com/ray-project/ray/pull/6523
So then it would be reasonable to turn all of my tasks into actors? That looks like a trivial thing to do.

BTW, why is method called `__ray_kill__`? This is generally reserved for Python standard method names for interfaces which have language-based syntax to access. Like if there would be `kill(actor)` function which would then internally call `__ray_kill__` or something.
That seems like a reasonable workaround for now. That's a good point about the API - what would you think of `ray.kill_actor(<actor>)` instead?
What about just `ray.kill(<actor>)`? In the future it could also support killing tasks.

BTW, any reason why it would not be as easy to expose API to kill tasks as well? I am not sure how much overhead is to have an actor instead of a task?
It would be as easy to expose the API, but not to actually implement it. This is because anyone with a handle to an actor knows its location (RPC address) and can send a kill message, but to kill a task given only an `ObjectID` you would first have to look up where the task is running, which is possible but somewhat complicated and not currently supported.
@mitar 
@suquark 

We'd love to setup a quick call to go over your requirements here and better understand your use case. We're sketching a design and it'd be helpful to have your input. Can we setup some time in the next week or so? shoot us an email my first name at anyscale.com

Resolving patch:
diff --git a/python/ray/worker.py b/python/ray/worker.py
--- a/python/ray/worker.py
+++ b/python/ray/worker.py
@@ -1683,7 +1683,7 @@ def kill(actor):
 
 
 def cancel(object_id, force=False):
-    """Cancels a locally-submitted task according to the following conditions.
+    """Cancels a task according to the following conditions.
 
     If the specified task is pending execution, it will not be executed. If
     the task is currently executing, the behavior depends on the ``force``
@@ -1702,8 +1702,7 @@ def cancel(object_id, force=False):
         force (boolean): Whether to force-kill a running task by killing
             the worker that is running the task.
     Raises:
-        ValueError: This is also raised for actor tasks, already completed
-            tasks, and non-locally submitted tasks.
+        TypeError: This is also raised for actor tasks.
     """
     worker = ray.worker.global_worker
     worker.check_connected()


Test patch:
diff --git a/python/ray/tests/test_cancel.py b/python/ray/tests/test_cancel.py
--- a/python/ray/tests/test_cancel.py
+++ b/python/ray/tests/test_cancel.py
@@ -228,5 +228,31 @@ def wait_for(y):
             assert isinstance(e, valid_exceptions(use_force))
 
 
+@pytest.mark.parametrize("use_force", [True, False])
+def test_remote_cancel(ray_start_regular, use_force):
+    signaler = SignalActor.remote()
+
+    @ray.remote
+    def wait_for(y):
+        return ray.get(y[0])
+
+    @ray.remote
+    def remote_wait(sg):
+        return [wait_for.remote([sg[0]])]
+
+    sig = signaler.wait.remote()
+
+    outer = remote_wait.remote([sig])
+    inner = ray.get(outer)[0]
+
+    with pytest.raises(RayTimeoutError):
+        ray.get(inner, 1)
+
+    ray.cancel(inner)
+
+    with pytest.raises(valid_exceptions(use_force)):
+        ray.get(inner, 10)
+
+
 if __name__ == "__main__":
     sys.exit(pytest.main(["-v", __file__]))