Repository: ray-project/ray

Issue:
`ray.get` on cluster mode sometimes does not return
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux	
- **Ray installed from (source or binary)**: wheels
- **Ray version**:  0.8.0.dev3
- **Python version**: 3.6
- **Exact command to reproduce**:
With 2 nodes:

```ipython3
In [1]: import ray

In [2]: ray.init(redis_address="localhost:6379")
2019-08-01 02:57:18,898	WARNING worker.py:1372 -- WARNING: Not updating worker name since `setproctitle` is not installed. Install this with `pip install setproctitle` (or ray[debug]) to enable monitoring of worker processes.
Out[2]:
{'node_ip_address': '172.31.95.217',
 'redis_address': '172.31.95.217:6379',
 'object_store_address': '/tmp/ray/session_2019-08-01_02-55-05_728763_2867/sockets/plasma_store',
 'raylet_socket_name': '/tmp/ray/session_2019-08-01_02-55-05_728763_2867/sockets/raylet',
 'webui_url': None,
 'session_dir': '/tmp/ray/session_2019-08-01_02-55-05_728763_2867'}

In [3]: @ray.remote
   ...: def test():
   ...:     print("hello!")
   ...:     return 123
   ...:
   ...:

In [4]: ray.get(test.remote())
(pid=2896) hello!
Out[4]: 123

In [5]: ray.get(test.remote())
(pid=2833, ip=172.31.89.59) hello!
```

Sometimes, `ray.get` does not return. 

```yaml
# An unique identifier for the head node and workers of this cluster.
cluster_name: sgd-pytorch

# The maximum number of workers nodes to launch in addition to the head
# node. This takes precedence over min_workers. min_workers default to 0.
min_workers: 1
initial_workers: 1
max_workers: 1

target_utilization_fraction: 0.9

# If a node is idle for this many minutes, it will be removed.
idle_timeout_minutes: 20
provider:
    type: aws
    region: us-east-1
    availability_zone: us-east-1f

auth:
    ssh_user: ubuntu

head_node:
    InstanceType: c5.xlarge
    ImageId: ami-0d96d570269578cd7

worker_nodes:
    InstanceType: c5.xlarge
    ImageId: ami-0d96d570269578cd7

setup_commands:
    - pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev3-cp36-cp36m-manylinux1_x86_64.whl

file_mounts: {}

# Custom commands that will be run on the head node after common setup.
head_setup_commands: []

# Custom commands that will be run on worker nodes after common setup.
worker_setup_commands: []

# # Command to start ray on the head node. You don't need to change this.
head_start_ray_commands:
    - ray stop
    - ray start --head --redis-port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml --object-store-memory=1000000000

# Command to start ray on worker nodes. You don't need to change this.
worker_start_ray_commands:
    - ray stop
    - ray start --redis-address=$RAY_HEAD_IP:6379 --object-manager-port=8076 --object-store-memory=1000000000
```


Discussion:
you can run `ray up example-sgd.yaml --restart-only` over again and `ray exec example-sgd.yaml  'ipython'` for fast testing.
I'm not really sure what's happening here, but it seems to have something to do with the gRPC change in the object manager. With `DEBUG` messages on, I see this on the driver's node:
```
I0801 05:08:49.677681  7593 object_manager.cc:202] Sending pull request from 2b40211fa4822dae803472cbffaa1964743f499c to 7ff8904fdde34990457c041b537723df072ed05b of object 9000448a466ea5adf7ea7c12b3b9828c01000000
I0801 05:08:49.677784  7593 object_manager.cc:797] Get rpc client, address: 172.31.83.159, port: 8076, local port: 8076
```
The remote node receives the `PullRequest` message and seems to send the object (which is already in the remote node's object store):
```
I0801 05:08:49.681157  6323 object_manager.cc:720] Received pull request from client 2b40211fa4822dae803472cbffaa1964743f499c for object [9000448a466ea5adf7ea7c12b3b9828c01000000].
I0801 05:08:49.681197  6312 object_manager.cc:329] Push on 7ff8904fdde34990457c041b537723df072ed05b to 2b40211fa4822dae803472cbffaa1964743f499c of object 9000448a466ea5adf7ea7c12b3b9828c01000000
I0801 05:08:49.681258  6312 object_manager.cc:797] Get rpc client, address: 172.31.86.139, port: 8076, local port: 8076
I0801 05:08:49.681265  6312 object_manager.cc:395] Sending object chunks of 9000448a466ea5adf7ea7c12b3b9828c01000000 to client 2b40211fa4822dae803472cbffaa1964743f499c, number of chunks: 1, total data size: 448
I0801 05:08:49.681928  6312 object_manager.cc:284] HandleSendFinished on 7ff8904fdde34990457c041b537723df072ed05b to 2b40211fa4822dae803472cbffaa1964743f499c of object 9000448a466ea5adf7ea7c12b3b9828c01000000 chunk 0, status: OK
```

But then the driver's node doesn't seem to receive the pushed object. I don't see any messages with `ReceiveObjectChunk` in them, and the driver's node eventually tries to send the `PullRequest` again.
@raulchen @jiangzihao2009
I traced down the real issue. So this bug only occurs after https://github.com/ray-project/ray/pull/5120. However, the real issue seems to be somehow `ray stop` is unable to stop the `default_worker` and `raylet` processes. I'm suspecting that `kill` failed for some reason; or gRPC thread doesn't respond to signals correctly. 

Checking... 
Steps to reproduce the ray stop bug 

https://gist.github.com/simon-mo/7194824e161f336d699d9a0bcb65c13e

Resolving patch:
diff --git a/python/ray/scripts/scripts.py b/python/ray/scripts/scripts.py
--- a/python/ray/scripts/scripts.py
+++ b/python/ray/scripts/scripts.py
@@ -397,8 +397,8 @@ def stop():
     ]
 
     for process in processes_to_kill:
-        command = ("kill $(ps aux | grep '" + process + "' | grep -v grep | " +
-                   "awk '{ print $2 }') 2> /dev/null")
+        command = ("kill -9 $(ps aux | grep '" + process +
+                   "' | grep -v grep | " + "awk '{ print $2 }') 2> /dev/null")
         subprocess.call([command], shell=True)
 
     # Find the PID of the jupyter process and kill it.


Test patch: