Repository: ray-project/ray

Issue:
Error message on ray init when run inside Jupyter: setpgrp failed, processes may not be cleaned up properly
### What is the problem?

*Ray version and other system information (Python version, TensorFlow version, OS):*

Ray 0.8.0
Python 3.6
Ubuntu 18.04

*Does the problem occur on the [latest wheels](https://ray.readthedocs.io/en/latest/installation.html)?*

Yes, with ray-0.9.0.dev0.

### Reproduction
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

If we cannot run your script, we cannot fix your issue.

I am running juypyterlab 1.2.4 within docker, using an 18.04 base image. To reproduce, simply import ray and call ray.init.

```python
import ray
ray.init()
```

And the error message will be generated.

```
2020-01-06 17:54:50,339	WARNING services.py:595 -- setpgrp failed, processes may not be cleaned up properly: [Errno 1] Operation not permitted.
```

PID, PGID, PPID, and SID all have the same value in this case. Outside of Jupyter, this error does not occur.


Discussion:
Thanks for reporting this @mc-allen. After looking into this a bit, I was only able to reproduce in jupyterlab and unfortunately am not able to find a workaround.

This shouldn't cause any significant problems, it just means that if the python process that called `ray.init()` (the jupyter notebook in this case) is terminated via `SIGKILL`, there may be some background processes left up. If this does happen for some reason, you can always clean these processes up manually using `ray stop`.
Is "ray stop" the same effect as "ray.shutdown()"?
`ray stop` is a bash command and will stop all ray processes running on the machine (started by `ray.init()` in python or `ray start` on the command line.

`ray.shutdown()` will stop all ray processes started by a previous `ray.init()` call in the current python process.
I see. I'm looking for something I can call from python that will release all ray resources, including object store shared memory, and ideally redis too, and kill all tasks. This is needed in the event of a crash or restart of a script.
If ray was started from within the process that crashed, resources should be cleaned up automatically.

Also, not sure if this exactly fits the bill, but if you call `ray start` with `--block`, it will clean up all processes started by ray of any of them die (e.g., if redis goes down or the raylet crashes) 
Sorry if this is the wrong place to continue this conversation, but...

In my experience, it seems like ray.shutdown() does not clear ray's usage of /dev/shm. Is there a surefire way to accomplish this?
The `/dev/shm` usage will be cleared once the object store goes down. This should be done by `ray.shutdown()` if you called `ray.init()` in the same process, otherwise you'll have to call `ray stop`.

This looks very similar to this [os.setsid](https://stackoverflow.com/questions/25701333/os-setsid-operation-not-permitted) question where python raises an exception if the value is already set.

Could this be a simple as changing
`    os.setpgrp()`
to
`    if os.getpgrp() != os.getpid(): os.setpgrp()`

This worked for me on Ubuntu 18.04.
Thanks @hubcity!

@mehrdadn do you think you could take a look when you have a chance?
Yeah I'll try to think about it, it seems like a nontrivial problem but there might be a solution.

Resolving patch:
diff --git a/python/ray/services.py b/python/ray/services.py
--- a/python/ray/services.py
+++ b/python/ray/services.py
@@ -1,4 +1,5 @@
 import collections
+import errno
 import json
 import logging
 import multiprocessing
@@ -282,10 +283,10 @@ def get_node_ip_address(address="8.8.8.8:53"):
         # connection.
         s.connect((ip_address, int(port)))
         node_ip_address = s.getsockname()[0]
-    except Exception as e:
+    except OSError as e:
         node_ip_address = "127.0.0.1"
         # [Errno 101] Network is unreachable
-        if e.errno == 101:
+        if e.errno == errno.ENETUNREACH:
             try:
                 # try get node ip address from host name
                 host_name = socket.getfqdn(socket.gethostname())
@@ -582,11 +583,15 @@ def start_reaper():
     try:
         os.setpgrp()
     except OSError as e:
-        logger.warning("setpgrp failed, processes may not be "
-                       "cleaned up properly: {}.".format(e))
-        # Don't start the reaper in this case as it could result in killing
-        # other user processes.
-        return None
+        if e.errno == errno.EPERM and os.getpgrp() == os.getpid():
+            # Nothing to do; we're already a session leader.
+            pass
+        else:
+            logger.warning("setpgrp failed, processes may not be "
+                           "cleaned up properly: {}.".format(e))
+            # Don't start the reaper in this case as it could result in killing
+            # other user processes.
+            return None
 
     reaper_filepath = os.path.join(
         os.path.dirname(os.path.abspath(__file__)), "ray_process_reaper.py")


Test patch: