Repository: ray-project/ray

Issue:
Same function assigned different IDs on driver and worker
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 17.10
- **Ray installed from (source or binary)**: Binary
- **Ray version**: 0.40
- **Python version**: Python 3.6.5
- **Exact command to reproduce**:

### Describe the problem

ray/worker.py:compute_function_id checks whether __main__ has the __file__ attribute to decide whether to include the source code in the function hash. When starting a Python executable using ```python -m mypkg.executable```, dependencies in mypkg can be imported before __main__ is loaded, resulting in this check failing (and source code being excluded) in the driver. When it comes to running the code, however, __main__ exists. This can lead to function IDs differing between the driver and worker in nested parallelism settings.

The error I get is similar to that of #1446 so it may have a common cause. After diagnosing this I think I've become convinced that it is best for executables to be in the top-level and not nested in the package structure; I'm posting this mostly as a warning to others as it is far from obvious what caused this error.

I think we can make this more robust by using a try-except statement rather than guessing when the source code is present; I'll submit a PR.

### Source code / logs
This bug is only triggered with a particular package structure, so this needs a few files. First, create a directory foo containing:

```python3
# __init__.py
from foo import bar
```

```python3
# main.py
import ray                                                                                                                                                                                            

from foo import bar

if __name__ == '__main__':
    ray.init(redirect_worker_output=True)
    bar.run()
```

```python3
# bar.py
import ray                                                                                                                                                                                            

@ray.remote
def f(x):
    return x

def g(x):
    return f.remote(x)

@ray.remote
def h(x):
    return ray.get(g(x))

def run():
    print(ray.get(h.remote(42)))
```

Running this produces:
```
python -m foo.main
Process STDOUT and STDERR is being redirected to /tmp/raylogs/.
Waiting for redis server at 127.0.0.1:39255 to respond...
Waiting for redis server at 127.0.0.1:20998 to respond...
Starting local scheduler with the following resources: {'CPU': 12, 'GPU': 2}.

======================================================================
View the web UI at http://localhost:8889/notebooks/ray_ui70409.ipynb?token=d6d894f5459ef32b1986d4cdf348574421d1787c71960fef
======================================================================

Traceback (most recent call last):
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/adam/dev/ray/foo/main.py", line 7, in <module>
    bar.run()
  File "/home/adam/dev/ray/foo/bar.py", line 15, in run
    print(ray.get(h.remote(42)))
Remote function foo.bar.h failed with:

Traceback (most recent call last):
  File "/home/adam/dev/ray/foo/bar.py", line 12, in h
    return ray.get(g(x))
  File "/home/adam/dev/ray/foo/bar.py", line 8, in g
    return f.remote(x)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/site-packages/ray/worker.py", line 2602, in func_call
    return _submit(args=args, kwargs=kwargs)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/site-packages/ray/worker.py", line 2622, in _submit
    resources=resources)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/site-packages/ray/worker.py", line 2430, in _submit_task
    return global_worker.submit_task(function_id, *args, **kwargs)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/site-packages/ray/worker.py", line 580, in submit_task
    self.task_driver_id.id()][function_id.id()]
KeyError: b'\x8d\x04\x11\xd9\xea\xd4\xbe\xad\x90\x0c\x0b\x10\x0f^\x0cq\xcc\xe3\xb1\x07'
```
Same function assigned different IDs on driver and worker
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 17.10
- **Ray installed from (source or binary)**: Binary
- **Ray version**: 0.40
- **Python version**: Python 3.6.5
- **Exact command to reproduce**:

### Describe the problem

ray/worker.py:compute_function_id checks whether __main__ has the __file__ attribute to decide whether to include the source code in the function hash. When starting a Python executable using ```python -m mypkg.executable```, dependencies in mypkg can be imported before __main__ is loaded, resulting in this check failing (and source code being excluded) in the driver. When it comes to running the code, however, __main__ exists. This can lead to function IDs differing between the driver and worker in nested parallelism settings.

The error I get is similar to that of #1446 so it may have a common cause. After diagnosing this I think I've become convinced that it is best for executables to be in the top-level and not nested in the package structure; I'm posting this mostly as a warning to others as it is far from obvious what caused this error.

I think we can make this more robust by using a try-except statement rather than guessing when the source code is present; I'll submit a PR.

### Source code / logs
This bug is only triggered with a particular package structure, so this needs a few files. First, create a directory foo containing:

```python3
# __init__.py
from foo import bar
```

```python3
# main.py
import ray                                                                                                                                                                                            

from foo import bar

if __name__ == '__main__':
    ray.init(redirect_worker_output=True)
    bar.run()
```

```python3
# bar.py
import ray                                                                                                                                                                                            

@ray.remote
def f(x):
    return x

def g(x):
    return f.remote(x)

@ray.remote
def h(x):
    return ray.get(g(x))

def run():
    print(ray.get(h.remote(42)))
```

Running this produces:
```
python -m foo.main
Process STDOUT and STDERR is being redirected to /tmp/raylogs/.
Waiting for redis server at 127.0.0.1:39255 to respond...
Waiting for redis server at 127.0.0.1:20998 to respond...
Starting local scheduler with the following resources: {'CPU': 12, 'GPU': 2}.

======================================================================
View the web UI at http://localhost:8889/notebooks/ray_ui70409.ipynb?token=d6d894f5459ef32b1986d4cdf348574421d1787c71960fef
======================================================================

Traceback (most recent call last):
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/adam/dev/ray/foo/main.py", line 7, in <module>
    bar.run()
  File "/home/adam/dev/ray/foo/bar.py", line 15, in run
    print(ray.get(h.remote(42)))
Remote function foo.bar.h failed with:

Traceback (most recent call last):
  File "/home/adam/dev/ray/foo/bar.py", line 12, in h
    return ray.get(g(x))
  File "/home/adam/dev/ray/foo/bar.py", line 8, in g
    return f.remote(x)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/site-packages/ray/worker.py", line 2602, in func_call
    return _submit(args=args, kwargs=kwargs)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/site-packages/ray/worker.py", line 2622, in _submit
    resources=resources)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/site-packages/ray/worker.py", line 2430, in _submit_task
    return global_worker.submit_task(function_id, *args, **kwargs)
  File "/home/adam/bin/anaconda3/envs/mypirl/lib/python3.6/site-packages/ray/worker.py", line 580, in submit_task
    self.task_driver_id.id()][function_id.id()]
KeyError: b'\x8d\x04\x11\xd9\xea\xd4\xbe\xad\x90\x0c\x0b\x10\x0f^\x0cq\xcc\xe3\xb1\x07'
```


Discussion:

Resolving patch:
diff --git a/python/ray/remote_function.py b/python/ray/remote_function.py
--- a/python/ray/remote_function.py
+++ b/python/ray/remote_function.py
@@ -14,15 +14,6 @@
 DEFAULT_REMOTE_FUNCTION_MAX_CALLS = 0
 
 
-def in_ipython():
-    """Return true if we are in an IPython interpreter and false otherwise."""
-    try:
-        __IPYTHON__
-        return True
-    except NameError:
-        return False
-
-
 def compute_function_id(function):
     """Compute an function ID for a function.
 
@@ -36,14 +27,14 @@ def compute_function_id(function):
     # Include the function module and name in the hash.
     function_id_hash.update(function.__module__.encode("ascii"))
     function_id_hash.update(function.__name__.encode("ascii"))
-    # If we are running a script or are in IPython, include the source code in
-    # the hash. If we are in a regular Python interpreter we skip this part
-    # because the source code is not accessible. If the function is a built-in
-    # (e.g., Cython), the source code is not accessible.
-    import __main__ as main
-    if (hasattr(main, "__file__") or in_ipython()) \
-            and inspect.isfunction(function):
-        function_id_hash.update(inspect.getsource(function).encode("ascii"))
+    try:
+        # If we are running a script or are in IPython, include the source code
+        # in the hash.
+        source = inspect.getsource(function).encode("ascii")
+        function_id_hash.update(source)
+    except (IOError, OSError, TypeError):
+        # Source code may not be available: e.g. Cython or Python interpreter.
+        pass
     # Compute the function ID.
     function_id = function_id_hash.digest()
     assert len(function_id) == 20


Test patch: