Repository: ray-project/ray

Issue:
ray.init() crashes if pyarrow was imported before ray
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS 10.13.1
- **Ray installed from (source or binary)**: Binary wheel (pip, `ray-0.3.0-cp36-cp36m-macosx_10_6_intel.whl`)
- **Ray version**: 0.3.0
- **pyarrow version**: 0.7.1 (pip, `pyarrow-0.7.1-cp36-cp36m-macosx_10_6_intel.whl`)
- **Python version**: 3.6
- **Exact command to reproduce**: `import pyarrow; import ray; ray.init()`

### Describe the problem
If I don't import pyarrow, or import ray then pyarrow, calling ray.init() will work. Otherwise the process crashes with a segmentation fault.

Also, `pyarrow` comes before `ray` in alphabetical order 😨

### Source code / logs
```
[py: ipython3] remram 12:08:01
randy /private/tmp$ python
Python 3.6.3 (default, Oct  7 2017, 02:03:21)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
>>> import ray
>>> ray.init()
Waiting for redis server at 127.0.0.1:55607 to respond...
Waiting for redis server at 127.0.0.1:11827 to respond...
Allowing the Plasma store to use up to 13.7439GB of memory.
Starting object store with directory /tmp and huge page support disabled
Starting local scheduler with 8 CPUs, 0 GPUs

======================================================================
View the web UI at http://localhost:8896/notebooks/ray_ui34099.ipynb?token=78475c82f783b6bd1684744c9105e11a6fa623da814fb515
======================================================================

Disconnecting client on fd 9
Segmentation fault: 11

[e: 139] [py: ipython3] remram 12:08:10
randy /private/tmp$ python
Python 3.6.3 (default, Oct  7 2017, 02:03:21)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>> import pyarrow
>>> ray.init()
Waiting for redis server at 127.0.0.1:61223 to respond...
Waiting for redis server at 127.0.0.1:24990 to respond...
Allowing the Plasma store to use up to 13.7439GB of memory.
Starting object store with directory /tmp and huge page support disabled
Starting local scheduler with 8 CPUs, 0 GPUs

======================================================================
View the web UI at http://localhost:8897/notebooks/ray_ui14621.ipynb?token=4a184fb11f4cbd65674d3a4910a2b8e9d20ddb4172128476
======================================================================

{'node_ip_address': '127.0.0.1', 'redis_address': '127.0.0.1:61223', 'object_store_addresses': [ObjectStoreAddress(name='/tmp/plasma_store36375821', manager_name='/tmp/plasma_manager51291755', manager_port=31461)], 'local_scheduler_socket_names': ['/tmp/scheduler78575961'], 'webui_url': 'http://localhost:8897/notebooks/ray_ui14621.ipynb?token=4a184fb11f4cbd65674d3a4910a2b8e9d20ddb4172128476'}
>>>
```


Discussion:
Thanks for reporting the issue.

Currently we ship our own version of pyarrow with Ray. When you call `import ray`, we modify the python path so that `import pyarrow` finds our version of pyarrow. See

https://github.com/ray-project/ray/blob/c21e18937137a17205d7ee0aa6158e7058ef6796/python/ray/__init__.py#L9-L11

When you do `import pyarrow` before `import ray`, you are actually getting a different version of pyarrow. At the moment, pyarrow is undergoing a lot of development and so we rely on using pyarrow at very specific commits. Down the road we'll just install pyarrow through pip and not ship it ourselves.

Maybe what we should do here (to prevent the crash) is to check if pyarrow has already been imported and raise an exception if so (when ray is imported).
That would be ideal. Also consider mentioning this in your documentation.

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -32,8 +32,6 @@
                 "tensorflow.python",
                 "tensorflow.python.client",
                 "tensorflow.python.util",
-                "pyarrow",
-                "pyarrow.plasma",
                 "smart_open",
                 "ray.local_scheduler",
                 "ray.plasma",
diff --git a/python/ray/__init__.py b/python/ray/__init__.py
--- a/python/ray/__init__.py
+++ b/python/ray/__init__.py
@@ -4,6 +4,12 @@
 
 import os
 import sys
+
+if "pyarrow" in sys.modules:
+    raise ImportError("Ray must be imported before pyarrow because Ray "
+                      "requires a specific version of pyarrow (which is "
+                      "packaged along with Ray).")
+
 # Add the directory containing pyarrow to the Python path so that we find the
 # pyarrow version packaged with ray and not a pre-existing pyarrow.
 pyarrow_path = os.path.join(os.path.abspath(os.path.dirname(__file__)),


Test patch: