Repository: ray-project/ray

Issue:
Not possible to install Ray from git
I want to install Ray for Python 3.

```
$ apt-get install -y cmake pkg-config python3-dev build-essential autoconf curl libtool libboost-all-dev unzip
$ pip3 install git+https://github.com/ray-project/ray.git@37282330c0ea687fd1b983176dce85731fcf189d#subdirectory=python
```

But this tries to install it for python2.7, and not python3, failing with:

```
    CMake Error at cmake_modules/FindNumPy.cmake:62 (message):
      NumPy import failure:
    
      Traceback (most recent call last):
    
        File "<string>", line 1, in <module>
    
      ImportError: No module named numpy
```

Because numpy is installed only for Python 3.


Discussion:
If you first `pip3 install numpy` then does it work?

I think there may be a couple problems here.

1. This line https://github.com/ray-project/ray/blob/master/python/setup.py#L28 in our `setup.py` needs numpy to compile a bunch of stuff, and that all happens before the setup.py has a chance to pull in the numpy dependency.

2. Even after addressing point 1 and moving the call to `build.sh` into the `build_ext` class, I'm not sure that the numpy dependency gets pulled before the `build_ext` runs, but maybe it does.
`numpy` is installed for Python 3, but not for Python 2. The issue seems to be that it is trying to install it for Python 2.7 despite being installed by `pip3`.
I see, good catch.

The problem is that when we compile Arrow, it is finding the wrong version of Python. Specifically, we need to make sure this call to `cmake` finds the right Python. https://github.com/ray-project/ray/blob/64eaaaebf02ea3e4eeab5d51cbb51d7191c191d2/src/thirdparty/build_thirdparty.sh#L34

Passing in `-DPYTHON_EXECUTABLE:FILEPATH=$PYTHON_EXECUTABLE` to the `cmake` call helps it find the right Python interpreter, but it still finds the wrong Python libraries. We can probably also pass in `-DPYTHON_LIBRARIES=`, though that's usually less easy to figure out I think.
This should be fixed on Linux by #820. However, I think it still does not work on Mac.. that may require patching the arrow cmake module for finding the Python libraries.
It does not seem to work. I still get inside Docker container:

```
pip3 install git+https://github.com/ray-project/ray.git@master#subdirectory=python
```

```
    Thread model: posix
    gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)
    
    INFOCompiler id: GNU
    Selected compiler gcc 5.4.0
    CMake Deprecation Warning at /usr/share/cmake-3.5/Modules/GenerateExportHeader.cmake:383 (message):
      The add_compiler_export_flags function is obsolete.  Use the
      CXX_VISIBILITY_PRESET and VISIBILITY_INLINES_HIDDEN target properties
      instead.
    Call Stack (most recent call first):
      CMakeLists.txt:307 (add_compiler_export_flags)
    
    
    -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
    -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
    -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
    -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
    -- Performing Test COMPILER_HAS_DEPRECATED_ATTR
    -- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
    -- Found cpplint executable at /tmp/pip-fu5ujtgs-build/src/thirdparty/arrow/cpp/build-support/cpplint.py
    -- Found PythonInterp: /usr/bin/python (found version "2.7.12")
    -- Searching for Python libs in /usr/lib64;/usr/lib;/usr/lib/python2.7/config-x86_64-linux-gnu
    -- Looking for python2.7
    -- Found Python lib /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so
    -- Found PythonLibs: /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so
    -- Looking for pthread.h
    -- Looking for pthread.h - found
    -- Looking for pthread_create
    -- Looking for pthread_create - not found
    -- Looking for pthread_create in pthreads
    -- Looking for pthread_create in pthreads - not found
    -- Looking for pthread_create in pthread
    -- Looking for pthread_create in pthread - found
    -- Found Threads: TRUE
    -- Searching for Python libs in /usr/lib64;/usr/lib;/usr/lib/python2.7/config-x86_64-linux-gnu
    -- Looking for python2.7
    -- Found Python lib /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so
    CMake Error at cmake_modules/FindNumPy.cmake:62 (message):
      NumPy import failure:
    
      Traceback (most recent call last):
    
        File "<string>", line 1, in <module>
    
      ImportError: No module named numpy
    
    Call Stack (most recent call first):
      CMakeLists.txt:780 (find_package)
    
    
    -- Configuring incomplete, errors occurred!
    See also "/tmp/pip-fu5ujtgs-build/src/thirdparty/arrow/cpp/build/CMakeFiles/CMakeOutput.log".
    See also "/tmp/pip-fu5ujtgs-build/src/thirdparty/arrow/cpp/build/CMakeFiles/CMakeError.log".
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-fu5ujtgs-build/python/setup.py", line 28, in <module>
        subprocess.check_call(["../build.sh", sys.executable])
      File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['../build.sh', '/usr/bin/python3']' returned non-zero exit status 1
```

I think the issue is that because of `libboost-all-dev`, python2.7 also gets installed. And then it finds that one. And not the one from which is being called.

If I forcefully remove python2.7 files, then it succeeds installing it.

I think just setting the path is not enough, because it might find python2.7 in the same path and continue using the wrong version.
Is `libboost-all-dev` really necessary dependency? I do not like it installing python2.7. :-(
Oh right.. if you have multiple python binaries in the same directory (both in `/usr/bin/` in this case), then you can't guarantee that it'll find the right one.

Ok, in that case the previous approach of `-DPYTHON_EXECUTABLE:FILEPATH=$PYTHON_EXECUTABLE` is preferable. Let me see if that works on Linux.

As for `libboost-all-dev`, I think we need it for compiling Arrow.
> As for libboost-all-dev, I think we need it for compiling Arrow.

But, which do you really need `all`? Or just some boost libraries.

I also asked here: https://askubuntu.com/questions/944035/installing-libboost-python-dev-for-python3-without-installing-python2-7
libboost-dev libboost-filesystem-dev libboost-system-dev should be sufficient
Perfect! Thanks. Then I can confirm that:

```
$ apt-get install -y cmake pkg-config python3-dev build-essential autoconf curl libtool libboost-dev libboost-filesystem-dev libboost-system-dev unzip
$ pip3 install git+https://github.com/ray-project/ray.git@master#subdirectory=python
```

Works.
Great! Are you interested in creating a PR that changes it? It could be changed in the locations shown below and would probably reduce travis times a bunch and make everybody's life a little happier :)

![all-libboost](https://user-images.githubusercontent.com/113316/29052000-a96c68f4-7b9b-11e7-9d3f-9128fecb6dc5.png)

Sure.
Done. See #823.

BTW, you use both Travis and Jenkins? Why?
Awesome, thanks! I'll merge as soon as the tests pass.

We use Travis because it is simple and supports Mac and Jenkins because for some tests we need more resources than are available in Travis (like more memory). Also our multinode docker tests are done in Jenkins.

Resolving patch:
diff --git a/python/setup.py b/python/setup.py
--- a/python/setup.py
+++ b/python/setup.py
@@ -81,12 +81,15 @@ def has_ext_modules(self):
       # The BinaryDistribution argument triggers build_ext.
       distclass=BinaryDistribution,
       install_requires=["numpy",
+                        "cython",
                         "funcsigs",
                         "click",
                         "colorama",
                         "psutil",
                         "redis",
                         "cloudpickle >= 0.2.2",
+                        # The six module is required by pyarrow.
+                        "six >= 1.0.0",
                         "flatbuffers"],
       entry_points={"console_scripts": ["ray=ray.scripts.scripts:main"]},
       include_package_data=True,


Test patch: