Repository: ray-project/ray

Issue:
Ray fails to compile from master branch with Python 3 only
Relevant log output:

```
    + git clone https://github.com/ray-project/catapult.git /tmp/pip-_r2ylpv0-build/src/thirdparty/catapult
    Cloning into '/tmp/pip-_r2ylpv0-build/src/thirdparty/catapult'...
    + break
    + pushd /tmp/pip-_r2ylpv0-build/src/thirdparty/catapult
    /tmp/pip-_r2ylpv0-build/src/thirdparty/catapult /tmp/pip-_r2ylpv0-build/python
    + git checkout 18cd334755701cf0c3b90b7172126c686d2eb787
    Note: checking out '18cd334755701cf0c3b90b7172126c686d2eb787'.
    
    You are in 'detached HEAD' state. You can look around, make experimental
    changes and commit them, and you can discard any commits you make in this
    state without impacting any branches by performing another checkout.
    
    If you want to create a new branch to retain commits you create, you may
    do so (now or later) by using -b with the checkout command again. Example:
    
      git checkout -b <new-branch-name>
    
    HEAD is now at 18cd334... Allow timeline scroll-to-zoom without holding ALT
    + popd
    /tmp/pip-_r2ylpv0-build/python
    + [[ ! -f /tmp/pip-_r2ylpv0-build/src/thirdparty/../../python/ray/core/src/catapult_files/index.html ]]
    + python2 /tmp/pip-_r2ylpv0-build/src/thirdparty/catapult/tracing/bin/vulcanize_trace_viewer --config chrome --output /tmp/pip-_r2ylpv0-build/src/thirdparty/../../python/ray/core/src/catapult_files/trace_viewer_full.html
    /tmp/pip-_r2ylpv0-build/src/thirdparty/build_ui.sh: line 52: python2: command not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-_r2ylpv0-build/python/setup.py", line 99, in <module>
        license="Apache 2.0")
      File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-_r2ylpv0-build/python/setup.py", line 38, in run
        subprocess.check_call(["../build.sh", sys.executable])
      File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['../build.sh', '/usr/local/bin/python3.6']' returned non-zero exit status 127.
```


Discussion:
Ah, in this case it is failing to find a `python2` executable. This could be fixed by doing `sudo apt-get install python` so that `which python2` succeeds.

Why does it need `python2`? Great question. We use it to generate the HTML page for the task timeline visualization in the web UI. This uses the catapult project https://github.com/catapult-project/catapult, which currently requires python 2. The relevant line in the codebase is

https://github.com/ray-project/ray/blob/0dcf36c91eb52b2a75d992af60c7d5ba13ba1eee/src/thirdparty/build_ui.sh#L52

Actually, now I'm confused about the error. It's failing at the line above, but it should have exited the script at 

https://github.com/ray-project/ray/blob/0dcf36c91eb52b2a75d992af60c7d5ba13ba1eee/src/thirdparty/build_ui.sh#L26

and never gotten to that line.

Could you try doing `export INCLUDE_UI=0` before building Ray?
Since we forked catapult anyway, we could just make the relevant script work with Python 3. It probably wouldn't be that hard, but I haven't looked into it yet.
It feels like catapult is not something which is really required for day-to-day use of Ray, no?
Line:

```
if [ ! type python2 > /dev/null ]; then
```

is buggy. This is not valid Bash.

`[` and `]` should not be there.

> Could you try doing `export INCLUDE_UI=0` before building Ray?

I could do it for testing, but in my case I am installing this through pip from `requirements.txt` so it might be a bit tricky to pass it all the way down.
After fixing that, the next problem is that the following files from `setup.py` are not available anymore so installation fails:

```
    "ray/core/src/catapult_files/index.html",
    "ray/core/src/catapult_files/trace_viewer_full.html",
```

Resolving patch:
diff --git a/python/setup.py b/python/setup.py
--- a/python/setup.py
+++ b/python/setup.py
@@ -23,11 +23,23 @@
     "ray/core/src/local_scheduler/local_scheduler",
     "ray/core/src/local_scheduler/liblocal_scheduler_library.so",
     "ray/core/src/global_scheduler/global_scheduler",
-    "ray/core/src/catapult_files/index.html",
-    "ray/core/src/catapult_files/trace_viewer_full.html",
     "ray/WebUI.ipynb"
 ]
 
+optional_ray_files = []
+
+ray_ui_files = [
+    "ray/core/src/catapult_files/index.html",
+    "ray/core/src/catapult_files/trace_viewer_full.html"
+]
+
+# The UI files are mandatory if the INCLUDE_UI environment variable equals 1.
+# Otherwise, they are optional.
+if "INCLUDE_UI" in os.environ and os.environ["INCLUDE_UI"] == "1":
+    ray_files += ray_ui_files
+else:
+    optional_ray_files += ray_ui_files
+
 
 class build_ext(_build_ext.build_ext):
     def run(self):
@@ -56,6 +68,14 @@ def run(self):
                 self.move_file(os.path.join(generated_python_directory,
                                             filename))
 
+        # Try to copy over the optional files.
+        for filename in optional_ray_files:
+            try:
+                self.move_file(filename)
+            except Exception as e:
+                print("Failed to copy optional file {}. This is ok."
+                      .format(filename))
+
     def move_file(self, filename):
         # TODO(rkn): This feels very brittle. It may not handle all cases. See
         # https://github.com/apache/arrow/blob/master/python/setup.py for an


Test patch: