Repository: ray-project/ray

Issue:
Cloning catapult takes way too long. Speed this up.
Currently when using the web UI, the first time you click on "View Task Timeline", the catapult repository https://github.com/catapult-project/catapult will be cloned. This takes on the order of **minutes**. This needs to be faster.

Some thoughts and questions:
1. A quick looks suggests that the catapult repo is about 270MB, and 265MB of that is in the .git file, so if we could just clone a snapshot of the repo at the right commit without getting the git history that would help a lot (the remaining 5MB seems to be scattered around large html files).
2. Naively packaging catapult in our wheels would make our wheels enormous and slow to download, however, it should be ok if we first remove the git history.
3. Question, how much of catapult do we actually need?

So, if possible, a great short-term fix would be to figure out how to get only a snapshot of the repo without the git history.

Later, we should just the repo with our wheels after we've removed the git history and ideally other unnecessary files as well. For people building Ray from source, we can clone catapult in `build.sh` and cache it.

@alanamarzoev @Wapaul1 any thoughts about this?


Discussion:
@robertnishihara afaik, you should only need catapult tracing. 

I think that just taking a snapshot of the repo should work.
Right. I googled a little bit and didn't find a way to do that, but I'm sure there is one..
`git clone --depth=1` on catapult results in a reduction of 160 MB in the `.git` directory to a total download size of 107 MB. While this is better, we may want to consider forking catapult and keeping all the important files.
It would also be nice to print "this might take a while" in the UI so that users don't get confused.
We should look into speeding it up by extracting/minifying the relevant parts and deploying with the python wheel, bower/vulcanize should have an option for that.
I think that either cloning catapult or extracting the relevant files from catapult and including them with the python wheel/in the build.sh script would be better than what we're currently doing. Even if it ends up not being significantly faster, it at least wouldn't interfere with the perceived interactivity of the UI. 

Resolving patch:
diff --git a/python/ray/experimental/ui.py b/python/ray/experimental/ui.py
--- a/python/ray/experimental/ui.py
+++ b/python/ray/experimental/ui.py
@@ -4,7 +4,6 @@
 import pprint
 import ray
 import shutil
-import subprocess
 import tempfile
 import time
 
@@ -289,65 +288,6 @@ def _get_temp_file_path(**kwargs):
     return os.path.relpath(temp_file_path)
 
 
-# Helper function that ensures that catapult is cloned to the correct location
-# and that the HTML files required for task trace embedding are in the same
-# directory as the web UI.
-def _setup_trace_dependencies():
-    catapult_home = "/tmp/ray/catapult"
-    catapult_commit = "33a9271eb3cf5caf925293ec6a4b47c94f1ac968"
-    try:
-        # Check if we're inside a git repo
-        cmd = ["git",
-               "-C",
-               catapult_home,
-               "rev-parse",
-               "--is-inside-work-tree"]
-        subprocess.check_call(cmd)
-
-    except subprocess.CalledProcessError:
-        # Error on non-zero exit code (e.g. - ".git not found")
-        if not os.path.exists(os.path.join(catapult_home)):
-            print(
-                "Cloning catapult to {} (this may take a while...)".format(
-                    catapult_home))
-            cmd = ["git",
-                   "clone",
-                   "https://github.com/catapult-project/catapult.git",
-                   catapult_home]
-            subprocess.check_call(cmd)
-
-        # Checks out the commit associated with allowing different arrow
-        # colors. This can and should be removed after catapult's next
-        # release.
-        print("Checking out commit {}.".format(catapult_commit))
-        cmd = ["git", "-C", catapult_home, "checkout", catapult_commit]
-        subprocess.check_call(cmd)
-
-    # Path to the embedded trace viewer HTML file.
-    embedded_trace_path = os.path.join(catapult_home,
-                                       "tracing",
-                                       "bin",
-                                       "index.html")
-    # Checks that the trace viewer renderer file exists, generates it if it
-    # doesn't.
-    if not os.path.exists("trace_viewer_full.html"):
-        vulcanize_bin = os.path.join(catapult_home,
-                                     "tracing",
-                                     "bin",
-                                     "vulcanize_trace_viewer")
-        # TODO(rkn): The vulcanize_trace_viewer script currently requires
-        # Python 2. Remove this dependency.
-        cmd = ["python2",
-               vulcanize_bin,
-               "--config",
-               "chrome",
-               "--output",
-               "trace_viewer_full.html"]
-        subprocess.check_call(cmd)
-
-    return catapult_home, embedded_trace_path
-
-
 def task_timeline():
     path_input = widgets.Button(description="View task timeline")
 
@@ -381,6 +321,14 @@ def task_timeline():
     display(widgets.HBox([label_options, breakdown_opt]))
     display(path_input)
 
+    # Check that the trace viewer renderer file is present, and copy it to the
+    # current working directory if it is not present.
+    if not os.path.exists("trace_viewer_full.html"):
+        shutil.copy(
+            os.path.join(os.path.dirname(os.path.abspath(__file__)),
+                         "../core/src/catapult_files/trace_viewer_full.html"),
+            "trace_viewer_full.html")
+
     def handle_submit(sender):
         json_tmp = tempfile.mktemp() + ".json"
 
@@ -425,9 +373,9 @@ def handle_submit(sender):
 
         print("Opening html file in browser...")
 
-        # Check that the catapult repo is cloned to the correct location
-        print(_setup_trace_dependencies())
-        catapult_home, trace_viewer_path = _setup_trace_dependencies()
+        trace_viewer_path = os.path.join(
+            os.path.dirname(os.path.abspath(__file__)),
+            "../core/src/catapult_files/index.html")
 
         html_file_path = _get_temp_file_path(suffix=".html")
         json_file_path = _get_temp_file_path(suffix=".json")
diff --git a/python/setup.py b/python/setup.py
--- a/python/setup.py
+++ b/python/setup.py
@@ -23,6 +23,8 @@
     "ray/core/src/local_scheduler/local_scheduler",
     "ray/core/src/local_scheduler/liblocal_scheduler_library.so",
     "ray/core/src/global_scheduler/global_scheduler",
+    "ray/core/src/catapult_files/index.html",
+    "ray/core/src/catapult_files/trace_viewer_full.html",
     "ray/WebUI.ipynb"
 ]
 


Test patch: