Repository: skypilot-org/skypilot
Pull request: Keep wheels build up-to-date
Actual testing is by manually launching and ssh into the cluster to see if the installed sky get updated.

A simplified testing procedure:

1. Under the directory containing `setup.py`, run `pip wheel --no-deps . --wheel-dir /tmp`. This generates the wheel `/tmp/sky-*.whl`
2. Use `unzip` command to see the content of the wheel.
3. Delete `build` in the current directory, the wheel and unzipped files in `/tmp`
4. Update your code, repeat 1-2 to see if the content in the wheel includes updated code.

Closes #296
Closes #260

Problem statement:
`sky launch` on existing cluster will not update the `ray_patches`
I am editing the files in skylet/ray_patches, but it seems we will not update the `ray_patches`. Do you have any idea of how to solve this @suquark ?

Ensure `sky launch` update skylet code on an existing cluster
@Michaelvll @suquark @infwinston

Hints from issue discussion:
related to #260

Commit message: keep wheels build up-to-date

Patch to review:
diff --git a/prototype/config/aws-ray.yml.j2 b/prototype/config/aws-ray.yml.j2
index 23e10f8b61f..4c6f4b1a989 100644
--- a/prototype/config/aws-ray.yml.j2
+++ b/prototype/config/aws-ray.yml.j2
@@ -89,7 +89,7 @@ setup_commands:
   # This AMI's system Python is version 2+.
   # _gang_schedule_ray_up() requires first two lines; do not modify.
   - pip3 install -U ray[default]=={{ray_version}} && mkdir -p ~/sky_workdir && mkdir -p ~/.sky/sky_app
-  - pip3 install {{sky_remote_path}}/*.whl && python3 -c "from sky.skylet.ray_patches import patch; patch()" # patch the buggy ray file
+  - pip3 uninstall sky -y &> /dev/null; pip3 install {{sky_remote_path}}/*.whl && python3 -c "from sky.skylet.ray_patches import patch; patch()" # patch the buggy ray file
 {%- if setup_sh_path is not none %}
   - cd ~/sky_workdir && bash /tmp/setup.sh  # FIXME: /tmp is volatile.
 {%- endif %}
diff --git a/prototype/config/azure-ray.yml.j2 b/prototype/config/azure-ray.yml.j2
index d4f8a6a373d..8402743abca 100644
--- a/prototype/config/azure-ray.yml.j2
+++ b/prototype/config/azure-ray.yml.j2
@@ -95,7 +95,7 @@ setup_commands:
   # This AMI's system Python is version 2+.
   # _gang_schedule_ray_up() requires first two lines; do not modify.
   - pip3 install -U ray[default]=={{ray_version}} && mkdir -p ~/sky_workdir && mkdir -p ~/.sky/sky_app && touch ~/.sudo_as_admin_successful
-  - pip3 install {{sky_remote_path}}/*.whl && python3 -c "from sky.skylet.ray_patches import patch; patch()" # patch the buggy ray file
+  - pip3 uninstall sky -y &> /dev/null; pip3 install {{sky_remote_path}}/*.whl'[azure]' && python3 -c "from sky.skylet.ray_patches import patch; patch()" # patch the buggy ray file
 {%- if setup_sh_path is not none %}
   - cd ~/sky_workdir && bash /tmp/setup.sh  # FIXME: /tmp is volatile.
 {%- endif %}
diff --git a/prototype/config/gcp-ray.yml.j2 b/prototype/config/gcp-ray.yml.j2
index 8048d52c06e..40a173e7d02 100644
--- a/prototype/config/gcp-ray.yml.j2
+++ b/prototype/config/gcp-ray.yml.j2
@@ -121,7 +121,7 @@ rsync_filter:
 setup_commands:
   # _gang_schedule_ray_up() requires first two lines; do not modify.
   - pip3 install -U ray[default]=={{ray_version}} && mkdir -p ~/sky_workdir && mkdir -p ~/.sky/sky_app
-  - pip3 install {{sky_remote_path}}/*.whl && python3 -c "from sky.skylet.ray_patches import patch; patch()" # patch the buggy ray file
+  - pip3 uninstall sky -y &> /dev/null; pip3 install {{sky_remote_path}}/*.whl && python3 -c "from sky.skylet.ray_patches import patch; patch()" # patch the buggy ray file
 {%- if setup_sh_path is not none %}
   - cd ~/sky_workdir && bash /tmp/setup.sh  # FIXME: /tmp is volatile.
 {%- endif %}
diff --git a/prototype/sky/backends/backend_utils.py b/prototype/sky/backends/backend_utils.py
index b7b8ddedc89..54fc323e506 100644
--- a/prototype/sky/backends/backend_utils.py
+++ b/prototype/sky/backends/backend_utils.py
@@ -13,6 +13,7 @@
 import uuid
 import yaml
 import zlib
+import shutil
 
 import jinja2
 
@@ -327,6 +328,11 @@ def _build_sky_wheel() -> pathlib.Path:
     norm_path = str(package_root) + os.sep
     username = getpass.getuser()
     wheel_dir = pathlib.Path(tempfile.gettempdir()) / f'sky_wheels_{username}'
+    # remove old wheels
+    for f in wheel_dir.glob('sky-*.whl'):
+        f.unlink()
+    # remove pip wheels build directory, otherwise we may include outdated code
+    shutil.rmtree(str(package_root / 'build'), ignore_errors=True)
     try:
         # TODO(suquark): For python>=3.7, 'subprocess.run' supports capture of
         # the output.


Code review comments:
diff_hunk: @@ -95,7 +95,7 @@ setup_commands:
   # This AMI's system Python is version 2+.
   # _gang_schedule_ray_up() requires first two lines; do not modify.
   - pip3 install -U ray[default]=={{ray_version}} && mkdir -p ~/sky_workdir && mkdir -p ~/.sky/sky_app && touch ~/.sudo_as_admin_successful
-  - pip3 install {{sky_remote_path}}/*.whl && python3 -c "from sky.skylet.ray_patches import patch; patch()" # patch the buggy ray file
+  - pip3 uninstall sky -y &> /dev/null; pip3 install {{sky_remote_path}}/*.whl'[azure]' && python3 -c "from sky.skylet.ray_patches import patch; patch()" # patch the buggy ray file
path: prototype/config/azure-ray.yml.j2
text: @user1:
Why do we need this `[azure]`? nit: Can we have a comment here?

@author:
Actually I will keep this fix in #316, this is where it should be.