Repository: skypilot-org/skypilot
Pull request: [k8s] Wait for apt installation to complete before proceeding with setup
As a part of nimbus (https://github.com/skypilot-org/skypilot/pull/4393) we parallelized dependency setup in the k8s pods. However, we did not add a wait for SSH setup to complete. Waiting for SSH is critical for rsync operations to succeed.

A recent archive.ubuntu.com outage caused delayed SSH installs, resulting in #5794 and #5792.

This PR fixes it by waiting for SSH install to complete. 

Note: we are not updating APT mirrors in this PR to avoid hardcoding mirrors. 

Closes #5794 and #5792.

Tested: 
 - [x] `sky launch -c test --infra kubernetes --image-id docker:nvcr.io/nvidia/pytorch:24.05-py3`
 - [x] `sky launch -c test --infra kubernetes --image-id docker:ubuntu:latest`
 - [x] Back compat -  `sky launch` with master, then `sky launch` again with this branch
Note: we are not updating the mirror 

Problem statement:
[k8s] `sky launch --infra k8s --image-id continuumio/miniconda3:latest` experience undeterministic ssh issue
```
E 05-28 23:26:07 sdk.py:1599] Using Python 3.10.17 environment at: /root/skypilot-runtime
--
  | E 05-28 23:26:07 sdk.py:1599] skypilot                  1.0.0.dev0
  | E 05-28 23:26:07 sdk.py:1599] === Skypilot wheel installation completed in 0 secs ===
  | E 05-28 23:26:07 sdk.py:1599] DefaultTasksMax=infinity
  | E 05-28 23:26:07 sdk.py:1599] === Setup system configs and fuse completed in 0 secs ===
  | E 05-28 23:26:07 sdk.py:1599] System has not been booted with systemd as init system (PID 1). Can't operate.
  | E 05-28 23:26:07 sdk.py:1599] Failed to connect to bus: Host is down
  | E 05-28 23:26:07 sdk.py:1599] ssh: unrecognized service
  | E 05-28 23:26:07 sdk.py:1599]
  | E 05-28 23:26:07 sdk.py:1599] ===== stderr =====command terminated with exit code 1
```

Hints from issue discussion:
Looks like archive.ubuntu.com has been having issues, causing SSH install to fail:

```
Get:32 http://security.ubuntu.com/ubuntu noble-updates/main amd64 openssh-server amd64 1:9.6p1-3ubuntu13.11 [509 kB]
Get:42 http://security.ubuntu.com/ubuntu noble-updates/main amd64 libglib2.0-0t64 amd64 2.80.0-6ubuntu3.4 [1544 kB]
Get:43 http://security.ubuntu.com/ubuntu noble-updates/main amd64 gir1.2-glib-2.0 amd64 2.80.0-6ubuntu3.4 [183 kB]
Get:46 http://security.ubuntu.com/ubuntu noble-updates/main amd64 libglib2.0-data all 2.80.0-6ubuntu3.4 [48.7 kB]
Get:49 http://security.ubuntu.com/ubuntu noble-updates/main amd64 libxml2 amd64 2.9.14+dfsg-1.3ubuntu3.3 [762 kB]
Get:55 http://security.ubuntu.com/ubuntu noble-updates/main amd64 python3-pkg-resources all 68.1.2-2ubuntu1.1 [168 kB]
Get:65 http://security.ubuntu.com/ubuntu noble-updates/main amd64 python3-cryptography amd64 41.0.7-4ubuntu0.1 [810 kB]
Fetched 11.8 MB in 1min 2s (192 kB/s)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python3-defaults/python3-minimal_3.12.3-0ubuntu2_amd64.deb  Unable to connect to archive.ubuntu.com:80:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/media-types/media-types_10.1.0_all.deb  Unable to connect to archive.ubuntu.com:80:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/n/netbase/netbase_6.4_all.deb  Unable to connect to archive.ubuntu.com:80:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/r/readline/readline-common_8.2-4build1_all.deb  Unable to connect to archive.ubuntu.com:80:
....
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Optional package openssh-server installation failed, skip.
```

Ping to `archive.ubuntu.com` is also borked:

<img width="540" alt="Image" src="https://github.com/user-attachments/assets/9208c245-8fd7-4bee-b405-82121f8d706b" />

Commit message: Remove mirror update

Patch to review:
diff --git a/sky/templates/kubernetes-ray.yml.j2 b/sky/templates/kubernetes-ray.yml.j2
index f2de21cc091..0c8e9ccee1e 100644
--- a/sky/templates/kubernetes-ray.yml.j2
+++ b/sky/templates/kubernetes-ray.yml.j2
@@ -395,6 +395,13 @@ available_node_types:
               # STEP 1: Run apt update, install missing packages, and set up ssh.
               (
                 (
+                # For backwards compatibility, we put a marker file in the pod
+                # to indicate that the apt ssh setup step will write a completion
+                # marker file (/tmp/apt_ssh_setup_complete) to the pod.
+                # TODO: Remove this marker file and it's usage in setup_commands
+                # after v0.11.0 release.
+                touch /tmp/apt_ssh_setup_started
+
                 DEBIAN_FRONTEND=noninteractive $(prefix_cmd) apt-get update > /tmp/apt-update.log 2>&1 || \
                 echo "Warning: apt-get update failed. Continuing anyway..." >> /tmp/apt-update.log
                 # Install both fuse2 and fuse3 for compatibility for all possible fuse adapters in advance,
@@ -402,7 +409,7 @@ available_node_types:
                 PACKAGES="rsync curl wget netcat gcc patch pciutils fuse fuse3 openssh-server";
 
                 # Separate packages into two groups: packages that are installed first
-                # so that curl, rsync and wget are available sooner to unblock the following
+                # so that curl, rsync, ssh and wget are available sooner to unblock the following
                 # conda installation and rsync.
                 # Also, we install fuse first to avoid confliction with fuse3.
                 set -e
@@ -494,6 +501,8 @@ available_node_types:
                 $(prefix_cmd) service ssh restart;
                 $(prefix_cmd) sed -i "s/mesg n/tty -s \&\& mesg n/" ~/.profile;
 
+                touch /tmp/apt_ssh_setup_complete
+                echo "=== SSH setup completed ==="
                 ) > /tmp/${STEPS[0]}.log 2>&1 || {
                   echo "Error: ${STEPS[0]} failed. Continuing anyway..." > /tmp/${STEPS[0]}.failed
                   cat /tmp/${STEPS[0]}.log
@@ -791,6 +800,15 @@ setup_commands:
     {%- endfor %}
     STEPS=("apt-ssh-setup" "runtime-setup" "env-setup")
     start_epoch=$(date +%s);
+    
+    # Wait for SSH setup to complete before proceeding
+    if [ -f /tmp/apt_ssh_setup_started ]; then
+      echo "=== Logs for asynchronous SSH setup ===";
+      [ -f /tmp/apt_ssh_setup_complete ] && cat /tmp/${STEPS[0]}.log ||
+      { tail -f -n +1 /tmp/${STEPS[0]}.log & TAIL_PID=$!; echo "Tail PID: $TAIL_PID"; until [ -f /tmp/apt_ssh_setup_complete ]; do sleep 0.5; done; kill $TAIL_PID || true; };
+      [ -f /tmp/${STEPS[0]}.failed ] && { echo "Error: ${STEPS[0]} failed. Exiting."; exit 1; } || true;
+    fi
+    
     echo "=== Logs for asynchronous ray and skypilot installation ===";
     if [ -f /tmp/skypilot_is_nimbus ]; then
       echo "=== Logs for asynchronous ray and skypilot installation ===";


Code review comments:
diff_hunk: @@ -395,14 +395,21 @@ available_node_types:
               # STEP 1: Run apt update, install missing packages, and set up ssh.
               (
                 (
+                # For backwards compatibility, we put a marker file in the pod
+                # to indicate that the apt ssh setup step will write a completion
+                # marker file (/tmp/apt_ssh_setup_complete) to the pod.
+                # TODO: Remove this marker file and it's usage in setup_commands
path: sky/templates/kubernetes-ray.yml.j2
text: @user1:
```suggestion
                # TODO: Remove this marker file and its usage in setup_commands
```