Repository: skypilot-org/skypilot
Pull request: [Core] Support setting sudo password when deploying remote clusters
<!-- Describe the changes in this PR -->
Fix #4795 

- Update doc to recommend using passwordless sudo when deploying remote clusters with `sky local up`
- If passwordless sudo is not available, support setting sudo password when deploying remote clusters with `sky local up`


<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: install pre-commit (auto-check on commit) or `bash format.sh`
- [x] Any manual or new tests for this PR (see detailed cases below)
- [ ] All smoke tests: `/smoke-test` (CI) or `pytest tests/test_smoke.py` (local)
- [ ] Relevant individual tests: `/smoke-test -k test_name` (CI) or `pytest tests/test_smoke.py::test_name` (local)
- [x] Backward compatibility: `/quicktest-core` (CI) or `pytest tests/smoke_tests/test_backward_compat.py` (local)

<!-- CI commands (/-prefixed) can only be triggered by repo members -->


Problem statement:
[k8s] Allow sudo with password for `sky local up`
Currently, if a user's sudo access requires password on the remote cluster, `sky local up` will fail. We can either:
1. Add support for password with sudo (better)
2. Add docs for setting up passwordless sudo on remote instance

Hints from issue discussion:
One way to support it is to just prompt the user to enter the password
Discussion summary:
1. We prefer the user to set up passwordless sudo in the remote server
2. If 1 cannot be done, we request the user to set a password in `sky local up` options

Commit message: update doc

Patch to review:
diff --git a/docs/source/reservations/existing-machines.rst b/docs/source/reservations/existing-machines.rst
index a54af88f4c4..0a0f4079ae0 100644
--- a/docs/source/reservations/existing-machines.rst
+++ b/docs/source/reservations/existing-machines.rst
@@ -47,7 +47,8 @@ Prerequisites
 **Remote machines (your cluster, optionally with GPUs):**
 
 * Debian-based OS (tested on Debian 11)
-* SSH access from local machine to all remote machines with key-based authentication and passwordless sudo
+* SSH access from local machine to all remote machines with key-based authentication
+* It's recommended to use passwordless sudo for all remote machines. If passwordless sudo cannot be used, all machines must use the same password for the SSH username to use sudo.
 * All machines must use the same SSH key and username
 * All machines must have network access to each other
 * Port 6443 must be accessible on at least one node from your local machine
@@ -70,6 +71,8 @@ Deploying SkyPilot
 
 2. Run ``sky local up`` and pass the ``ips.txt`` file, SSH username, and SSH key as arguments:
 
+   If passwordless sudo is available:
+
    .. code-block:: bash
 
       IP_FILE=ips.txt
@@ -78,6 +81,17 @@ Deploying SkyPilot
       CONTEXT_NAME=mycluster  # Optional, sets the context name in the kubeconfig. Defaults to "default".
       sky local up --ips $IP_FILE --ssh-user $SSH_USER --ssh-key-path $SSH_KEY --context-name $CONTEXT_NAME
 
+   If passwordless sudo is not available:
+
+   .. code-block:: bash
+
+      IP_FILE=ips.txt
+      SSH_USER=username
+      SSH_KEY=path/to/ssh/key
+      PASSWORD=password
+      CONTEXT_NAME=mycluster  # Optional, sets the context name in the kubeconfig. Defaults to "default".
+      sky local up --ips $IP_FILE --ssh-user $SSH_USER --ssh-key-path $SSH_KEY --context-name $CONTEXT_NAME --password $PASSWORD
+
    SkyPilot will deploy a Kubernetes cluster on the remote machines, set up GPU support, configure Kubernetes credentials on your local machine, and set up SkyPilot to operate with the new cluster.
 
    Example output of ``sky local up``:
@@ -143,6 +157,8 @@ Cleanup
 
 To clean up all state created by SkyPilot on your machines, use the ``--cleanup`` flag:
 
+If passwordless sudo is available:
+
 .. code-block:: bash
 
     IP_FILE=ips.txt
@@ -150,6 +166,16 @@ To clean up all state created by SkyPilot on your machines, use the ``--cleanup`
     SSH_KEY=path/to/ssh/key
     sky local up --ip $IP_FILE --ssh-user SSH_USER --ssh-key-path $SSH_KEY --cleanup
 
+If passwordless sudo is not available:
+
+.. code-block:: bash
+
+    IP_FILE=ips.txt
+    SSH_USER=username
+    SSH_KEY=path/to/ssh/key
+    PASSWORD=password
+    sky local up --ip $IP_FILE --ssh-user SSH_USER --ssh-key-path $SSH_KEY --password $PASSWORD --cleanup
+
 This will stop all Kubernetes services on the remote machines.
 
 
diff --git a/sky/cli.py b/sky/cli.py
index 9a8ce8c7267..5c66c08413b 100644
--- a/sky/cli.py
+++ b/sky/cli.py
@@ -5455,11 +5455,16 @@ def local():
     type=str,
     required=False,
     help='Name to use for the kubeconfig context. Defaults to "default".')
+@click.option('--password',
+              type=str,
+              required=False,
+              help='Password for the ssh-user to execute sudo commands.')
 @local.command('up', cls=_DocumentedCodeCommand)
 @_add_click_options(_COMMON_OPTIONS)
 @usage_lib.entrypoint
 def local_up(gpus: bool, ips: str, ssh_user: str, ssh_key_path: str,
-             cleanup: bool, context_name: Optional[str], async_call: bool):
+             cleanup: bool, context_name: Optional[str],
+             password: Optional[str], async_call: bool):
     """Creates a local or remote cluster."""
 
     def _validate_args(ips, ssh_user, ssh_key_path, cleanup):
@@ -5505,7 +5510,7 @@ def _validate_args(ips, ssh_user, ssh_key_path, cleanup):
                 f'Failed to read SSH key file {ssh_key_path}: {str(e)}')
 
     request_id = sdk.local_up(gpus, ip_list, ssh_user, ssh_key, cleanup,
-                              context_name)
+                              context_name, password)
     _async_call_or_wait(request_id, async_call, request_name='local up')
 
 
diff --git a/sky/client/sdk.py b/sky/client/sdk.py
index 5505ef4820d..11bf9046af7 100644
--- a/sky/client/sdk.py
+++ b/sky/client/sdk.py
@@ -1295,7 +1295,8 @@ def local_up(gpus: bool,
              ssh_user: Optional[str],
              ssh_key: Optional[str],
              cleanup: bool,
-             context_name: Optional[str] = None) -> server_common.RequestId:
+             context_name: Optional[str] = None,
+             password: Optional[str] = None) -> server_common.RequestId:
     """Launches a Kubernetes cluster on local machines.
 
     Returns:
@@ -1314,7 +1315,8 @@ def local_up(gpus: bool,
                                 ssh_user=ssh_user,
                                 ssh_key=ssh_key,
                                 cleanup=cleanup,
-                                context_name=context_name)
+                                context_name=context_name,
+                                password=password)
     response = requests.post(f'{server_common.get_server_url()}/local_up',
                              json=json.loads(body.model_dump_json()))
     return server_common.get_request_id(response)
diff --git a/sky/core.py b/sky/core.py
index 82a4a462867..af2f446fa7c 100644
--- a/sky/core.py
+++ b/sky/core.py
@@ -1062,6 +1062,7 @@ def local_up(gpus: bool,
              ips: Optional[List[str]],
              ssh_user: Optional[str],
              ssh_key: Optional[str],
+             password: Optional[str],
              cleanup: bool,
              context_name: Optional[str] = None) -> None:
     """Creates a local or remote cluster."""
@@ -1089,7 +1090,8 @@ def _validate_args(ips, ssh_user, ssh_key, cleanup):
     if ips:
         assert ssh_user is not None and ssh_key is not None
         kubernetes_deploy_utils.deploy_remote_cluster(ips, ssh_user, ssh_key,
-                                                      cleanup, context_name)
+                                                      password, cleanup,
+                                                      context_name)
     else:
         # Run local deployment (kind) if no remote args are specified
         kubernetes_deploy_utils.deploy_local_cluster(gpus)
diff --git a/sky/server/requests/payloads.py b/sky/server/requests/payloads.py
index 2b1a23632c5..a1d7d202038 100644
--- a/sky/server/requests/payloads.py
+++ b/sky/server/requests/payloads.py
@@ -464,6 +464,7 @@ class LocalUpBody(RequestBody):
     ssh_key: Optional[str] = None
     cleanup: bool = False
     context_name: Optional[str] = None
+    password: Optional[str] = None
 
 
 class ServeTerminateReplicaBody(RequestBody):
diff --git a/sky/utils/kubernetes/deploy_remote_cluster.sh b/sky/utils/kubernetes/deploy_remote_cluster.sh
index 02dada9918c..c950c62cbb9 100755
--- a/sky/utils/kubernetes/deploy_remote_cluster.sh
+++ b/sky/utils/kubernetes/deploy_remote_cluster.sh
@@ -12,6 +12,7 @@ NC='\033[0m' # No color
 CLEANUP=false
 INSTALL_GPU=false
 POSITIONAL_ARGS=()
+PASSWORD=""
 
 # Process all arguments
 while [[ $# -gt 0 ]]; do
@@ -20,6 +21,11 @@ while [[ $# -gt 0 ]]; do
             CLEANUP=true
             shift
             ;;
+        --password)
+            PASSWORD=$2
+            shift
+            shift
+            ;;
         *)
             POSITIONAL_ARGS+=("$1")
             shift
@@ -40,7 +46,7 @@ K3S_TOKEN=mytoken  # Any string can be used as the token
 # Basic argument checks
 if [ -z "$IPS_FILE" ] || [ -z "$USER" ] || [ -z "$SSH_KEY" ]; then
     >&2 echo -e "${RED}Error: Missing required arguments.${NC}"
-    >&2 echo "Usage: ./deploy_remote_cluster.sh ips.txt username path/to/ssh/key [context-name] [--cleanup]"
+    >&2 echo "Usage: ./deploy_remote_cluster.sh ips.txt username path/to/ssh/key [context-name] [--cleanup] [--password password]"
     exit 1
 fi
 
@@ -89,9 +95,19 @@ cleanup_server_node() {
     local NODE_IP=$1
     echo -e "${YELLOW}Cleaning up head node $NODE_IP...${NC}"
     run_remote "$NODE_IP" "
+        # Create temporary askpass script
+        ASKPASS_SCRIPT=\$(mktemp)
+        trap 'rm -f \$ASKPASS_SCRIPT' EXIT INT TERM ERR QUIT
+cat > \$ASKPASS_SCRIPT << EOF
+#!/bin/bash
+echo "$PASSWORD"
+EOF
+        chmod 700 \$ASKPASS_SCRIPT
+        # Use askpass
+        export SUDO_ASKPASS=\$ASKPASS_SCRIPT
         echo 'Uninstalling k3s...' &&
-        /usr/local/bin/k3s-uninstall.sh || true &&
-        sudo rm -rf /etc/rancher /var/lib/rancher /var/lib/kubelet /etc/kubernetes ~/.kube
+        sudo -A /usr/local/bin/k3s-uninstall.sh || true &&
+        sudo -A rm -rf /etc/rancher /var/lib/rancher /var/lib/kubelet /etc/kubernetes ~/.kube
     "
     echo -e "${GREEN}Node $NODE_IP cleaned up successfully.${NC}"
 }
@@ -101,9 +117,19 @@ cleanup_agent_node() {
     local NODE_IP=$1
     echo -e "${YELLOW}Cleaning up node $NODE_IP...${NC}"
     run_remote "$NODE_IP" "
+        # Create temporary askpass script
+        ASKPASS_SCRIPT=\$(mktemp)
+        trap 'rm -f \$ASKPASS_SCRIPT' EXIT INT TERM ERR QUIT
+cat > \$ASKPASS_SCRIPT << EOF
+#!/bin/bash
+echo "$PASSWORD"
+EOF
+        chmod 700 \$ASKPASS_SCRIPT
+        # Use askpass
+        export SUDO_ASKPASS=\$ASKPASS_SCRIPT
         echo 'Uninstalling k3s...' &&
-        /usr/local/bin/k3s-agent-uninstall.sh || true &&
-        sudo rm -rf /etc/rancher /var/lib/rancher /var/lib/kubelet /etc/kubernetes ~/.kube
+        sudo -A /usr/local/bin/k3s-agent-uninstall.sh || true &&
+        sudo -A rm -rf /etc/rancher /var/lib/rancher /var/lib/kubelet /etc/kubernetes ~/.kube
     "
     echo -e "${GREEN}Node $NODE_IP cleaned up successfully.${NC}"
 }
@@ -151,10 +177,20 @@ fi
 # Step 1: Install k3s on the head node
 progress_message "Deploying Kubernetes on head node ($HEAD_NODE)..."
 run_remote "$HEAD_NODE" "
-    curl -sfL https://get.k3s.io | K3S_TOKEN=$K3S_TOKEN sh - &&
+    # Create temporary askpass script
+    ASKPASS_SCRIPT=\$(mktemp)
+    trap 'rm -f \$ASKPASS_SCRIPT' EXIT INT TERM ERR QUIT
+cat > \$ASKPASS_SCRIPT << EOF
+#!/bin/bash
+echo "$PASSWORD"
+EOF
+    chmod 700 \$ASKPASS_SCRIPT
+    # Use askpass
+    export SUDO_ASKPASS=\$ASKPASS_SCRIPT
+    curl -sfL https://get.k3s.io | K3S_TOKEN=$K3S_TOKEN sudo -E -A sh - &&
     mkdir -p ~/.kube &&
-    sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config &&
-    sudo chown \$(id -u):\$(id -g) ~/.kube/config &&
+    sudo -A cp /etc/rancher/k3s/k3s.yaml ~/.kube/config &&
+    sudo -A chown \$(id -u):\$(id -g) ~/.kube/config &&
     for i in {1..3}; do
         if kubectl wait --for=condition=ready node --all --timeout=2m --kubeconfig ~/.kube/config; then
             break
@@ -163,7 +199,7 @@ run_remote "$HEAD_NODE" "
             sleep 5
         fi
     done
-    if [ $i -eq 3 ]; then
+    if [ \$i -eq 3 ]; then
         echo 'Failed to wait for nodes to be ready after 3 attempts'
         exit 1
     fi"
@@ -184,7 +220,17 @@ echo -e "${GREEN}Master node internal IP: $MASTER_ADDR${NC}"
 for NODE in $WORKER_NODES; do
     progress_message "Deploying Kubernetes on worker node ($NODE)..."
     run_remote "$NODE" "
-        curl -sfL https://get.k3s.io | K3S_URL=https://$MASTER_ADDR:6443 K3S_TOKEN=$K3S_TOKEN sh -"
+        # Create temporary askpass script
+        ASKPASS_SCRIPT=\$(mktemp)
+        trap 'rm -f \$ASKPASS_SCRIPT' EXIT INT TERM ERR QUIT
+cat > \$ASKPASS_SCRIPT << EOF
+#!/bin/bash
+echo "$PASSWORD"
+EOF
+        chmod 700 \$ASKPASS_SCRIPT
+        # Use askpass
+        export SUDO_ASKPASS=\$ASKPASS_SCRIPT
+        curl -sfL https://get.k3s.io | K3S_URL=https://$MASTER_ADDR:6443 K3S_TOKEN=$K3S_TOKEN sudo -E -A sh -"
     success_message "Kubernetes deployed on worker node ($NODE)."
 
     # Check if worker node has a GPU
@@ -247,12 +293,22 @@ echo "Cluster deployment completed. You can now run 'kubectl get nodes' to verif
 if [ "$INSTALL_GPU" == "true" ]; then
     echo -e "${YELLOW}GPU detected in the cluster. Installing Nvidia GPU Operator...${NC}"
     run_remote "$HEAD_NODE" "
+        # Create temporary askpass script
+        ASKPASS_SCRIPT=\$(mktemp)
+        trap 'rm -f \$ASKPASS_SCRIPT' EXIT INT TERM ERR QUIT
+cat > \$ASKPASS_SCRIPT << EOF
+#!/bin/bash
+echo "$PASSWORD"
+EOF
+        chmod 700 \$ASKPASS_SCRIPT
+        # Use askpass
+        export SUDO_ASKPASS=\$ASKPASS_SCRIPT
         curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 &&
         chmod 700 get_helm.sh &&
         ./get_helm.sh &&
         helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update &&
         kubectl create namespace gpu-operator --kubeconfig ~/.kube/config || true &&
-        sudo ln -s /sbin/ldconfig /sbin/ldconfig.real || true &&
+        sudo -A ln -s /sbin/ldconfig /sbin/ldconfig.real || true &&
         helm install gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator \
         --set 'toolkit.env[0].name=CONTAINERD_CONFIG' \
         --set 'toolkit.env[0].value=/var/lib/rancher/k3s/agent/etc/containerd/config.toml' \
diff --git a/sky/utils/kubernetes/kubernetes_deploy_utils.py b/sky/utils/kubernetes/kubernetes_deploy_utils.py
index 24e11b913b1..680cdc6e158 100644
--- a/sky/utils/kubernetes/kubernetes_deploy_utils.py
+++ b/sky/utils/kubernetes/kubernetes_deploy_utils.py
@@ -23,6 +23,7 @@
 def deploy_remote_cluster(ip_list: List[str],
                           ssh_user: str,
                           ssh_key: str,
+                          password: Optional[str],
                           cleanup: bool,
                           context_name: Optional[str] = None):
     success = False
@@ -47,6 +48,8 @@ def deploy_remote_cluster(ip_list: List[str],
                           f'{ssh_user} {key_file.name}')
         if context_name is not None:
             deploy_command += f' {context_name}'
+        if password is not None:
+            deploy_command += f' --password {password}'
         if cleanup:
             deploy_command += ' --cleanup'
 


Code review comments:
diff_hunk: @@ -101,9 +117,19 @@ cleanup_agent_node() {
     local NODE_IP=$1
     echo -e "${YELLOW}Cleaning up node $NODE_IP...${NC}"
     run_remote "$NODE_IP" "
+        # Create temporary askpass script
+        ASKPASS_SCRIPT=\$(mktemp)
+        trap 'rm -f \$ASKPASS_SCRIPT' EXIT INT TERM ERR QUIT
+cat > \$ASKPASS_SCRIPT << EOF
+#!/bin/bash
+echo "$PASSWORD"
+EOF
+        chmod 700 \$ASKPASS_SCRIPT
+        # Use askpass
+        export SUDO_ASKPASS=\$ASKPASS_SCRIPT
path: sky/utils/kubernetes/deploy_remote_cluster.sh
text: @user1:
Can we make these a function? Looks like we just repeat it multiple times

@author:
Yes, I have updated this.

diff_hunk: @@ -5455,11 +5455,16 @@ def local():
     type=str,
     required=False,
     help='Name to use for the kubeconfig context. Defaults to "default".')
+@click.option('--password',
+              type=str,
+              required=False,
+              help='Password for the ssh-user to execute sudo commands.')
path: sky/cli.py
text: @user1:
```suggestion
              help='Password for the ssh-user to execute sudo commands. Required only if passwordless sudo is not setup.')
```

@author:
Updated

diff_hunk: @@ -247,12 +293,22 @@ echo "Cluster deployment completed. You can now run 'kubectl get nodes' to verif
 if [ "$INSTALL_GPU" == "true" ]; then
     echo -e "${YELLOW}GPU detected in the cluster. Installing Nvidia GPU Operator...${NC}"
     run_remote "$HEAD_NODE" "
+        # Create temporary askpass script
+        ASKPASS_SCRIPT=\$(mktemp)
+        trap 'rm -f \$ASKPASS_SCRIPT' EXIT INT TERM ERR QUIT
+cat > \$ASKPASS_SCRIPT << EOF
+#!/bin/bash
+echo "$PASSWORD"
+EOF
+        chmod 700 \$ASKPASS_SCRIPT
+        # Use askpass
+        export SUDO_ASKPASS=\$ASKPASS_SCRIPT
path: sky/utils/kubernetes/deploy_remote_cluster.sh
text: @user1:
This code snippet is used pretty often in this file, can we have a macro/function that can be reused instead of copy pasting? 

@author:
Yes, sorry for this.