Repository: skypilot-org/skypilot
Pull request: [controller] fix kubectl installation
<!-- Describe the changes in this PR -->
This regressed in #4835.
Fixes #5186.


<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [ ] Code formatting: install pre-commit (auto-check on commit) or `bash format.sh`
- [ ] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `/smoke-test` (CI) or `pytest tests/test_smoke.py` (local)
- [ ] Relevant individual tests: `/smoke-test -k test_name` (CI) or `pytest tests/test_smoke.py::test_name` (local)
- [ ] Backward compatibility: `/quicktest-core` (CI) or `pytest tests/smoke_tests/test_backward_compat.py` (local)

<!-- CI commands (/-prefixed) can only be triggered by repo members -->


Problem statement:
[Jobs] `kubectl` not found on jobs controller
To reproduce:
1. set up a k8s cluster with nodes that does not have enough CPUs for jobs controller
2. When running `sky jobs launch --cpus 2`
3. The controller starts on a cloud, but `kubectl` is not installed

Hints from issue discussion:
https://github.com/skypilot-org/skypilot/pull/4835/files#diff-8acc88bd4d951508e5ea2f822f65dbc0bfaaed24fd6f8c18027affe8fda7ce1fL250-R257 is broken

Commit message: [controller] fix kubectl installation

Patch to review:
diff --git a/sky/utils/controller_utils.py b/sky/utils/controller_utils.py
index 4267e4eed21..8fc3bfce344 100644
--- a/sky/utils/controller_utils.py
+++ b/sky/utils/controller_utils.py
@@ -262,8 +262,10 @@ def _get_cloud_dependencies_installation_commands(
                 '  ARCH="amd64"; '
                 'fi && '
                 '(command -v kubectl &>/dev/null || '
-                '("https://dl.k8s.io/release/v1.31.6/bin/linux/$ARCH/kubectl" '
-                '&& sudo install -o root -g root -m 0755 '
+                '(curl -s -LO "https://dl.k8s.io/release/'
+                '$(curl -L -s https://dl.k8s.io/release/stable.txt)'
+                '/bin/linux/$ARCH/kubectl" && '
+                'sudo install -o root -g root -m 0755 '
                 'kubectl /usr/local/bin/kubectl))')
         elif isinstance(cloud, clouds.Cudo):
             step_prefix = prefix_str.replace('<step>', str(len(commands) + 1))


Code review comments:
diff_hunk: @@ -262,8 +262,10 @@ def _get_cloud_dependencies_installation_commands(
                 '  ARCH="amd64"; '
                 'fi && '
                 '(command -v kubectl &>/dev/null || '
-                '("https://dl.k8s.io/release/v1.31.6/bin/linux/$ARCH/kubectl" '
-                '&& sudo install -o root -g root -m 0755 '
+                '(curl -s -LO "https://dl.k8s.io/release/'
+                '$(curl -L -s https://dl.k8s.io/release/stable.txt)'
path: sky/utils/controller_utils.py
text: @user1:
Should we stick with the fixed version as our other places did?

Also, can we add a test for this?

@author:
tbh, I'm not sure how this isn't caught in our existing tests - maybe only affects cross-cloud use (e.g. AWS controller -> k8s job)

@user2:
Our k8s base image has kubectl already baked into it, so a bad kubectl install attempt won't break anything. If you used a different base image for the controller (or ran on a different cloud, like @author said), you'll run into this

@author:
I think adding a test will be more difficult. Prefer to just merge this and follow up with cross-cloud tests later.