Repository: skypilot-org/skypilot
Pull request: [K8s] Add user hash to the kind config for multi-user system permission issue
<!-- Describe the changes in this PR -->

Fixes #3533

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [ ] Code formatting: `bash format.sh`
- [ ] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `pytest tests/test_smoke.py` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [ ] Backward compatibility tests: `conda deactivate; bash -i tests/backward_compatibility_tests.sh`


Problem statement:
[K8S] `sky local up` failed on a multi-user system
<!-- Describe the bug report / feature request here -->

On commit `7f30ce5e`

Currently the `sky local up` will create a config yaml under `/tmp` and does not clean it up after local cluster creation. If another user on the same Linux OS tries to run `sky local up`, it will show no permission to write on that fail and abort the creation. Should we cleanup the config or make the name of it user-specific to avoid conflict? Logs:

```bash
Creating local cluster...
To view detailed progress: tail -n100 -f ~/sky_logs/sky-2024-05-09-19-57-51-741127/local_up.log
Failed to create local cluster. Full log: ~/sky_logs/sky-2024-05-09-19-57-51-741127/local_up.log
Error: Traceback (most recent call last):
  File "/home/memory/install/miniconda3/envs/sky/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/memory/install/miniconda3/envs/sky/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/memory/skypilot/sky/utils/kubernetes/generate_kind_config.py", line 93, in <module>
    generate_kind_config(args.path, args.port_start, args.port_end,
  File "/home/memory/skypilot/sky/utils/kubernetes/generate_kind_config.py", line 55, in generate_kind_config
    with open(path, 'w', encoding='utf-8') as f:
PermissionError: [Errno 13] Permission denied: '/tmp/skypilot-kind.yaml'
```

<!-- If relevant, fill in versioning info to help us troubleshoot -->

Hints from issue discussion:
We can probably attach a user hash to the yaml name : )

Commit message: fix

Patch to review:
diff --git a/sky/cli.py b/sky/cli.py
index c538c99aeb3..b9756b63083 100644
--- a/sky/cli.py
+++ b/sky/cli.py
@@ -5097,7 +5097,8 @@ def _deploy_local_cluster(gpus: bool):
 
     # Get directory of script and run it from there
     cwd = os.path.dirname(os.path.abspath(up_script_path))
-    run_command = up_script_path + ' --gpus' if gpus else up_script_path
+    run_command = up_script_path + f' --user {common_utils.get_user_hash()}'
+    run_command = run_command + ' --gpus' if gpus else run_command
     run_command = shlex.split(run_command)
 
     # Setup logging paths
diff --git a/sky/utils/kubernetes/create_cluster.sh b/sky/utils/kubernetes/create_cluster.sh
index 52bbd1804e8..7c5c4cea57f 100755
--- a/sky/utils/kubernetes/create_cluster.sh
+++ b/sky/utils/kubernetes/create_cluster.sh
@@ -12,9 +12,11 @@ IMAGE_GPU="us-central1-docker.pkg.dev/skypilot-375900/skypilotk8s/skypilot-gpu:l
 PORT_RANGE_START=30000
 PORT_RANGE_END=30100
 
+USER_HASH=$1
+
 # Check for GPU flag
 ENABLE_GPUS=false
-if [[ "$1" == "--gpus" ]]; then
+if [[ "$2" == "--gpus" ]]; then
     ENABLE_GPUS=true
 fi
 
@@ -88,16 +90,17 @@ if kind get clusters | grep -q skypilot; then
 fi
 
 # Generate cluster YAML
-echo "Generating /tmp/skypilot-kind.yaml"
+YAML_PATH="/tmp/skypilot-kind-$USER_HASH.yaml"
+echo "Generating $YAML_PATH"
 
 # Add GPUs flag to the generate_kind_config.py command if GPUs are enabled
 if $ENABLE_GPUS; then
-    python -m sky.utils.kubernetes.generate_kind_config --path /tmp/skypilot-kind.yaml --port-start ${PORT_RANGE_START} --port-end ${PORT_RANGE_END} --gpus
+    python -m sky.utils.kubernetes.generate_kind_config --path $YAML_PATH --port-start ${PORT_RANGE_START} --port-end ${PORT_RANGE_END} --gpus
 else
-  python -m sky.utils.kubernetes.generate_kind_config --path /tmp/skypilot-kind.yaml --port-start ${PORT_RANGE_START} --port-end ${PORT_RANGE_END}
+  python -m sky.utils.kubernetes.generate_kind_config --path $YAML_PATH --port-start ${PORT_RANGE_START} --port-end ${PORT_RANGE_END}
 fi
 
-kind create cluster --config /tmp/skypilot-kind.yaml --name skypilot
+kind create cluster --config $YAML_PATH --name skypilot
 
 echo "Kind cluster created."
 
diff --git a/sky/utils/kubernetes/generate_kind_config.py b/sky/utils/kubernetes/generate_kind_config.py
index f9c954bc97a..2406b94d02f 100644
--- a/sky/utils/kubernetes/generate_kind_config.py
+++ b/sky/utils/kubernetes/generate_kind_config.py
@@ -71,7 +71,6 @@ def generate_kind_config(path: str,
                                      ' from host to container')
     parser.add_argument('--path',
                         type=str,
-                        default='/tmp/skypilot-kind.yaml',
                         help='Path to generate the config file at')
     parser.add_argument('--port-start',
                         type=int,


Code review comments:
diff_hunk: @@ -5097,7 +5097,8 @@ def _deploy_local_cluster(gpus: bool):
 
     # Get directory of script and run it from there
     cwd = os.path.dirname(os.path.abspath(up_script_path))
-    run_command = up_script_path + ' --gpus' if gpus else up_script_path
+    run_command = up_script_path + f' --user {common_utils.get_user_hash()}'
path: sky/cli.py
text: @user1:
This created ` /tmp/skypilot-kind---user.yaml`. Maybe we want to remove the `--user` here?

```
No kind clusters found.
Generating /tmp/skypilot-kind---user.yaml
D 10-07 22:21:55 skypilot_config.py:228] Using config path: /Users/romilb/.sky/config.yaml
D 10-07 22:21:55 skypilot_config.py:233] Config loaded:
D 10-07 22:21:55 skypilot_config.py:233] {'allowed_clouds': ['kubernetes', 'gcp', 'lambda', 'aws'],
D 10-07 22:21:55 skypilot_config.py:233]  'jobs': {'controller': {'resources': {'cloud': 'kubernetes',
D 10-07 22:21:55 skypilot_config.py:233]                                        'cpus': '4+',
D 10-07 22:21:55 skypilot_config.py:233]                                        'memory': '4+'}}},
D 10-07 22:21:55 skypilot_config.py:233]  'kubernetes': {'provision_timeout': -1},
D 10-07 22:21:55 skypilot_config.py:233]  'serve': {'controller': {'resources': {'cloud': 'kubernetes',
D 10-07 22:21:55 skypilot_config.py:233]                                         'cpus': '4+',
D 10-07 22:21:55 skypilot_config.py:233]                                         'memory': '4+'}}}}
D 10-07 22:21:55 skypilot_config.py:245] Config syntax check passed.
Creating cluster "skypilot" ...
 • Ensuring node image (kindest/node:v1.25.3) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 • Preparing nodes 📦   ...
 ✓ Preparing nodes 📦 
 • Writing configuration 📜  ...
 ✓ Writing configuration 📜
 • Starting control-plane 🕹️  ...
 ✓ Starting control-plane 🕹️
 • Installing CNI 🔌  ...
 ✓ Installing CNI 🔌
 • Installing StorageClass 💾  ...
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-skypilot"
You can now use your cluster with:

kubectl cluster-info --context kind-skypilot
```

@author:
Thanks for pointing it out! I've done some command argument reordering but forgot to test it after. Updated and tested now!

```bash
No kind clusters found.
Generating /tmp/skypilot-kind-4a0782e9.yaml
Creating cluster "skypilot" ...
 • Ensuring node image (kindest/node:v1.27.3) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 • Preparing nodes 📦   ...
 ✓ Preparing nodes 📦 
 • Writing configuration 📜  ...
 ✓ Writing configuration 📜
 • Starting control-plane 🕹️  ...
 ✓ Starting control-plane 🕹️
 • Installing CNI 🔌  ...
 ✓ Installing CNI 🔌
 • Installing StorageClass 💾  ...
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-skypilot"
```

diff_hunk: @@ -71,7 +71,6 @@ def generate_kind_config(path: str,
                                      ' from host to container')
     parser.add_argument('--path',
                         type=str,
-                        default='/tmp/skypilot-kind.yaml',
path: sky/utils/kubernetes/generate_kind_config.py
text: @user1:
Should we leave the default here?