Repository: skypilot-org/skypilot
Pull request: Fix sky start for AMI without conda
<!-- Describe the changes in this PR -->

Describe the changes in this PR:

Fixes https://github.com/skypilot-org/skypilot/issues/1517 and add one smoke test to cover this case in which an AMI without conda is used.
The bug can be reproduced by the following yaml.
```
resources:
  cloud: aws
  instance_type: g4dn.xlarge
  region: us-west-2
  image_id: ami-0fe5af21074ad2a10  # Deep learning AMI with CUDA 11.6

setup: |
  echo 

run: |
  nvidia-smi
```

```
> sky launch -c test test.yaml
> sky stop test
> sky start test
...
Command 'ray' not found, did you mean:
```

This AMI doesn't contain conda and we install conda during `sky launch`. But later `sky start` detects conda is installed and set `conda config --set auto_activate_base false` which disables conda again. I believe it should not be needed because we still use conda python even with this command + pre-installed conda AMI.
Using master to do `sky launch` an AWS VM:
```
ubuntu@ip-172-31-90-40:~$ which python
/opt/conda/bin/python
ubuntu@ip-172-31-90-40:~$ which pip3
/opt/conda/bin/pip3
```
(note that `gcp-ray.yml.j2` and `azure-ray.yml.j2` also don't have `conda config --set auto_activate_base false`)
If we confirm this config command is not needed. simply remove it can fix the issue.



<!-- Describe tests ran in a "Tested:" section -->
<!-- Unit tests (tests/test_*.py) are part of Github CI; the below are additional tests. -->

Tested (run the relevant ones):

- [x] Any manual or new tests for this the PR
- [x] All smoke tests: `bash tests/run_smoke_tests.sh` 
- [x] Backward compatibility tests: `bash tests/backward_comaptibility_tests.sh`


Problem statement:
Error when restarting cluster
Hi, I'm running into the following error when restarting a cluster: `ModuleNotFoundError: No module named 'sky'`.
Steps to reproduce:
1. `sky launch -C my_cluster config.yaml`
2. `sky stop my_cluster`
3. `sky start my_cluster`

The instance is successfully provisioned on AWS, and I am able to SSH using the IP address. I'm not sure if this is relevant, but I noticed that after running `sky start` and SSHing, the terminal would not be running inside an active Conda environment.

I'm also using the AWS Deep Learning AMI and storing data in S3.

Full error log:
```
Restarting 1 cluster: sky-b4da-peter. Proceed? [Y/n]:
I 12-13 13:58:55 cloud_vm_ray_backend.py:903] To view detailed progress: tail -n100 -f /home/peter/sky_logs/sky-2022-12-13-13-58-55-599628/provision.log
I 12-13 13:58:55 cloud_vm_ray_backend.py:744] Cluster 'sky-b4da-peter' (status: STOPPED) was previously launched in AWS (us-west-2). Relaunching in that region.
I 12-13 13:58:56 cloud_vm_ray_backend.py:1152] Launching on AWS us-west-2 (us-west-2a,us-west-2b,us-west-2c,us-west-2d)
I 12-13 14:00:13 log_utils.py:45] Head node is up.
I 12-13 14:00:47 cloud_vm_ray_backend.py:996] Successfully provisioned or found existing VM.
E 12-13 14:00:48 subprocess_utils.py:69] Traceback (most recent call last):
E 12-13 14:00:48 subprocess_utils.py:69]   File "<string>", line 1, in <module>
E 12-13 14:00:48 subprocess_utils.py:69] ModuleNotFoundError: No module named 'sky'
E 12-13 14:00:48 subprocess_utils.py:69]
sky.exceptions.CommandError: Command python3 -u -c 'import os;from sky.skylet import job_lib, log_lib;job_lib.fail_all_jobs_in_progress()' failed with return code 1.
Failed to set previously in-progress jobs to FAILED
```

Hints from issue discussion:
Thank you for reporting the issue @pschafhalter! The SkyPilot runtime seems to have been wrongly set up on your remote cluster. Do you mind sharing the resources field of your `config.yaml` or the `image_id` you are using for the cluster to make it easier for us to debug the problem? 

PS, I tried the latest master branch d97ffed6, with the default AMI, but failed to reproduce the issue. 
Here's my YAML. I've changed the name of the python script and the S3 bucket. I believe I've run into the same issue with other instance types, so it might be a problem with the AMI.

```yaml
resources:
  cloud: aws
  instance_type: p4de.24xlarge
  region: us-west-2
  image_id: ami-0fe5af21074ad2a10  # Deep learning AMI with CUDA 11.6

workdir: .

file_mounts:
  /data:
    name: my_s3_bucket
    mode: MOUNT

setup: |
  ls data/  # Set up the S3 bucket
  bash scripts/setup.sh

run: |
  python3 my_script.py
```
~~It might be the system of the AMI that causes the problem. Do you want to try the Deep Learning AMI with Ubuntu 20.04 or Ubuntu 18.04?~~
It seems to be ubuntu 20.04
```
> aws ec2 describe-images  --region us-west-2 --image-ids ami-0fe5af21074ad2a10
{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2022-09-07T12:07:11.000Z",
            "ImageId": "ami-0fe5af21074ad2a10",
            "ImageLocation": "amazon/Deep Learning AMI GPU TensorFlow 2.9.1 (Ubuntu 20.04) 20220907",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "898082745236",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/sda1",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "Iops": 3000,
                        "SnapshotId": "snap-011bfe9ee94b58676",
                        "VolumeSize": 35,
                        "VolumeType": "gp3",
                        "Throughput": 125,
                        "Encrypted": false
                    }
                },
                {
                    "DeviceName": "/dev/sdb",
                    "VirtualName": "ephemeral0"
                },
                {
                    "DeviceName": "/dev/sdc",
                    "VirtualName": "ephemeral1"
                }
            ],
            "Description": "Built with AWS optimized TensorFlow, NVIDIA CUDA, cuDNN, NCCL, GPU Driver, Docker, NVIDIA-Docker and EFA support. For a fully managed experience, check: https://aws.amazon.com/sagemaker",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "ImageOwnerAlias": "amazon",
            "Name": "Deep Learning AMI GPU TensorFlow 2.9.1 (Ubuntu 20.04) 20220907",
            "RootDeviceName": "/dev/sda1",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm",
            "DeprecationTime": "2024-09-07T12:07:11.000Z"
        }
    ]
}
```
@pschafhalter Thanks for reporting this issue. I'm able to reproduce it if using the same AMI.
Looks like the issue is specific to this AMI in which conda is not installed by default.
Though SkyPilot installs conda if not exist, it seems not correctly activated after `sky start`.
I'm investigating it now.
@pschafhalter we have an ongoing PR https://github.com/skypilot-org/skypilot/pull/1532 for this issue. To unblock your use case, you may modify a line in `sky/templates/aws-ray.yml.j2` as below for now. 
https://github.com/skypilot-org/skypilot/pull/1532/files#diff-bea9d257f5081c10250926eda9b6a4b996c764f4359647b5020afe41066a6a8e

We'll soon review this PR and merge it to master. thanks again!

Commit message: rename

Patch to review:
diff --git a/examples/tests/test_yamls/no_conda_ami.yaml b/examples/tests/test_yamls/no_conda_ami.yaml
new file mode 100644
index 00000000000..5aae1f70a55
--- /dev/null
+++ b/examples/tests/test_yamls/no_conda_ami.yaml
@@ -0,0 +1,9 @@
+resources:
+  cloud: aws
+  instance_type: g4dn.xlarge
+  region: us-west-2
+  image_id: ami-0fe5af21074ad2a10  # Deep learning AMI with CUDA 11.6
+
+run: |
+  nvidia-smi
+
diff --git a/examples/tpu/tpuvm_mnist.yaml b/examples/tpu/tpuvm_mnist.yaml
index 67a69f6d13c..f25932331d8 100644
--- a/examples/tpu/tpuvm_mnist.yaml
+++ b/examples/tpu/tpuvm_mnist.yaml
@@ -20,6 +20,7 @@ setup: |
     pip install "jax[tpu]>=0.2.16" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html
     pip install --upgrade clu
     pip install -e flax
+    pip install tensorflow tensorflow-datasets
   fi
 
 
diff --git a/sky/templates/aws-ray.yml.j2 b/sky/templates/aws-ray.yml.j2
index 4646ff12241..e10609cf319 100644
--- a/sky/templates/aws-ray.yml.j2
+++ b/sky/templates/aws-ray.yml.j2
@@ -110,7 +110,7 @@ setup_commands:
     pip3 --version > /dev/null 2>&1 || (curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python3 get-pip.py && echo "PATH=$HOME/.local/bin:$PATH" >> ~/.bashrc);
     (type -a python | grep -q python3) || echo 'alias python=python3' >> ~/.bashrc;
     (type -a pip | grep -q pip3) || echo 'alias pip=pip3' >> ~/.bashrc;
-    (which conda > /dev/null 2>&1 && conda init > /dev/null && conda config --set auto_activate_base false) || (wget -nc https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b && eval "$(~/miniconda3/bin/conda shell.bash hook)" && conda init && conda config --set auto_activate_base true);
+    (which conda > /dev/null 2>&1 && conda init > /dev/null) || (wget -nc https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b && eval "$(~/miniconda3/bin/conda shell.bash hook)" && conda init && conda config --set auto_activate_base true);
     source ~/.bashrc;
     (pip3 list | grep ray | grep {{ray_version}} 2>&1 > /dev/null || pip3 install -U ray[default]=={{ray_version}}) && mkdir -p ~/sky_workdir && mkdir -p ~/.sky/sky_app;
     (pip3 list | grep skypilot && [ "$(cat {{sky_remote_path}}/current_sky_wheel_hash)" == "{{sky_wheel_hash}}" ]) || (pip3 uninstall skypilot -y; pip3 install "$(echo {{sky_remote_path}}/{{sky_wheel_hash}}/skypilot-{{sky_version}}*.whl)[aws]" && echo "{{sky_wheel_hash}}" > {{sky_remote_path}}/current_sky_wheel_hash || exit 1);
diff --git a/tests/test_smoke.py b/tests/test_smoke.py
index c3b536f5a35..ac0d432f016 100644
--- a/tests/test_smoke.py
+++ b/tests/test_smoke.py
@@ -262,6 +262,24 @@ def test_image_id_dict_with_zone():
     run_one_test(test)
 
 
+def test_image_no_conda():
+    name = _get_cluster_name()
+    test = Test(
+        'image_no_conda',
+        [
+            # Use image id dict.
+            f'sky launch -y -c {name} examples/tests/test_yamls/no_conda_ami.yaml',
+            f'sky logs {name} 1 --status',
+            f'sky stop {name} -y',
+            f'sky start {name} -y',
+            f'sky exec {name} examples/tests/test_yamls/no_conda_ami.yaml',
+            f'sky logs {name} 2 --status',
+        ],
+        f'sky down -y {name}',
+    )
+    run_one_test(test)
+
+
 def test_stale_job():
     name = _get_cluster_name()
     test = Test(


Code review comments:
diff_hunk: @@ -0,0 +1,9 @@
+resources:
+  cloud: aws
+  instance_type: g4dn.xlarge
+  region: us-west-2
+  image_id: ami-0fe5af21074ad2a10  # Deep learning AMI with CUDA 11.6
path: examples/tests/test_yamls/no_conda_ami.yaml
text: @user1:
```suggestion
  image_id: ami-0fe5af21074ad2a10  # Deep learning AMI with CUDA 11.6 without conda installed
```