Repository: skypilot-org/skypilot
Pull request: [GCP] Use the latest debian image to align python 3.10
<!-- Describe the changes in this PR -->

Fixes #1604 

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: `bash format.sh`
- [x] Any manual or new tests for this PR (please specify below)
  - [x] `sky launch -c test-new-debian --cloud gcp --cpus 2+`
  - [x] `sky launch -c test-new-debian-gpu --clou gcp --gpus T4 nvidia-smi`
- [x] All smoke tests: `pytest tests/test_smoke.py`  (except for #2309)
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [ ] Backward compatibility tests: `bash tests/backward_comaptibility_tests.sh`


Problem statement:
Python version mismatch when migrating from AWS to GCP
# Description of Problem
I recently migrated my workload from AWS to GCP and ran into issues with python versions. AWS's deep learning AMI is based on LTS versions of Ubuntu with Python 3.8+, whereas the default SkyPilot GCP image uses Debian 10 with Python 3.7. This resulted in an error as some libraries in my application relied on features from Python 3.8+. 

Note that the default SkyPilot image for AWS is based on Ubuntu 22.04 and uses Python 3.9.

I believe this is a SkyPilot issue because running workloads on any cloud requires a consistent environment. This can be addressed by launching SkyPilot tasks in images that are uniform across clouds with tools such as docker or the cloud's image marketplace.

# Current workaround
I've currently made some changes to my YAML files to set up and activate conda environments for the specific Python version I need:
```yaml
setup: |
  # Create an environment with Python 3.8.
  conda create -y -n my_task python=3.8
  source activate my_task
  # Install required Python packages.

run: |
  source activate experiments
  # Run the workload script.
```

Note that this workaround only addresses Python version mismatches. Version mismatches in other libraries and programs (e.g. `gcc`) will require a different solution. 


Posted at the request of @romilbhardwaj.

Hints from issue discussion:
I have similar problems with the python versions, so I am following up here. Several tools I use have bumped their minimum python version to py3.8, while the GCP sky-VM default python is still py3.7. I can do a similar workaround like @pschafhalter wrote above. 

I wonder the SkyPilot has some built-in solutions for this, or would it be the user's job to handle python versions?

Besides, another very common change I have to do to the default VM is that I use **`mamba` instead of `conda`** to install packages (especially many bioinfo-specific ones). `conda` takes a ridiculously long time to resolve dependencies (hours), which makes it impossible for me to use. I wonder could the SkyPilot consider adding `mamba` as an option?

My current workaround code in the `setup` section looks like this:
I first download the latest mambaforge (like a miniconda install), I then add mamba to the `$PATH`, then I need to install skypilot again otherwise there will be an error saying `sky` not found (`sky` was installed in the default conda env). I can then install my heavy duty packages and everything seems works fine with me so far.
```{yaml}
setup: |
  if [ -f $HOME/mambaforge/bin/mamba ]
  then
    echo "Skip setup"
  else
    wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh -O $HOME/Mambaforge-Linux-x86_64.sh
    bash $HOME/Mambaforge-Linux-x86_64.sh -b -p $HOME/mambaforge
    echo 'export PATH=$HOME/mambaforge/bin:$PATH' >> $HOME/.bashrc
    source $HOME/.bashrc
    
    pip install --upgrade pip
    pip install "skypilot[gcp]"
    
    gsutil cp gs://ecker-hanqing-wmb-us-west1/wmb/sky/env.yaml ./
    mamba env update -f env.yaml
  fi
```


Another user asks for the ubuntu os system. We need to prioritize this a bit.
Ran into this too. HF Transformers required >= 3.8 ([ref](https://github.com/huggingface/transformers/blob/main/setup.py#L152)). Had to create a conda env manually rather than using sys python (3.7.x) when on the GCP image.

Commit message: Merge branch 'master' of github.com:skypilot-org/skypilot into align-py310-gcp

Patch to review:
diff --git a/sky/clouds/gcp.py b/sky/clouds/gcp.py
index a6d32751928..1b2d395ee83 100644
--- a/sky/clouds/gcp.py
+++ b/sky/clouds/gcp.py
@@ -344,7 +344,7 @@ def make_deploy_resources_variables(
         # --no-standard-images
         # We use the debian image, as the ubuntu image has some connectivity
         # issue when first booted.
-        image_id = 'skypilot:cpu-debian-10'
+        image_id = 'skypilot:cpu-debian-11'
 
         r = resources
         # Find GPU spec, if any.
@@ -389,15 +389,9 @@ def make_deploy_resources_variables(
                     # versions of CUDA as noted below.
                     # CUDA driver version 470.57.02, CUDA Library 11.4
                     image_id = 'skypilot:k80-debian-10'
-                elif acc == 'L4':
-                    # CUDA driver version 525.105.17, CUDA Library 11.8
-                    image_id = 'skypilot:cuda118-debian-11'
                 else:
-                    # Though the image is called cu113, it actually has later
-                    # versions of CUDA as noted below.
-                    # CUDA driver version 510.47.03, CUDA Library 11.6
-                    # Does not support torch==1.13.0 with cu117
-                    image_id = 'skypilot:gpu-debian-10'
+                    # CUDA driver version 525.105.17, CUDA Library 11.8
+                    image_id = 'skypilot:gpu-debian-11'
 
         if resources.image_id is not None:
             if None in resources.image_id:
diff --git a/sky/clouds/service_catalog/common.py b/sky/clouds/service_catalog/common.py
index 040a640a2c5..4491e339194 100644
--- a/sky/clouds/service_catalog/common.py
+++ b/sky/clouds/service_catalog/common.py
@@ -67,7 +67,7 @@ def read_catalog(filename: str,
     """
     assert filename.endswith('.csv'), 'The catalog file must be a CSV file.'
     assert (pull_frequency_hours is None or
-            pull_frequency_hours > 0), pull_frequency_hours
+            pull_frequency_hours >= 0), pull_frequency_hours
     catalog_path = get_catalog_path(filename)
     cloud = cloud_lib.CLOUD_REGISTRY.from_str(os.path.dirname(filename))
 
diff --git a/sky/clouds/service_catalog/gcp_catalog.py b/sky/clouds/service_catalog/gcp_catalog.py
index 8719b9e0224..3a26b8f38ed 100644
--- a/sky/clouds/service_catalog/gcp_catalog.py
+++ b/sky/clouds/service_catalog/gcp_catalog.py
@@ -27,6 +27,11 @@
 _image_df = common.read_catalog('gcp/images.csv',
                                 pull_frequency_hours=_PULL_FREQUENCY_HOURS)
 
+if _image_df[_image_df['Tag'] == 'skypilot:gpu-debian-11'].empty:
+    # Update the image catalog if it does not include the updated images
+    # https://github.com/skypilot-org/skypilot-catalog/pull/25.
+    _image_df = common.read_catalog('gcp/images.csv', pull_frequency_hours=0)
+
 _quotas_df = common.read_catalog('gcp/accelerator_quota_mapping.csv')
 
 _TPU_REGIONS = [


Code review comments:
diff_hunk: @@ -27,6 +27,11 @@
 _image_df = common.read_catalog('gcp/images.csv',
                                 pull_frequency_hours=_PULL_FREQUENCY_HOURS)
 
+if _image_df[_image_df['Tag'] == 'skypilot:gpu-debian-11'].empty:
path: sky/clouds/service_catalog/gcp_catalog.py
text: @user1:
Q: do we expect doing this for future image tags? Or can we have some auto update mechanism in the future?

@author:
Good point! I changed it to automatically refresh when the tag is failed to find in the catalog. Wdyt?