Repository: skypilot-org/skypilot
Pull request: Update GCP image for OS consistency and fix GCP's spot CPU
Close https://github.com/sky-proj/sky/issues/607.

Before:
```
ubuntu@ray-k80-gcp-old-head-4b2251cf-compute:~$ lsb_release -d
Description:    Debian GNU/Linux 10 (buster)

ubuntu@ray-k80-gcp-old-head-4b2251cf-compute:~$ nvidia-smi
Fri Mar 18 23:51:52 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
```

Now:
```
gcpuser@ray-v100-gcp-head-209ae428-compute:~$ lsb_release -d
Description:    Ubuntu 20.04.4 LTS

gcpuser@ray-v100-gcp-head-209ae428-compute:~$ nvidia-smi
Sat Mar 19 00:10:33 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
```
CUDA driver version slightly upgraded but should be fine. Tested normally on K80/V100.
CUDA runtime library upgraded from 11.0 to 11.3, which seems to be the latest version we can use according to
```
gcloud compute images list         --project deeplearning-platform-release 
```

However, two changes in this PR:

1) it looks like `ssh_user=='ubuntu'` does not work with the ubuntu image `common-cu113-ubuntu-2004`. Somehow GCP failed to create an `ubuntu` account and we couldn't ssh-login to the machine. Therefore we have to change the `ssh_user` in Ray yaml to `gcpuser` to make it work..
2) @Michaelvll and I found a bug in our jinja template for GCP. It looks like `use_spot` option would not be honored when CPU node is used.

Problem statement:
Inconsistent system version for different cloud
For different clouds, we are using inconsistent systems. I think it would be fine to be on different small versions of `ubuntu 20`, but GCP's `Debian` would be a bit surprising.

AWS:
```
ubuntu@ip-172-31-1-185:~$ lsb_release -d
Description:    Ubuntu 20.04.4 LTS
```

GCP:
```
ubuntu@ray-gcp-cpunode-head-97a5a5a1-compute:~$ lsb_release -d
Description:    Debian GNU/Linux 10 (buster)
```

Azure:
```
azureuser@ray-azure-cpunode-head-6f29a63e0:~$ lsb_release -d
Description:    Ubuntu 20.04.3 LTS
```

Hints from issue discussion:

Commit message: Fix worker

Patch to review:
diff --git a/sky/clouds/gcp.py b/sky/clouds/gcp.py
index 4e820d5067f..4252b4c620d 100644
--- a/sky/clouds/gcp.py
+++ b/sky/clouds/gcp.py
@@ -186,6 +186,7 @@ def make_deploy_resources_variables(self, resources):
             'tpu': None,
             'custom_resources': None,
             'use_spot': r.use_spot,
+            'image_name': 'common-cpu',
         }
         accelerators = r.accelerators
         if accelerators is not None:
@@ -204,6 +205,8 @@ def make_deploy_resources_variables(self, resources):
                 # https://cloud.google.com/compute/docs/gpus
                 resources_vars['gpu'] = 'nvidia-tesla-{}'.format(acc.lower())
                 resources_vars['gpu_count'] = acc_count
+                # CUDA driver version 470.103.01, CUDA Library 11.3
+                resources_vars['image_name'] = 'common-cu113-ubuntu-2004'
 
         return resources_vars
 
diff --git a/sky/templates/gcp-ray.yml.j2 b/sky/templates/gcp-ray.yml.j2
index e02119f0e2d..28862bb1011 100644
--- a/sky/templates/gcp-ray.yml.j2
+++ b/sky/templates/gcp-ray.yml.j2
@@ -20,7 +20,7 @@ provider:
   cache_stopped_nodes: True
 
 auth:
-  ssh_user: ubuntu
+  ssh_user: gcpuser
 
 available_node_types:
   ray_head_default:
@@ -35,9 +35,9 @@ available_node_types:
             diskSizeGb: {{disk_size}}
             # See https://cloud.google.com/deep-learning-vm/docs/images
 {%- if gpu is none %}
-            sourceImage: projects/deeplearning-platform-release/global/images/family/common-cpu
+            sourceImage: projects/deeplearning-platform-release/global/images/family/{{image_name}}
 {%- else %}
-            sourceImage: projects/deeplearning-platform-release/global/images/family/common-cu110
+            sourceImage: projects/deeplearning-platform-release/global/images/family/{{image_name}}
       guestAccelerators:
         - acceleratorType: projects/intercloud-320520/zones/{{zones}}/acceleratorTypes/{{gpu}}
           acceleratorCount: {{gpu_count}}
@@ -45,12 +45,12 @@ available_node_types:
         items:
           - key: install-nvidia-driver
             value: "True"
+{%- endif %}
       scheduling:
 {%- if use_spot %}
         - preemptible: true
 {%- endif %}
         - onHostMaintenance: TERMINATE  # Required for GPU-attached VMs.
-{%- endif %}
 {% if num_nodes > 1 %}
   ray_worker_default:
     min_workers: {{num_nodes - 1}}
@@ -66,9 +66,9 @@ available_node_types:
             diskSizeGb: {{disk_size}}
             # See https://cloud.google.com/deep-learning-vm/docs/images
 {%- if gpu is none %}
-            sourceImage: projects/deeplearning-platform-release/global/images/family/common-cpu
+            sourceImage: projects/deeplearning-platform-release/global/images/family/{{image_name}}
 {%- else %}
-            sourceImage: projects/deeplearning-platform-release/global/images/family/common-cu110
+            sourceImage: projects/deeplearning-platform-release/global/images/family/{{image_name}}
       guestAccelerators:
         - acceleratorType: projects/intercloud-320520/zones/{{zones}}/acceleratorTypes/{{gpu}}
           acceleratorCount: {{gpu_count}}
@@ -76,13 +76,13 @@ available_node_types:
         items:
           - key: install-nvidia-driver
             value: "True"
+{%- endif %}
       scheduling:
 {%- if use_spot %}
         - preemptible: true
 {%- endif %}
         - onHostMaintenance: TERMINATE  # Required for GPU-attached VMs.
 {%- endif %}
-{%- endif %}
 
 head_node_type: ray_head_default
 


Code review comments:
diff_hunk: @@ -186,6 +186,7 @@ def make_deploy_resources_variables(self, resources):
             'tpu': None,
             'custom_resources': None,
             'use_spot': r.use_spot,
+            'image_name': 'common-cpu',
path: sky/clouds/gcp.py
text: @user1:
is this `ubuntu 20.04`?

@author:
Ah my bad. Just updated to the correct image name.