Repository: skypilot-org/skypilot
Pull request: [Azure] Downgrade the image version for K80 instances
Fixes #449 and probably closes #133 

I found that the MS-DSVM image v21.08.30 uses NVIDIA driver 470, which does not incur the GPU loading error on Ubuntu machines. Let's use this version of image until Azure releases a new fixed image for K80.



Problem statement:
Azure K80 provision fails
Azure will raise the following error when launching an instance with K80. We may need to update the `azure-ray.yml.j2` to avoid this error.
```bash
Exception Details:	(BadRequest) {
	  "error": {
	    "code": "BadRequest",
	    "message": "The selected VM size 'Standard_NC6_Promo' cannot boot Hypervisor Generation '2'. If this was a Create operation please check that the Hypervisor Generation of the Image matches the Hypervisor Generation of the selected VM Size. If this was an Update operation please select a Hypervisor Generation '2' VM Size."
	  }
	}
	Code: BadRequest
	Message: {
	  "error": {
	    "code": "BadRequest",
	    "message": "The selected VM size 'Standard_NC6_Promo' cannot boot Hypervisor Generation '2'. If this was a Create operation please check that the Hypervisor Generation of the Image matches the Hypervisor Generation of the selected VM Size. If this was an Update operation please select a Hypervisor Generation '2' VM Size."
	  }
	}
```

[Azure] Cuda driver fails on Azure's K80 gpunode
Looks like the image `microsoft-dsvm` we're based on doesn't work on Azure's K80 instance. Thanks @WoosukKwon for reporting this.
```
azureuser@ray-sky-gpunode-weichiang-head-9ecb80c80:~$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
```

Hints from issue discussion:
+1, have noticed this as well.
Found some related discussion
https://docs.microsoft.com/en-us/answers/questions/658624/nc6-dsvm-ubuntu-1804-gen-1-nvidia-smi-has-failed-b.html
looks like DSVM image is broken on K80.
One possible quick workaround is downgrading the Nvidia driver by executing:
```bash
sudo apt purge libnvidia*
sudo apt update
sudo apt install nvidia-driver-470
sudo reboot
```
After rebooting, I could use k80 for simple PyTorch programs.
I don't know how this will affect other software/program though.
Yes that's definitely one way of fixing this issue but it'd also delay the machine setup time quite a bit. I think a better solution might be to find an Azure image that directly works on K80.

Commit message: Remove a reference link

Patch to review:
diff --git a/sky/clouds/azure.py b/sky/clouds/azure.py
index f66460bccf4..68f87fa3641 100644
--- a/sky/clouds/azure.py
+++ b/sky/clouds/azure.py
@@ -79,8 +79,21 @@ def _get_image_config(cls, gen_version, instance_type):
             'image_version': '21.11.04'
         }
 
-        # ubuntu-2004 does not work on A100
+        # ubuntu-2004 v21.10.21 and v21.11.04 do not work on K80
+        # due to an NVIDIA driver issue.
         if instance_type in [
+                'Standard_NC6',
+                'Standard_NC6_Promo',
+                'Standard_NC12',
+                'Standard_NC12_Promo',
+                'Standard_NC24',
+                'Standard_NC24_Promo',
+                'Standard_NC24r',
+                'Standard_NC24r_Promo',
+        ]:
+            image_config['image_version'] = '21.08.30'
+        # ubuntu-2004 does not work on A100
+        elif instance_type in [
                 'Standard_ND96asr_v4', 'Standard_ND96amsr_A100_v4'
         ]:
             image_config['image_offer'] = 'ubuntu-hpc'


Code review comments:
diff_hunk: @@ -79,8 +79,21 @@ def _get_image_config(cls, gen_version, instance_type):
             'image_version': '21.11.04'
         }
 
-        # ubuntu-2004 does not work on A100
+        # ubuntu-2004 v21.10.21 and v21.11.04 do not work on K80
+        # due to an NVIDIA driver issue.
         if instance_type in [
+                'Standard_NC6',
+                'Standard_NC6_Promo',
+                'Standard_NC12',
+                'Standard_NC12_Promo',
+                'Standard_NC24',
+                'Standard_NC24_Promo',
+                'Standard_NC24r',
+                'Standard_NC24r_Promo',
+        ]:
path: sky/clouds/azure.py
text: @user1:
Instead of listing all K80 instance names, would the code logic be cleaner if we pass the `accelerator_name` into `_get_image_config` and do sth like
```
if acc_name == 'K80':
    image_config['image_version'] = '21.08.30'
```

@author:
Sounds good! Will fix that right away.