Repository: skypilot-org/skypilot
Pull request: Add auto disk tier failover in Azure
<!-- Describe the changes in this PR -->

This PR introduces auto disk tier failover in Azure and fixes one problem in #618 .

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [ ] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `pytest tests/test_smoke.py` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [ ] Backward compatibility tests: `bash tests/backward_comaptibility_tests.sh`


Problem statement:
Failed to launch Azure gpunode
GCP cpunode is unstable. I tried to `sky launch` 10 cpu nodes. But only 2 out of 10 succeeded.

Errors happened during `ray up`.
```
  [2/7] Processing file mounts
Shared connection to 35.233.131.63 closed.
    ~/.sky/sky_wheels/ from /tmp/sky_wheels_weichiang/
Shared connection to 35.233.131.63 closed.
    ~/.aws/ from /home/eecs/weichiang/.aws/
Shared connection to 35.233.131.63 closed.
    ~/.config/gcloud/ from /home/eecs/weichiang/.config/gcloud/
Shared connection to 35.233.131.63 closed.
    ~/.azure/ from /home/eecs/weichiang/.azure/
Shared connection to 35.233.131.63 closed.
Shared connection to 35.233.131.63 closed.
  New status: update-failed
  !!!
  SSH command failed.
  !!!
  
  Failed to setup head node.
```

I rerun it with `2>&1` to get more logs. Looks like the cause is rsync was broken due to some unknown reason.

```
Shared connection to 34.83.35.163 closed.
Shared connection to 34.83.35.163 closed.
rsync: connection unexpectedly closed (28 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]
2022-03-18 22:20:50,873 PANIC commands.py:724 -- Failed to setup head node.
Error: Failed to setup head node.
```
Weirdly, when I tried `sky gpunode` 8 times. All of them were successful.

My best guess is VM might be unstable during the first few mins after it launched. Since setup time for gpunode is longer, the problem is not triggered. As per @Michaelvll's suggestion, I'm trying to see if adding `sleep` after node launch can solve this..

Hints from issue discussion:
OK... After I added `time.sleep(100)` https://github.com/sky-proj/sky/blob/f9b1d842f48d86610f4b3b2255a4f60a76a63e6d/sky/skylet/providers/gcp/node_provider.py#L193

and launched 10 cpu nodes. All of them becomes successful.
Our above hypothesis of VM instability seems to be true. 
Not sure if it's related to https://github.com/ray-project/ray/issues/16539, https://github.com/ray-project/ray/issues/18755.
Good observation! I just met similar issue with an azure cpunode. After `sky cpunode --cloud azure` and the instance launched successfully, I got kicked out from the instance due to some ssh connection issue, but when I ssh into it again, it worked. 
Yeah I experienced same issue with Azure before. I feel like the ssh connection is unstable when VMs are just launched.
After an offline discussion with Wei-Lin, the `ssh disconnected` problem still causes the `ray up` losing worker node when launching multi-node cluster, even if we add a sleep after instance creation.
I've left a comment in a related Ray issue https://github.com/ray-project/ray/issues/16539#issuecomment-1073138982. They have reopened it but not sure if they're planning to fix this. I'm not sure why `sleep` works for single-node but not multi-node. Still investigating it.
This issue seems to disappear after rolling back to debian-based image https://github.com/sky-proj/sky/pull/636. Closing this for now.
Getting this error while trying to restart a stopped gpunode with `sky gpunode -c flax`.

```
2022-03-24 12:08:30,935	INFO util.py:282 -- setting max workers for head node type to 0
ssh: connect to host 20.228.225.109 port 22: Operation timed out
ssh: connect to host 20.228.225.109 port 22: Operation timed out
ssh: connect to host 20.228.225.109 port 22: Operation timed out
ssh: connect to host 20.228.225.109 port 22: Operation timed out
ssh: connect to host 20.228.225.109 port 22: Operation timed out
ssh: connect to host 20.228.225.109 port 22: Connection refused
ssh: connect to host 20.228.225.109 port 22: Connection refused
Warning: Permanently added '20.228.225.109' (ED25519) to the list of known hosts.
 19:12:12 up 1 min,  1 user,  load average: 5.83, 1.95, 0.70
Shared connection to 20.228.225.109 closed.
Shared connection to 20.228.225.109 closed.
  Failed to setup head node.
Cluster: flax

Loaded cached provider configuration
If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
No head node found. Launching a new cluster. Confirm [y/N]: y [automatic, due to --yes]

Acquiring an up-to-date head node
  Launched a new head node
  Fetching the new head node

<1/1> Setting up head node
  Prepared bootstrap config
  New status: waiting-for-ssh
  [1/7] Waiting for SSH to become available
    Running `uptime` as a test.
    Fetched IP: 20.228.225.109
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    Success.
  Updating cluster configuration. [hash=20b6cd1a2aef4774d3695208a878375c2ac4bc93]
  New status: syncing-files
  [2/7] Processing file mounts
  New status: update-failed
  !!!
  SSH command failed.
  !!!
```

This puts the cluster back into the `INIT` state and now when I try again, the optimizer gets triggered:
```
sky ❯ sky gpunode -c flax

I 03-24 12:18:49 optimizer.py:363] Optimizer - plan minimizing cost (~$0.2):
I 03-24 12:18:49 optimizer.py:378]
I 03-24 12:18:49 optimizer.py:378] TASK     BEST_RESOURCE
I 03-24 12:18:49 optimizer.py:378] gpunode  Azure(Standard_NC6_Promo, {'K80': 1})
I 03-24 12:18:49 optimizer.py:378]
I 03-24 12:18:49 optimizer.py:327] Multiple Azure instances satisfy K80:1. The cheapest Azure(Standard_NC6_Promo, {'K80': 1}) is considered among:
I 03-24 12:18:49 optimizer.py:327] ['Standard_NC6_Promo', 'Standard_NC6'].
I 03-24 12:18:49 optimizer.py:327]
I 03-24 12:18:49 optimizer.py:333] To list more details, run 'sky show-gpus K80'.
I 03-24 12:18:49 cloud_vm_ray_backend.py:782] To view detailed progress: tail -n100 -f /Users/gautam/sky_logs/sky-2022-03-24-12-18-49-294457/provision.log
I 03-24 12:18:49 cloud_vm_ray_backend.py:638] Cluster 'flax' (status: INIT) was previously launched in Azure (eastus). Relaunching in that region.
I 03-24 12:18:50 cloud_vm_ray_backend.py:953] Launching on Azure eastus (all zones)
⠴ Launching
```

in reference to #605 and based on offline convo with @infwinston 
Looks like Azure is experiencing the same SSH connection failure problem... I confirm this happens to me too. I'm suspecting it's the ubuntu-20-04 causing such issue.

```
> sky gpunode -c flax
...
  [3/7] No worker file mounts to sync
  New status: setting-up
  [4/7] No initialization commands to run.
  [5/7] Initalizing command runner
  [6/7] Running setup commands
    (0/4) (which pip3 > /dev/null 2>&1 |...
    (1/4) pip3 uninstall sky -y &> /dev/...
  New status: update-failed
  !!!
  SSH command failed.
  !!!
```
However, it's still mysterious that Azure's image has been upgraded for one month but we didn't meet this issue until now. Not sure if there's other reason causing this trouble..

Commit message: add failover to only specify instance type case

Patch to review:
diff --git a/sky/clouds/azure.py b/sky/clouds/azure.py
index 8227ab8e881..267cdda4f09 100644
--- a/sky/clouds/azure.py
+++ b/sky/clouds/azure.py
@@ -245,6 +245,17 @@ def make_deploy_resources_variables(
         }
 
     def get_feasible_launchable_resources(self, resources):
+
+        def failover_disk_tier(instance_type: str,
+                               disk_tier: Optional[str]) -> Optional[str]:
+            ok, _ = Azure.check_disk_tier(
+                instance_type, disk_tier or clouds.Cloud._DEFAULT_DISK_TIER)
+            if not ok and disk_tier is None:
+                # Auto failover to low disk tier when disk tier
+                # are not specified
+                return 'low'
+            return disk_tier
+
         if resources.use_spot:
             # TODO(zhwu): our azure subscription offer ID does not support spot.
             # Need to support it.
@@ -252,23 +263,28 @@ def get_feasible_launchable_resources(self, resources):
         if resources.instance_type is not None:
             assert resources.is_launchable(), resources
             # Treat Resources(AWS, p3.2x, V100) as Resources(AWS, p3.2x).
-            resources = resources.copy(accelerators=None)
+            resources = resources.copy(
+                accelerators=None,
+                disk_tier=failover_disk_tier(resources.instance_type,
+                                             resources.disk_tier),
+            )
             return ([resources], [])
 
         def _make(instance_list):
             resource_list = []
             for instance_type in instance_list:
-                if Azure.check_disk_tier(instance_type, resources.disk_tier)[0]:
-                    r = resources.copy(
-                        cloud=Azure(),
-                        instance_type=instance_type,
-                        # Setting this to None as Azure doesn't separately bill /
-                        # attach the accelerators.  Billed as part of the VM type.
-                        accelerators=None,
-                        cpus=None,
-                        memory=None,
-                    )
-                    resource_list.append(r)
+                r = resources.copy(
+                    cloud=Azure(),
+                    instance_type=instance_type,
+                    disk_tier=failover_disk_tier(instance_type,
+                                                 resources.disk_tier),
+                    # Setting this to None as Azure doesn't separately bill /
+                    # attach the accelerators.  Billed as part of the VM type.
+                    accelerators=None,
+                    cpus=None,
+                    memory=None,
+                )
+                resource_list.append(r)
             return resource_list
 
         # Currently, handle a filter on accelerators only.


Code review comments:
diff_hunk: @@ -245,30 +245,46 @@ def make_deploy_resources_variables(
         }
 
     def get_feasible_launchable_resources(self, resources):
+
+        def failover_disk_tier(instance_type: str,
+                               disk_tier: Optional[str]) -> Optional[str]:
+            ok, _ = Azure.check_disk_tier(
+                instance_type, disk_tier or clouds.Cloud._DEFAULT_DISK_TIER)
+            if not ok and disk_tier is None:
+                # Auto failover to low disk tier when disk tier
+                # are not specified
+                return 'low'
path: sky/clouds/azure.py
text: @user1:
nit: 
```suggestion
            if disk_tier is None:
                ok, _ = Azure.check_disk_tier(
                    instance_type, clouds.Cloud._DEFAULT_DISK_TIER)
                if not ok:
                    # Auto failover to low disk tier when disk tier
                    # are not specified
                    return 'low'
            # The disk_tier specified by the user will be checked in the
            # initialization of the Resources.
```