Repository: skypilot-org/skypilot
Pull request: GCP/provisioner: Handle the RESOURCE_NOT_FOUND error.
<!-- Describe the changes in this PR -->

Fixes #1797.

I've tried a few days and finally got a set_trace inside `need_ray_up()`, tested that the `re.search()` there works. This doesn't test things end-to-end, however.
```
sky launch -y --gpus A100-80GB:8 --cloud gcp --down -c dbg -r 
```

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [ ] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `pytest tests/test_smoke.py` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [ ] Backward compatibility tests: `bash tests/backward_comaptibility_tests.sh`


Problem statement:
RESOURCE_NOT_FOUND error in launching a GCP VM
I met this error when launching an 8xA100 VM on GCP. The VM was initiated despite the error.

```
I 03-19 17:18:05 optimizer.py:617] == Optimizer ==
I 03-19 17:18:05 optimizer.py:628] Target: minimizing cost
I 03-19 17:18:05 optimizer.py:640] Estimated cost: $29.4 / hour
I 03-19 17:18:05 optimizer.py:640] 
I 03-19 17:18:05 optimizer.py:712] Considered resources (1 node):
I 03-19 17:18:05 optimizer.py:760] --------------------------------------------------------------------------------------------
I 03-19 17:18:05 optimizer.py:760]  CLOUD   INSTANCE        vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN   
I 03-19 17:18:05 optimizer.py:760] --------------------------------------------------------------------------------------------
I 03-19 17:18:05 optimizer.py:760]  GCP     a2-highgpu-8g   96      680       A100:8         us-central1   29.39         ✔     
I 03-19 17:18:05 optimizer.py:760] --------------------------------------------------------------------------------------------
I 03-19 17:18:05 optimizer.py:760] 
Launching a new cluster 'woosuk-test'. Proceed? [Y/n]: y
I 03-19 17:18:09 cloud_vm_ray_backend.py:3327] Creating a new cluster: "woosuk-test" [1x GCP(a2-highgpu-8g, {'A100': 8}, disk_size=1024)].
I 03-19 17:18:09 cloud_vm_ray_backend.py:3327] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 03-19 17:18:10 cloud_vm_ray_backend.py:1156] To view detailed progress: tail -n100 -f /Users/woosuk/sky_logs/sky-2023-03-19-17-18-05-710862/provision.log
I 03-19 17:18:11 cloud_vm_ray_backend.py:1480] Launching on GCP us-central1 (us-central1-a)
W 03-19 17:18:25 cloud_vm_ray_backend.py:621] Got return code ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS in us-central1-a (message: The zone 'projects/xxx/zones/us-central1-a' does not have enough resources available to fulfill the request.  '(resource type:compute)'.)
I 03-19 17:18:35 cloud_vm_ray_backend.py:1480] Launching on GCP us-central1 (us-central1-b)
W 03-19 17:20:16 cloud_vm_ray_backend.py:621] Got return code RESOURCE_NOT_FOUND in us-central1-b (message: The resource 'projects/xxx/zones/us-central1-b/instances/ray-woosuk-test-head-570d9247-compute' was not found)
Clusters
NAME              LAUNCHED     RESOURCES                                           STATUS   AUTOSTOP  COMMAND                       
woosuk-test       41 secs ago  1x GCP(a2-highgpu-8g, {'A100': 8}, disk_size=1024)  INIT     -         sky launch --cloud gcp ll...          

Traceback (most recent call last):
...
  File "/Users/woosuk/workspace/research/sky-proj/skypilot/sky/backends/cloud_vm_ray_backend.py", line 661, in _update_blocklist_on_gcp_error
    assert False, error
AssertionError: {'code': 'RESOURCE_NOT_FOUND', 'message': "The resource 'projects/xxx/zones/us-central1-b/instances/ray-woosuk-test-head-570d9247-compute' was not found"}
```

It seems there exists a GCP error code that is not handled by SkyPilot.

Hints from issue discussion:
Got a similar problem as well!
Got reports that this happened again on spot A100:8 on GCP. @WoosukKwon @Michaelvll do you remember when you encountered the problem, were the on-demand VMs ssh-able?

If so, we can retry `ray up`  on this transient error.
Got the same error again for launching on-demand A100-80GB:8, and checked the console. Seems there is no VM actually launched, and `sky launch` again will just return `RESOURCE_POOL_EXHAUSTED_WITH_DETAILS`. I am guessing this is an implementation bug of GCP, where the resource allocation is not well-locked, causing the system to assign the same resources to multiple users concurrently.

Commit message: GPC/provisioner: Mitigate RESOURCE_NOT_FOUND error for A100s.

Patch to review:
diff --git a/sky/backends/cloud_vm_ray_backend.py b/sky/backends/cloud_vm_ray_backend.py
index d5eb9ca59b0..be0577b8c0d 100644
--- a/sky/backends/cloud_vm_ray_backend.py
+++ b/sky/backends/cloud_vm_ray_backend.py
@@ -657,6 +657,15 @@ def _update_blocklist_on_gcp_error(
                     # {'code': 8, 'message': 'There is no more capacity in the zone "europe-west4-a"; you can try in another zone where Cloud TPU Nodes are offered (see https://cloud.google.com/tpu/docs/regions) [EID: 0x1bc8f9d790be9142]'} # pylint: disable=line-too-long
                     self._blocked_resources.add(
                         launchable_resources.copy(zone=zone.name))
+                elif code == 'RESOURCE_NOT_FOUND':
+                    # https://github.com/skypilot-org/skypilot/issues/1797
+                    # The VM may be alive on console. In the inner provision
+                    # loop we have used retries to recover but failed. The
+                    # provision loop will terminate the potentially live VMs
+                    # and move onto the next zone. Since the VM may have been
+                    # provisioned in this zone, it doesn't seem right to block
+                    # the current zone.
+                    pass
                 else:
                     assert False, error
         elif len(httperror_str) >= 1:
@@ -1524,6 +1533,18 @@ def need_ray_up(
                         'Retrying due to list request rate limit exceeded.')
                     return True
 
+                # https://github.com/skypilot-org/skypilot/issues/1797
+                # "The resource 'projects/xxx/zones/us-central1-b/instances/ray-yyy-head-<hash>-compute' was not found" # pylint: disable=line-too-long
+                pattern = (r'\'code\': \'RESOURCE_NOT_FOUND\'.*The resource'
+                           r'.*instances\/.*-compute\' was not found')
+                result = re.search(pattern, stderr)
+                if result is not None:
+                    # Retry. Unlikely will succeed if it's due to no capacity.
+                    logger.info(
+                        'Retrying due to the possibly flaky RESOURCE_NOT_FOUND '
+                        'error.')
+                    return True
+
             if ('Processing file mounts' in stdout and
                     'Running setup commands' not in stdout and
                     'Failed to setup head node.' in stderr):


Code review comments:
diff_hunk: @@ -657,6 +657,15 @@ def _update_blocklist_on_gcp_error(
                     # {'code': 8, 'message': 'There is no more capacity in the zone "europe-west4-a"; you can try in another zone where Cloud TPU Nodes are offered (see https://cloud.google.com/tpu/docs/regions) [EID: 0x1bc8f9d790be9142]'} # pylint: disable=line-too-long
                     self._blocked_resources.add(
                         launchable_resources.copy(zone=zone.name))
+                elif code == 'RESOURCE_NOT_FOUND':
+                    # https://github.com/skypilot-org/skypilot/issues/1797
+                    # The VM may be alive on console. In the inner provision
+                    # loop we have used retries to recover but failed. The
+                    # provision loop will terminate the potentially live VMs
+                    # and move onto the next zone. Since the VM may have been
+                    # provisioned in this zone, it doesn't seem right to block
+                    # the current zone.
+                    pass
path: sky/backends/cloud_vm_ray_backend.py
text: @user1:
Since we have done the retry in `_gang_schedule_ray_up` by adding that error in the `need_ray_up`, we should add the zone to the block list here, to avoid the outer failover loop go to that zone again.

@author:
IIRC, without `-r` the outer loop will do a single pass over all zones, so this zone will not be tried again. So blocking vs. not blocking has the same effect. With `-r`, it's desirable to retry this zone. Is this the case?

@user1:
Our yield zones will not loop through all the zones in a region for spot, as we currently rely on the optimizer to generate optimization result for per-zone. That is to say the failover for zones will happen [here](https://github.com/skypilot-org/skypilot/blob/2f5fa21b1d0b1e537ae03a845e214b454a17ad5e/sky/backends/cloud_vm_ray_backend.py#L1795-L1799), but not [here](https://github.com/skypilot-org/skypilot/blob/2f5fa21b1d0b1e537ae03a845e214b454a17ad5e/sky/backends/cloud_vm_ray_backend.py#L1171).
However, since we have add the `to_provision` in the block list [here](https://github.com/skypilot-org/skypilot/blob/2f5fa21b1d0b1e537ae03a845e214b454a17ad5e/sky/backends/cloud_vm_ray_backend.py#L1769), it should explain why it does not go to that zone again. I still prefer to add that to the block list here for consistency with the other parts. Wdyt?

@author:
Oh, that's right. I got caught up thinking `_yield_zones()` will go through the zones. Added now.