Repository: skypilot-org/skypilot
Pull request: [AWS Catalog] Handle the exceptions when the user does not have access to the region 
<!-- Describe the changes in this PR -->

Describe the changes in this PR:

Fixes #1459 

Our previous `fetch_availability_zone_mappings` does not handle the case where the user does not have access to some of the regions, causing the problem in #1459, where the @iojw fails to fetch his own mapping.

**The previous problem could affect all the AWS users without the `ap-east-1` access.**

It also adds the integrity check to make sure that the auto-update of the catalog will always fetch all the regions.


<!-- Describe tests ran in a "Tested:" section -->
<!-- Unit tests (tests/test_*.py) are part of Github CI; the below are additional tests. -->

Tested (run the relevant ones):

- [x] `python -m sky.clouds.service_catalog.data_fetchers.fetch_aws --strict-all-regions`
- [x] `python -m sky.clouds.service_catalog.data_fetchers.fetch_aws` using an account without access to all regions
- [x] `sky launch -c test-mapping --cloud aws` on an account without access to all regions
- [x] @iojw, could you test with this PR to see if this will correctly generate the `az_mapping` for your account?


Problem statement:
No usable subnets matching availability zone
For the `us-west-1` region on AWS, my account only has access to the `us-west-1b` and `us-west-1c` zones, but SkyPilot does not take this into account. This causes the following error when running some of the smoke tests:

```
+ sky status --refresh | grep test-autostop-a64f6128-dd | grep STOPPED
Error: No usable subnets matching availability zone ['us-west-1a'] found for node type ray.head.default.
Choose a different availability zone or try manually creating an instance in your specified region to populate the list of subnets and trying this again.
```

The `aws ec2 describe-availability-zones --region region-name` command can determine what zones a user has access to. e.g. for me:

```
❯ aws ec2 describe-availability-zones --region us-west-1
{
    "AvailabilityZones": [
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "us-west-1",
            "ZoneName": "us-west-1b",
            "ZoneId": "usw1-az3",
            "GroupName": "us-west-1",
            "NetworkBorderGroup": "us-west-1",
            "ZoneType": "availability-zone"
        },
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "us-west-1",
            "ZoneName": "us-west-1c",
            "ZoneId": "usw1-az1",
            "GroupName": "us-west-1",
            "NetworkBorderGroup": "us-west-1",
            "ZoneType": "availability-zone"
        }
    ]
}
```

Hints from issue discussion:
cc @WoosukKwon @infwinston 
thanks @iojw I think this issue could be related to https://github.com/skypilot-org/skypilot/issues/1461.

My AZ looks like this:
```
weichiang@mbp tpu-example % aws ec2 describe-availability-zones --region us-west-1
{
    "AvailabilityZones": [
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "us-west-1",
            "ZoneName": "us-west-1a",
            "ZoneId": "usw1-az1",
            "GroupName": "us-west-1",
            "NetworkBorderGroup": "us-west-1",
            "ZoneType": "availability-zone"
        },
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "us-west-1",
            "ZoneName": "us-west-1c",
            "ZoneId": "usw1-az3",
            "GroupName": "us-west-1",
            "NetworkBorderGroup": "us-west-1",
            "ZoneType": "availability-zone"
        }
    ]
}
```

The available AZ IDs are `usw1-az1` and `usw1-az3`. For my account, they are named `us-west-1a` and `us-west-1c`.
But even so, I'm not exactly sure why an incorrect zone is returned as the code we use to get the zone name should be per account basis.
https://github.com/skypilot-org/skypilot/blob/master/sky/clouds/aws.py#L144
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html

Could you paste the entire logs for the smoke test?
@infwinston Here's one of the logs:

```
❯ cat /var/folders/m_/gym1_4y53m95hb50x8qc1myw0000gn/T/image_id_dict_with_region-kky84fu6.log
+ sky launch -y -c test-image-id-dict-with-zone-a64f6128-c4 --zone us-west-2b examples/per_region_images.yaml && exit 1 || true
Task from YAML spec: examples/per_region_images.yaml
W 11-29 18:38:59 resources.py:654] image_id in resources is experimental. It only supports AWS/GCP.
ValueError: image_id {'us-east-1': 'ami-0729d913a335efca7', 'us-west-1': 'skypilot:gpu-ubuntu-1804'} should contain the image for the specified region us-west-2.
+ sky status | grep test-image-id-dict-with-zone-a64f6128-c4 && exit 1 || true
+ sky launch -y -c test-image-id-dict-with-zone-a64f6128-c4 --zone us-west-1a examples/per_region_images.yaml
Task from YAML spec: examples/per_region_images.yaml
W 11-29 18:39:01 resources.py:654] image_id in resources is experimental. It only supports AWS/GCP.
D 11-29 18:39:02 optimizer.py:232] #### Task(run='conda env list\n')
D 11-29 18:39:02 optimizer.py:232]   resources: {AWS(None, image_id={'us-west-1': 'skypilot:gpu-ubuntu-1804'})} ####
D 11-29 18:39:02 optimizer.py:261] Defaulting the task's estimated time to 1 hour.
D 11-29 18:39:02 optimizer.py:278] resources: AWS(m6i.2xlarge, image_id={'us-west-1': 'skypilot:gpu-ubuntu-1804'})
D 11-29 18:39:02 optimizer.py:287]   estimated_runtime: 3600 s (1.0 hr)
D 11-29 18:39:02 optimizer.py:291]   estimated_cost (not incl. egress): $0.4
I 11-29 18:39:02 optimizer.py:606] == Optimizer ==
I 11-29 18:39:02 optimizer.py:629] Estimated cost: $0.4 / hour
I 11-29 18:39:02 optimizer.py:629]
I 11-29 18:39:02 optimizer.py:685] Considered resources (1 node):
I 11-29 18:39:02 optimizer.py:714] ------------------------------------------------------------------
I 11-29 18:39:02 optimizer.py:714]  CLOUD   INSTANCE      vCPUs   ACCELERATORS   COST ($)   CHOSEN
I 11-29 18:39:02 optimizer.py:714] ------------------------------------------------------------------
I 11-29 18:39:02 optimizer.py:714]  AWS     m6i.2xlarge   8       -              0.38          ✔
I 11-29 18:39:02 optimizer.py:714] ------------------------------------------------------------------
I 11-29 18:39:02 optimizer.py:714]
Running task on cluster test-image-id-dict-with-zone-a64f6128-c4...
I 11-29 18:39:02 cloud_vm_ray_backend.py:2862] Creating a new cluster: "test-image-id-dict-with-zone-a64f6128-c4" [1x AWS(m6i.2xlarge, image_id={'us-west-1': 'skypilot:gpu-ubuntu-1804'})].
I 11-29 18:39:02 cloud_vm_ray_backend.py:2862] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 11-29 18:39:03 cloud_vm_ray_backend.py:999] To view detailed progress: tail -n100 -f /Users/isaac/sky_logs/sky-2022-11-29-18-39-02-140854/provision.log
I 11-29 18:39:03 cloud_vm_ray_backend.py:1259] Launching on AWS us-west-1 (us-west-1a)
D 11-29 18:39:10 cloud_vm_ray_backend.py:1338] Ray up takes 7.055246114730835 seconds with 1 retries.
I 11-29 18:39:10 cloud_vm_ray_backend.py:684] ====== stdout ======
2022-11-29 18:39:07,567	INFO commands.py:289 -- Cluster: test-image-id-dict-with-zone-a64f6128-c4
2022-11-29 18:39:08,192	INFO commands.py:366 -- Checking External environment settings
2022-11-29 18:39:08,192	VINFO utils.py:149 -- Creating AWS resource `ec2` in `us-west-1`
2022-11-29 18:39:09,448	VINFO utils.py:149 -- Creating AWS resource `iam` in `us-west-1`
2022-11-29 18:39:09,996	VINFO utils.py:149 -- Creating AWS resource `ec2` in `us-west-1`

I 11-29 18:39:10 cloud_vm_ray_backend.py:687] ====== stderr ======
2022-11-29 18:39:07,567	WARNING util.py:241 -- Dropping the empty legacy field head_node. head_nodeis not supported for ray>=2.0.0. It is recommended to removehead_node from the cluster config.
2022-11-29 18:39:07,567	WARNING util.py:241 -- Dropping the empty legacy field worker_nodes. worker_nodesis not supported for ray>=2.0.0. It is recommended to removeworker_nodes from the cluster config.
2022-11-29 18:39:07,567	INFO util.py:364 -- setting max workers for head node type to 0
2022-11-29 18:39:10,141	PANIC config.py:517 -- No usable subnets matching availability zone ['us-west-1a'] found for node type ray.head.default.
Choose a different availability zone or try manually creating an instance in your specified region to populate the list of subnets and trying this again.
Error: No usable subnets matching availability zone ['us-west-1a'] found for node type ray.head.default.
Choose a different availability zone or try manually creating an instance in your specified region to populate the list of subnets and trying this again.

Clusters
NAME                                        LAUNCHED        RESOURCES                                                                  STATUS  AUTOSTOP  COMMAND
test-stale-job-a64f6128-6b                  a few secs ago  1x GCP(n1-highmem-8)                                                       INIT    -         sky launch -y -c test-sta...
test-huggingface-a64f6128-99                a few secs ago  1x GCP(n1-highmem-8, {'V100': 1})                                          INIT    -         sky launch -y -c test-hug...
test-large-job-queue-a64f6128-97            a few secs ago  1x GCP(n1-highmem-8)                                                       INIT    -         sky launch -y -c test-lar...
test-image-id-dict-with-zone-a64f6128-c4    a few secs ago  1x AWS(m6i.2xlarge, image_id={'us-west-1': 'skypilot:gpu-ubuntu-1804'}...  INIT    -         sky launch -y -c test-ima...
test-image-id-dict-a64f6128-85              a few secs ago  1x AWS(m6i.2xlarge, image_id={'us-east-1': 'ami-0729d913a335efca7'})       INIT    -         sky launch -y -c test-ima...
test-multi-echo-a64f6128-10                 a few secs ago  2x AWS(p2.xlarge, {'K80': 1})                                              INIT    -         examples/multi_echo.py te...
test-logs-a64f6128-d2                       a few secs ago  2x AWS(m6i.2xlarge)                                                        INIT    -         sky launch -y -c test-log...
test-image-id-dict-with-region-a64f6128-d0  a few secs ago  1x AWS(m6i.2xlarge, image_id={'us-west-1': 'skypilot:gpu-ubuntu-1804'}...  INIT    -         sky launch -y -c test-ima...
test-stale-job-manual-restart-a64f6128-10   a few secs ago  1x AWS(m6i.2xlarge)                                                        INIT    -         sky launch -y -c test-sta...
test-job-queue-a64f6128-45                  a few secs ago  1x AWS(p2.xlarge, {'K80': 1})                                              INIT    -         sky launch -y -c test-job...
test-file-mounts-a64f6128-75                a few secs ago  2x AWS(m6i.2xlarge)                                                        INIT    -         sky launch -y -c test-fil...
test-region-a64f6128-1c                     a few secs ago  1x AWS(m6i.2xlarge)                                                        INIT    -         sky launch -y -c test-reg...
test-zone-a64f6128-8a                       a few secs ago  1x AWS(m6i.2xlarge)                                                        INIT    -         sky launch -y -c test-zon...
test-env-check-a64f6128-75                  a few secs ago  2x AWS(m6i.2xlarge)                                                        INIT    -         sky launch -y -c test-env...
test-n-node-job-queue-a64f6128-d6           a few secs ago  2x AWS(p2.xlarge, {'K80': 1})                                              INIT    -         sky launch -y -c test-n-n...
test-minimal-a64f6128-a5                    a few secs ago  1x AWS(m6i.2xlarge, image_id={'us-east-1': 'skypilot:gpu-ubuntu-1804'}...  INIT    -         sky launch -y -c test-min...

RuntimeError: Errors occurred during provision; check logs above.
```

This also causes the cluster to be stuck in INIT status, with the same error occurring if I try to down it.
```
❯ sky down test-image-id-dict-with-zone-a64f6128-c4
Terminating 1 cluster: test-image-id-dict-with-zone-a64f6128-c4. Proceed? [Y/n]: y
Terminating 1 cluster ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--E 11-29 18:42:57 cloud_vm_ray_backend.py:2687]
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] Failed to terminate test-image-id-dict-with-zone-a64f6128-c4. If you want to ignore this error and remove the cluster from the status table, use `sky down --purge`.
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] **** STDOUT ****
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] 2022-11-29 18:42:55,921	INFO commands.py:366 -- Checking External environment settings
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] 2022-11-29 18:42:55,921	VINFO utils.py:149 -- Creating AWS resource `ec2` in `us-west-1`
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] 2022-11-29 18:42:56,574	VINFO utils.py:149 -- Creating AWS resource `iam` in `us-west-1`
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] 2022-11-29 18:42:56,897	VINFO utils.py:149 -- Creating AWS resource `ec2` in `us-west-1`
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687]
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] **** STDERR ****
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] 2022-11-29 18:42:55,729	WARNING util.py:241 -- Dropping the empty legacy field head_node. head_nodeis not supported for ray>=2.0.0. It is recommended to removehead_node from the cluster config.
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] 2022-11-29 18:42:55,729	WARNING util.py:241 -- Dropping the empty legacy field worker_nodes. worker_nodesis not supported for ray>=2.0.0. It is recommended to removeworker_nodes from the cluster config.
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] 2022-11-29 18:42:55,729	INFO util.py:364 -- setting max workers for head node type to 0
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] 2022-11-29 18:42:57,006	PANIC config.py:517 -- No usable subnets matching availability zone ['us-west-1a'] found for node type ray.head.default.
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] Choose a different availability zone or try manually creating an instance in your specified region to populate the list of subnets and trying this again.
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] Error: No usable subnets matching availability zone ['us-west-1a'] found for node type ray.head.default.
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687] Choose a different availability zone or try manually creating an instance in your specified region to populate the list of subnets and trying this again.
E 11-29 18:42:57 cloud_vm_ray_backend.py:2687]
Terminating cluster test-image-id-dict-with-zone-a64f6128-c4...done.
Terminating 1 cluster ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
```

Commit message: fix except

Patch to review:
diff --git a/sky/clouds/service_catalog/data_fetchers/fetch_aws.py b/sky/clouds/service_catalog/data_fetchers/fetch_aws.py
index 1b5ba766815..0e291ade9ca 100644
--- a/sky/clouds/service_catalog/data_fetchers/fetch_aws.py
+++ b/sky/clouds/service_catalog/data_fetchers/fetch_aws.py
@@ -5,7 +5,7 @@
 import datetime
 import os
 import subprocess
-from typing import List, Tuple, Union
+from typing import List, Optional, Tuple, Union
 
 import numpy as np
 import pandas as pd
@@ -13,7 +13,10 @@
 
 from sky.adaptors import aws
 
-# Turn off the regions disabled for a new AWS account by default.
+# Enable all the regions. The regions enabled by the user's AWS account will be
+# filtered by joining with the availability zone mapping in `aws_catalog`.
+# TODO(zhwu): fix the regions with no supported AMI (maybe by finding AMIs
+# similar to the Deep Learning AMI).
 ALL_REGIONS = [
     'us-east-1',
     'us-east-2',
@@ -21,20 +24,20 @@
     'us-west-2',
     'ca-central-1',
     'eu-central-1',
-    # 'eu-central-2',
+    # 'eu-central-2', # no supported AMI
     'eu-west-1',
     'eu-west-2',
-    # 'eu-south-1',
-    # 'eu-south-2',
+    'eu-south-1',
+    # 'eu-south-2', # no supported AMI
     'eu-west-3',
     'eu-north-1',
-    # 'me-south-1',
-    # 'me-central-1',
-    # 'af-south-1',
-    'ap-east-1',  # enable this non-default region due to user request
-    # 'ap-southeast-3',
+    'me-south-1',
+    'me-central-1',
+    'af-south-1',
+    'ap-east-1',
+    'ap-southeast-3',
     'ap-south-1',
-    # 'ap-south-2',
+    # 'ap-south-2', # no supported AMI
     'ap-northeast-3',
     'ap-northeast-2',
     'ap-southeast-1',
@@ -69,10 +72,14 @@ def _get_instance_types(region: str) -> pd.DataFrame:
 
 
 @ray.remote
-def _get_availability_zones(region: str) -> pd.DataFrame:
-    client = aws.client('ec2', region_name=region)
-    zones = []
-    response = client.describe_availability_zones()
+def _get_availability_zones(region: str) -> Optional[pd.DataFrame]:
+    try:
+        client = aws.client('ec2', region_name=region)
+        zones = []
+        response = client.describe_availability_zones()
+    except aws.client_exception():
+        # The user's AWS account may not have access to this region.
+        return None
     for resp in response['AvailabilityZones']:
         zones.append({
             'AvailabilityZoneName': resp['ZoneName'],
@@ -133,9 +140,14 @@ def _get_spot_pricing_table(region: str) -> pd.DataFrame:
 @ray.remote
 def _get_instance_types_df(region: str) -> Union[str, pd.DataFrame]:
     try:
-        df, zone_df, pricing_df, spot_pricing_df = ray.get([
+        # Fetch the zone info first to make sure the account has access to the
+        # region.
+        zone_df = ray.get(_get_availability_zones.remote(region))
+        if zone_df is None:
+            raise RuntimeError(f'No access to region {region}')
+
+        df, pricing_df, spot_pricing_df = ray.get([
             _get_instance_types.remote(region),
-            _get_availability_zones.remote(region),
             _get_pricing_table.remote(region),
             _get_spot_pricing_table.remote(region),
         ])
@@ -302,6 +314,8 @@ def fetch_availability_zone_mappings() -> pd.DataFrame:
     """
     az_mappings = [_get_availability_zones.remote(r) for r in ALL_REGIONS]
     az_mappings = ray.get(az_mappings)
+    # Remove the regions that the user does not have access to.
+    az_mappings = [m for m in az_mappings if m is not None]
     az_mappings = pd.concat(az_mappings)
     return az_mappings
 
@@ -320,18 +334,35 @@ def fetch_availability_zone_mappings() -> pd.DataFrame:
     parser.add_argument('--no-az-mappings',
                         dest='az_mappings',
                         action='store_false')
+    parser.add_argument('--check-regions-integrity',
+                        action='store_true',
+                        help='Check the region integrity of the fetched data.')
     parser.set_defaults(az_mappings=True)
     args = parser.parse_args()
 
     region_filter = ALL_REGIONS if args.all_regions else US_REGIONS
 
+    def _check_regions_integrity(df: pd.DataFrame, name: str):
+        fetched_regions = set(df['Region'].unique())
+        requested_regions = set(region_filter)
+        if fetched_regions != requested_regions:
+            raise RuntimeError(
+                f'Fetched regions {fetched_regions} does not match '
+                f'requested regions {requested_regions} for {name}.')
+
     ray.init()
     instance_df = get_all_regions_instance_types_df(region_filter)
+    if args.check_regions_integrity:
+        _check_regions_integrity(instance_df, 'instance types')
+
     os.makedirs('aws', exist_ok=True)
     instance_df.to_csv('aws/vms.csv', index=False)
     print('AWS Service Catalog saved to aws/vms.csv')
 
     image_df = get_all_regions_images_df()
+    if args.check_regions_integrity:
+        _check_regions_integrity(image_df, 'images')
+
     image_df.to_csv('aws/images.csv', index=False)
     print('AWS Images saved to aws/images.csv')
 


Code review comments:
diff_hunk: @@ -69,10 +72,14 @@ def _get_instance_types(region: str) -> pd.DataFrame:
 
 
 @ray.remote
-def _get_availability_zones(region: str) -> pd.DataFrame:
-    client = aws.client('ec2', region_name=region)
-    zones = []
-    response = client.describe_availability_zones()
+def _get_availability_zones(region: str) -> Optional[pd.DataFrame]:
+    try:
+        client = aws.client('ec2', region_name=region)
+        zones = []
path: sky/clouds/service_catalog/data_fetchers/fetch_aws.py
text: @user1:
nit: move outside

diff_hunk: @@ -320,18 +334,35 @@ def fetch_availability_zone_mappings() -> pd.DataFrame:
     parser.add_argument('--no-az-mappings',
                         dest='az_mappings',
                         action='store_false')
+    parser.add_argument('--check-regions-integrity',
+                        action='store_true',
+                        help='Check the region integrity of the fetched data.')
     parser.set_defaults(az_mappings=True)
     args = parser.parse_args()
 
     region_filter = ALL_REGIONS if args.all_regions else US_REGIONS
 
+    def _check_regions_integrity(df: pd.DataFrame, name: str):
+        fetched_regions = set(df['Region'].unique())
+        requested_regions = set(region_filter)
+        if fetched_regions != requested_regions:
path: sky/clouds/service_catalog/data_fetchers/fetch_aws.py
text: @user1:
When will this happen? Worth a comment?


@author:
It can happen if we fail to fetch any of the regions in `region_filter`, due to network issues, AWS API glitch, etc.

@user1:
I see. If users manually run this fetcher, is it likely such errors happen to them (I've seen some IncompleteRead)? If so, is not checking okay? Asking as I'm thinking whether this flag should be removed and always default to yes.

@author:
Good point! I changed logic of the fetch_aws:
1. We check the fetched regions contains all the regions enabled with the user's account by default
2. We always fetch all the regions, instead of the US regions
3. If `--strict-all-regions` is specified, we will check if the fetched regions match the `ALL_REGIONS` list despite the enabled regions of the user's account.
PTAL.

diff_hunk: @@ -320,18 +334,35 @@ def fetch_availability_zone_mappings() -> pd.DataFrame:
     parser.add_argument('--no-az-mappings',
                         dest='az_mappings',
                         action='store_false')
+    parser.add_argument('--check-regions-integrity',
+                        action='store_true',
+                        help='Check the region integrity of the fetched data.')
path: sky/clouds/service_catalog/data_fetchers/fetch_aws.py
text: @user1:
At first glance not sure what "region integrity" means. 

@author:
Fixed the helpstr and changed option name. How does that look?