Repository: skypilot-org/skypilot
Pull request: [Dependency] Remove AWS dependency by default
<!-- Describe the changes in this PR -->

Fixes #2810

This PR removes AWS dependency from the default installation dependency.

Reason:
1. AWS dependency is not actually lightweight - with AWS dependency: 39.5s, without AWS dependency: 31.1s (20% difference)
2. AWS depdency has a strict version requirement `colorama < 0.4.5` which is quite old and can conflict with users' existing environment.
3. `sky show-gpus` should be available without awscli installed.

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [ ] Code formatting: `bash format.sh`
- [x] Any manual or new tests for this PR (please specify below)
  - [x] `sky show-gpus` without `aws` dependencies installed
- [ ] All smoke tests: `pytest tests/test_smoke.py` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [ ] Backward compatibility tests: `bash tests/backward_comaptibility_tests.sh`


Problem statement:
[Core] Don't include AWS dependencies by default
When SkyPilot was first released, it was a convenience to include AWS dependencies by default upon install, and according to the comment in setup.py, was a bit of a "why not" because it installed quickly. However, wow there are many backends supported, including some which don't use a cloud provider at all, and the included dependencies are actually causing breakage.

First of all, awscli 2 or higher has no pypi install, and isn't even recognized by Pypi. So a user uses Sagemaker APIs and requires AWSCLI >2, when they install SkyPilot it will overwrite their awscli installation with a 1.xxx version, and they will only figure out what's happening if they manually pip uninstall the 1.xxx version. Essentially, this makes projects which use both SkyPilot and SageMaker (like Runhouse) impossible. Second, even if you manually uninstall the awscli 1.xxxx version after installing SkyPilot, your environment **still breaks,** like so:

```
**$ pip uninstall awscli**
Found existing installation: awscli 1.30.2
Uninstalling awscli-1.30.2:
  Would remove:
    /Users/donny/miniconda3/envs/runhouse/bin/aws
    /Users/donny/miniconda3/envs/runhouse/bin/aws.cmd
    /Users/donny/miniconda3/envs/runhouse/bin/aws_bash_completer
    /Users/donny/miniconda3/envs/runhouse/bin/aws_completer
    /Users/donny/miniconda3/envs/runhouse/bin/aws_zsh_completer.sh
    /Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/awscli-1.30.2.dist-info/*
    /Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/awscli/*
Proceed (Y/n)? y
  Successfully uninstalled awscli-1.30.2

**$ aws --version**
aws-cli/2.13.20 Python/3.11.5 Darwin/22.4.0 exe/x86_64 prompt/off

**$ runhouse start**
Traceback (most recent call last):
  File "/Users/donny/miniconda3/envs/runhouse/bin/runhouse", line 33, in <module>
    sys.exit(load_entry_point('runhouse', 'console_scripts', 'runhouse')())
  File "/Users/donny/miniconda3/envs/runhouse/bin/runhouse", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/donny/code/runhouse/runhouse/__init__.py", line 1, in <module>
    from runhouse.resources.blobs import blob, Blob, file, File
  File "/Users/donny/code/runhouse/runhouse/resources/blobs/__init__.py", line 1, in <module>
    from .blob import Blob, blob
  File "/Users/donny/code/runhouse/runhouse/resources/blobs/blob.py", line 5, in <module>
    from runhouse.resources.envs import _get_env_from, Env
  File "/Users/donny/code/runhouse/runhouse/resources/envs/__init__.py", line 1, in <module>
    from .conda_env import CondaEnv
  File "/Users/donny/code/runhouse/runhouse/resources/envs/conda_env.py", line 8, in <module>
    from runhouse.globals import obj_store
  File "/Users/donny/code/runhouse/runhouse/globals.py", line 8, in <module>
    from runhouse.servers.obj_store import ObjStore
  File "/Users/donny/code/runhouse/runhouse/servers/obj_store.py", line 6, in <module>
    import ray
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/ray/__init__.py", line 123, in <module>
    from ray._private.worker import (  # noqa: E402,F401
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/ray/_private/worker.py", line 47, in <module>
    import ray._private.parameter
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/ray/_private/parameter.py", line 4, in <module>
    import pkg_resources
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3260, in <module>
    def _initialize_master_working_set():
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3272, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/pkg_resources/__init__.py", line 581, in _build_master
    ws.require(__requires__)
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/pkg_resources/__init__.py", line 909, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/donny/miniconda3/envs/runhouse/lib/python3.10/site-packages/pkg_resources/__init__.py", line 795, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'awscli>=1.27.10' distribution was not found and is required by skypilot
```

Lastly, the awscli, botocore, and boto3 package versions are highly coupled and historically have created challenges for those who directly depend on them. If I don't plan to use SkyPilot with AWS, but I use the AWS libraries elsewhere, the AWS extras just create install pain with my existing libraries. In the past when Runhouse was depending on both SkyPilot and s3fs, we saw pip searching through hundreds of versions of these libraries to find a compatible set. We actually had to find a workaround to drop s3fs essentially for this reason.

Unfortunately, there is no way to create a default extras string so this change doesn't disrupt anyone who is depending on `pip install skypilot` to inherit these dependencies. I would argue that you'll eventually need to make this change anyway, because enterprise users will be demanding about the install set and won't want you to install cloud-specific extras they don't use or need. If you really really want to include aws installs by default, perhaps you could publish a skypilot-minimal package to Pypi alongside the core one so we can depend on that instead.

Hints from issue discussion:
Just adding to this issue, as there's frequent dependency issues for numerous other libraries, I'd suggest not having `awscli` in any released package. I consider best practice to use `boto3`, and would push for refactoring any existing code depending on `awscli` to use `boto3`, as the former is built on top of the latter. Pinning versions for other conflicting libraries as they're reported will garner unnecessary development creep.

Internally at my work, I've pushed for installing `awscli` through [official](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) methods, so that there would be no interference between this dev tool dependency and other projects/ internal tool releases.

Commit message: Fix aws check credentials

Patch to review:
diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst
index 70b6f9c4ab6..464cda0a99d 100644
--- a/docs/source/getting-started/installation.rst
+++ b/docs/source/getting-started/installation.rst
@@ -23,7 +23,7 @@ Install SkyPilot using pip:
 
           pip install skypilot-nightly
 
-          # Or, choose an extra (defaults to [aws]):
+          # Or, choose an extra:
           pip install "skypilot-nightly[aws]"
           pip install "skypilot-nightly[gcp]"
           pip install "skypilot-nightly[azure]"
@@ -46,7 +46,7 @@ Install SkyPilot using pip:
 
           pip install skypilot
 
-          # Or, choose an extra (defaults to [aws]):
+          # Or, choose an extra:
           pip install "skypilot[aws]"
           pip install "skypilot[gcp]"
           pip install "skypilot[azure]"
@@ -71,7 +71,7 @@ Install SkyPilot using pip:
           cd skypilot
           pip install -e .
 
-          # Or, choose an extra (defaults to [aws]):
+          # Or, choose an extra:
           pip install -e ".[aws]"
           pip install -e ".[gcp]"
           pip install -e ".[azure]"
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 541f8d6f91e..a7e798d01a8 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -165,3 +165,10 @@ Documentation
    reference/cli
    reference/api
    reference/config
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Developer Guides
+
+   contribute/contributing
+   contribute/new-cloud
diff --git a/sky/clouds/aws.py b/sky/clouds/aws.py
index b524c3a9d96..066b5480df8 100644
--- a/sky/clouds/aws.py
+++ b/sky/clouds/aws.py
@@ -106,7 +106,6 @@ class AWS(clouds.Cloud):
     _INDENT_PREFIX = '    '
     _STATIC_CREDENTIAL_HELP_STR = (
         'Run the following commands:'
-        f'\n{_INDENT_PREFIX}  $ pip install boto3'
         f'\n{_INDENT_PREFIX}  $ aws configure'
         f'\n{_INDENT_PREFIX}  $ aws configure list  # Ensure that this shows identity is set.'
         f'\n{_INDENT_PREFIX}For more info: '
@@ -463,7 +462,7 @@ def check_credentials(cls) -> Tuple[bool, Optional[str]]:
                 'AWS CLI is not installed properly. '
                 'Run the following commands:'
                 f'\n{cls._INDENT_PREFIX}  $ pip install skypilot[aws]'
-                f'{cls._INDENT_PREFIX}Credentials may also need to be set. '
+                f'\n{cls._INDENT_PREFIX}Credentials may also need to be set. '
                 f'{cls._STATIC_CREDENTIAL_HELP_STR}')
 
         # Checks if AWS credentials 1) exist and 2) are valid.
diff --git a/sky/clouds/service_catalog/aws_catalog.py b/sky/clouds/service_catalog/aws_catalog.py
index 89334d7cb21..eb0f76f77d8 100644
--- a/sky/clouds/service_catalog/aws_catalog.py
+++ b/sky/clouds/service_catalog/aws_catalog.py
@@ -156,7 +156,7 @@ def _get_df() -> pd.DataFrame:
         if _user_df is None:
             try:
                 _user_df = _fetch_and_apply_az_mapping(_default_df)
-            except RuntimeError as e:
+            except (RuntimeError, ImportError) as e:
                 if config.get_use_default_catalog_if_failed():
                     logger.warning('Failed to fetch availability zone mapping. '
                                    f'{common_utils.format_exception(e)}')
diff --git a/sky/setup_files/setup.py b/sky/setup_files/setup.py
index 5cbc801b2e8..52b1a0a1d30 100644
--- a/sky/setup_files/setup.py
+++ b/sky/setup_files/setup.py
@@ -238,10 +238,6 @@ def parse_readme(readme: str) -> str:
 
 extras_require['all'] = sum(extras_require.values(), [])
 
-# Install aws requirements by default, as it is the most common cloud provider,
-# and the installation is quick.
-install_requires += extras_require['aws']
-
 long_description = ''
 readme_filepath = 'README.md'
 # When sky/backends/wheel_utils.py builds wheels, it will not contain the


Code review comments:
diff_hunk: @@ -23,7 +23,7 @@ Install SkyPilot using pip:
 
           pip install skypilot-nightly
 
-          # Or, choose an extra (defaults to [aws]):
+          # Or, choose an extra:
path: docs/source/getting-started/installation.rst
text: @user1:
Do we expect installing `pip install skypilot-nightly` alone is useful? Feels like the most common case for people is to directly go to an extra. If that's the case, maybe remove the no-extra case and hint "Choose your cloud"?

```
To use a subset of the clouds, combine some of the pip extras above to
reduce the dependencies:
```
may also need to change, e.g., "To use a few clouds, combine the pip extras:"

@author:
Good call! Updated. PTAL