Repository: skypilot-org/skypilot
Pull request #5348 (2025-04-24 10:08:39)

supress warning by setting default value of `asyncio_default_fixture_loop_scope`
<!-- Describe the changes in this PR -->

We see these test warning recently:

```bash
D 04-24 09:58:10 skypilot_config.py:384]
  | /home/buildkite/miniconda3/lib/python3.10/site-packages/pytest_asyncio/plugin.py:217: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
  | The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
  |  
  | warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
  | bringing up nodes.../home/buildkite/miniconda3/lib/python3.10/site-packages/pytest_asyncio/plugin.py:217: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
  | The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
  |  
  | warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
  | bringing up nodes.../home/buildkite/miniconda3/lib/python3.10/site-packages/pytest_asyncio/plugin.py:217: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
  | The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
  |  
  | warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
  | bringing up nodes.../home/buildkite/miniconda3/lib/python3.10/site-packages/pytest_asyncio/plugin.py:217: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
  | The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
  |  
  | warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
  | bringing up nodes.../home/buildkite/miniconda3/lib/python3.10/site-packages/pytest_asyncio/plugin.py:217: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
  | The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
  |  
  | warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
  | /home/buildkite/miniconda3/lib/python3.10/site-packages/pytest_asyncio/plugin.py:217: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
  | The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
  |  
  | warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
  | bringing up nodes.../home/buildkite/miniconda3/lib/python3.10/site-packages/pytest_asyncio/plugin.py:217: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
  | The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
  |  
  | warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
  | bringing up nodes.../home/buildkite/miniconda3/lib/python3.10/site-packages/pytest_asyncio/plugin.py:217: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
  | The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
  |  

<br class="Apple-interchange-newline">
```

[buildkite link](https://buildkite.com/skypilot-1/smoke-tests/builds/798)

Set default value of `asyncio_default_fixture_loop_scope` to supress warning

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: install pre-commit (auto-check on commit) or `bash format.sh`
- [x] Relevant individual tests: `/smoke-test --aws -k test_multi_echo` (CI) 

<!-- CI commands (/-prefixed) can only be triggered by repo members -->


zpoint: /smoke-test --aws -k test_multi_echo

Patch:
diff --git a/.github/workflows/dashboard.yml b/.github/workflows/dashboard.yml
index 94af4b90f..2809cf6a9 100644
--- a/.github/workflows/dashboard.yml
+++ b/.github/workflows/dashboard.yml
@@ -23,3 +23,4 @@ jobs:
         npm --prefix sky/dashboard install
         npm --prefix sky/dashboard run lint
         npm --prefix sky/dashboard run format:check
+        npm --prefix sky/dashboard run build
diff --git a/.github/workflows/release-build.yml b/.github/workflows/release-build.yml
index 225e78c0e..c8f57c306 100644
--- a/.github/workflows/release-build.yml
+++ b/.github/workflows/release-build.yml
@@ -65,6 +65,9 @@ jobs:
           echo "pypi_base_branch=${PYPI_BASE_BRANCH}" >> $GITHUB_OUTPUT
           echo "Determined PyPI base branch for comparison: ${PYPI_BASE_BRANCH}"
 
+          # Output the latest PyPI version for subsequent steps
+          echo "latest_pypi_version=${LATEST_PYPI_VERSION}" >> $GITHUB_OUTPUT
+
           # Parse latest PyPI version
           PYPI_MAJOR=$(echo $LATEST_PYPI_VERSION | cut -d. -f1)
           PYPI_MINOR=$(echo $LATEST_PYPI_VERSION | cut -d. -f2)
@@ -85,6 +88,97 @@ jobs:
             exit 1
           fi
 
+      - name: Install the latest version of uv
+        uses: astral-sh/setup-uv@v4
+        with:
+          version: "latest"
+          python-version: '3.10'
+
+      - name: Verify API Version Compatibility
+        id: verify_api_version
+        run: |
+          RELEASE_VERSION="${{ github.event.inputs.release_version }}"
+          LATEST_PYPI_VERSION="${{ steps.verify_version.outputs.latest_pypi_version }}"
+          MANUAL_CREATED_RELEASE_BRANCH="${{ steps.validate_input_version.outputs.manual_created_release_branch }}"
+
+          # Checkout the manual created release branch to read the correct API version
+          echo "Checking out ${MANUAL_CREATED_RELEASE_BRANCH} to verify API version..."
+          git checkout ${MANUAL_CREATED_RELEASE_BRANCH}
+
+          echo "Reading current API version from the current checkout..."
+          # Extract current API version from the current checked-out state
+          CURRENT_API_VERSION=$(grep "^API_VERSION = " sky/server/constants.py | sed "s/API_VERSION = '\\(.*\\)'/\\1/")
+          echo "Current API version (from checkout): ${CURRENT_API_VERSION}"
+
+          # Create and activate uv virtual environment
+          echo "Creating uv virtual environment..."
+          uv venv --seed ~/pypi-check-env
+          echo "Activating virtual environment..."
+          source ~/pypi-check-env/bin/activate
+
+          echo "Installing skypilot==${LATEST_PYPI_VERSION} from PyPI using uv..."
+          uv pip install --prerelease=allow "azure-cli>=2.65.0" "omegaconf>=2.4.0dev3"
+          uv pip install skypilot==${LATEST_PYPI_VERSION}
+
+          echo "Extracting API version from installed PyPI package..."
+          # Store current directory and change to a temporary one to avoid importing local sky package
+          ORIGINAL_DIR=$(pwd)
+          TEMP_DIR=$(mktemp -d)
+          cd "${TEMP_DIR}"
+          echo "Changed directory to ${TEMP_DIR} to ensure correct package import."
+
+          # Python command now runs within the activated venv and outside the project directory
+          PYPI_API_VERSION=$(python -c "import sky.server.constants; print(sky.server.constants.API_VERSION)")
+
+          # Change back to original directory and clean up temp dir
+          cd "${ORIGINAL_DIR}"
+          rm -rf "${TEMP_DIR}"
+          echo "Restored original directory: ${ORIGINAL_DIR}"
+
+          # Deactivate environment
+          deactivate
+
+          if [[ -z "${PYPI_API_VERSION}" ]]; then
+            echo "Error: Could not fetch API_VERSION from the installed PyPI package. Python command likely failed."
+            exit 1
+          fi
+          echo "Latest PyPI API version: ${PYPI_API_VERSION}"
+
+          # Assert current API version >= PyPI API version
+          if [[ "${CURRENT_API_VERSION}" -lt "${PYPI_API_VERSION}" ]]; then
+            echo "Error: Current API version (${CURRENT_API_VERSION}) is less than the latest PyPI API version (${PYPI_API_VERSION})."
+            exit 1
+          fi
+          echo "Assertion passed: Current API version (${CURRENT_API_VERSION}) >= PyPI API version (${PYPI_API_VERSION})."
+
+          # If API version changed, ensure it's not just a patch release
+          if [[ "${CURRENT_API_VERSION}" -gt "${PYPI_API_VERSION}" ]]; then
+            echo "API version has increased from ${PYPI_API_VERSION} to ${CURRENT_API_VERSION}."
+
+            # Parse versions
+            REL_MAJOR=$(echo $RELEASE_VERSION | cut -d. -f1)
+            REL_MINOR=$(echo $RELEASE_VERSION | cut -d. -f2)
+
+            PYPI_MAJOR=$(echo $LATEST_PYPI_VERSION | cut -d. -f1)
+            PYPI_MINOR=$(echo $LATEST_PYPI_VERSION | cut -d. -f2)
+
+            IS_MINOR_OR_MAJOR_UPGRADE=false
+            if [[ "${REL_MAJOR}" -gt "${PYPI_MAJOR}" ]]; then
+              IS_MINOR_OR_MAJOR_UPGRADE=true
+            elif [[ "${REL_MAJOR}" -eq "${PYPI_MAJOR}" && "${REL_MINOR}" -gt "${PYPI_MINOR}" ]]; then
+              IS_MINOR_OR_MAJOR_UPGRADE=true
+            fi
+
+            if [[ "${IS_MINOR_OR_MAJOR_UPGRADE}" == "false" ]]; then
+              echo "Error: API version changed (${PYPI_API_VERSION} -> ${CURRENT_API_VERSION}), but the release (${RELEASE_VERSION}) is only a patch upgrade from the latest PyPI version (${LATEST_PYPI_VERSION}). API version changes require a minor or major version bump."
+              exit 1
+            else
+              echo "API version change is accompanied by a minor or major version bump (${LATEST_PYPI_VERSION} -> ${RELEASE_VERSION}). Proceeding."
+            fi
+          else
+            echo "API version has not changed (${CURRENT_API_VERSION}). Proceeding."
+          fi
+
       - name: Set release version and commit changes
         id: commit_changes
         run: |
diff --git a/.github/workflows/release-publish.yml b/.github/workflows/release-publish.yml
index aa427458d..e4a73e47e 100644
--- a/.github/workflows/release-publish.yml
+++ b/.github/workflows/release-publish.yml
@@ -79,21 +79,19 @@ jobs:
           repository-url: https://test.pypi.org/legacy/
           skip-existing: true
 
-      - name: Publish distribution to PyPI
-        uses: pypa/gh-action-pypi-publish@release/v1
-        with:
-          password: ${{ secrets.PYPI_API_TOKEN }}
-          skip-existing: true
-
       - name: Validate published package
         run: |
           # Wait a moment for package to be available
           sleep 30
+          export SKYPILOT_DISABLE_USAGE_COLLECTION=1
+          pip install fastapi
           # Install skypilot from Test PyPI with fallback to PyPI for dependencies
           pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple skypilot
           # Verify installation
           sky -v
           sky -c
+          sky check
+          sky show-gpus
           # Verify version
           EXPECTED_VERSION="${{ needs.extract-version.outputs.version }}"
           INSTALLED_VERSION=$(pip show skypilot | grep Version | awk '{print $2}')
@@ -104,6 +102,12 @@ jobs:
           fi
           echo "Version verified successfully!"
 
+      - name: Publish distribution to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          password: ${{ secrets.PYPI_API_TOKEN }}
+          skip-existing: true
+
   trigger-docker-build:
     needs: [extract-version, publish-to-pypi]
     uses: ./.github/workflows/docker-build.yaml
diff --git a/Dockerfile b/Dockerfile
index ddb22c86d..43e0f75e0 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -25,7 +25,7 @@ RUN conda install -c conda-forge google-cloud-sdk && \
     rm kubectl && \
     # Install uv and skypilot
     curl -LsSf https://astral.sh/uv/install.sh | sh && \
-    ~/.local/bin/uv pip install --prerelease allow azure-cli --system && \
+    ~/.local/bin/uv pip install --prerelease allow 'azure-cli>=2.65.0' 'omegaconf>=2.4.0dev3' --system && \
     ~/.local/bin/uv pip install skypilot-nightly[all] --system && \
     # Cleanup all caches to reduce the image size
     conda clean -afy && \
diff --git a/docs/source/conf.py b/docs/source/conf.py
index f32cda89f..bbfcb05f9 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -117,7 +117,7 @@ html_theme_options = {
         'icon': 'fab fa-github',
     }],
     'use_edit_page_button': True,
-    'announcement': '<div style="padding: 24px; text-align: center; font-size: 0.8rem; line-height: 1;">👋 Join us for the <b>SkyPilot AI Infra Meetup</b> in San Francisco on April 24! <a href="https://lu.ma/r8ht5829?utm_source=skydocs">Register here</a></div>',
+    # 'announcement': '<div style="padding: 24px; text-align: center; font-size: 0.8rem; line-height: 1;">Put meetup and other announcements here.</div>',
     'secondary_sidebar_items': [
         'page-toc',
         'edit-this-page',
diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst
index 125c5e058..8f4a7518f 100644
--- a/docs/source/getting-started/installation.rst
+++ b/docs/source/getting-started/installation.rst
@@ -132,10 +132,10 @@ Installing via ``uv`` is also supported:
 .. code-block:: shell
 
   uv venv --seed --python 3.10
-  uv pip install "skypilot[kubernetes,aws,gcp]"
-  # Azure CLI has an issue with uv, and requires '--prerelease allow'.
-  uv pip install --prerelease allow azure-cli
-  uv pip install "skypilot[all]"
+  uv pip install --prerelease allow 'azure-cli>=2.65.0'
+  # Explicitly install prerelease dependency to work around https://docs.astral.sh/uv/pip/compatibility/#pre-release-compatibility
+  # Optionally only install specific clouds - e.g. 'skypilot[aws,gcp,kubernetes]'
+  uv pip install 'omegaconf>=2.4.0dev3' 'skypilot[all]'
 
 
 Alternatively, we also provide a :ref:`Docker image <docker-image>` as a quick way to try out SkyPilot.
diff --git a/docs/source/reference/config-sources.rst b/docs/source/reference/config-sources.rst
index 8a2136026..e03a6bed1 100644
--- a/docs/source/reference/config-sources.rst
+++ b/docs/source/reference/config-sources.rst
@@ -78,8 +78,6 @@ Server configuration
 
 If you are using a remote :ref:`SkyPilot API server <sky-api-server>`, it looks for ``~/.sky/config.yaml`` in the API server instance/container to find the server configuration.
 
-To specify a different file, set ``SKYPILOT_GLOBAL_CONFIG`` environment variable to the desired path.
-
 If you are using a local API server, you can use :ref:`User configuration<config-client-user-config>` to set global configuration.
 
 .. _config-client-user-config:
@@ -89,8 +87,6 @@ User configuration
 
 SkyPilot client looks for ``~/.sky/config.yaml`` to find the user configuration.
 
-To specify a different file, set ``SKYPILOT_GLOBAL_CONFIG`` environment variable to the desired path.
-
 .. _config-client-project-config:
 
 Project configuration
diff --git a/pyproject.toml b/pyproject.toml
index 39d1cc344..a8fd2403a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -16,6 +16,7 @@ env = [
     "SKYPILOT_DISABLE_USAGE_COLLECTION=1"
 ]
 addopts = "-s -n 16 -q --tb=short --dist loadgroup --disable-warnings"
+asyncio_default_fixture_loop_scope = "function"
 
 [tool.mypy]
 python_version = "3.8"
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 361151d04..66874fb09 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -40,3 +40,6 @@ aiohttp==3.9.3
 
 # For mocking AWS
 moto==5.1.2
+
+# boto3 type checking
+boto3-stubs[s3,ec2,sts,iam,service-quotas]
diff --git a/sky/adaptors/aws.py b/sky/adaptors/aws.py
index 27cf3fcc5..a225f9121 100644
--- a/sky/adaptors/aws.py
+++ b/sky/adaptors/aws.py
@@ -31,12 +31,22 @@ This is informed by the following boto3 docs:
 import logging
 import threading
 import time
-from typing import Any, Callable
+import typing
+from typing import Callable, Literal, Optional, TypeVar
 
 from sky.adaptors import common
 from sky.utils import annotations
 from sky.utils import common_utils
 
+if typing.TYPE_CHECKING:
+    import boto3
+    _ = boto3  # Supress pylint use before assignment error
+    import mypy_boto3_ec2
+    import mypy_boto3_iam
+    import mypy_boto3_s3
+    import mypy_boto3_service_quotas
+    import mypy_boto3_sts
+
 _IMPORT_ERROR_MESSAGE = ('Failed to import dependencies for AWS. '
                          'Try pip install "skypilot[aws]"')
 boto3 = common.LazyImport('boto3', import_error_message=_IMPORT_ERROR_MESSAGE)
@@ -44,6 +54,8 @@ botocore = common.LazyImport('botocore',
                              import_error_message=_IMPORT_ERROR_MESSAGE)
 _LAZY_MODULES = (boto3, botocore)
 
+T = TypeVar('T')
+
 logger = logging.getLogger(__name__)
 _session_creation_lock = threading.RLock()
 
@@ -73,8 +85,8 @@ def _assert_kwargs_builtin_type(kwargs):
         f'kwargs should not contain none built-in types: {kwargs}')
 
 
-def _create_aws_object(creation_fn_or_cls: Callable[[], Any],
-                       object_name: str) -> Any:
+def _create_aws_object(creation_fn_or_cls: Callable[[], T],
+                       object_name: str) -> T:
     """Create an AWS object.
 
     Args:
@@ -123,6 +135,25 @@ def session(check_credentials: bool = True):
     return s
 
 
+# New typing overloads can be added as needed.
+@typing.overload
+def resource(service_name: Literal['ec2'],
+             **kwargs) -> 'mypy_boto3_ec2.ServiceResource':
+    ...
+
+
+@typing.overload
+def resource(service_name: Literal['s3'],
+             **kwargs) -> 'mypy_boto3_s3.ServiceResource':
+    ...
+
+
+@typing.overload
+def resource(service_name: Literal['iam'],
+             **kwargs) -> 'mypy_boto3_iam.ServiceResource':
+    ...
+
+
 # Avoid caching the resource/client objects. If we are using the assumed role,
 # the credentials will be automatically rotated, but the cached resource/client
 # object will only refresh the credentials with a fixed 15 minutes interval,
@@ -142,7 +173,7 @@ def resource(service_name: str, **kwargs):
     """
     _assert_kwargs_builtin_type(kwargs)
 
-    max_attempts = kwargs.pop('max_attempts', None)
+    max_attempts: Optional[int] = kwargs.pop('max_attempts', None)
     if max_attempts is not None:
         config = botocore_config().Config(
             retries={'max_attempts': max_attempts})
@@ -158,6 +189,28 @@ def resource(service_name: str, **kwargs):
             service_name, **kwargs), 'resource')
 
 
+# New typing overloads can be added as needed.
+@typing.overload
+def client(service_name: Literal['s3'], **kwargs) -> 'mypy_boto3_s3.Client':
+    pass
+
+
+@typing.overload
+def client(service_name: Literal['ec2'], **kwargs) -> 'mypy_boto3_ec2.Client':
+    pass
+
+
+@typing.overload
+def client(service_name: Literal['sts'], **kwargs) -> 'mypy_boto3_sts.Client':
+    pass
+
+
+@typing.overload
+def client(service_name: Literal['service-quotas'],
+           **kwargs) -> 'mypy_boto3_service_quotas.Client':
+    pass
+
+
 def client(service_name: str, **kwargs):
     """Create an AWS client of a certain service.
 
diff --git a/sky/adaptors/common.py b/sky/adaptors/common.py
index 89c08f9c4..c3877d886 100644
--- a/sky/adaptors/common.py
+++ b/sky/adaptors/common.py
@@ -2,10 +2,11 @@
 import functools
 import importlib
 import threading
+import types
 from typing import Any, Callable, Optional, Tuple
 
 
-class LazyImport:
+class LazyImport(types.ModuleType):
     """Lazy importer for modules.
 
     This is mainly used in two cases:
diff --git a/sky/adaptors/kubernetes.py b/sky/adaptors/kubernetes.py
index 1e8607870..27343bc15 100644
--- a/sky/adaptors/kubernetes.py
+++ b/sky/adaptors/kubernetes.py
@@ -70,21 +70,26 @@ def _load_config(context: Optional[str] = None):
             kubernetes.config.load_kube_config(context=context)
         except kubernetes.config.config_exception.ConfigException as e:
             suffix = common_utils.format_exception(e, use_bracket=True)
+            context_name = '(current-context)' if context is None else context
             # Check if exception was due to no current-context
             if 'Expected key current-context' in str(e):
-                err_str = (
-                    f'Failed to load Kubernetes configuration for {context!r}. '
-                    'Kubeconfig does not contain any valid context(s).'
-                    f'\n{suffix}\n'
-                    '    If you were running a local Kubernetes '
-                    'cluster, run `sky local up` to start the cluster.')
+                err_str = ('Failed to load Kubernetes configuration for '
+                           f'{context_name!r}. '
+                           'Kubeconfig does not contain any valid context(s).'
+                           f'\n{suffix}\n'
+                           '    If you were running a local Kubernetes '
+                           'cluster, run `sky local up` to start the cluster.')
             else:
                 kubeconfig_path = os.environ.get('KUBECONFIG', '~/.kube/config')
                 err_str = (
-                    f'Failed to load Kubernetes configuration for {context!r}. '
-                    'Please check if your kubeconfig file exists at '
-                    f'{kubeconfig_path} and is valid.\n{suffix}')
+                    f'Failed to load Kubernetes configuration for '
+                    f'{context_name!r}. Please check if your kubeconfig file '
+                    f'exists at {kubeconfig_path} and is valid.\n{suffix}')
             err_str += '\nTo disable Kubernetes for SkyPilot: run `sky check`.'
+            if context is None:  # kubernetes defaults to current-context.
+                err_str += (
+                    '\nHint: Kubernetes attempted to query the current-context '
+                    'set in kubeconfig. Check if the current-context is valid.')
             with ux_utils.print_exception_no_traceback():
                 raise ValueError(err_str) from None
 
diff --git a/sky/cli.py b/sky/cli.py
index 435efb95c..f43a0c570 100644
--- a/sky/cli.py
+++ b/sky/cli.py
@@ -5910,7 +5910,7 @@ def api_info():
     api_server_info = sdk.api_info()
     user_name = os.getenv(constants.USER_ENV_VAR, getpass.getuser())
     user_hash = common_utils.get_user_hash()
-    dashboard_url = f'{url}/dashboard'
+    dashboard_url = server_common.get_dashboard_url(url)
     click.echo(f'Using SkyPilot API server: {url} Dashboard: {dashboard_url}\n'
                f'{ux_utils.INDENT_SYMBOL}Status: {api_server_info["status"]}, '
                f'commit: {api_server_info["commit"]}, '
diff --git a/sky/client/sdk.py b/sky/client/sdk.py
index 4e22ac6b7..1b4ea654d 100644
--- a/sky/client/sdk.py
+++ b/sky/client/sdk.py
@@ -301,7 +301,7 @@ def validate(
 def dashboard() -> None:
     """Starts the dashboard for SkyPilot."""
     api_server_url = server_common.get_server_url()
-    url = f'{api_server_url}/dashboard'
+    url = server_common.get_dashboard_url(api_server_url)
     logger.info(f'Opening dashboard in browser: {url}')
     webbrowser.open(url)
 
@@ -1725,9 +1725,11 @@ def api_start(
     if foreground:
         # Explain why current process exited
         logger.info('API server is already running:')
-    dashboard_msg = f'Dashboard: {server_common.get_server_url(host)}/dashboard'
+    api_server_url = server_common.get_server_url(host)
+    dashboard_url = server_common.get_dashboard_url(api_server_url)
+    dashboard_msg = f'Dashboard: {dashboard_url}'
     logger.info(f'{ux_utils.INDENT_SYMBOL}SkyPilot API server: '
-                f'{server_common.get_server_url(host)} {dashboard_msg}\n'
+                f'{api_server_url} {dashboard_msg}\n'
                 f'{ux_utils.INDENT_LAST_SYMBOL}'
                 f'View API server logs at: {constants.API_SERVER_LOGS}')
 
@@ -1835,7 +1837,8 @@ def api_login(endpoint: Optional[str] = None) -> None:
             config = skypilot_config.get_user_config()
             config.set_nested(('api_server', 'endpoint'), endpoint)
         common_utils.dump_yaml(str(config_path), dict(config))
-        dashboard_msg = f'Dashboard: {endpoint}/dashboard'
+        dashboard_url = server_common.get_dashboard_url(endpoint)
+        dashboard_msg = f'Dashboard: {dashboard_url}'
         click.secho(
             f'Logged in to SkyPilot API server at {endpoint}.'
             f' {dashboard_msg}',
diff --git a/sky/clouds/aws.py b/sky/clouds/aws.py
index a3c1882eb..b1fedc586 100644
--- a/sky/clouds/aws.py
+++ b/sky/clouds/aws.py
@@ -312,13 +312,12 @@ class AWS(clouds.Cloud):
             'Example: ami-0729d913a335efca7')
         try:
             client = aws.client('ec2', region_name=region)
-            image_info = client.describe_images(ImageIds=[image_id])
-            image_info = image_info.get('Images', [])
+            image_info = client.describe_images(ImageIds=[image_id]).get(
+                'Images', [])
             if not image_info:
                 with ux_utils.print_exception_no_traceback():
                     raise ValueError(image_not_found_message)
-            image_info = image_info[0]
-            image_size = image_info['BlockDeviceMappings'][0]['Ebs'][
+            image_size = image_info[0]['BlockDeviceMappings'][0]['Ebs'][
                 'VolumeSize']
         except (aws.botocore_exceptions().NoCredentialsError,
                 aws.botocore_exceptions().ProfileNotFound):
diff --git a/sky/clouds/gcp.py b/sky/clouds/gcp.py
index 0a88000aa..2c49f2df1 100644
--- a/sky/clouds/gcp.py
+++ b/sky/clouds/gcp.py
@@ -174,7 +174,10 @@ class GCP(clouds.Cloud):
         # Install the Google Cloud SDK:
         f'{_INDENT_PREFIX}  $ pip install google-api-python-client\n'
         f'{_INDENT_PREFIX}  $ conda install -c conda-forge '
-        'google-cloud-sdk -y')
+        'google-cloud-sdk -y\n'
+        f'{_INDENT_PREFIX} If gcloud was recently installed with wget, API server'
+        ' may need to be restarted with following commands:\n'
+        f'{_INDENT_PREFIX}  $ sky api stop; sky api start')
 
     _CREDENTIAL_HINT = (
         'Run the following commands:\n'
@@ -476,7 +479,7 @@ class GCP(clouds.Cloud):
             'custom_resources': None,
             'use_spot': r.use_spot,
             'gcp_project_id': self.get_project_id(dryrun),
-            **GCP._get_disk_specs(_failover_disk_tier()),
+            **GCP._get_disk_specs(r.instance_type, _failover_disk_tier()),
         }
         accelerators = r.accelerators
         if accelerators is not None:
@@ -776,7 +779,7 @@ class GCP(clouds.Cloud):
                         raise FileNotFoundError(file)
             except FileNotFoundError as e:
                 return False, (
-                    f'Credentails are not set. '
+                    f'Credentials are not set. '
                     f'{cls._CREDENTIAL_HINT}\n'
                     f'{cls._INDENT_PREFIX}Details: '
                     f'{common_utils.format_exception(e, use_bracket=True)}')
@@ -1012,20 +1015,23 @@ class GCP(clouds.Cloud):
         if disk_tier != resources_utils.DiskTier.ULTRA or instance_type is None:
             return True, ''
         # Ultra disk tier (pd-extreme) only support m2, m3 and part of n2
-        # instance types, so we failover to lower tiers for other instance
-        # types. Reference:
+        # instance types. For a3 instances, we map the ULTRA tier to
+        # hyperdisk-balanced. For all other instance types, we failover to
+        # lower tiers. Reference:
         # https://cloud.google.com/compute/docs/disks/extreme-persistent-disk#machine_shape_support  # pylint: disable=line-too-long
         series = instance_type.split('-')[0]
-        if series in ['m2', 'm3', 'n2']:
+        if series in ['m2', 'm3', 'n2', 'a3']:
             if series == 'n2':
                 num_cpus = int(instance_type.split('-')[2])
                 if num_cpus < 64:
                     return False, ('n2 series with less than 64 vCPUs are '
                                    'not supported with pd-extreme.')
             return True, ''
-        return False, (f'{series} series is not supported with pd-extreme. '
-                       'Only m2, m3 series and n2 series with 64 or more vCPUs '
-                       'are supported.')
+        return False, (f'{series} series is not supported with pd-extreme '
+                       'or hyperdisk-balanced. Only m2, m3 series and n2 '
+                       'series with 64 or more vCPUs are supported with '
+                       'pd-extreme. Also, only a3 is supported with '
+                       'hyperdisk-balanced.')
 
     @classmethod
     def check_disk_tier_enabled(cls, instance_type: Optional[str],
@@ -1036,7 +1042,7 @@ class GCP(clouds.Cloud):
                 raise exceptions.NotSupportedError(msg)
 
     @classmethod
-    def _get_disk_type(cls,
+    def _get_disk_type(cls, instance_type: Optional[str],
                        disk_tier: Optional[resources_utils.DiskTier]) -> str:
         tier = cls._translate_disk_tier(disk_tier)
         tier2name = {
@@ -1045,13 +1051,23 @@ class GCP(clouds.Cloud):
             resources_utils.DiskTier.MEDIUM: 'pd-balanced',
             resources_utils.DiskTier.LOW: 'pd-standard',
         }
+
+        # Remap series-specific disk types
+        series = instance_type.split('-')[0]  # type: ignore
+        if series == 'a3':
+            tier2name[resources_utils.DiskTier.LOW] = tier2name[
+                resources_utils.DiskTier.MEDIUM]
+            tier2name[resources_utils.DiskTier.ULTRA] = 'hyperdisk-balanced'
+
         return tier2name[tier]
 
     @classmethod
     def _get_disk_specs(
-            cls,
+            cls, instance_type: Optional[str],
             disk_tier: Optional[resources_utils.DiskTier]) -> Dict[str, Any]:
-        specs: Dict[str, Any] = {'disk_tier': cls._get_disk_type(disk_tier)}
+        specs: Dict[str, Any] = {
+            'disk_tier': cls._get_disk_type(instance_type, disk_tier)
+        }
         if disk_tier == resources_utils.DiskTier.ULTRA:
             # Only pd-extreme supports custom iops.
             # see https://cloud.google.com/compute/docs/disks#disk-types
diff --git a/sky/clouds/runpod.py b/sky/clouds/runpod.py
index f90d99dcb..9bfbe75a7 100644
--- a/sky/clouds/runpod.py
+++ b/sky/clouds/runpod.py
@@ -67,13 +67,18 @@ class RunPod(clouds.Cloud):
                               accelerators: Optional[Dict[str, int]],
                               use_spot: bool, region: Optional[str],
                               zone: Optional[str]) -> List[clouds.Region]:
-        assert zone is None, 'RunPod does not support zones.'
-        del accelerators, zone  # unused
+        del accelerators  # unused
         regions = service_catalog.get_region_zones_for_instance_type(
             instance_type, use_spot, 'runpod')
 
         if region is not None:
             regions = [r for r in regions if r.name == region]
+
+        if zone is not None:
+            for r in regions:
+                assert r.zones is not None, r
+                r.set_zones([z for z in r.zones if z.name == zone])
+            regions = [r for r in regions if r.zones]
         return regions
 
     @classmethod
@@ -93,7 +98,7 @@ class RunPod(clouds.Cloud):
         instance_type: str,
         accelerators: Optional[Dict[str, int]] = None,
         use_spot: bool = False,
-    ) -> Iterator[None]:
+    ) -> Iterator[Optional[List['clouds.Zone']]]:
         del num_nodes  # unused
         regions = cls.regions_with_offering(instance_type,
                                             accelerators,
@@ -101,7 +106,7 @@ class RunPod(clouds.Cloud):
                                             region=region,
                                             zone=None)
         for r in regions:
-            assert r.zones is None, r
+            assert r
             yield r.zones
 
     def instance_type_to_hourly_cost(self,
@@ -158,7 +163,10 @@ class RunPod(clouds.Cloud):
             zones: Optional[List['clouds.Zone']],
             num_nodes: int,
             dryrun: bool = False) -> Dict[str, Optional[str]]:
-        del zones, dryrun, cluster_name  # unused
+        del dryrun, cluster_name  # unused
+        assert zones is not None, (region, zones)
+
+        zone_names = [zone.name for zone in zones]
 
         r = resources
         acc_dict = self.get_accelerators_from_instance_type(r.instance_type)
@@ -187,6 +195,7 @@ class RunPod(clouds.Cloud):
             'instance_type': instance_type,
             'custom_resources': custom_resources,
             'region': region.name,
+            'availability_zone': ','.join(zone_names),
             'image_id': image_id,
             'use_spot': use_spot,
             'bid_per_gpu': str(hourly_cost),
diff --git a/sky/clouds/service_catalog/data_fetchers/fetch_aws.py b/sky/clouds/service_catalog/data_fetchers/fetch_aws.py
index f5d108063..1fe3e31a9 100644
--- a/sky/clouds/service_catalog/data_fetchers/fetch_aws.py
+++ b/sky/clouds/service_catalog/data_fetchers/fetch_aws.py
@@ -13,7 +13,7 @@ import sys
 import textwrap
 import traceback
 import typing
-from typing import Dict, List, Optional, Set, Tuple, Union
+from typing import List, Optional, Set, Tuple, Union
 
 import numpy as np
 
@@ -24,6 +24,7 @@ from sky.utils import log_utils
 from sky.utils import ux_utils
 
 if typing.TYPE_CHECKING:
+    from mypy_boto3_ec2 import type_defs as ec2_type_defs
     import pandas as pd
 else:
     pd = adaptors_common.LazyImport('pandas')
@@ -192,7 +193,7 @@ def _get_spot_pricing_table(region: str) -> 'pd.DataFrame':
     paginator = client.get_paginator('describe_spot_price_history')
     response_iterator = paginator.paginate(ProductDescriptions=['Linux/UNIX'],
                                            StartTime=datetime.datetime.utcnow())
-    ret: List[Dict[str, str]] = []
+    ret: List['ec2_type_defs.SpotPriceTypeDef'] = []
     for response in response_iterator:
         # response['SpotPriceHistory'] is a list of dicts, each dict is like:
         # {
diff --git a/sky/clouds/service_catalog/runpod_catalog.py b/sky/clouds/service_catalog/runpod_catalog.py
index 7fbc46206..764fb4c23 100644
--- a/sky/clouds/service_catalog/runpod_catalog.py
+++ b/sky/clouds/service_catalog/runpod_catalog.py
@@ -8,7 +8,6 @@ import typing
 from typing import Dict, List, Optional, Tuple, Union
 
 from sky.clouds.service_catalog import common
-from sky.utils import ux_utils
 
 if typing.TYPE_CHECKING:
     from sky.clouds import cloud
@@ -23,9 +22,6 @@ def instance_type_exists(instance_type: str) -> bool:
 def validate_region_zone(
         region: Optional[str],
         zone: Optional[str]) -> Tuple[Optional[str], Optional[str]]:
-    if zone is not None:
-        with ux_utils.print_exception_no_traceback():
-            raise ValueError('RunPod does not support zones.')
     return common.validate_region_zone_impl('runpod', _df, region, zone)
 
 
@@ -34,9 +30,6 @@ def get_hourly_cost(instance_type: str,
                     region: Optional[str] = None,
                     zone: Optional[str] = None) -> float:
     """Returns the cost, or the cheapest cost among all zones for spot."""
-    if zone is not None:
-        with ux_utils.print_exception_no_traceback():
-            raise ValueError('RunPod does not support zones.')
     return common.get_hourly_cost_impl(_df, instance_type, use_spot, region,
                                        zone)
 
@@ -69,9 +62,6 @@ def get_instance_type_for_accelerator(
         region: Optional[str] = None,
         zone: Optional[str] = None) -> Tuple[Optional[List[str]], List[str]]:
     """Returns a list of instance types that have the given accelerator."""
-    if zone is not None:
-        with ux_utils.print_exception_no_traceback():
-            raise ValueError('RunPod does not support zones.')
     return common.get_instance_type_for_accelerator_impl(df=_df,
                                                          acc_name=acc_name,
                                                          acc_count=acc_count,
diff --git a/sky/dashboard/src/hooks/usemobile.js b/sky/dashboard/src/hooks/useMobile.js
similarity index 100%
rename from sky/dashboard/src/hooks/usemobile.js
rename to sky/dashboard/src/hooks/useMobile.js
diff --git a/sky/data/storage.py b/sky/data/storage.py
index ea4fc96c6..d9faee9e3 100644
--- a/sky/data/storage.py
+++ b/sky/data/storage.py
@@ -38,8 +38,8 @@ from sky.utils import status_lib
 from sky.utils import ux_utils
 
 if typing.TYPE_CHECKING:
-    import boto3  # type: ignore
     from google.cloud import storage  # type: ignore
+    import mypy_boto3_s3
 
 logger = sky_logging.init_logger(__name__)
 
@@ -1363,7 +1363,7 @@ class S3Store(AbstractStore):
                  is_sky_managed: Optional[bool] = None,
                  sync_on_reconstruction: bool = True,
                  _bucket_sub_path: Optional[str] = None):
-        self.client: 'boto3.client.Client'
+        self.client: 'mypy_boto3_s3.Client'
         self.bucket: 'StorageHandle'
         # TODO(romilb): This is purely a stopgap fix for
         #  https://github.com/skypilot-org/skypilot/issues/3405
@@ -3295,7 +3295,7 @@ class R2Store(AbstractStore):
                  is_sky_managed: Optional[bool] = None,
                  sync_on_reconstruction: Optional[bool] = True,
                  _bucket_sub_path: Optional[str] = None):
-        self.client: 'boto3.client.Client'
+        self.client: 'mypy_boto3_s3.Client'
         self.bucket: 'StorageHandle'
         super().__init__(name, source, region, is_sky_managed,
                          sync_on_reconstruction, _bucket_sub_path)
@@ -4700,7 +4700,7 @@ class NebiusStore(AbstractStore):
                  is_sky_managed: Optional[bool] = None,
                  sync_on_reconstruction: bool = True,
                  _bucket_sub_path: Optional[str] = None):
-        self.client: 'boto3.client.Client'
+        self.client: 'mypy_boto3_s3.Client'
         self.bucket: 'StorageHandle'
         super().__init__(name, source, region, is_sky_managed,
                          sync_on_reconstruction, _bucket_sub_path)
diff --git a/sky/exceptions.py b/sky/exceptions.py
index c4712371c..4b6a033d2 100644
--- a/sky/exceptions.py
+++ b/sky/exceptions.py
@@ -293,6 +293,11 @@ class ClusterDoesNotExist(ValueError):
     pass
 
 
+class CachedClusterUnavailable(Exception):
+    """Raised when a cached cluster record is unavailable."""
+    pass
+
+
 class NotSupportedError(Exception):
     """Raised when a feature is not supported."""
     pass
diff --git a/sky/jobs/server/core.py b/sky/jobs/server/core.py
index 9c630f22e..dc178fe27 100644
--- a/sky/jobs/server/core.py
+++ b/sky/jobs/server/core.py
@@ -14,6 +14,7 @@ from sky import backends
 from sky import core
 from sky import exceptions
 from sky import execution
+from sky import global_user_state
 from sky import provision as provision_lib
 from sky import sky_logging
 from sky import task as task_lib
@@ -64,6 +65,8 @@ def launch(
         ValueError: cluster does not exist. Or, the entrypoint is not a valid
             chain dag.
         sky.exceptions.NotSupportedError: the feature is not supported.
+        sky.exceptions.CachedClusterUnavailable: cached jobs controller cluster
+            is unavailable
 
     Returns:
       job_id: Optional[int]; the job ID of the submitted job. None if the
@@ -103,6 +106,31 @@ def launch(
     with rich_utils.safe_status(
             ux_utils.spinner_message('Initializing managed job')):
 
+        # Check whether cached jobs controller cluster is accessible
+        cluster_name = (
+            controller_utils.Controllers.JOBS_CONTROLLER.value.cluster_name)
+        record = global_user_state.get_cluster_from_name(cluster_name)
+        if record is not None:
+            # there is a cached jobs controller cluster
+            try:
+                # TODO: do something with returned status?
+                _, _ = backend_utils.refresh_cluster_status_handle(
+                    cluster_name=cluster_name,
+                    force_refresh_statuses=set(status_lib.ClusterStatus),
+                    acquire_per_cluster_status_lock=False)
+            except (exceptions.ClusterOwnerIdentityMismatchError,
+                    exceptions.CloudUserIdentityError,
+                    exceptions.ClusterStatusFetchingError) as e:
+                # we weren't able to refresh the cluster for its status.
+                with ux_utils.print_exception_no_traceback():
+                    raise exceptions.CachedClusterUnavailable(
+                        f'Cached jobs controller cluster '
+                        f'{cluster_name} cannot be refreshed. Please check if '
+                        'the cluster is accessible. If the cluster was '
+                        'removed, consider removing the cluster from SkyPilot '
+                        f'with:\n\n`sky down {cluster_name} --purge`\n\n'
+                        f'Reason: {common_utils.format_exception(e)}')
+
         local_to_controller_file_mounts = {}
 
         if storage_lib.get_cached_enabled_storage_cloud_names_or_refresh():
@@ -142,11 +170,11 @@ def launch(
         remote_user_config_path = f'{prefix}/{dag.name}-{dag_uuid}.config_yaml'
         remote_env_file_path = f'{prefix}/{dag.name}-{dag_uuid}.env'
         controller_resources = controller_utils.get_controller_resources(
-            controller=controller_utils.Controllers.JOBS_CONTROLLER,
+            controller=controller,
             task_resources=sum([list(t.resources) for t in dag.tasks], []))
         controller_idle_minutes_to_autostop, controller_down = (
             controller_utils.get_controller_autostop_config(
-                controller=controller_utils.Controllers.JOBS_CONTROLLER))
+                controller=controller))
 
         vars_to_fill = {
             'remote_user_yaml_path': remote_user_yaml_path,
@@ -162,7 +190,7 @@ def launch(
             'dashboard_setup_cmd': managed_job_constants.DASHBOARD_SETUP_CMD,
             'dashboard_user_id': common.SERVER_ID,
             **controller_utils.shared_controller_vars_to_fill(
-                controller_utils.Controllers.JOBS_CONTROLLER,
+                controller,
                 remote_user_config_path=remote_user_config_path,
                 local_user_config=mutated_user_config,
             ),
diff --git a/sky/provision/aws/config.py b/sky/provision/aws/config.py
index a3ad89c07..694a3b5b2 100644
--- a/sky/provision/aws/config.py
+++ b/sky/provision/aws/config.py
@@ -11,6 +11,7 @@ import copy
 import json
 import logging
 import time
+import typing
 from typing import Any, Dict, List, Optional, Set, Tuple
 
 import colorama
@@ -23,6 +24,10 @@ from sky.provision.aws import utils
 from sky.utils import annotations
 from sky.utils import common_utils
 
+if typing.TYPE_CHECKING:
+    import mypy_boto3_ec2
+    from mypy_boto3_ec2 import type_defs as ec2_type_defs
+
 logger = sky_logging.init_logger(__name__)
 
 RAY = 'ray-autoscaler'
@@ -223,7 +228,8 @@ def _configure_iam_role(iam) -> Dict[str, Any]:
 
 
 @annotations.lru_cache(scope='request', maxsize=128)  # Keep bounded.
-def _get_route_tables(ec2, vpc_id: Optional[str], region: str,
+def _get_route_tables(ec2: 'mypy_boto3_ec2.ServiceResource',
+                      vpc_id: Optional[str], region: str,
                       main: bool) -> List[Any]:
     """Get route tables associated with a VPC and region
 
@@ -248,7 +254,8 @@ def _get_route_tables(ec2, vpc_id: Optional[str], region: str,
         'RouteTables', [])
 
 
-def _is_subnet_public(ec2, subnet_id, vpc_id: Optional[str]) -> bool:
+def _is_subnet_public(ec2: 'mypy_boto3_ec2.ServiceResource', subnet_id,
+                      vpc_id: Optional[str]) -> bool:
     """Checks if a subnet is public by existence of a route to an IGW.
 
     Conventionally, public subnets connect to a IGW, and private subnets to a
@@ -441,10 +448,14 @@ def _usable_subnets(
     return subnets, first_subnet_vpc_id
 
 
-def _vpc_id_from_security_group_ids(ec2, sg_ids: List[str]) -> Any:
+def _vpc_id_from_security_group_ids(ec2: 'mypy_boto3_ec2.ServiceResource',
+                                    sg_ids: List[str]) -> Any:
     # sort security group IDs to support deterministic unit test stubbing
     sg_ids = sorted(set(sg_ids))
-    filters = [{'Name': 'group-id', 'Values': sg_ids}]
+    filters: List['ec2_type_defs.FilterTypeDef'] = [{
+        'Name': 'group-id',
+        'Values': sg_ids
+    }]
     security_groups = ec2.security_groups.filter(Filters=filters)
     vpc_ids = [sg.vpc_id for sg in security_groups]
     vpc_ids = list(set(vpc_ids))
@@ -462,7 +473,8 @@ def _vpc_id_from_security_group_ids(ec2, sg_ids: List[str]) -> Any:
     return vpc_ids[0]
 
 
-def _get_vpc_id_by_name(ec2, vpc_name: str, region: str) -> str:
+def _get_vpc_id_by_name(ec2: 'mypy_boto3_ec2.ServiceResource', vpc_name: str,
+                        region: str) -> str:
     """Returns the VPC ID of the unique VPC with a given name.
 
     Exits with code 1 if:
@@ -470,7 +482,10 @@ def _get_vpc_id_by_name(ec2, vpc_name: str, region: str) -> str:
       - More than 1 VPC with the given name are found in the current region.
     """
     # Look in the 'Name' tag (shown as Name column in console).
-    filters = [{'Name': 'tag:Name', 'Values': [vpc_name]}]
+    filters: List['ec2_type_defs.FilterTypeDef'] = [{
+        'Name': 'tag:Name',
+        'Values': [vpc_name]
+    }]
     vpcs = list(ec2.vpcs.filter(Filters=filters))
     if not vpcs:
         _skypilot_log_error_and_exit_for_failover(
@@ -486,8 +501,9 @@ def _get_vpc_id_by_name(ec2, vpc_name: str, region: str) -> str:
     return vpcs[0].id
 
 
-def _get_subnet_and_vpc_id(ec2, security_group_ids: Optional[List[str]],
-                           region: str, availability_zone: Optional[str],
+def _get_subnet_and_vpc_id(ec2: 'mypy_boto3_ec2.ServiceResource',
+                           security_group_ids: Optional[List[str]], region: str,
+                           availability_zone: Optional[str],
                            use_internal_ips: bool,
                            vpc_name: Optional[str]) -> Tuple[Any, str]:
     if vpc_name is not None:
@@ -514,7 +530,8 @@ def _get_subnet_and_vpc_id(ec2, security_group_ids: Optional[List[str]],
     return subnets, vpc_id
 
 
-def _configure_security_group(ec2, vpc_id: str, expected_sg_name: str,
+def _configure_security_group(ec2: 'mypy_boto3_ec2.ServiceResource',
+                              vpc_id: str, expected_sg_name: str,
                               extended_ip_rules: List) -> List[str]:
     security_group = _get_or_create_vpc_security_group(ec2, vpc_id,
                                                        expected_sg_name)
@@ -551,7 +568,8 @@ def _configure_security_group(ec2, vpc_id: str, expected_sg_name: str,
     return sg_ids
 
 
-def _get_or_create_vpc_security_group(ec2, vpc_id: str,
+def _get_or_create_vpc_security_group(ec2: 'mypy_boto3_ec2.ServiceResource',
+                                      vpc_id: str,
                                       expected_sg_name: str) -> Any:
     """Find or create a security group in the specified VPC.
 
@@ -612,7 +630,8 @@ def _get_or_create_vpc_security_group(ec2, vpc_id: str,
     return security_group
 
 
-def _get_security_group_from_vpc_id(ec2, vpc_id: str,
+def _get_security_group_from_vpc_id(ec2: 'mypy_boto3_ec2.ServiceResource',
+                                    vpc_id: str,
                                     group_name: str) -> Optional[Any]:
     """Get security group by VPC ID and group name."""
     existing_groups = list(
diff --git a/sky/provision/aws/instance.py b/sky/provision/aws/instance.py
index dc3387a63..01ebbe870 100644
--- a/sky/provision/aws/instance.py
+++ b/sky/provision/aws/instance.py
@@ -9,6 +9,7 @@ import logging
 from multiprocessing import pool
 import re
 import time
+import typing
 from typing import Any, Callable, Dict, List, Optional, Set, TypeVar
 
 from sky import sky_logging
@@ -23,6 +24,11 @@ from sky.utils import resources_utils
 from sky.utils import status_lib
 from sky.utils import ux_utils
 
+if typing.TYPE_CHECKING:
+    from botocore import waiter as botowaiter
+    import mypy_boto3_ec2
+    from mypy_boto3_ec2 import type_defs as ec2_type_defs
+
 logger = sky_logging.init_logger(__name__)
 
 _T = TypeVar('_T')
@@ -55,7 +61,9 @@ _RESUME_PER_INSTANCE_TIMEOUT = 120  # 2 minutes
 # https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer_within_the_same_AWS_Region
 
 
-def _default_ec2_resource(region: str, check_credentials: bool = True) -> Any:
+def _default_ec2_resource(
+        region: str,
+        check_credentials: bool = True) -> 'mypy_boto3_ec2.ServiceResource':
     if not hasattr(aws, 'version'):
         # For backward compatibility, reload the module if the aws module was
         # imported before and stale. Used for, e.g., a live jobs controller
@@ -99,7 +107,8 @@ def _default_ec2_resource(region: str, check_credentials: bool = True) -> Any:
                         check_credentials=check_credentials)
 
 
-def _cluster_name_filter(cluster_name_on_cloud: str) -> List[Dict[str, Any]]:
+def _cluster_name_filter(
+        cluster_name_on_cloud: str) -> List['ec2_type_defs.FilterTypeDef']:
     return [{
         'Name': f'tag:{constants.TAG_RAY_CLUSTER_NAME}',
         'Values': [cluster_name_on_cloud],
@@ -282,7 +291,7 @@ def run_instances(region: str, cluster_name_on_cloud: str,
 
     # sort tags by key to support deterministic unit test stubbing
     tags = dict(sorted(copy.deepcopy(config.tags).items()))
-    filters = [{
+    filters: List['ec2_type_defs.FilterTypeDef'] = [{
         'Name': 'instance-state-name',
         'Values': ['pending', 'running', 'stopping', 'stopped'],
     }, {
@@ -551,7 +560,8 @@ def run_instances(region: str, cluster_name_on_cloud: str,
                                   created_instance_ids=created_instance_ids)
 
 
-def _filter_instances(ec2, filters: List[Dict[str, Any]],
+def _filter_instances(ec2: 'mypy_boto3_ec2.ServiceResource',
+                      filters: List['ec2_type_defs.FilterTypeDef'],
                       included_instances: Optional[List[str]],
                       excluded_instances: Optional[List[str]]):
     instances = ec2.instances.filter(Filters=filters)
@@ -616,7 +626,7 @@ def stop_instances(
     assert provider_config is not None, (cluster_name_on_cloud, provider_config)
     region = provider_config['region']
     ec2 = _default_ec2_resource(region)
-    filters: List[Dict[str, Any]] = [
+    filters: List['ec2_type_defs.FilterTypeDef'] = [
         {
             'Name': 'instance-state-name',
             'Values': ['pending', 'running'],
@@ -653,7 +663,7 @@ def terminate_instances(
     managed_by_skypilot = provider_config['security_group'].get(
         'ManagedBySkyPilot', True)
     ec2 = _default_ec2_resource(region)
-    filters = [
+    filters: List['ec2_type_defs.FilterTypeDef'] = [
         {
             'Name': 'instance-state-name',
             # exclude 'shutting-down' or 'terminated' states
@@ -751,7 +761,7 @@ def open_ports(
     region = provider_config['region']
     ec2 = _default_ec2_resource(region)
     sg_name = provider_config['security_group']['GroupName']
-    filters = [
+    filters: List['ec2_type_defs.FilterTypeDef'] = [
         {
             'Name': 'instance-state-name',
             # exclude 'shutting-down' or 'terminated' states
@@ -789,6 +799,7 @@ def open_ports(
                 range(existing_rule['FromPort'], existing_rule['ToPort'] + 1))
         elif existing_rule['IpProtocol'] == '-1':
             # For AWS, IpProtocol = -1 means all traffic
+            all_traffic_allowed: bool = False
             for group_pairs in existing_rule['UserIdGroupPairs']:
                 if group_pairs['GroupId'] != sg.id:
                     # We skip the port opening when the rule allows access from
@@ -797,8 +808,10 @@ def open_ports(
                     # The security group created by SkyPilot allows all traffic
                     # from the same security group, which should not be skipped.
                     existing_ports.add(-1)
+                    all_traffic_allowed = True
                     break
-            break
+            if all_traffic_allowed:
+                break
 
     ports_to_open = []
     # Do not need to open any ports when all traffic is already allowed.
@@ -875,7 +888,7 @@ def wait_instances(region: str, cluster_name_on_cloud: str,
     ec2 = _default_ec2_resource(region)
     client = ec2.meta.client
 
-    filters = [
+    filters: List['ec2_type_defs.FilterTypeDef'] = [
         {
             'Name': f'tag:{constants.TAG_RAY_CLUSTER_NAME}',
             'Values': [cluster_name_on_cloud],
@@ -904,6 +917,7 @@ def wait_instances(region: str, cluster_name_on_cloud: str,
         raise RuntimeError(
             f'No instances found for cluster {cluster_name_on_cloud}.')
 
+    waiter: 'botowaiter.Waiter'
     if state == status_lib.ClusterStatus.UP:
         waiter = client.get_waiter('instance_running')
     elif state == status_lib.ClusterStatus.STOPPED:
@@ -922,7 +936,7 @@ def get_cluster_info(
         provider_config: Optional[Dict[str, Any]] = None) -> common.ClusterInfo:
     """See sky/provision/__init__.py"""
     ec2 = _default_ec2_resource(region)
-    filters = [
+    filters: List['ec2_type_defs.FilterTypeDef'] = [
         {
             'Name': 'instance-state-name',
             'Values': ['running'],
diff --git a/sky/provision/runpod/instance.py b/sky/provision/runpod/instance.py
index ce02f923b..0bfef6ede 100644
--- a/sky/provision/runpod/instance.py
+++ b/sky/provision/runpod/instance.py
@@ -70,13 +70,14 @@ def run_instances(region: str, cluster_name_on_cloud: str,
                 f'Cluster {cluster_name_on_cloud} has no head node.')
         logger.info(f'Cluster {cluster_name_on_cloud} already has '
                     f'{len(exist_instances)} nodes, no need to start more.')
-        return common.ProvisionRecord(provider_name='runpod',
-                                      cluster_name=cluster_name_on_cloud,
-                                      region=region,
-                                      zone=None,
-                                      head_instance_id=head_instance_id,
-                                      resumed_instance_ids=[],
-                                      created_instance_ids=[])
+        return common.ProvisionRecord(
+            provider_name='runpod',
+            cluster_name=cluster_name_on_cloud,
+            region=region,
+            zone=config.provider_config['availability_zone'],
+            head_instance_id=head_instance_id,
+            resumed_instance_ids=[],
+            created_instance_ids=[])
 
     created_instance_ids = []
     for _ in range(to_start_count):
@@ -87,6 +88,7 @@ def run_instances(region: str, cluster_name_on_cloud: str,
                 node_type=node_type,
                 instance_type=config.node_config['InstanceType'],
                 region=region,
+                zone=config.provider_config['availability_zone'],
                 disk_size=config.node_config['DiskSize'],
                 image_name=config.node_config['ImageId'],
                 ports=config.ports_to_open_on_launch,
@@ -118,13 +120,14 @@ def run_instances(region: str, cluster_name_on_cloud: str,
 
         time.sleep(POLL_INTERVAL)
     assert head_instance_id is not None, 'head_instance_id should not be None'
-    return common.ProvisionRecord(provider_name='runpod',
-                                  cluster_name=cluster_name_on_cloud,
-                                  region=region,
-                                  zone=None,
-                                  head_instance_id=head_instance_id,
-                                  resumed_instance_ids=[],
-                                  created_instance_ids=created_instance_ids)
+    return common.ProvisionRecord(
+        provider_name='runpod',
+        cluster_name=cluster_name_on_cloud,
+        region=region,
+        zone=config.provider_config['availability_zone'],
+        head_instance_id=head_instance_id,
+        resumed_instance_ids=[],
+        created_instance_ids=created_instance_ids)
 
 
 def wait_instances(region: str, cluster_name_on_cloud: str,
diff --git a/sky/provision/runpod/utils.py b/sky/provision/runpod/utils.py
index aa9d294fa..2ab39b616 100644
--- a/sky/provision/runpod/utils.py
+++ b/sky/provision/runpod/utils.py
@@ -264,8 +264,9 @@ def _create_template_for_docker_login(
 
 
 def launch(cluster_name: str, node_type: str, instance_type: str, region: str,
-           disk_size: int, image_name: str, ports: Optional[List[int]],
-           public_key: str, preemptible: Optional[bool], bid_per_gpu: float,
+           zone: str, disk_size: int, image_name: str,
+           ports: Optional[List[int]], public_key: str,
+           preemptible: Optional[bool], bid_per_gpu: float,
            docker_login_config: Optional[Dict[str, str]]) -> str:
     """Launches an instance with the given parameters.
 
@@ -332,6 +333,7 @@ def launch(cluster_name: str, node_type: str, instance_type: str, region: str,
         'min_memory_in_gb': gpu_specs['memoryInGb'] * gpu_quantity,
         'gpu_count': gpu_quantity,
         'country_code': region,
+        'data_center_id': zone,
         'ports': ports_str,
         'support_public_ip': True,
         'docker_args': docker_args,
diff --git a/sky/server/common.py b/sky/server/common.py
index eaf766729..a0524db3c 100644
--- a/sky/server/common.py
+++ b/sky/server/common.py
@@ -13,6 +13,7 @@ import sys
 import time
 import typing
 from typing import Any, Dict, Optional
+from urllib import parse
 import uuid
 
 import colorama
@@ -150,6 +151,23 @@ def get_server_url(host: Optional[str] = None) -> str:
     return url.rstrip('/')
 
 
+@annotations.lru_cache(scope='global')
+def get_dashboard_url(server_url: str) -> str:
+    # The server_url may include username or password with the
+    # format of https://username:password@example.com:8080/path
+    # We need to remove the username and password and only
+    # return `https://example.com:8080/path`
+    parsed = parse.urlparse(server_url)
+    # Reconstruct the URL without credentials but keeping the scheme
+    dashboard_url = f'{parsed.scheme}://{parsed.hostname}'
+    if parsed.port:
+        dashboard_url = f'{dashboard_url}:{parsed.port}'
+    if parsed.path:
+        dashboard_url = f'{dashboard_url}{parsed.path}'
+    dashboard_url = dashboard_url.rstrip('/')
+    return f'{dashboard_url}/dashboard'
+
+
 @annotations.lru_cache(scope='global')
 def is_api_server_local():
     return get_server_url() in AVAILABLE_LOCAL_API_SERVER_URLS
@@ -314,8 +332,9 @@ def _start_api_server(deploy: bool = False,
             else:
                 break
 
-        dashboard_msg = (f'Dashboard: {get_server_url(host)}/dashboard')
-        api_server_info = get_api_server_status(get_server_url(host))
+        server_url = get_server_url(host)
+        dashboard_msg = (f'Dashboard: {get_dashboard_url(server_url)}')
+        api_server_info = get_api_server_status(server_url)
         if api_server_info.version == _DEV_VERSION:
             dashboard_msg += (
                 f'\n{colorama.Style.RESET_ALL}{ux_utils.INDENT_SYMBOL}'
@@ -323,11 +342,13 @@ def _start_api_server(deploy: bool = False,
             if not os.path.isdir(server_constants.DASHBOARD_DIR):
                 dashboard_msg += (
                     'Dashboard is not built, '
-                    'to build: npm --prefix sky/dashboard run build')
+                    'to build: npm --prefix sky/dashboard install '
+                    '&& npm --prefix sky/dashboard run build')
             else:
                 dashboard_msg += (
                     'Dashboard may be stale when installed from source, '
-                    'to rebuild: npm --prefix sky/dashboard run build')
+                    'to rebuild: npm --prefix sky/dashboard install '
+                    '&& npm --prefix sky/dashboard run build')
             dashboard_msg += f'{colorama.Style.RESET_ALL}'
         logger.info(
             ux_utils.finishing_message(
diff --git a/sky/templates/runpod-ray.yml.j2 b/sky/templates/runpod-ray.yml.j2
index bfd91a30e..c74b6d61c 100644
--- a/sky/templates/runpod-ray.yml.j2
+++ b/sky/templates/runpod-ray.yml.j2
@@ -9,6 +9,7 @@ provider:
   type: external
   module: sky.provision.runpod
   region: "{{region}}"
+  availability_zone: "{{availability_zone}}"
   disable_launch_config_check: true
   # For RunPod, we directly set the image id for the docker as runtime environment
   # support, thus we need to avoid the DockerInitializer detects the docker field


Test patch:
diff --git a/tests/smoke_tests/smoke_tests_utils.py b/tests/smoke_tests/smoke_tests_utils.py
index 676ff943c..30f105566 100644
--- a/tests/smoke_tests/smoke_tests_utils.py
+++ b/tests/smoke_tests/smoke_tests_utils.py
@@ -212,13 +212,19 @@ _WAIT_UNTIL_JOB_STATUS_CONTAINS_MATCHING_JOB_NAME = _WAIT_UNTIL_JOB_STATUS_CONTA
 
 
 def get_cmd_wait_until_job_status_contains_matching_job_id(
-        cluster_name: str, job_id: str, job_status: List[sky.JobStatus],
-        timeout: int):
-    return _WAIT_UNTIL_JOB_STATUS_CONTAINS_MATCHING_JOB_ID.format(
+        cluster_name: str,
+        job_id: str,
+        job_status: List[sky.JobStatus],
+        timeout: int,
+        all_users: bool = False):
+    cmd = _WAIT_UNTIL_JOB_STATUS_CONTAINS_MATCHING_JOB_ID.format(
         cluster_name=cluster_name,
         job_id=job_id,
         job_status=_statuses_to_str(job_status),
         timeout=timeout)
+    if all_users:
+        cmd = cmd.replace('sky queue ', 'sky queue -u ')
+    return cmd
 
 
 def get_cmd_wait_until_job_status_contains_without_matching_job(
diff --git a/tests/smoke_tests/test_backward_compat.py b/tests/smoke_tests/test_backward_compat.py
index 19109a1ef..a38d6279c 100644
--- a/tests/smoke_tests/test_backward_compat.py
+++ b/tests/smoke_tests/test_backward_compat.py
@@ -146,11 +146,13 @@ class TestBackwardCompatibility:
                 cluster_name=f"{cluster_name}",
                 job_id=2,
                 job_status=[sky.JobStatus.SUCCEEDED],
-                timeout=120)} && {smoke_tests_utils.get_cmd_wait_until_job_status_contains_matching_job_id(
+                timeout=120,
+                all_users=True)} && {smoke_tests_utils.get_cmd_wait_until_job_status_contains_matching_job_id(
                 cluster_name=f"{cluster_name}",
                 job_id=3,
                 job_status=[sky.JobStatus.SUCCEEDED],
-                timeout=120)}
+                timeout=120,
+                all_users=True)}
             """
             f'{self.ACTIVATE_CURRENT} && result="$(sky queue -u {cluster_name})"; echo "$result"; echo "$result" | grep SUCCEEDED | wc -l | grep 4'
         ]
diff --git a/tests/unit_tests/test_sky/server/test_common.py b/tests/unit_tests/test_sky/server/test_common.py
index f13885a49..a176d2624 100644
--- a/tests/unit_tests/test_sky/server/test_common.py
+++ b/tests/unit_tests/test_sky/server/test_common.py
@@ -213,3 +213,40 @@ allowed_clouds:
     )
     assert skypilot_config.get_nested(keys=('allowed_clouds',),
                                       default_value=None) == ['gcp']
+
+
+def test_get_dashboard_url():
+    """Test get_dashboard_url with default URL."""
+    common.get_server_url.cache_clear()
+    assert common.get_dashboard_url(server_url='http://127.0.0.1:46580'
+                                   ) == 'http://127.0.0.1:46580/dashboard'
+    """Test get_dashboard_url with basic URL."""
+    common.get_server_url.cache_clear()
+    assert common.get_dashboard_url(server_url='http://example.com:8080'
+                                   ) == 'http://example.com:8080/dashboard'
+    """Test get_dashboard_url with URL containing path."""
+    common.get_server_url.cache_clear()
+    assert common.get_dashboard_url(server_url='http://example.com:8080/api/'
+                                   ) == 'http://example.com:8080/api/dashboard'
+    """Test get_dashboard_url with URL containing credentials."""
+    common.get_server_url.cache_clear()
+    assert common.get_dashboard_url(
+        server_url='https://user:pass@example.com:8080'
+    ) == 'https://example.com:8080/dashboard'
+    """Test get_dashboard_url with URL containing username."""
+    common.get_server_url.cache_clear()
+    assert common.get_dashboard_url(server_url='https://user@example.com:8080'
+                                   ) == 'https://example.com:8080/dashboard'
+    """Test get_dashboard_url with host parameter."""
+    common.get_server_url.cache_clear()
+    assert common.get_dashboard_url(server_url='http://custom-host:8080'
+                                   ) == 'http://custom-host:8080/dashboard'
+    """Test get_dashboard_url with complex path."""
+    common.get_server_url.cache_clear()
+    assert common.get_dashboard_url(
+        server_url='https://user:pass@example.com:8080/api/v1'
+    ) == 'https://example.com:8080/api/v1/dashboard'
+    """Test get_dashboard_url without port."""
+    common.get_server_url.cache_clear()
+    assert common.get_dashboard_url(
+        server_url='http://example.com') == 'http://example.com/dashboard'