Repository: skypilot-org/skypilot
Pull request #4911 (2025-03-07 00:37:00)

[Docs] Add a few more examples for k8s.
<!-- Describe the changes in this PR -->



<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [ ] Code formatting: `bash format.sh`
- [ ] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `pytest tests/test_smoke.py` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [ ] Backward compatibility tests: `conda deactivate; bash -i tests/backward_compatibility_tests.sh`



Patch:
diff --git a/README.md b/README.md
index 71ffa414f..95a88779b 100644
--- a/README.md
+++ b/README.md
@@ -48,12 +48,12 @@
 
 SkyPilot is a framework for running AI and batch workloads on any infra, offering unified execution, high cost savings, and high GPU availability.
 
-SkyPilot **abstracts away infra burdens**:
+SkyPilot **abstracts away AI infra burden**:
 - Launch [clusters](https://docs.skypilot.co/en/latest/examples/interactive-development.html), [jobs](https://docs.skypilot.co/en/latest/examples/managed-jobs.html), and [serving](https://docs.skypilot.co/en/latest/serving/sky-serve.html) on any infra
 - Easy job management: queue, run, and auto-recover many jobs
 
-SkyPilot **supports multiple clusters, clouds, and hardware** ([the Sky](https://arxiv.org/abs/2205.07147)):
-- Bring your reserved GPUs, Kubernetes clusters, or 12+ clouds
+SkyPilot **supports multiple clusters, clouds, and hardware**:
+- Bring reserved GPUs, Kubernetes clusters, or 15+ clouds
 - [Flexible provisioning](https://docs.skypilot.co/en/latest/examples/auto-failover.html) of GPUs, TPUs, CPUs, with auto-retry
 
 SkyPilot **cuts your cloud costs & maximizes GPU availability**:
@@ -75,7 +75,9 @@ pip install "skypilot-nightly[kubernetes,aws,gcp,azure,oci,lambda,runpod,fluidst
 ```
 
 
-[Current supported infra](https://docs.skypilot.co/en/latest/getting-started/installation.html) (Kubernetes; AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack, RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai, VMware vSphere, Nebius):
+Current supported infra: Kubernetes, AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack,
+RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai,
+VMware vSphere, Nebius.
 <p align="center">
   <picture>
     <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/skypilot-org/skypilot/master/docs/source/images/cloud-logos-dark.png">
diff --git a/docs/source/docs/index.rst b/docs/source/docs/index.rst
index 885d9a839..bc1ae32b2 100644
--- a/docs/source/docs/index.rst
+++ b/docs/source/docs/index.rst
@@ -22,7 +22,6 @@ Welcome to SkyPilot!
    <p style="text-align:center">
    <a class="github-button" href="https://github.com/skypilot-org/skypilot" data-show-count="true" data-size="large" aria-label="Star skypilot-org/skypilot on GitHub">Star</a>
    <a class="github-button" href="https://github.com/skypilot-org/skypilot/subscription" data-icon="octicon-eye" data-size="large" aria-label="Watch skypilot-org/skypilot on GitHub">Watch</a>
-   <a class="github-button" href="https://github.com/skypilot-org/skypilot/fork" data-icon="octicon-repo-forked" data-size="large" aria-label="Fork skypilot-org/skypilot on GitHub">Fork</a>
    <a class="reference external image-reference" style="vertical-align:9.5px" href="http://slack.skypilot.co"><img src="https://img.shields.io/badge/SkyPilot-Join%20Slack-blue?logo=slack" style="height:27px"></a>
    <script async defer src="https://buttons.github.io/buttons.js"></script>
    </p>
@@ -30,26 +29,28 @@ Welcome to SkyPilot!
 
 SkyPilot is a framework for running AI and batch workloads on any infra, offering unified execution, high cost savings, and high GPU availability.
 
-SkyPilot **abstracts away infra burdens**:
+SkyPilot **abstracts away AI infra burden**:
 
-- Launch :ref:`clusters <dev-cluster>`, :ref:`jobs <managed-jobs>`, and :ref:`serving <sky-serve>` on any infra
+- Launch clusters, jobs, and serving on any infra
 - Easy job management: queue, run, and auto-recover many jobs
 
-SkyPilot **supports multiple clusters, clouds, and hardware** (`the Sky <https://arxiv.org/abs/2205.07147>`_):
+SkyPilot **supports multiple clusters, clouds, and hardware**:
 
-- Bring your reserved GPUs, Kubernetes clusters, or 12+ clouds
+- Bring reserved GPUs, Kubernetes clusters, or 15+ clouds
 - :ref:`Flexible provisioning <auto-failover>` of GPUs, TPUs, CPUs, with auto-retry
 
 SkyPilot **cuts your cloud costs & maximizes GPU availability**:
 
-* :ref:`Autostop <auto-stop>`: automatic cleanup of idle resources
+* Autostop: automatic cleanup of idle resources
 * :ref:`Managed Spot <managed-jobs>`: 3-6x cost savings using spot instances, with preemption auto-recovery
-* :ref:`Optimizer <auto-failover>`: 2x cost savings by auto-picking the cheapest & most available infra
+* Optimizer: 2x cost savings by auto-picking the cheapest & most available infra
 
 SkyPilot supports your existing GPU, TPU, and CPU workloads, with no code changes.
 
 
-:ref:`Current supported infra <installation>` (Kubernetes; AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack, RunPod, Cudo, Paperspace, Digital Ocean, Cloudflare, Samsung, IBM, Vast.ai, VMware vSphere):
+Current supported infra: Kubernetes, AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack,
+RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai,
+VMware vSphere, Nebius.
 
 .. raw:: html
 
diff --git a/docs/source/examples/frameworks/index.rst b/docs/source/examples/frameworks/index.rst
index 399d9c501..bad1434cb 100644
--- a/docs/source/examples/frameworks/index.rst
+++ b/docs/source/examples/frameworks/index.rst
@@ -6,5 +6,6 @@ Frameworks
 
    Airflow <airflow>
    DVC <dvc>
-   MPI <mpi>
    Jupyter <jupyter>
+   Kueue <https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/tutorials-and-examples/skypilot/dws-and-kueue>
+   MPI <mpi>
diff --git a/docs/source/examples/training/index.rst b/docs/source/examples/training/index.rst
index c66b38a83..342c8514e 100644
--- a/docs/source/examples/training/index.rst
+++ b/docs/source/examples/training/index.rst
@@ -14,6 +14,4 @@ Training
    Ray <ray.md>
    Training on TPUs <tpu.md>
    Unsloth <unsloth.md>
-
-
-.. Hugging Face <https://github.com/skypilot-org/skypilot/blob/master/examples/huggingface_glue_imdb_app.yaml>
+   Vertex AI <https://medium.com/google-cloud/streamline-ai-ml-model-development-on-gke-with-skypilot-and-vertex-ai-workbench-453729a8897c>
diff --git a/docs/source/reference/kubernetes/examples/index.rst b/docs/source/reference/kubernetes/examples/index.rst
new file mode 100644
index 000000000..4b4607993
--- /dev/null
+++ b/docs/source/reference/kubernetes/examples/index.rst
@@ -0,0 +1,11 @@
+.. _kubernetes-examples:
+
+Kubernetes Examples
+===================
+
+.. toctree::
+   :maxdepth: 1
+
+   Dynamic Workload Scheduler <https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/tutorials-and-examples/skypilot/dws-and-kueue>
+   Kueue <https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/tutorials-and-examples/skypilot/dws-and-kueue>
+   Multi-region Kubernetes <https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/tutorials-and-examples/skypilot>
diff --git a/docs/source/reference/kubernetes/index.rst b/docs/source/reference/kubernetes/index.rst
index 10815ba0a..54c3b6792 100644
--- a/docs/source/reference/kubernetes/index.rst
+++ b/docs/source/reference/kubernetes/index.rst
@@ -105,3 +105,4 @@ Table of Contents
    kubernetes-troubleshooting
    multi-kubernetes
    SkyPilot vs. Vanilla Kubernetes <skypilot-and-vanilla-k8s>
+   Examples <examples/index>
diff --git a/docs/source/reservations/reservations.rst b/docs/source/reservations/reservations.rst
index c46784cec..40be303d7 100644
--- a/docs/source/reservations/reservations.rst
+++ b/docs/source/reservations/reservations.rst
@@ -169,6 +169,8 @@ In case you want to specify the DWS configuration for each job/cluster, you can
 
     num_nodes: 4
 
+.. _dws-on-gke:
+
 Using DWS on GKE with Kueue
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/sky/backends/wheel_utils.py b/sky/backends/wheel_utils.py
index 805117ee2..84274007b 100644
--- a/sky/backends/wheel_utils.py
+++ b/sky/backends/wheel_utils.py
@@ -17,7 +17,7 @@ import re
 import shutil
 import subprocess
 import tempfile
-from typing import Tuple
+from typing import Optional, Tuple
 
 import filelock
 from packaging import version
@@ -149,16 +149,29 @@ def build_sky_wheel() -> Tuple[pathlib.Path, str]:
         - wheel_hash: The wheel content hash.
     """
 
-    def _get_latest_modification_time(path: pathlib.Path) -> float:
+    def _get_latest_modification_time(path: pathlib.Path) -> Optional[float]:
+        max_time = -1.
         if not path.exists():
-            return -1.
-        try:
-            return max(
-                os.path.getmtime(os.path.join(root, f))
-                for root, dirs, files in os.walk(path)
-                for f in (*dirs, *files))
-        except ValueError:
-            return -1.
+            return max_time
+        for root, dirs, files in os.walk(path):
+            # Prune __pycache__ directories to prevent walking into them and
+            # exclude them from processing
+            if '__pycache__' in dirs:
+                dirs.remove('__pycache__')
+            # Filter out .pyc files
+            filtered_files = [f for f in files if not f.endswith('.pyc')]
+            # Process remaining directories and files
+            for entry in (*dirs, *filtered_files):
+                entry_path = os.path.join(root, entry)
+                try:
+                    mtime = os.path.getmtime(entry_path)
+                    if mtime > max_time:
+                        max_time = mtime
+                except OSError:
+                    # Handle cases where file might have been deleted after
+                    # listing
+                    return None
+        return max_time
 
     # This lock prevents that the wheel is updated while being copied.
     # Although the current caller already uses a lock, we still lock it here
@@ -170,10 +183,12 @@ def build_sky_wheel() -> Tuple[pathlib.Path, str]:
         last_modification_time = _get_latest_modification_time(SKY_PACKAGE_PATH)
         last_wheel_modification_time = _get_latest_modification_time(WHEEL_DIR)
 
-        # Only build wheels if the wheel is outdated or wheel does not exist
-        # for the requested version.
-        if (last_wheel_modification_time < last_modification_time) or not any(
-                WHEEL_DIR.glob(f'**/{_WHEEL_PATTERN}')):
+        # Only build wheels if the wheel is outdated, wheel does not exist
+        # for the requested version, or files were deleted during checking.
+        if ((last_modification_time is None or
+             last_wheel_modification_time is None) or
+            (last_wheel_modification_time < last_modification_time) or
+                not any(WHEEL_DIR.glob(f'**/{_WHEEL_PATTERN}'))):
             if not WHEEL_DIR.exists():
                 WHEEL_DIR.mkdir(parents=True, exist_ok=True)
             latest_wheel = _build_sky_wheel()
diff --git a/sky/utils/rich_utils.py b/sky/utils/rich_utils.py
index 54e9672f0..2f4dae300 100644
--- a/sky/utils/rich_utils.py
+++ b/sky/utils/rich_utils.py
@@ -230,15 +230,53 @@ def decode_rich_status(
     decoding_status = None
     try:
         last_line = ''
+        # Buffer to store incomplete UTF-8 bytes between chunks
+        undecoded_buffer = b''
+
         # Iterate over the response content in chunks. We do not use iter_lines
         # because it will strip the trailing newline characters, causing the
         # progress bar ending with `\r` becomes a pyramid.
-        for encoded_msg in response.iter_content(chunk_size=None):
-            if encoded_msg is None:
+        for chunk in response.iter_content(chunk_size=None):
+            if chunk is None:
                 return
-            encoded_msg = encoded_msg.decode('utf-8')
+
+            # Append the new chunk to any leftover bytes from previous iteration
+            current_bytes = undecoded_buffer + chunk
+            undecoded_buffer = b''
+
+            # Try to decode the combined bytes
+            try:
+                encoded_msg = current_bytes.decode('utf-8')
+            except UnicodeDecodeError as e:
+                # Check if this is potentially an incomplete sequence at the end
+                if e.start > 0:
+                    # Decode the valid part
+                    encoded_msg = current_bytes[:e.start].decode('utf-8')
+
+                    # Check if the remaining bytes are likely a partial char
+                    # or actually invalid UTF-8
+                    remaining_bytes = current_bytes[e.start:]
+                    if len(remaining_bytes) < 4:  # Max UTF-8 char is 4 bytes
+                        # Likely incomplete - save for next chunk
+                        undecoded_buffer = remaining_bytes
+                    else:
+                        # Likely invalid - replace with replacement character
+                        encoded_msg += remaining_bytes.decode('utf-8',
+                                                              errors='replace')
+                        undecoded_buffer = b''
+                else:
+                    # Error at the very beginning of the buffer - invalid UTF-8
+                    encoded_msg = current_bytes.decode('utf-8',
+                                                       errors='replace')
+                    undecoded_buffer = b''
+
             lines = encoded_msg.splitlines(keepends=True)
 
+            # Skip processing if lines is empty to avoid IndexError
+            if not lines:
+                continue
+
+            # Append any leftover text from previous chunk to first line
             lines[0] = last_line + lines[0]
             last_line = lines[-1]
             # If the last line is not ended with `\r` or `\n` (with ending


Test patch:
diff --git a/tests/unit_tests/sky/utils/test_rich_utils.py b/tests/unit_tests/sky/utils/test_rich_utils.py
new file mode 100644
index 000000000..36239cb3c
--- /dev/null
+++ b/tests/unit_tests/sky/utils/test_rich_utils.py
@@ -0,0 +1,83 @@
+import unittest
+from unittest import mock
+
+import requests
+
+from sky.utils import rich_utils
+
+
+class RichUtilsTest(unittest.TestCase):
+
+    def test_decode_rich_status_with_split_utf8(self):
+        """Test that decode_rich_status handles UTF-8 characters split across chunks."""
+        # This is a progress bar with UTF-8 box drawing characters
+        # The character █ is a multibyte UTF-8 character (0xE2 0x96 0x88)
+        progress_bar = "20%|████        | 115/581 [00:12<00:41, 11.20it/s]\n"
+
+        # Create a response object with a custom content iterator
+        mock_response = mock.MagicMock(spec=requests.Response)
+
+        # Convert to bytes and deliberately split in the middle of a multibyte character
+        # The character █ is encoded as three bytes: \xe2\x96\x88
+        progress_bytes = progress_bar.encode('utf-8')
+
+        # Find position of the first █ character and split between its bytes
+        pos = progress_bytes.find(b'\xe2\x96\x88')
+
+        # Split the bytes so that \xe2 is in one chunk and \x96\x88 is in the next
+        chunk1 = progress_bytes[:pos + 1]  # Include the first byte of █ (\xe2)
+        chunk2 = progress_bytes[pos +
+                                1:]  # Rest of the bytes including \x96\x88
+
+        # Configure the mock to return these chunks when iterated
+        mock_response.iter_content.return_value = [chunk1, chunk2]
+
+        # The function should handle the split character correctly
+        result = list(rich_utils.decode_rich_status(mock_response))
+
+        # Verify the output contains the correctly decoded progress bar
+        self.assertEqual(len(result), 1)
+        self.assertEqual(result[0], progress_bar)
+
+    def test_decode_rich_status_with_multiple_split_characters(self):
+        """Test handling multiple UTF-8 characters split across several chunks."""
+        # Progress bar with multiple multibyte characters
+        progress_bar = "50%|█████████████████████████████░░░░░░░░░░░░░| 250/500 [01:15<01:15, 3.33it/s]\n"
+        progress_bytes = progress_bar.encode('utf-8')
+
+        # Create chunks that split at several different points
+        # Find positions of several █ characters and split between their bytes
+        multibyte_char = '█'.encode('utf-8')  # \xe2\x96\x88
+        positions = []
+        start = 0
+        while True:
+            pos = progress_bytes.find(multibyte_char, start)
+            if pos == -1 or len(positions) >= 3:
+                break
+            positions.append(pos)
+            start = pos + len(multibyte_char)
+
+        # Create chunks that split at different multibyte character boundaries
+        chunks = []
+        last_pos = 0
+        for i, pos in enumerate(positions):
+            if i % 2 == 0:  # For even indices, split after first byte of multibyte char
+                chunks.append(progress_bytes[last_pos:pos + 1])
+                last_pos = pos + 1
+            else:  # For odd indices, split normally
+                chunks.append(progress_bytes[last_pos:pos])
+                last_pos = pos
+
+        # Add the final chunk
+        chunks.append(progress_bytes[last_pos:])
+
+        # Configure mock response
+        mock_response = mock.MagicMock(spec=requests.Response)
+        mock_response.iter_content.return_value = chunks
+
+        # Should handle multiple split characters correctly
+        result = list(rich_utils.decode_rich_status(mock_response))
+
+        # Verify the output
+        joined_result = ''.join(r for r in result if r is not None)
+        self.assertEqual(joined_result, progress_bar)