Repository: skypilot-org/skypilot
Pull request: [Spot] Update docs for auto-translation
This PR updates the docs for managed spot jobs with auto-translation added in #1081. Also, fixes #1269 

Tested:
- renderred locally.

Problem statement:
[Docs] Spot controller missing from docs
The docs lack information on how to manage the spot controller, especially on how to terminate the controller. We need to answer questions like:

1. Is the spot controller shared among tasks?
2. How long does the spot controller keep running?
3. I am uninstalling SkyPilot or I do not want to keep the controller stopped. How do I terminate the spot controller?

Hints from issue discussion:

Commit message: update

Patch to review:
diff --git a/docs/source/examples/spot-jobs.rst b/docs/source/examples/spot-jobs.rst
index 836439ca89b..5b282af1224 100644
--- a/docs/source/examples/spot-jobs.rst
+++ b/docs/source/examples/spot-jobs.rst
@@ -6,57 +6,89 @@ Managed Spot Jobs
 SkyPilot supports managed spot jobs that can **automatically recover from preemptions**.
 This feature **saves significant cost** (e.g., up to 70\% for GPU VMs) by making preemptible spot instances practical for long-running jobs.
 
-To maximize availability, SkyPilot automatically finds available spot resources across regions and clouds.
-Here is an example of BERT training job failing over different regions across AWS and GCP.
+SkyPilot automatically finds available spot resources across regions and clouds to maximize availability.
+Here is an example of a BERT training job failing over different regions across AWS and GCP.
 
 .. image:: ../images/spot-training.png
   :width: 600
   :alt: BERT training on Spot V100
 
-Below are requirements for using managed spot jobs:
+Below are the requirements for using managed spot jobs:
 
-(1) **Mounting code and datasets**: Local file mounts/workdir are not supported. Cloud buckets should be used to hold code and datasets, which can be satisfied by using :ref:`SkyPilot Storage <sky-storage>`.
+(1) **Task YAML**: Managed Spot requires a YAML to describe the job, tested with :code:`sky launch`.
+(2) **Checkpointing and recovery**(optional): For job recovery with less progress loss, application code can checkpoint periodically to a
+:ref:`SkyPilot Storage <sky-storage>`-mounted cloud bucket. The program can reload the latest checkpoint when restarted.
 
-(2) **Saving and loading checkpoints**: (For ML jobs) Application code should save checkpoints periodically to a :ref:`SkyPilot Storage <sky-storage>`-mounted cloud bucket. For job recovery,  the program should try to reload a latest checkpoint from that path when it starts.
 
-We explain them in detail below.
+Task YAML
+---------
 
+To launch a spot job, you can simply reuse your task YAML (recommended to test it with :code:`sky launch` first).
+For example, we found the BERT fine-tuning YAML works with :code:`sky launch`, and want to
+launch it with SkyPilot managed spot jobs. 
 
-Mounting code and datasets
---------------------------------
+We can launch it with the following:
 
-To launch a spot job, users should upload their codebase and data to cloud buckets through :ref:`SkyPilot Storage <sky-storage>`.
-Note that the cloud buckets can be mounted to VMs in different regions/clouds and thus enable enables transparent job relaunching without user's intervention.
-The YAML below shows an example.
+.. code-block:: console
+
+    $ sky spot launch -n bert-qa bert_qa.yaml
 
 .. code-block:: yaml
 
-  file_mounts:
-    /code:
-      name: # NOTE: Fill in your bucket name
-      source: /path/to/your/codebase
-      persistent: true
-      mode: COPY
+  # bert_qa.yaml
+  name: bert_qa
+  
+  resources:
+    accelerators: V100:1
 
-    # If data is stored locally
-    /data:
-      name: # NOTE: Fill in your bucket name
-      source: /path/to/your/dataset
-      persistent: true
-      mode: COPY
+  # Assume your working directory is under `~/transformers`.
+  # To make this example work, please run the following command:
+  # git clone https://github.com/huggingface/transformers.git ~/transformers
+  workdir: ~/transformers
+
+  setup: |
+    # Fill in your wandb key: copy from https://wandb.ai/authorize
+    # Alternatively, you can use `--env WANDB_API_KEY=$WANDB_API_KEY`
+    # to pass the key in the command line, during `sky spot launch`.
+    echo export WANDB_API_KEY=[YOUR-WANDB-API-KEY] >> ~/.bashrc
+
+    git checkout v4.18.0
+    pip install -e .
+    cd examples/pytorch/question-answering/
+    pip install -r requirements.txt
+    pip install wandb
 
-    # If data is already on S3
-    /data2: s3://your-s3-bucket-name
+  run: |
+    cd ./examples/pytorch/question-answering/
+    python run_qa.py \
+    --model_name_or_path bert-base-uncased \
+    --dataset_name squad \
+    --do_train \
+    --do_eval \
+    --per_device_train_batch_size 12 \
+    --learning_rate 3e-5 \
+    --num_train_epochs 50 \
+    --max_seq_length 384 \
+    --doc_stride 128 \
+    --report_to wandb
 
 .. note::
 
-  Currently :ref:`workdir <sync-code-artifacts>` and :ref:`file mounts with local files <sync-code-artifacts>` are not
-  supported for spot jobs. You can convert them to :ref:`SkyPilot Storage <sky-storage>`.
+  :ref:`workdir <sync-code-artifacts>` and :ref:`file mounts with local files <sync-code-artifacts>` will be automatically uploaded to
+  :ref:`SkyPilot Storage <sky-storage>`. Cloud bucket will be created during the job running time, and cleaned up after the job
+  finishes.
+
+SkyPilot will launch and start monitoring the spot job. When a preemption happens, SkyPilot will automatically
+search for resources across regions and clouds to re-launch the job.
+
+In this example, the job will be restarted from scratch after each preemption recovery.
+To resume the job from previous states, user's application needs to implement checkpointing and recovery
+
 
-Saving and loading checkpoints
---------------------------------
+Checkpointing and recovery
+--------------------------
 
-To allow spot recovery, another cloud bucket is typically needed for storing states of the job (e.g., model checkpoints).
+To allow spot recovery, a cloud bucket is typically needed to store the job's states (e.g., model checkpoints).
 Below is an example of mounting a bucket to :code:`/checkpoint`.
 
 .. code-block:: yaml
@@ -71,36 +103,29 @@ Note that the application code should save program checkpoints periodically and
 This is typically achieved by reloading the latest checkpoint at the beginning of your program.
 
 An end-to-end example
---------------------------------
+---------------------
 
-Below we show an `example <https://github.com/skypilot-org/skypilot/blob/master/examples/spot/bert_qa.yaml>`_ for fine-tuning a bert model on a question answering task with HuggingFace.
+Below we show an `example <https://github.com/skypilot-org/skypilot/blob/master/examples/spot/bert_qa.yaml>`_ for fine-tuning a BERT model on a question-answering task with HuggingFace.
 
 .. code-block:: yaml
+  :linenos:
+  :emphasize-lines: 12-15,42-45
 
   # bert_qa.yaml
   name: bert_qa
 
   resources:
     accelerators: V100:1
-    # NOTE: `use_spot` and `spot_recovery` are optional when using `sky spot launch`.
-    use_spot: true
-    # When a spot cluster is preempted, this strategy recovers by first waiting for
-    # the resources in the current region for a while (default: 3 minutes), and
-    # then failing over to other regions and clouds, until the resources are launched.
-    spot_recovery: FAILOVER
+
+  # Assume your working directory is under `~/transformers`.
+  # To make this example work, please run the following command:
+  # git clone https://github.com/huggingface/transformers.git ~/transformers
+  workdir: ~/transformers
 
   file_mounts:
     /checkpoint:
       name: # NOTE: Fill in your bucket name
       mode: MOUNT
-    /code:
-      name: # NOTE: Fill in your bucket name
-      # Assume your working directory is under `~/transformers`.
-      # To make this example work, please run the following command:
-      # git clone https://github.com/huggingface/transformers.git ~/transformers
-      source: ~/transformers
-      persistent: false
-      mode: COPY
 
   setup: |
     # Fill in your wandb key: copy from https://wandb.ai/authorize
@@ -108,14 +133,14 @@ Below we show an `example <https://github.com/skypilot-org/skypilot/blob/master/
     # to pass the key in the command line, during `sky spot launch`.
     echo export WANDB_API_KEY=[YOUR-WANDB-API-KEY] >> ~/.bashrc
 
-    cd /code && git checkout v4.18.0
+    git checkout v4.18.0
     pip install -e .
     cd examples/pytorch/question-answering/
     pip install -r requirements.txt
     pip install wandb
 
   run: |
-    cd /code/examples/pytorch/question-answering/
+    cd ./examples/pytorch/question-answering/
     python run_qa.py \
     --model_name_or_path bert-base-uncased \
     --dataset_name squad \
@@ -126,9 +151,9 @@ Below we show an `example <https://github.com/skypilot-org/skypilot/blob/master/
     --num_train_epochs 50 \
     --max_seq_length 384 \
     --doc_stride 128 \
-    --output_dir /checkpoint/bert_qa/ \
     --report_to wandb \
     --run_name $SKYPILOT_RUN_ID \
+    --output_dir /checkpoint/bert_qa/ \
     --save_total_limit 10 \
     --save_steps 1000
 
@@ -141,21 +166,23 @@ As HuggingFace has built-in support for periodically checkpointing, we only need
 .. note::
   You may also refer to another example `here <https://github.com/skypilot-org/skypilot/tree/master/examples/spot/resnet_ddp>`_ for periodically checkpointing with PyTorch.
 
-With the above changes, you are ready to launch a spot job with ``sky spot launch``!
+With the above changes, the managed spot job can now resume training after preemption with ``sky spot launch``! We can enjoy the benefits of
+cost savings from spot instances without worrying about preemption or losing progress.
 
 .. code-block:: console
 
     $ sky spot launch -n bert-qa bert_qa.yaml
 
-SkyPilot will launch and start monitoring the spot job. When a preemption happens, SkyPilot will automatically
-search for resources across regions and clouds to re-launch the job. 
 
 .. note::
-  To identify the same job across multiple recoveries, use the environment variable :code:`$SKYPILOT_RUN_ID` in the task's
-  :code:`run` commands or directly in the program itself (e.g., access via :code:`os.environ` and pass to Weights & Biases
-  for tracking purposes in your training script). It is made available to the task whenever it is invoked. The run ID
-  is kept identical across all recoveries of the same spot job.
+  The environment variable :code:`$SKYPILOT_RUN_ID` can be used to identify the same job, i.e., it is kept identical across all
+  recoveries of the job. It can be accessed in the task's :code:`run` commands or directly in the program itself (e.g., access
+  via :code:`os.environ` and pass to Weights & Biases for tracking purposes in your training script). It is made available to
+  the task whenever it is invoked.
+
 
+Useful CLIs
+-----------
 
 Here are some commands for managed spot jobs. Check :code:`sky spot --help` for more details.
 
@@ -175,3 +202,17 @@ Here are some commands for managed spot jobs. Check :code:`sky spot --help` for
     # Cancel a spot job by name
     $ sky spot cancel -n bert-qa
 
+
+Advanced note (spot controller)
+-------------------------------
+
+There will be a single spot controller cluster (a small on-demand CPU VM) running in the background to manage all the spot jobs.
+It will be autostopped after 30 minutes of inactivity. Normally, **no user intervention** is needed. 
+You can find the cluster with :code:`sky status -a`, and refresh the status with :code:`sky status -ar`.
+
+Although, the cost of the stopped spot controller is negligible (less than $0.1 per day), you can still tear it down manually with 
+:code:`sky down -p sky-spot-controller-<hash>`, where the ``<hash>`` can be found in the output of :code:`sky status -a`.
+
+.. note::
+  Tearing down the spot controller when there are still spot jobs running will cause resource leakage of those spot VMs.
+
diff --git a/docs/source/reference/benchmark/cli.rst b/docs/source/reference/benchmark/cli.rst
index 7ccaa6247ca..d235feb0198 100644
--- a/docs/source/reference/benchmark/cli.rst
+++ b/docs/source/reference/benchmark/cli.rst
@@ -1,10 +1,10 @@
 .. _benchmark-cli:
 
 CLI
-=============
+===
 
 Workflow
---------------------------------
+--------
 
 You can use SkyPilot Benchmark by simply replacing your ``sky launch`` command with ``sky bench launch``:
 
diff --git a/docs/source/reference/benchmark/config.rst b/docs/source/reference/benchmark/config.rst
index 1d82cc1a892..b1865c8db26 100644
--- a/docs/source/reference/benchmark/config.rst
+++ b/docs/source/reference/benchmark/config.rst
@@ -1,7 +1,7 @@
 .. _benchmark-yaml:
 
 YAML Configuration
-===================
+==================
 
 The resources to benchmark can be configured in the SkyPilot YAML interface.
 Below we provide an example:
diff --git a/docs/source/reference/interactive-nodes.rst b/docs/source/reference/interactive-nodes.rst
index 68c789de0c2..1ba83100d9f 100644
--- a/docs/source/reference/interactive-nodes.rst
+++ b/docs/source/reference/interactive-nodes.rst
@@ -15,7 +15,7 @@ Interactive nodes are normal SkyPilot clusters.  They allow fast access to insta
 without requiring a task YAML specification.
 
 Workflow
--------------------------------
+--------
 
 Use :code:`sky gpunode` to get a node with GPU(s):
 
diff --git a/examples/spot/bert_qa.yaml b/examples/spot/bert_qa.yaml
index cc0459578e3..0273ee1549c 100644
--- a/examples/spot/bert_qa.yaml
+++ b/examples/spot/bert_qa.yaml
@@ -2,11 +2,11 @@ name: bert_qa
 
 resources:
     accelerators: V100:1
-    cloud: aws
-    use_spot: true
-    spot_recovery: FAILOVER
 
-num_nodes: 1
+# Assume your working directory is under `~/transformers`.
+# To make this example work, please run the following command:
+# git clone https://github.com/huggingface/transformers.git ~/transformers
+workdir: ~/transformers
 
 file_mounts:
     /checkpoint:
@@ -17,15 +17,14 @@ setup: |
     # Fill in your wandb key: copy from https://wandb.ai/authorize
     echo export WANDB_API_KEY=[YOUR-WANDB-API-KEY] >> ~/.bashrc
 
-    git clone https://github.com/huggingface/transformers.git
-    cd transformers && git checkout v4.18.0
+    git checkout v4.18.0
     pip install -e .
     cd examples/pytorch/question-answering/
     pip install -r requirements.txt
     pip install wandb
 
 run: |
-    cd transformers/examples/pytorch/question-answering/
+    cd examples/pytorch/question-answering/
     python run_qa.py \
     --model_name_or_path bert-base-uncased \
     --dataset_name squad \
@@ -36,7 +35,8 @@ run: |
     --num_train_epochs 50 \
     --max_seq_length 384 \
     --doc_stride 128 \
-    --output_dir /checkpoint/bert_qa/ \
     --report_to wandb \
     --run_name $SKYPILOT_RUN_ID \
-    --save_total_limit 10
+    --output_dir /checkpoint/bert_qa/ \
+    --save_total_limit 10 \
+    --save_steps 1000


Code review comments:
diff_hunk: @@ -6,57 +6,89 @@ Managed Spot Jobs
 SkyPilot supports managed spot jobs that can **automatically recover from preemptions**.
 This feature **saves significant cost** (e.g., up to 70\% for GPU VMs) by making preemptible spot instances practical for long-running jobs.
 
-To maximize availability, SkyPilot automatically finds available spot resources across regions and clouds.
-Here is an example of BERT training job failing over different regions across AWS and GCP.
+SkyPilot automatically finds available spot resources across regions and clouds to maximize availability.
+Here is an example of a BERT training job failing over different regions across AWS and GCP.
 
 .. image:: ../images/spot-training.png
   :width: 600
   :alt: BERT training on Spot V100
 
-Below are requirements for using managed spot jobs:
+Below are the requirements for using managed spot jobs:
 
-(1) **Mounting code and datasets**: Local file mounts/workdir are not supported. Cloud buckets should be used to hold code and datasets, which can be satisfied by using :ref:`SkyPilot Storage <sky-storage>`.
+(1) **Task YAML**: Managed Spot requires a YAML to describe the job, tested with :code:`sky launch`.
+(2) **Checkpointing and recovery**(optional): For job recovery with less progress loss, application code can checkpoint periodically to a
path: docs/source/examples/spot-jobs.rst
text: @user1:
Need a space before (optional)? for the bold font to correctly show

@user1:
Is it better?: For job recovery with less progress loss -> For job recovery with progress resuming

@author:
Updated. Thanks!

diff_hunk: @@ -175,3 +202,17 @@ Here are some commands for managed spot jobs. Check :code:`sky spot --help` for
     # Cancel a spot job by name
     $ sky spot cancel -n bert-qa
 
+
+Advanced note (spot controller)
path: docs/source/examples/spot-jobs.rst
text: @user1:
Should we do the other way around?:
Spot Controller (advanced note)

@author:
Good point! Changed. Thanks!

diff_hunk: @@ -175,3 +202,17 @@ Here are some commands for managed spot jobs. Check :code:`sky spot --help` for
     # Cancel a spot job by name
     $ sky spot cancel -n bert-qa
 
+
+Advanced note (spot controller)
+-------------------------------
+
+There will be a single spot controller cluster (a small on-demand CPU VM) running in the background to manage all the spot jobs.
+It will be autostopped after 30 minutes of inactivity. Normally, **no user intervention** is needed. 
+You can find the cluster with :code:`sky status -a`, and refresh the status with :code:`sky status -ar`.
path: docs/source/examples/spot-jobs.rst
text: @user1:
> the cluster

the controller?

@author:
Fixed. Thanks!