Repository: skypilot-org/skypilot
Pull request: Update documentation with provisioning quickstart
Adds new subsection to quickstart with provisioning introduction and addresses issues found during first user onboarding.

Problem statement:
[Storage] User tends to use the default file_mounts
Though we support cloud storage for large files, the user tends to use the default one. The reasons might be the following:
1. The default file_mounts is more intuitive.
2. We do not have a good doc to guide the user using cloud storage.

A quick fix could be finding a better place (maybe in the quick start) to tell the user to use the cloud storage in our document.

Hints from issue discussion:
Note, however, object stores are not a panacea and may surface other issues:
1. It costs money & the cleanups may not be automatic
2. It can run into bucket name conflicts, which will fail the setup.
3. Cloud storage is much more complex than rsync - there must be many error messages we're not handling (ACL, bucket policy, etc.) which will surface as bugs

It may be best to clearly distinguish "Small files via rsync" vs. "Large files via Cloud Storage" in docs.



For the same reasons, not sure it's reasonable to "auto switch" to cloud storage (#391). 

Yes, we need to clearly distinguish small files vs. big file upload patterns. In our first user onboarding we saw that the user was not aware of these differences and would have uploaded their dataset with `workdir` had they not known about this.
This issue is a matter of documentation I believe. Another solution is further simplify the storage API (or take away some parameters in the minimal example).

Commit message: transfer artifacts

Patch to review:
diff --git a/README.md b/README.md
index 2952926e930..d4a1bc55e79 100644
--- a/README.md
+++ b/README.md
@@ -40,7 +40,7 @@ Please refer to our [documentation](https://sky-proj-sky.readthedocs-hosted.com/
 # Clone the sky codebase
 git clone ssh://git@github.com/sky-proj/sky.git
 cd sky
-# Sky requires python >= 3.6.
+# Sky requires python >= 3.6. 3.10+ is currently NOT supported.
 pip install ".[all]"
 ```
 
@@ -54,6 +54,8 @@ tasks in the clouds, configure access to at least one cloud:
 
 **AWS**:
 
+To get the **AWS Access Key** required by the `aws configure`, please refer to the [AWS manual](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey). The **Default region name [None]:** and **Default output format [None]:** are optional.
+
 ```bash
 # Install boto
 pip install boto3
@@ -62,8 +64,6 @@ pip install boto3
 aws configure
 ```
 
-To get the **AWS Access Key** required by the `aws configure`, please refer to the [AWS manual](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey). The **Default region name [None]:** and **Default output format [None]:** are optional.
-
 **GCP**:
 
 ```bash
diff --git a/docs/source/examples/auto-failover.rst b/docs/source/examples/auto-failover.rst
index 2b796451d16..b4b5484c872 100644
--- a/docs/source/examples/auto-failover.rst
+++ b/docs/source/examples/auto-failover.rst
@@ -1,3 +1,4 @@
+.. _auto-failover:
 Auto-provisioning GPUs
 ==========================
 
diff --git a/docs/source/examples/iterative-dev-project.rst b/docs/source/examples/iterative-dev-project.rst
index d8ae0984648..334abddd5d7 100644
--- a/docs/source/examples/iterative-dev-project.rst
+++ b/docs/source/examples/iterative-dev-project.rst
@@ -1,3 +1,4 @@
+.. _iter-dev:
 Iteratively Developing a Project
 ====================================
 
diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst
index d2dbd1c2d16..9cfcc838e25 100644
--- a/docs/source/getting-started/installation.rst
+++ b/docs/source/getting-started/installation.rst
@@ -10,7 +10,7 @@ Install Sky using pip:
    $ # Clone the sky codebase
    $ git clone ssh://git@github.com/sky-proj/sky.git
    $ cd sky
-   $ # Sky requires python >= 3.6.
+   $ # Sky requires python >= 3.6. 3.10+ is currently NOT supported.
    $ pip install ".[all]"
 
 If you only want the dependencies for certain clouds, you can also use
@@ -24,6 +24,8 @@ tasks in the clouds, configure access to at least one cloud:
 
 **AWS**:
 
+To get the **AWS Access Key** required by the :code:`aws configure`, please refer to the `AWS manual <https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey>`_. The **Default region name [None]:** and **Default output format [None]:** are optional.
+
 .. code-block:: console
 
    $ # Install boto
@@ -32,7 +34,6 @@ tasks in the clouds, configure access to at least one cloud:
    $ # Configure your AWS credentials
    $ aws configure
 
-To get the **AWS Access Key** required by the :code:`aws configure`, please refer to the `AWS manual <https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey>`_. The **Default region name [None]:** and **Default output format [None]:** are optional.
 
 **GCP**:
 
diff --git a/docs/source/getting-started/quickstart.rst b/docs/source/getting-started/quickstart.rst
index 83029485f6d..73c0f128257 100644
--- a/docs/source/getting-started/quickstart.rst
+++ b/docs/source/getting-started/quickstart.rst
@@ -19,19 +19,46 @@ Key Features
 - **No cloud lock-in** - transparently run your code across AWS, Google Cloud, and Azure
 
 
+Provisioning your first cluster
+--------------------------------
+We'll start by launching our first cluster on Sky using an interactive node.
+Interactive nodes are standalone machines that can be used like any other VM instance,
+but are easy to configure without any additional setup. Sky also handles provisioning
+these nodes with your specified resources as cheaply and quickly as possible using an
+:ref:`auto-failover provisioner <auto-failover>`.
+
+Let's provision an instance with a single K80 GPU.
+
+.. code-block:: bash
+
+    # Provisions/reuses an interactive node with a single K80 GPU.
+    sky gpunode -c mygpu --gpus K80
+
+    # View the machine in the cluster table.
+    sky status
+
+    NAME   LAUNCHED        RESOURCES                     COMMAND                          STATUS
+    mygpu  a few secs ago  1x Azure(Standard_NC6_Promo)  sky gpunode -c mygpu --gpus K80  UP
+
+Sky can also provision interactive CPU and TPU nodes with :code:`cpunode` and :code:`tpunode`.
+Please see our :ref:`CLI reference <cli>` for all configuration options. For more information on
+using and managing interactive nodes, check out our :ref:`reference documentation <interactive-nodes>`.
+
+
 Hello, Sky!
 -----------
-We'll start by launching our first cluster on Sky by defining a task.
+You can also define tasks to be executed by Sky. We'll define our very first task
+to be a simple hello world program.
 
 We can specify the following task attributes with a YAML file:
 
 - :code:`resources` (optional): what cloud resources the task must be run on (e.g., accelerators, instance type, etc.)
-- :code:`workdir` (optional): specifies work directory that is synced with the provisioned instance(s)
+- :code:`workdir` (optional): specifies working directory containing project code that is synced with the provisioned instance(s). **NOTE**: Sky does not currently support large, multi-gigabyte workdirs (e.g. do not store your large datasets in your working directory) as the files are synced to the remote VM with :code:`rsync`. Please consider using :ref:`Sky Storage <sky-storage>` to transfer large datasets and files.
 - :code:`setup` (optional): commands that must be run before the task is executed
 - :code:`run` (optional): specifies the commands that must be run as the actual ask
 
 Below is a minimal task YAML that prints "hello sky!" and shows installed Conda environments,
-requiring an NVIDIA Tesla K80 GPU on AWS. (See more example yaml files in the `repo <https://github.com/sky-proj/sky/tree/master/examples>`_, with a fully-complete example documented :ref:`here <yaml-spec>`.)
+requiring an NVIDIA Tesla K80 GPU on AWS. See more example yaml files in the `repo <https://github.com/sky-proj/sky/tree/master/examples>`_, with a fully-complete example documented :ref:`here <yaml-spec>`.
 
 .. code-block:: yaml
 
@@ -44,14 +71,16 @@ requiring an NVIDIA Tesla K80 GPU on AWS. (See more example yaml files in the `r
      # Get 1 K80 GPU.  Use <name>:<n> to get more (e.g., "K80:8").
      accelerators: K80
 
+   # Working directory (optional) containing the project codebase.
+   # This directory will be synced with the provisioned cluster.
    workdir: .
 
+   # Typical use: pip install -r requirements.txt
    setup: |
-     # Typical use: pip install -r requirements.txt
      echo "running setup"
 
+   # Typical use: make use of resources, such as running training.
    run: |
-     # Typical use: make use of resources, such as running training.
      echo "hello sky!"
      conda env list
 
@@ -91,7 +120,14 @@ If you would like to log into the a cluster, Sky provides convenient SSH access
 
    $ ssh mycluster
 
+If you would like to transfer files to and from the cluster, *rsync* or *scp* can be used:
+
+.. code-block:: console
+
+    $ scp -r /local/path/source mycluster:/remote/dest  # copy files to remote VM
+    $ scp -r mycluster:/remote/source /local/dest       # copy files from remote VM
+
 Sky is more than a tool for easily provisioning and managing multiple clusters
-on different clouds.  It also comes with features for storing and moving data,
-queueing multiple jobs, iterative development, and interactive nodes for
+on different clouds.  It also comes with features for :ref:`storing and moving data <sky-storage>`,
+:ref:`queueing multiple jobs <job-queue>`, :ref:`iterative development <iter-dev>`, and :ref:`interactive nodes <interactive-nodes>` for
 debugging.
diff --git a/docs/source/getting-started/tutorial.rst b/docs/source/getting-started/tutorial.rst
index b1dda49a028..f3f458994ad 100644
--- a/docs/source/getting-started/tutorial.rst
+++ b/docs/source/getting-started/tutorial.rst
@@ -79,7 +79,7 @@ If we wish to view the output for each run after it has completed we can use:
    $ # View the jobs in the queue
    $ sky queue lm-cluster
 
-   ID  NAME         USER  SUBMITTED    STARTED     STATUS   
+   ID  NAME         USER  SUBMITTED    STARTED     STATUS
    5   job5         user  10 mins ago  10 mins ago RUNNING
    4   job4         user  10 mins ago  -           PENDING
    3   job3         user  10 mins ago  9 mins ago  RUNNING
@@ -92,3 +92,13 @@ If we wish to view the output for each run after it has completed we can use:
 
    $ # Cancel job job3 (ID: 3)
    $ sky cancel lm-cluster 3
+
+
+Transferring Checkpoints and Artifacts
+--------------------------------------
+To transfer a checkpoint or artifact from the remote VM to the local VM, you can use `scp` or `rsync`:
+
+.. code-block:: console
+
+    $ scp -r local_artifacts/ lm-cluster:/path/to/destination  # copy files to remote VM
+    $ scp -r lm-cluster:/path/to/checkpoints local_artifacts/  # copy files from remote VM
diff --git a/docs/source/reference/cli.rst b/docs/source/reference/cli.rst
index b4d396049a5..3daf6ecda43 100644
--- a/docs/source/reference/cli.rst
+++ b/docs/source/reference/cli.rst
@@ -1,3 +1,4 @@
+.. _cli:
 CLI Reference
 =============
 
diff --git a/docs/source/reference/interactive-nodes.rst b/docs/source/reference/interactive-nodes.rst
index 5179f8e0805..1aaf06ada5b 100644
--- a/docs/source/reference/interactive-nodes.rst
+++ b/docs/source/reference/interactive-nodes.rst
@@ -1,3 +1,4 @@
+.. _interactive-nodes:
 Interactive Nodes
 =================
 
diff --git a/docs/source/reference/job-queue.rst b/docs/source/reference/job-queue.rst
index 666d729acf9..4508a31acd2 100644
--- a/docs/source/reference/job-queue.rst
+++ b/docs/source/reference/job-queue.rst
@@ -1,3 +1,4 @@
+.. _job-queue:
 Job Queue
 =========
 
diff --git a/docs/source/reference/storage.rst b/docs/source/reference/storage.rst
index c29f2556df1..9c17cab7875 100644
--- a/docs/source/reference/storage.rst
+++ b/docs/source/reference/storage.rst
@@ -1,3 +1,4 @@
+.. _sky-storage:
 Storage
 =======
 
diff --git a/docs/source/reference/yaml-spec.rst b/docs/source/reference/yaml-spec.rst
index ee394199997..2e5fdbadc5e 100644
--- a/docs/source/reference/yaml-spec.rst
+++ b/docs/source/reference/yaml-spec.rst
@@ -13,6 +13,9 @@ describe all fields available.
 
     # Working directory (optional), synced each time launch or exec is run
     # with the yaml file.
+    # NOTE: Sky does not currently support large, multi-gigabyte workdirs as the
+    # files are synced to the remote VM with `rsync`. Please consider using Sky
+    # Storage to transfer large datasets and files.
     workdir: ~/my-task-code
 
     # Number of nodes (optional) to launch including the head node. If not


Code review comments:
diff_hunk: @@ -19,19 +19,46 @@ Key Features
 - **No cloud lock-in** - transparently run your code across AWS, Google Cloud, and Azure
 
 
+Provisioning your first cluster
+--------------------------------
+We'll start by launching our first cluster on Sky using an interactive node.
+Interactive nodes are standalone machines that can be used like any other VM instance,
+but are easy to configure without any additional setup. Sky also handles provisioning
+these nodes with your specified resources as cheaply and quickly as possible using an
+:ref:`auto-failover provisioner <auto-failover>`.
+
+Let's provision an instance with a single K80 GPU.
+
+.. code-block:: bash
+
+    # Provisions/reuses an interactive node with a single K80 GPU.
+    sky gpunode -c mygpu --gpus K80
path: docs/source/getting-started/quickstart.rst
text: @user1:
Doesn't this command drop us in a terminal at the gpunode? I think we should show that here (and possibly run something like `nvidia-smi` there) for people who aren't following the quickstart in their terminal but are curious about sky.

diff_hunk: @@ -91,7 +120,14 @@ If you would like to log into the a cluster, Sky provides convenient SSH access
 
    $ ssh mycluster
 
+If you would like to transfer files to and from the cluster, *rsync* or *scp* can be used:
+
+.. code-block:: console
+
+    $ scp -r /local/path/source mycluster:/remote/dest  # copy files to remote VM
+    $ scp -r mycluster:/remote/source /local/dest       # copy files from remote VM
path: docs/source/getting-started/quickstart.rst
text: @user1:
Correct me if I am wrong, I think our hint message in `sky` uses `rsync`.

@user2:
+1 for recommending rsync here

diff_hunk: @@ -92,3 +92,13 @@ If we wish to view the output for each run after it has completed we can use:
 
    $ # Cancel job job3 (ID: 3)
    $ sky cancel lm-cluster 3
+
+
+Transferring Checkpoints and Artifacts
+--------------------------------------
+To transfer a checkpoint or artifact from the remote VM to the local VM, you can use `scp` or `rsync`:
+
+.. code-block:: console
+
+    $ scp -r local_artifacts/ lm-cluster:/path/to/destination  # copy files to remote VM
+    $ scp -r lm-cluster:/path/to/checkpoints local_artifacts/  # copy files from remote VM
path: docs/source/getting-started/tutorial.rst
text: @user1:
It would be better to use rsync for large files? `rsync -Pavz`

diff_hunk: @@ -19,19 +19,46 @@ Key Features
 - **No cloud lock-in** - transparently run your code across AWS, Google Cloud, and Azure
 
 
+Provisioning your first cluster
+--------------------------------
+We'll start by launching our first cluster on Sky using an interactive node.
+Interactive nodes are standalone machines that can be used like any other VM instance,
+but are easy to configure without any additional setup. Sky also handles provisioning
+these nodes with your specified resources as cheaply and quickly as possible using an
+:ref:`auto-failover provisioner <auto-failover>`.
+
+Let's provision an instance with a single K80 GPU.
+
+.. code-block:: bash
+
+    # Provisions/reuses an interactive node with a single K80 GPU.
+    sky gpunode -c mygpu --gpus K80
+
+    # View the machine in the cluster table.
+    sky status
+
+    NAME   LAUNCHED        RESOURCES                     COMMAND                          STATUS
+    mygpu  a few secs ago  1x Azure(Standard_NC6_Promo)  sky gpunode -c mygpu --gpus K80  UP
path: docs/source/getting-started/quickstart.rst
text: @user1:
Should we also mention how to stop or down the cluster here?

@author:
@user2 and I talked about this offline earlier today and decided that adding too much detail could risk making the quickstart too long (it's already quite long to be considered "quick"). I've linked to the relevant pages where sky stop/down is mentioned to account for this.

@user3:
I strongly feel we should end this block with suggesting `sky down mygpu`. The way many people interact with quickstart guides is skimming through them and running instructions as is in their terminal. I don't think many people are going to take the time to think about the implications of having launched a cluster (i.e., you need to shut it down too) without an explicit prompt from us. 

Something very brief is also fine - 
`After you are done, run  sky down mygpu to terminate your cluster. Find more details on managing the lifecycle of your cluster here<link>`.

@user3:
Same below, at the end of the task example