Repository: ray-project/kuberay
Pull request #3564: [DOCS] Apiserver improve docs readability
Author: login: machichima
Labels: name: docs
description: Improvements or additions to documentation
name: apiserver
name: 1.4.0
Created: 2025-05-08T15:15:05Z | Merged: 2025-05-28T21:05:36Z

- Mainly update following docs to make the example easy to follow:
  - Autoscaling.md
  - CreatingServe.md
  - HACluster.md
  - JobSubmission.md
  - Monitoring.md
  - SecuringImplementation.md

- Add dangling docs to README's "Advanced Usage" section


## Why are these changes needed?

Some of the documents for apiserver is a bit hard to follow, we should improve their readability.

<!-- Please give a short summary of the change and the problem this solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've made sure the tests are passing.
- Testing Strategy
  - [ ] Unit tests
  - [ ] Manual tests
  - [ ] This PR is not tested :(


Linked issue #1234: [Refactor] Separate the logic for retrieving Serve application statuses and checking Serve application statuses
### Search before asking

- [X] I had searched in the [issues](https://github.com/ray-project/kuberay/issues) and found no similar feature requirement.


### Description

Separate the logic for retrieving Serve application statuses from checking Serve application statuses into two separate functions.

https://github.com/ray-project/kuberay/pull/1230#discussion_r1260241124

### Use case

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

Diff:
diff --git a/apiserver/Autoscaling.md b/apiserver/Autoscaling.md
index bb69c532f04..852bdb12c12 100644
--- a/apiserver/Autoscaling.md
+++ b/apiserver/Autoscaling.md
@@ -1,122 +1,99 @@
-# Creating Autoscaling clusters using API server
+# Creating Autoscaling clusters using APIServer
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains how to set up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the APIServer and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to the [README](README.md) for setting up the KubeRay operator and APIServer.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before proceeding with the example, remove any running RayClusters to ensure a successful
+execution of the steps below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+> [!IMPORTANT]
+> All the following guidance requires you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Install this [ConfigMap], which contains the code for our example. Simply download
+the file and run:
+
+```sh
+kubectl apply -f test/cluster/cluster/detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the ConfigMap is successfully created. You should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, deploy a RayCluster with the following command:
 
-```shell
+```sh
+# Create compute template
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
 --header 'Content-Type: application/json' \
---data '{
-  "name": "default-template",
-  "namespace": "default",
-  "cpu": 2,
-  "memory": 4
-}'
+--data  @docs/api-example/compute_template.json
+
+# Create RayCluster
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 --header 'Content-Type: application/json' \
---data '{
-  "name": "test-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "clusterSpec": {
-    "enableInTreeAutoscaling": true,
-    "autoscalerOptions": {
-        "upscalingMode": "Default",
-        "idleTimeoutSeconds": 30,
-        "cpu": "500m",
-        "memory": "512Mi"
-    },
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0"
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-            "detached_actor.py": "detached_actor.py",
-            "terminate_detached_actor.py": "terminate_detached_actor.py"
-           }
-         }
-       ]
-    },
-    "workerGroupSpec": [
-      {
-        "groupName": "small-wg",
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "replicas": 0,
-        "minReplicas": 0,
-        "maxReplicas": 5,
-        "rayStartParams": {
-           "node-ip-address": "$MY_POD_IP"
-         },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-example",
-            "items": {
-                "detached_actor.py": "detached_actor.py",
-                "terminate_detached_actor.py": "terminate_detached_actor.py"
-            }
-          }
-        ]
-      }
-    ]
-  }
-}'
+--data @docs/api-example/autoscaling_clusters.json
 ```
 
-## Validate that Ray cluster is deployed correctly
+This command performs two main operations:
 
-Run:
+1. Creates a compute template `default-template` that specifies resources to use during
+   scale-up (2 CPUs and 4 GiB memory).
 
-```shell
-kubectl get pods
-```
+2. Deploys a RayCluster (test-cluster) with:
+    - A head pod that manages the cluster
+    - A worker group configured to scale between 0 and 5 replicas
+
+The worker group uses the following autoscalerOptions to control scaling behavior:
+
+- **`upscalingMode: "Default"`**: Default scaling behavior. Ray will scale up only as
+needed.
+- **`idleTimeoutSeconds: 30`**: If a worker pod remains idle (i.e., not running any tasks)
+for 30 seconds, it will be automatically removed.
+- **`cpu: "500m"`, `memory: "512Mi"`**: Defines the **minimum resource unit** Ray uses to
+assess scaling needs.  If no worker pod has at least this much free capacity, Ray will
+trigger a scale-up and launch a new worker pod.
 
-You should get something like this:
+> **Note:** These values **do not determine the actual size** of the worker pod. The
+> pod size comes from the `computeTemplate` (in this case, 2 CPUs and 4 GiB memory).
 
-```shell
-test-cluster-head-pr25j             2/2     Running   0          2m49s
+### Validate that RayCluster is deployed correctly
+
+Run the following command to get a list of pods running. You should see something like below:
+
+```sh
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# kuberay-operator-545586d46c-f9grr   1/1     Running   0          49m
+# test-cluster-head                   2/2     Running   0          3m1s
 ```
 
-Note that only head pod is running and it has 2 containers
+Note that there is no worker for `test-cluster` as we set its initial replicas to 0. You
+will only see head pod with 2 containers for `test-cluster`.
 
-## Trigger RayCluster scale-up
+### Trigger RayCluster scale-up
 
-Create a detached actor:
+Create a detached actor to trigger scale-up with the following command:
 
 ```sh
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobs' \
@@ -132,24 +109,26 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobs' \
 }'
 ```
 
-Because we have specified `num_cpu: 0` for head node, this will cause creation of a worker node. Run:
+The `detached_actor.py` file is defined in the [ConfigMap] we installed earlier and
+mounted to the head node, which requires `num_cpus=1`. Recall that initially there is no
+worker pod exists, RayCluster needs to scale up a worker for running this actor.
 
-```shell
-kubectl get pods
-```
+Check if a worker is created. You should see a worker `test-cluster-small-wg-worker` spin
+up.
 
-You should get something like this:
+```sh
+kubectl get pods
 
-```shell
-test-cluster-head-pr25j              2/2     Running   0          15m
-test-cluster-worker-small-wg-qrjfm   1/1     Running   0          2m48s
+# NAME                                 READY   STATUS      RESTARTS   AGE
+# create-actor-tsvfc                   0/1     Completed   0          99s
+# kuberay-operator-545586d46c-f9grr    1/1     Running     0          55m
+# test-cluster-head                    2/2     Running     0          9m37s
+# test-cluster-small-wg-worker-j54xf   1/1     Running     0          88s
 ```
 
-You can see that a worker node have been created.
+### Trigger RayCluster scale-down
 
-## Trigger RayCluster scale-down
-
-Run:
+Run the following command to delete the actor we created earlier:
 
 ```sh
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobs' \
@@ -165,16 +144,28 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobs' \
 }'
 ```
 
-A worker Pod will be deleted after `idleTimeoutSeconds` (default 60s, we specified 30) seconds. Run:
+Once the actor is deleted, the worker is no longer needed. The worker pod will be deleted
+after `idleTimeoutSeconds` (default 60; we specified 30) seconds.
+
+List all pods to verify that the worker pod is deleted:
 
-```shell
+```sh
 kubectl get pods
+
+# NAME                                READY   STATUS      RESTARTS   AGE
+# create-actor-tsvfc                  0/1     Completed   0          6m37s
+# delete-actor-89z8c                  0/1     Completed   0          83s
+# kuberay-operator-545586d46c-f9grr   1/1     Running     0          60m
+# test-cluster-head                   2/2     Running     0          14m
+
 ```
 
-And you should see only head node (worker node is deleted)
+### Clean up
 
-```shell
-test-cluster-head-pr25j             2/2     Running   0          27m
+```sh
+make clean-cluster
+# Remove apiserver from helm
+helm uninstall kuberay-apiserver
 ```
 
 [document]: https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/configuring-autoscaling.html
diff --git a/apiserver/CreatingServe.md b/apiserver/CreatingServe.md
index ba35d4dfe1a..65e59ebb66e 100644
--- a/apiserver/CreatingServe.md
+++ b/apiserver/CreatingServe.md
@@ -1,9 +1,13 @@
 # Creating a cluster with RayServe support
 
-Up until rescently the only way to create a Ray cluster supporting RayServe was by using `Create ray
-service` APIs. Although it does work, quite often you want to create cluster supporting Ray serve so
-that you can experiment with serve APIs directly. Now it is possible by adding the following
-annotation to the cluster:
+Currently, there are two ways to create a RayCluster with RayService support:
+
+1. Create a RayCluster first, then use [Create RayService](./HttpRequestSpec.md#Create-ray-service-in-a-given-namespace)
+   APIs to create a service.
+2. Directly create a cluster with RayServe support (described in this document)
+
+To create a cluster with RayService support, simply add the following annotation when
+creating the cluster:
 
 ```json
 "annotations" : {
@@ -11,94 +15,78 @@ annotation to the cluster:
   },
 ```
 
-the complete curl command to creation such cluster is as follows:
+## Setup
 
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "test-cluster",
-    "namespace": "default",
-    "user": "boris",
-    "annotations" : {
-      "ray.io/enable-serve-service": "true"
-    },
-    "clusterSpec": {
-      "headGroupSpec": {
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "serviceType": "ClusterIP",
-        "rayStartParams": {
-           "dashboard-host": "0.0.0.0",
-           "metrics-export-port": "8080",
-           "dashboard-agent-listen-port": "52365"
-         },
-         "volumes": [
-           {
-             "name": "code-sample",
-             "mountPath": "/home/ray/samples",
-             "volumeType": "CONFIGMAP",
-             "source": "ray-job-code-sample",
-             "items": {"sample_code.py" : "sample_code.py"}
-           }
-         ]
-      },
-      "workerGroupSpec": [
-        {
-          "groupName": "small-wg",
-          "computeTemplate": "default-template",
-          "image": "rayproject/ray:2.9.0-py310",
-          "replicas": 1,
-          "minReplicas": 0,
-          "maxReplicas": 5,
-          "rayStartParams": {
-             "node-ip-address": "$MY_POD_IP"
-           },
-          "volumes": [
-            {
-              "name": "code-sample",
-              "mountPath": "/home/ray/samples",
-              "volumeType": "CONFIGMAP",
-              "source": "ray-job-code-sample",
-              "items": {"sample_code.py" : "sample_code.py"}
-            }
-          ]
-        }
-      ]
-    }
-  }'
+Refer to the [README](README.md) for setting up the KubeRay operator and APIServer.
+
+## Example
+
+This example walks through how to create a cluster with RayService support.
+
+Before proceeding with the example, remove any running RayClusters to ensure a successful
+execution of the steps below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Note, that before creating a cluster you need to install this [configmap] and
-create default template using the following command:
+> [!IMPORTANT]
+> All the following guidance requires you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
+Install this [ConfigMap], which contains the code for our example. Simply download
+the file and run:
+
+```sh
+kubectl apply -f test/job/code.yaml
+```
+
+Check if the ConfigMap is successfully created. You should see `ray-job-code-sample` in
+the list:
+
+```sh
+kubectl get configmaps
+# NAME                  DATA   AGE
+# ray-job-code-sample   1      13s
+```
+
+### Create RayCluster
+
+Use the following commands to create a compute template and a RayCluster with RayService support:
+
+```sh
+# Create compute tempalte
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+--header 'Content-Type: application/json' \
+--data  @docs/api-example/compute_template.json
+
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
   --header 'Content-Type: application/json' \
-  --data '{
-    "name": "default-template",
-    "namespace": "default",
-    "cpu": 2,
-    "memory": 4
-  }'
+  --data @docs/api-example/create_serving_clusters.json
 ```
 
-To confirm that the cluster is created correctly, check created services using that following command:
+To confirm that the cluster is created correctly, check the created services using the following command:
 
-```shell
+```sh
 kubectl get service
+# NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                   AGE
+# test-cluster-head-svc    ClusterIP   None            <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP   7s
+# test-cluster-serve-svc   ClusterIP   10.96.161.26    <none>        8000/TCP                                                  7s
 ```
 
-that should return the following:
+You should see two services created: one for the head node to access the dashboard and manage the
+cluster, and the other to submit the serve request.
 
-```shell
-test-cluster-head-svc    ClusterIP   10.96.19.185    <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP
-test-cluster-serve-svc   ClusterIP   10.96.144.162   <none>        8000/TCP
-```
+Note that we set the 52365 port for the dashboard agent in the above curl command. This port is
+used internally by Ray Serve.
 
-As you can see, in this case two services are created - one for the head node to be able to see the
-dashboard and configure the cluster and one for submission of the serve requests.
+### Clean up
 
-For the head node service note that the additional port - 52365 is created for serve configuration.
+```sh
+make clean-cluster
+# Remove APIServer from helm
+helm uninstall kuberay-apiserver
+```
 
-[configmap]: test/job/code.yaml
+[ConfigMap]: test/job/code.yaml
diff --git a/apiserver/HACluster.md b/apiserver/HACluster.md
index c5fcfea8296..2002f34e88a 100644
--- a/apiserver/HACluster.md
+++ b/apiserver/HACluster.md
@@ -1,217 +1,223 @@
-# Creating HA cluster with API Server
+# Creating HA cluster with APIServer
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
+One of the issues with long-running Ray applications (e.g., RayServe) is that if the Ray head node
+dies, the whole cluster has to be restarted. Fortunately, the KubeRay cluster solves this by
+introducing the [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
 
-## Creating external Redis cluster
+The RayCluster with high availability can also be created with the APIServer, which aims to
+ensure high availability of Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which lacks fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS fault tolerance,
+we should have a highly available Redis so that when GCS restarts, it can resume its
+state by retrieving previous data from the Redis instance.
 
-A comprehensive documentation on creating Redis cluster on Kubernetes can be found
-[here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+We will provide a detailed example on how to create this highly available APIServer.
+
+## Setup
+
+### Setup Ray Operator and APIServer
+
+Refer to the [README](README.md) for setting up the KubeRay operator and APIServer.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
+
+Comprehensive documentation on creating a Redis cluster on Kubernetes can be found
+[here](https://www.dragonflydb.io/guides/redis-kubernetes). For this example, we will use a simple
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
-Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
-you to stand up, run, manage and monitor your own version of redis.
+Alternatively, if you run on the cloud, you can use a managed version of HA Redis, which will not require
+you to set up, run, manage, and monitor your own version of Redis.
 
-## Creating Redis password secret
+Check if Redis is successfully set up with the following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create a secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external Redis, please download the [Secret] and
+run the following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+We will use this [ConfigMap], which contains code for our example. For real-world
+cases, it is recommended to pack user code in an image.
+
+Please download the config map and deploy it with the following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use the following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, add the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we also added the following content in `rayStartParams` of `headGroupSpec`,
+which sets the Redis password and the number of CPUs. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
+{
+    "redis-password:: "$REDIS_PASSWORD"
+    "num-cpu": "0"
+}
+```
+
+Here, the `$REDIS_PASSWORD` is defined in the `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
     },
-    "workerGroupSpec": [
-      {
-        "groupName": "small-wg",
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "replicas": 1,
-        "minReplicas": 0,
-        "maxReplicas": 5,
-        "rayStartParams": {
-           "node-ip-address": "$MY_POD_IP",
-           "metrics-export-port": "8080"
-        },
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-example",
-            "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-          }
-        ]
-      }
-    ]
-  }
-}'
-```
-
-Note that computeTemplate here has to be created using this [command](test/cluster//template/simple)
-
-Lets discuss the important pieces here:
-You need to specify annotation, that tells Ray that this is cluster with GCS fault tolerance
+    "valuesFrom": {
+        "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+        }
+    }
+},
+```
 
-```sh
-ray.io/ft-enabled: "true"
+For the `workerGroupSpecs`, we set the `gcs_rpc_server_reconnect_timeout` environment
+variable, which controls the GCS heartbeat timeout (default 60 seconds). This controls how
+long after the head node dies we kill the worker node. While it takes time to restart
+the head node, we want these values to be large enough to prevent the worker node from being
+killed during the restarting period.
+
+```json
+"environment": {
+    "values": {
+        "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
+    }
+},
 ```
 
-For the `headGroupSpec` you need the following. In the `rayStartParams` you need to add information about Redis
-password.
+### Validate that RayCluster is deployed correctly
+
+Run the following command to get a list of pods running. You should see one head and one worker node
+as shown below:
 
 ```sh
-"redis-password:: "$REDIS_PASSWORD"
-"num-cpu": "0"
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     1/1     Running   0          2m36s
+# ha-cluster-small-wg-worker-22lbx    1/1     Running   0          2m36s
 ```
 
-Where the value of `REDIS_PASSWORD` comes from environment variable (below). Additionally `num-cpus:
-0` ensures that no application code runs on a head node.
+### Create an Actor
+
+Before we try to trigger the restoration, we need to find a way to validate that our GCS restore
+is working correctly. We will validate this by creating a detached actor. If it still
+exists and functions after the head node deletion and restoration, we can confirm that the
+GCS data is restored correctly.
 
-The following environment variable have to be added here:
+Run the following command to create a detached actor. Please change `ha-cluster-head` to
+your head node's name. Note that the `detached_actor.py` file is defined in the
+[ConfigMap] we installed earlier and mounted to the head node:
 
 ```sh
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-```
-
-For the `workerGroupSpecs` you might want to increase `gcs_rpc_server_reconnect_timeout` by specifying the following
-environment variable:
+kubectl exec -it ha-cluster-head -- python3 /home/ray/samples/detached_actor.py
+```
+
+Then, open a new terminal and use port-forward to enable access to the Ray dashboard.
+The dashboard can be accessed through `http://localhost:8265`:
 
 ```sh
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
+kubectl port-forward pod/ha-cluster-head 8265:8265
 ```
 
-This environment variable allows to increase GCS heartbeat timeout, which is 60 sec by default. The reason for
-increasing it is because restart of the head node can take some time, and we want to make sure that the worker node
-will not be killed during this time.
+In the dashboard, you can see two nodes in the Cluster pane, which are the head and worker:
+
+![hacluster-dashboard-cluster](img/hacluster-dashboard-cluster.png)
+
+If you go to the Actor pane, you can see the actor that we created earlier:
+
+![hacluster-dashboard-actor](img/hacluster-dashboard-actor.png)
 
-## Testing resulting cluster
+### Trigger the GCS restore
+
+To trigger the restoration, simply delete the head node with:
+
+```sh
+kubectl delete pods ha-cluster-head
+```
 
-Once the cluster is created, we can validate that it is working correctly. To do this first create a detached actor.
-To do this, note the name of the head node and create a detached actor using the following command:
+If you list the pods now, you can see a new head node is recreated
 
 ```sh
-kubectl exec -it <head node pod name> -- python3 /home/ray/samples/detached_actor.py
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     0/1     Running   0          5s
+# ha-cluster-small-wg-worker-tpgqs    1/1     Running   0          9m19s
 ```
 
-Once this is done, open Ray dashboard (using port-forward). In the cluster tab you should see 2 nodes and in the
-Actor's pane you should see created actor.
+Note that only the head node will be recreated, while the worker node remains unchanged.
 
-Now you can delete head node pod:
+Port-forward again and access the dashboard through `http://localhost:8265`:
 
 ```sh
-kubectl delete pods <head node pod name>
+kubectl port-forward pod/ha-cluster-head 8265:8265
 ```
 
-The operator will recreate it. Make sure that only head node is recreated (note that it now has a different name),
-while worker node stays as is. Now you can go to the dashboard and make sure that in the Cluster tab you still see
-2 nodes and in the Actor's pane you still see created actor.
+You can see one pod marked as "DEAD" in the Cluster pane, and the actor in the Actors pane
+is still running.
 
-For additional test run the following command:
+### Clean up
 
 ```sh
-kubectl exec -it <head node pod name> -- python3 /home/ray/samples/increment_counter.py
+make clean-cluster
+# Remove apiserver from helm
+helm uninstall kuberay-apiserver
 ```
 
-and make sure that it executes correctly. Note that the name of the head node here is different
+[RedisYAML]: test/cluster/redis/redis.yaml
+[Secret]: test/cluster/redis/redis_passwrd.yaml
+[ConfigMap]: test/cluster/code_configmap.yaml
diff --git a/apiserver/JobSubmission.md b/apiserver/JobSubmission.md
index bd7fa7a5304..993a4fec9a5 100644
--- a/apiserver/JobSubmission.md
+++ b/apiserver/JobSubmission.md
@@ -1,111 +1,78 @@
-# Job Submission using API server
+# Job Submission using APIServer
 
-Ray provides very convinient and powerful [Job Submission APIs]. The issue is that it needs to
-access cluster's dashboard Ingress, which currently does not have any security implementations.
-Alternatively you can use Ray cluster head service for this, but in this case your Ray job
-management code has to run in the same kubernetes cluster as Ray. Job submission implementation in
-the API server leverages already exposed URL of the API server to locate cluster and use its head
-service to implement Job management functionality. Because you can access API server running on the
-remote kubernetes cluster you can use these APIs for managing remote Ray clusters without exposing
-them via Ingress.
+Ray provides convenient and powerful [Job Submission APIs]. However, it requires access to
+the cluster's dashboard Ingress, which currently lacks security implementations.
+Alternatively, you can use the Ray cluster head service for this, but in that case, your
+Ray job management code must run in the same Kubernetes cluster as Ray. The job submission
+implementation in the APIServer leverages the already exposed URL of the APIServer to
+locate the cluster and use its head service to implement job management functionality.
+Since you can access the APIServer running on a remote Kubernetes cluster, you can use
+these APIs to manage remote Ray clusters without exposing them via Ingress.
 
 ## Using Job Submission APIs
 
-Note that job submission APIs will only work if you are running API server within kubernetes
-cluster. Local Development option of the API server will not work.
+Note that job submission APIs will only work if you are running the APIServer within a
+Kubernetes cluster. The local development option of the APIServer will not work.
 
-The first step is to deploy KubeRay operator and API server.
+Before proceeding with the example, remove any running RayClusters to ensure the successful
+execution of the steps below.
 
-### Deploy KubeRay operator and API server
+```sh
+kubectl delete raycluster --all
+```
 
-Reffer to [readme](README.md) for setting up KubRay operator and API server.
+### Deploy KubeRay operator and APIServer
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
-```
+Refer to [README](README.md) for setting up the KubeRay operator and APIServer.
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+### Install ConfigMap
 
-### Deploy Ray cluster
+Note that this cluster is mounting a volume from a ConfigMap. This ConfigMap should be
+created prior to cluster creation using [this YAML].
 
-Execute the following commands:
+We will use this [ConfigMap], which contains the code for our example. Please download the
+ConfigMap and deploy it with the following command:
 
-```shell
+```sh
+kubectl apply -f code_configmap.yaml
+```
+
+### Deploy RayCluster
+
+Execute the following commands to deploy the RayCluster:
+
+```sh
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "default-template",
-    "namespace": "default",
-    "cpu": 2,
-    "memory": 4
-  }'
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
 
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
   --header 'Content-Type: application/json' \
-  --data '{
-    "name": "test-cluster",
-    "namespace": "default",
-    "user": "boris",
-    "clusterSpec": {
-      "headGroupSpec": {
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "serviceType": "NodePort",
-        "rayStartParams": {
-          "dashboard-host": "0.0.0.0",
-          "metrics-export-port": "8080"
-        },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-job-code-sample",
-            "items": {"sample_code.py" : "sample_code.py"}
-          }
-        ]
-      },
-      "workerGroupSpec": [
-        {
-          "groupName": "small-wg",
-          "computeTemplate": "default-template",
-          "image": "rayproject/ray:2.9.0-py310",
-          "replicas": 1,
-          "minReplicas": 1,
-          "maxReplicas": 1,
-          "rayStartParams": {
-            "node-ip-address": "$MY_POD_IP"
-          },
-          "volumes": [
-            {
-              "name": "code-sample",
-              "mountPath": "/home/ray/samples",
-              "volumeType": "CONFIGMAP",
-              "source": "ray-job-code-sample",
-              "items": {"sample_code.py" : "sample_code.py"}
-            }
-          ]
-        }
-      ]
-    }
-  }'
+  --data @docs/api-example/jobsubmission_clusters.json
 ```
 
-Note that this cluster is mounting a volume from a configmap. This config map should be created
-prior to cluster creation using [this YAML].
+To verify if the RayCluster is set up correctly, list all pods with the following command.
+You should see a head and worker node for `test-cluster` running:
+
+```sh
+kubectl get pods
+# NAME                                 READY   STATUS    RESTARTS   AGE
+# test-cluster-head                    1/1     Running   0          4m9s
+# test-cluster-small-wg-worker-c8t2w   1/1     Running   0          4m9s
+```
 
 ### Submit Ray Job
 
-Once the cluster is up and running, you can submit a job to the cluster using the following command:
+Once the cluster is up and running, submit a job to the cluster using the following command:
 
-```shell
+```sh
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster' \
-  --header 'Content-Type: application/json' \
-  --data '{
+    --header 'Content-Type: application/json' \
+    --data '{
     "entrypoint": "python /home/ray/samples/sample_code.py",
     "runtimeEnv": "pip:\n  - requests==2.26.0\n  - pendulum==2.1.2\nenv_vars:\n  counter_name: test_counter\n",
     "numCpus": ".5"
-  }'
+    }'
 ```
 
 This should return the following:
@@ -116,19 +83,19 @@ This should return the following:
 }
 ```
 
-Note that the `submissionId` value that you will get is different
+Note that the `submissionId` value you receive will be different.
 
 ### Get job details
 
-Once the job is submitted, the following command can be used to get job's details. Note that
-submission ID returned during job creation should be used here.
+Once the job is submitted, use the following command to get the job's details. Please
+replace the `submissionId` with the one returned during job creation.
 
 ```shell
-curl -X GET 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster/raysubmit_KWZLwme56esG3Wcr' \
+curl -X GET 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster/<submissionID>' \
   --header 'Content-Type: application/json'
 ```
 
-This should return JSON similar to the one below
+This should return JSON similar to the example below:
 
 ```json
 {
@@ -148,15 +115,15 @@ This should return JSON similar to the one below
 
 ### Get Job log
 
-You can also get job execution log using the following command. Note that submission ID returned
-during job creation should be used here.
+You can also retrieve the job execution log using the following command. Please replace
+the `<submissionID>` with the one returned during job creation.
 
 ```shell
-curl -X GET 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster/log/raysubmit_KWZLwme56esG3Wcr' \
+curl -X GET 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster/log/<submissionID>' \
   --header 'Content-Type: application/json'
 ```
 
-This will return execution log, that will look something like the following
+This will return the execution log, which will look something like the following:
 
 ```text
 2023-11-08 03:24:31,904\tINFO worker.py:1329 -- Using address 10.244.2.2:6379 set in the environment variable RAY_ADDRESS
@@ -169,19 +136,19 @@ test_counter got 4
 test_counter got 5
 ```
 
-Note that this command always returns execution log from the begining (no streaming support) till
-the current moment.
+Note that this command always returns the execution log from the beginning (no streaming
+support) up to the current moment.
 
 ### List jobs
 
-You can also list all the jobs (in any state) in the Ray cluster using the following command:
+You can also list all jobs (in any state) in the Ray cluster using the following command:
 
 ```shell
 curl -X GET 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster' \
   --header 'Content-Type: application/json'
 ```
 
-This should return the list of the submissions, that looks as follows:
+This should return a list of submissions, which looks as follows:
 
 ```json
 {
@@ -205,25 +172,33 @@ This should return the list of the submissions, that looks as follows:
 
 ### Stop Job
 
-Execution of the job can be stoped using the following command. Note that submission ID returned
-during job creation should be used here.
+A job can be stopped using the following command. Please replace the `<submissionID>` with
+the one returned during job creation.
 
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster/raysubmit_KWZLwme56esG3Wcr' \
+```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster/<submissionID>' \
   --header 'Content-Type: application/json'
 ```
 
 ### Delete Job
 
-Finally, you can delete job using the following command. Note that submission ID returned during job
-creation should be used here.
+Finally, you can delete a job using the following command. Please replace the
+`<submissionID>` with the one returned during job creation.
 
-```shell
-curl -X DELETE 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster/raysubmit_KWZLwme56esG3Wcr' \
+```sh
+curl -X DELETE 'localhost:31888/apis/v1/namespaces/default/jobsubmissions/test-cluster/<submissionID>' \
   --header 'Content-Type: application/json'
 ```
 
-You can validate job deletion by looking at the Ray dashboard (jobs pane) and ensuring that it was removed
+You can confirm the job deletion by listing jobs again. You should see an empty list.
+
+### Clean up
+
+```sh
+make clean-cluster
+# Remove APIServer from helm
+helm uninstall kuberay-apiserver
+```
 
 [Job Submission APIs]: https://docs.ray.io/en/latest/cluster/running-applications/job-submission/rest.html
-[this YAML]: /test/job/code.yaml
+[ConfigMap]: test/job/code.yaml
diff --git a/apiserver/Makefile b/apiserver/Makefile
index 2fb87bd0094..028fa427395 100644
--- a/apiserver/Makefile
+++ b/apiserver/Makefile
@@ -163,9 +163,9 @@ endif
 .PHONY: load-security-proxy-image
 load-security-proxy-image: ## Load the security proxy image to the kind cluster created with make cluster.
 ifneq ($(SECURITY_IMAGE_TAG), latest)
-	$(ENGINE) pull kuberay/security-proxy:$(SECURITY_IMAGE_TAG)
+	$(ENGINE) pull quay.io/kuberay/security-proxy:$(SECURITY_IMAGE_TAG)
 endif
-	$(KIND) load docker-image kuberay/security-proxy:$(SECURITY_IMAGE_TAG) -n $(KIND_CLUSTER_NAME)
+	$(KIND) load docker-image quay.io/kuberay/security-proxy:$(SECURITY_IMAGE_TAG) -n $(KIND_CLUSTER_NAME)
 
 .PHONY: load-ray-test-image
 load-ray-test-image: ## Load the ray test images.
diff --git a/apiserver/Monitoring.md b/apiserver/Monitoring.md
index fa3332b0e83..25ed103d246 100644
--- a/apiserver/Monitoring.md
+++ b/apiserver/Monitoring.md
@@ -1,68 +1,104 @@
-# Monitoring of the API server and created Ray clusters
+# Monitoring of the APIServer and created RayClusters
 
-In order to ensure a proper functioning of the API server and created Ray clusters, it is typically necessary to
-monitor the API server itself and created Ray clusters. This document describes how to instrument both API server
-and created clusters with Prometheus and Grafana
+To ensure the proper functioning of the APIServer and created RayClusters, it is
+typically necessary to monitor them. This document describes how to monitor both the APIServer
+and created clusters with Prometheus and Grafana.
 
-## Monitoring of the API server
+The current implementation of the APIServer provides a flag `collectMetricsFlag` that defines whether to collect and
+expose Prometheus metrics, which is set to `true` by default. To disable metrics collection, this flag needs to
+be set to `false`, and the APIServer image needs to be rebuilt.
 
-Current implementation of the API server provides a flag `collectMetricsFlag` that defines whether to collect and
-expose Prometheus metrics, which is by default set to `true`. To disable metrics collection, this flag needs to
-be set to `false` and the API server image needs to be rebuild.
+If this flag is enabled, an `http` port at the `/metrics` endpoint provides APIServer metrics in Prometheus format.
 
-If this flag is enabled, an `http` port at `/metrics` endpoint provides API server metrics in Prometheus format.
+## Monitoring of the APIServer
 
-### Testing monitoring of the API server
+### Deploy KubeRay operator and APIServer
 
-On the kind server Install Kubernetes Prometheus Stack via Helm chart using the following
-[command](../install/prometheus/install.sh). The script additionally creates `ray-head-monitor` and
-`ray-workers-monitor` in the `prometheus-system` namespace, that we do not need. We can delete them using:
+Refer to [README](README.md) for setting up the KubeRay operator and APIServer. This will
+set the flag `collectMetricsFlag` to `true`, which enables metrics collection.
+
+> [!IMPORTANT]
+> All the following guidance requires you to switch your working directory to the KubeRay project root.
+
+### Install Kubernetes Prometheus Stack via Helm Chart
+
+Please navigate to the path `install/prometheus` under the KubeRay repository and execute the
+install script there to install the Prometheus Stack on the kind cluster.
+
+```sh
+cd install/prometheus
+bash install.sh
+```
+
+You can see multiple pods are created in the `prometheus-system` namespace:
+
+```sh
+kubectl get pods -n prometheus-system
+# NAME                                                     READY   STATUS            RESTARTS   AGE
+# alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          2m7s
+# prometheus-grafana-84ccb68cc-g9br2                       3/3     Running   0          2m22s
+# prometheus-kube-prometheus-operator-895b579fc-f55f6      1/1     Running   0          2m22s
+# prometheus-kube-state-metrics-77b6c5d54b-wd2ml           1/1     Running   0          2m22s
+# prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          2m7s
+# prometheus-prometheus-node-exporter-fs642                1/1     Running   0          2m22s
+```
+
+The `ray-head-monitor` and `ray-workers-monitor` in the `prometheus-system` namespace
+created by the script will not be used in this example. We can safely delete them with:
 
 ```shell
 kubectl delete servicemonitor ray-head-monitor -n prometheus-system
 kubectl delete podmonitor ray-workers-monitor -n prometheus-system
 ```
 
-To install API server, please follow the instructions from the [README.md](README.md) on installing
-API server using Helm
+### Testing monitoring of the APIServer
 
-Now we can install a [service monitor](deploy/prometheus/api_server_service_monitor.yaml) to scrape Api Server metrics into
+Now we can install a [service monitor](deploy/prometheus/api_server_service_monitor.yaml) to scrape APIServer metrics into
 Prometheus using the following command:
 
-```shell
-kubectl apply -f deploy/prometheus/api_server_service_monitor.yaml
+```sh
+# Assume you are already in the KubeRay project root
+kubectl apply -f apiserver/deploy/prometheus/api_server_service_monitor.yaml
 ```
 
-With this in place, you can use port-forward to expose Prometheus:
+Then, please open a new terminal and use port-forwarding to expose Prometheus with:
 
 ```shell
 kubectl port-forward svc/prometheus-operated -n prometheus-system 9090
 ```
 
-Now you can point your browser to `http://localhost:9090/` to get a PromQL panel. Start typing `apiserver` and
-you will see all of the api server metrics in Prometheus.
+Now you can point your browser to `http://localhost:9090/` to get a PromQL panel. Start
+typing `apiserver` in the search bar, and you will see all of the APIServer metrics in
+Prometheus.
 
-## Monitoring of the Ray Cluster created by the API server
+### Monitoring of the Ray Cluster created by the APIServer
 
-Ray provides [documentation](https://docs.ray.io/en/master/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html#kuberay-prometheus-grafana)
-describing how to monitor Ray cluster created using KubeRay operator. Because API server is using KubeRay operator
-to create the cluster. this documentation can be used directly. Here we will show a slightly simpler approach to
-monitor cluster. Instead of creating `service monitor` for scraping head node and `pod monitor` for scraping worker
-nodes we suggest we create a single [pod monitor](deploy/prometheus/ray_cluster_pod_monitor.yaml) that can be installed using
-the following command:
+Ray provides
+[documentation](https://docs.ray.io/en/master/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html#kuberay-prometheus-grafana)
+describing how to monitor Ray clusters created using the KubeRay operator. As the APIServer is
+using the KubeRay operator to create the cluster, this documentation can be used directly.
 
-```shell
-kubectl apply -f deploy/prometheus/ray_cluster_pod_monitor.yaml
+Instead of creating a `service monitor` for scraping the head node and a `pod monitor` for
+scraping worker nodes, we will utilize a simpler approach by creating a single [pod
+monitor](deploy/prometheus/ray_cluster_pod_monitor.yaml) that can be installed using the
+following command:
+
+```sh
+kubectl apply -f apiserver/deploy/prometheus/ray_cluster_pod_monitor.yaml
 ```
 
-With this in place, you can again use port-forward to expose Prometheus:
+Now you can go back to the PromQL panel at `http://localhost:9090/`. Go to the Status > Targets
+pane from the top bar, and you should be able to see `podMonitor/prometheus-system/ray-workers-monitor/0` in the list.
 
-```shell
-kubectl port-forward svc/prometheus-operated -n prometheus-system 9090
-```
+Also, take a look at the Ray [documentation](https://docs.ray.io/en/master/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html#kuberay-prometheus-grafana)
+for additional monitoring features, including recording rules, alerts, and Grafana integration.
 
-Now you can point your browser to `http://localhost:9090/` to get a PromQL panel. Start typing `ray` and
-you will see all of the Ray cluster metrics in Prometheus.
+### Clean up
 
-Also take a look at the Ray [documentation](https://docs.ray.io/en/master/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html#kuberay-prometheus-grafana)
-for additional monitoring features, including Recording rules, Alerts and Grafana integration.
+```sh
+make clean-cluster
+# Remove APIServer from Helm
+helm uninstall kuberay-apiserver
+# Remove Prometheus stack from Helm
+helm uninstall kube-prometheus-stack -n prometheus-system
+```
diff --git a/apiserver/README.md b/apiserver/README.md
index bf25e12b824..350a09367e8 100644
--- a/apiserver/README.md
+++ b/apiserver/README.md
@@ -1,23 +1,23 @@
 <!-- markdownlint-disable MD013 -->
 # KubeRay APIServer
 
-The KubeRay APIServer provides gRPC and HTTP APIs to manage KubeRay resources.
+The KubeRay APIServer offers gRPC and HTTP APIs to manage KubeRay resources.
 
 ## Introduction
 
-The KubeRay APIServer is an optional component. It provides a layer of simplified configuration for KubeRay resources. The KubeRay API server is used internally by some organizations to back user interfaces for KubeRay resource management.
+The KubeRay APIServer is an optional component that provides a layer of simplified configuration for KubeRay resources. Some organizations use the KubeRay APIServer internally to support user interfaces for managing KubeRay resources.
 
 ## Installation
 
 ### Start a local apiserver
 
-You could build and start apiserver from scratch on your local environment in one simple command. It will deploy all the necessities to a local kind cluster.
+You can build and start the APIServer from scratch in your local environment with a single command. This will deploy all the necessary components to a local kind cluster.
 
 ```sh
 make start-local-apiserver
 ```
 
-apiserver supports HTTP request, so you could easily check whether it's started successfully by issuing two simple curl requests.
+The APIServer supports HTTP requests, so you can easily verify its successful startup by issuing two simple curl commands.
 
 ```sh
 # Create complete template.
@@ -40,7 +40,7 @@ curl --silent -X 'GET' \
 
 ### Install with Helm
 
-Make sure the version of Helm is v3+. Currently, [existing CI tests](https://github.com/ray-project/kuberay/blob/master/.github/workflows/helm-lint.yaml) are based on Helm v3.4.1 and v3.9.4.
+Ensure that the version of Helm is v3+. Currently, [existing CI tests](https://github.com/ray-project/kuberay/blob/master/.github/workflows/helm-lint.yaml) are based on Helm v3.4.1 and v3.9.4.
 
 ```sh
 helm version
@@ -48,29 +48,11 @@ helm version
 
 #### Install KubeRay Operator
 
-- Install a stable version via Helm repository (only supports KubeRay v0.4.0+)
-
-  ```sh
-    # Install the KubeRay helm repo
-    helm repo add kuberay https://ray-project.github.io/kuberay-helm/
-
-    # Install KubeRay Operator v1.1.0.
-    helm install kuberay-operator kuberay/kuberay-operator --version v1.1.0
-
-    # Check the KubeRay Operator Pod in `default` namespace
-    kubectl get pods
-    # NAME                                             READY   STATUS    RESTARTS   AGE
-    # kuberay-operator-7456c6b69b-t6pt7                1/1     Running   0          172m
-  ```
+Refer to [this document](https://docs.ray.io/en/master/cluster/kubernetes/getting-started/kuberay-operator-installation.html#kuberay-operator-deploy) to install the latest stable KubeRay operator.
 
 #### Install KubeRay APIServer
 
-```text
-Please note that examples show here will only work with the nightly builds of the api-server. `v1.0.0` does not yet contain critical fixes
-to the api server that would allow Kuberay Serve endpoints to work properly
-```
-
-- Install a stable version via Helm repository (only supports KubeRay v0.4.0+)
+- Install a stable version via the Helm repository (supports KubeRay v0.4.0+ only)
 
   ```sh
   # Install the KubeRay helm repo
@@ -98,7 +80,7 @@ to the api server that would allow Kuberay Serve endpoints to work properly
   helm install kuberay-apiserver .
   ```
 
-- Install the current (working branch) version
+- Install the current working branch version
 
   ```sh
   # Step1: Clone KubeRay repository
@@ -112,7 +94,7 @@ to the api server that would allow Kuberay Serve endpoints to work properly
 
 #### List the chart
 
-To list the deployments:
+To list the deployed charts:
 
 ```sh
 helm ls
@@ -134,17 +116,17 @@ kubectl get pods
 
 ## Usage
 
-After the deployment we may use the `{{baseUrl}}` to access the service. See [swagger support section](https://ray-project.github.io/kuberay/components/apiserver/#swagger-support) to get the complete definitions of APIs.
+After deployment, you can use the `{{baseUrl}}` to access the service. Refer to the [Swagger support section](https://ray-project.github.io/kuberay/components/apiserver/#swagger-support) for complete API definitions.
 
-- (default) for nodeport access, use port `31888` for connection
+- (default) for NodePort access, use port `31888` for connection
 
 - for ingress access, you will need to create your own ingress
 
-The requests parameters detail can be seen in [KubeRay swagger](https://github.com/ray-project/kuberay/tree/master/proto/swagger), this document only presents basic examples.
+Details of the request parameters can be found in [KubeRay Swagger](https://github.com/ray-project/kuberay/tree/master/proto/swagger). This document only provides basic examples.
 
 ### Setup a smoke test
 
-The following steps allow you to validate that the KubeRay API Server components and KubeRay Operator integrate in your environment.
+The following steps allow you to validate the integration of the KubeRay APIServer components and the KubeRay Operator in your environment.
 
 1. (Optional) You may use your local kind cluster or minikube
 
@@ -193,7 +175,7 @@ The following steps allow you to validate that the KubeRay API Server components
     helm -n ray-system install kuberay-apiserver kuberay/kuberay-apiserver -n ray-system --create-namespace
     ```
 
-3. The APIServer expose service using `NodePort` by default. You can test access by your host and port, the default port is set to `31888`. The examples below assume a kind (localhost) deployment. If Kuberay API server is deployed on another type of cluster, you'll need to adjust the hostname to match your environment.
+3. The APIServer exposes its service using `NodePort` by default. You can test access via your host and port; the default port is set to `31888`. The examples below assume a kind (localhost) deployment. If the KubeRay APIServer is deployed on another type of cluster, you'll need to adjust the hostname to match your environment.
 
     ```sh
     curl localhost:31888
@@ -204,16 +186,26 @@ The following steps allow you to validate that the KubeRay API Server components
     }
     ```
 
-4. You can create `RayCluster`, `RayJobs` or `RayService` by dialing the endpoints.
+4. You can create `RayCluster`, `RayJobs`, or `RayService` by accessing the endpoints.
 
 ## Swagger Support
 
-Kuberay API server has support for Swagger UI. The swagger page can be reached at:
+The KubeRay APIServer supports Swagger UI. The Swagger page can be accessed at:
 
 - [localhost:31888/swagger-ui](localhost:31888/swagger-ui) for local kind deployments
 - [localhost:8888/swagger-ui](localhost:8888/swagger-ui) for instances started with `make run` (development machine builds)
-- `<host name>:31888/swagger-ui` for nodeport deployments
+- `<host name>:31888/swagger-ui` for NodePort deployments
 
 ## HTTP definition endpoints
 
-apiserver supports HTTP requests, for detailed specification checkout [full spec doc](HttpRequestSpec.md).
+The APIServer supports HTTP requests. For detailed specifications, check out the [full spec document](HttpRequestSpec.md).
+
+## Advanced Usage
+
+- [Creating Autoscaling clusters using API server](./Autoscaling.md)
+- [Creating a cluster with RayServe support](./CreatingServe.md)
+- [Setting Up a Highly Available Cluster via the API Server](./HACluster.md)
+- [RayJob Submission using API server](./JobSubmission.md)
+- [Monitoring API server and created RayClusters](./Monitoring.md)
+- [Setup API Server with Security](./SecuringImplementation.md)
+- [Different Volumes support for RayCluster by the API server](./Volumes.md)
diff --git a/apiserver/SecuringImplementation.md b/apiserver/SecuringImplementation.md
index 3c644c26748..f67ed1c10d5 100644
--- a/apiserver/SecuringImplementation.md
+++ b/apiserver/SecuringImplementation.md
@@ -1,30 +1,34 @@
 <!-- markdownlint-disable MD013 -->
-# Securing API server
+# Setup APIServer with Security
 
-Currently the KubeRay API server deployed on a publicly accessible cluster is directly exposed to the internet with no authentication/authorization. To protect its endpoint we need to introduce security.
+Currently, the KubeRay APIServer deployed on a publicly accessible cluster is directly exposed to the internet without authentication or authorization. To protect its endpoint, we need to introduce security.
 The solution is based on the architecture below:
 
 ![Overall security implementation](img/authorization.png)
 
-It basically adds Authorization sidecar to the KubeRay API server pod. This architecture is extremely flexible and allows users to plug sidecar implementations that adhere to their security requirements, that can differ significantly across multiple organizations.
+It essentially adds an Authorization sidecar to the KubeRay APIServer pod. This architecture is highly flexible and allows users to integrate sidecar implementations that meet their specific security requirements, which can vary significantly across organizations.
 
-Here we will use a very simple [sidecar implementation](../experimental/cmd/main.go) a reverse proxy using token based authorization. This is a very simple authorization based on the string token, shared between proxy and client. This implementation is not meant for production, but rather is here as a demonstration for the overall implementation. Additional examples of reverse proxy implementation can be found [here](https://github.com/blublinsky/auth-reverse-proxy). There is also a wealth of open source implementations, for example, [oauth2-proxy](https://github.com/oauth2-proxy/oauth2-proxy) and many commercial offerings.
+Here, we will use a very simple [sidecar implementation](../experimental/cmd/main.go) with
+a reverse proxy using token-based authorization. This is a basic authorization mechanism
+based on a string token shared between the proxy and the client. This implementation is not intended for
+production but serves as a demonstration. Additional examples of reverse proxy
+implementations can be found [here](https://github.com/blublinsky/auth-reverse-proxy).
+There is also a wealth of open-source implementations, for example,
+[oauth2-proxy](https://github.com/oauth2-proxy/oauth2-proxy) and many commercial
+offerings.
 
 ## Basic token-based authentication reverse proxy
 
-A simple token-based authentication reverse proxy [implementation](../experimental/cmd/main.go) is provided in the project along with make targets to build a docker image for it and push this image to the image repository. There is also a pre-build image `kuberay/security-proxy:nightly` that you can use for experimenting with security.
+A simple token-based authentication reverse proxy [implementation](../experimental/cmd/main.go) is provided in the project, along with make targets to build a Docker image for it and push this image to the image repository. There is also a pre-built image `kuberay/security-proxy:nightly` that you can use for experimenting with security.
 
 ## Installation
 
-Setting up kind cluster with all required things can be done using the following command:
+### Set `security.proxy.tag`
 
-```shell
-make operator-image docker-image security-proxy-image cluster load-operator-image load-image load-security-proxy-image deploy-operator deploy
-```
-
-Note. To use this command, please modify [values.yaml](../helm-chart/kuberay-apiserver/values.yaml) to set the parameter `security.proxy.tag` to `latest`.
+Before installation, please set `security.proxy.tag` to `latest` in
+[values.yaml](../helm-chart/kuberay-apiserver/values.yaml) file.
 
-The API server helm chart is updated to support both insecure and secure installations. To achieve this [valumes.yaml file](../helm-chart/kuberay-apiserver/values.yaml) is extended to include security configuration:
+Note that in this `values.yaml` file, there is a security configuration:
 
 ```yaml
 security:
@@ -40,40 +44,49 @@ security:
     ENABLE_GRPC: "true"
 ```
 
-removing security configuration there will run API server without security
+Removing this section will run the APIServer without security.
+
+### Deploy KubeRay operator and APIServer with security
 
-with security configuration in place you can install API server using the following helm command (assuming that you are in the `helm-chart` directory of the project):
+Setting up a kind cluster with all required components can be done using the following command:
+
+```sh
+make cluster operator-image docker-image security-proxy-image cluster load-operator-image load-image load-security-proxy-image deploy-operator deploy
+```
 
-```shell
+Alternatively, to install only the APIServer with the security configuration, you can use the
+following Helm command:
+
+```sh
+# Navigate to helm-chart/ directory if haven't
+cd helm-chart
 helm install apiserver kuberay-apiserver
 ```
 
-## Testing
+## Example
 
-Once the API server is installed, the command below
+Once the APIServer is installed, execute the following command:
 
-```shell
+```sh
 curl --silent -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "default-template",
-  "namespace": "default",
-  "cpu": 2,
-  "memory": 4
-}'
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
 ```
 
-fails with result `Unauthorised`
-To make it work we need to add an authorization header to the request:
+This fails with the result `Unauthorized`. To make it work, we need to add an authorization
+header to the request:
 
-```shell
+```sh
 curl --silent -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
---header 'Content-Type: application/json' \
---header 'Authorization: 12345' \
---data '{
-  "name": "default-template",
-  "namespace": "default",
-  "cpu": 2,
-  "memory": 4
-}'
+    --header 'Content-Type: application/json' \
+    --header 'Authorization: 12345' \
+    --data @docs/api-example/compute_template.json
+```
+
+### Clean up
+
+```sh
+make clean-cluster
+# Remove APIServer from Helm
+helm uninstall kuberay-apiserver
 ```
diff --git a/apiserver/docs/api-example/autoscaling_clusters.json b/apiserver/docs/api-example/autoscaling_clusters.json
new file mode 100644
index 00000000000..1fc06bc3265
--- /dev/null
+++ b/apiserver/docs/api-example/autoscaling_clusters.json
@@ -0,0 +1,61 @@
+{
+  "name": "test-cluster",
+  "namespace": "default",
+  "user": "boris",
+  "clusterSpec": {
+    "enableInTreeAutoscaling": true,
+    "autoscalerOptions": {
+      "upscalingMode": "Default",
+      "idleTimeoutSeconds": 30,
+      "cpu": "500m",
+      "memory": "512Mi"
+    },
+    "headGroupSpec": {
+      "computeTemplate": "default-template",
+      "image": "rayproject/ray:2.9.0-py310",
+      "serviceType": "NodePort",
+      "rayStartParams": {
+        "dashboard-host": "0.0.0.0",
+        "metrics-export-port": "8080",
+        "num-cpus": "0"
+      },
+      "volumes": [
+        {
+          "name": "code-sample",
+          "mountPath": "/home/ray/samples",
+          "volumeType": "CONFIGMAP",
+          "source": "ray-example",
+          "items": {
+            "detached_actor.py": "detached_actor.py",
+            "terminate_detached_actor.py": "terminate_detached_actor.py"
+          }
+        }
+      ]
+    },
+    "workerGroupSpec": [
+      {
+        "groupName": "small-wg",
+        "computeTemplate": "default-template",
+        "image": "rayproject/ray:2.9.0-py310",
+        "replicas": 0,
+        "minReplicas": 0,
+        "maxReplicas": 5,
+        "rayStartParams": {
+          "node-ip-address": "$MY_POD_IP"
+        },
+        "volumes": [
+          {
+            "name": "code-sample",
+            "mountPath": "/home/ray/samples",
+            "volumeType": "CONFIGMAP",
+            "source": "ray-example",
+            "items": {
+              "detached_actor.py": "detached_actor.py",
+              "terminate_detached_actor.py": "terminate_detached_actor.py"
+            }
+          }
+        ]
+      }
+    ]
+  }
+}
diff --git a/apiserver/docs/api-example/compute_template.json b/apiserver/docs/api-example/compute_template.json
new file mode 100644
index 00000000000..2b5406e2c04
--- /dev/null
+++ b/apiserver/docs/api-example/compute_template.json
@@ -0,0 +1,6 @@
+{
+  "name": "default-template",
+  "namespace": "default",
+  "cpu": 2,
+  "memory": 4
+}
diff --git a/apiserver/docs/api-example/create_serving_clusters.json b/apiserver/docs/api-example/create_serving_clusters.json
new file mode 100644
index 00000000000..96fc21b6d54
--- /dev/null
+++ b/apiserver/docs/api-example/create_serving_clusters.json
@@ -0,0 +1,55 @@
+{
+  "name": "test-cluster",
+  "namespace": "default",
+  "user": "boris",
+  "annotations": {
+    "ray.io/enable-serve-service": "true"
+  },
+  "clusterSpec": {
+    "headGroupSpec": {
+      "computeTemplate": "default-template",
+      "image": "rayproject/ray:2.9.0-py310",
+      "serviceType": "ClusterIP",
+      "rayStartParams": {
+        "dashboard-host": "0.0.0.0",
+        "metrics-export-port": "8080",
+        "dashboard-agent-listen-port": "52365"
+      },
+      "volumes": [
+        {
+          "name": "code-sample",
+          "mountPath": "/home/ray/samples",
+          "volumeType": "CONFIGMAP",
+          "source": "ray-job-code-sample",
+          "items": {
+            "sample_code.py": "sample_code.py"
+          }
+        }
+      ]
+    },
+    "workerGroupSpec": [
+      {
+        "groupName": "small-wg",
+        "computeTemplate": "default-template",
+        "image": "rayproject/ray:2.9.0-py310",
+        "replicas": 1,
+        "minReplicas": 0,
+        "maxReplicas": 5,
+        "rayStartParams": {
+          "node-ip-address": "$MY_POD_IP"
+        },
+        "volumes": [
+          {
+            "name": "code-sample",
+            "mountPath": "/home/ray/samples",
+            "volumeType": "CONFIGMAP",
+            "source": "ray-job-code-sample",
+            "items": {
+              "sample_code.py": "sample_code.py"
+            }
+          }
+        ]
+      }
+    ]
+  }
+}
diff --git a/apiserver/docs/api-example/ha_clusters.json b/apiserver/docs/api-example/ha_clusters.json
new file mode 100644
index 00000000000..fccb6373c4d
--- /dev/null
+++ b/apiserver/docs/api-example/ha_clusters.json
@@ -0,0 +1,78 @@
+{
+  "name": "ha-cluster",
+  "namespace": "default",
+  "user": "boris",
+  "version": "2.9.0",
+  "environment": "DEV",
+  "annotations": {
+    "ray.io/ft-enabled": "true"
+  },
+  "clusterSpec": {
+    "headGroupSpec": {
+      "computeTemplate": "default-template",
+      "image": "rayproject/ray:2.9.0-py310",
+      "serviceType": "NodePort",
+      "rayStartParams": {
+        "dashboard-host": "0.0.0.0",
+        "metrics-export-port": "8080",
+        "num-cpus": "0",
+        "redis-password": "$REDIS_PASSWORD"
+      },
+      "environment": {
+        "values": {
+          "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
+        },
+        "valuesFrom": {
+          "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+          }
+        }
+      },
+      "volumes": [
+        {
+          "name": "code-sample",
+          "mountPath": "/home/ray/samples",
+          "volumeType": "CONFIGMAP",
+          "source": "ray-example",
+          "items": {
+            "detached_actor.py": "detached_actor.py",
+            "increment_counter.py": "increment_counter.py"
+          }
+        }
+      ]
+    },
+    "workerGroupSpec": [
+      {
+        "groupName": "small-wg",
+        "computeTemplate": "default-template",
+        "image": "rayproject/ray:2.9.0-py310",
+        "replicas": 1,
+        "minReplicas": 0,
+        "maxReplicas": 5,
+        "rayStartParams": {
+          "node-ip-address": "$MY_POD_IP",
+          "metrics-export-port": "8080"
+        },
+        "environment": {
+          "values": {
+            "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
+          }
+        },
+        "volumes": [
+          {
+            "name": "code-sample",
+            "mountPath": "/home/ray/samples",
+            "volumeType": "CONFIGMAP",
+            "source": "ray-example",
+            "items": {
+              "detached_actor.py": "detached_actor.py",
+              "increment_counter.py": "increment_counter.py"
+            }
+          }
+        ]
+      }
+    ]
+  }
+}
diff --git a/apiserver/docs/api-example/jobsubmission_clusters.json b/apiserver/docs/api-example/jobsubmission_clusters.json
new file mode 100644
index 00000000000..ebc221ad986
--- /dev/null
+++ b/apiserver/docs/api-example/jobsubmission_clusters.json
@@ -0,0 +1,51 @@
+{
+  "name": "test-cluster",
+  "namespace": "default",
+  "user": "boris",
+  "clusterSpec": {
+    "headGroupSpec": {
+      "computeTemplate": "default-template",
+      "image": "rayproject/ray:2.9.0-py310",
+      "serviceType": "NodePort",
+      "rayStartParams": {
+        "dashboard-host": "0.0.0.0",
+        "metrics-export-port": "8080"
+      },
+      "volumes": [
+        {
+          "name": "code-sample",
+          "mountPath": "/home/ray/samples",
+          "volumeType": "CONFIGMAP",
+          "source": "ray-job-code-sample",
+          "items": {
+            "sample_code.py": "sample_code.py"
+          }
+        }
+      ]
+    },
+    "workerGroupSpec": [
+      {
+        "groupName": "small-wg",
+        "computeTemplate": "default-template",
+        "image": "rayproject/ray:2.9.0-py310",
+        "replicas": 1,
+        "minReplicas": 1,
+        "maxReplicas": 1,
+        "rayStartParams": {
+          "node-ip-address": "$MY_POD_IP"
+        },
+        "volumes": [
+          {
+            "name": "code-sample",
+            "mountPath": "/home/ray/samples",
+            "volumeType": "CONFIGMAP",
+            "source": "ray-job-code-sample",
+            "items": {
+              "sample_code.py": "sample_code.py"
+            }
+          }
+        ]
+      }
+    ]
+  }
+}
diff --git a/apiserver/img/hacluster-dashboard-actor.png b/apiserver/img/hacluster-dashboard-actor.png
new file mode 100644
index 00000000000..4d1410b1fd1
Binary files /dev/null and b/apiserver/img/hacluster-dashboard-actor.png differ
diff --git a/apiserver/img/hacluster-dashboard-cluster.png b/apiserver/img/hacluster-dashboard-cluster.png
new file mode 100644
index 00000000000..375c0873254
Binary files /dev/null and b/apiserver/img/hacluster-dashboard-cluster.png differ


Reviews:
user: login: dentiny
body: Thanks for the effort, looks really great!
Just finish first iteration, will take a deep look later tonight.
Left some comments and questions for discussion :)
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-10T23:03:12Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-11T11:14:11Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T12:50:15Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T12:54:08Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T13:10:21Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T13:14:37Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T13:39:22Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T13:40:15Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T13:42:24Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T13:48:50Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T04:42:06Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T05:27:05Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T14:53:16Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T14:56:54Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:21:48Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:32:33Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:37:01Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:37:23Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:39:17Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:40:50Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:43:25Z
user: login: dentiny
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:44:53Z
user: login: dentiny
body: Thanks!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T23:49:06Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T14:04:51Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T14:13:33Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T14:15:10Z
user: login: rueian
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T19:17:01Z
user: login: machichima
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T01:36:50Z
user: login: rueian
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-23T22:10:36Z
user: login: kevin85421
body: I decide to merge this PR first, and I will open a follow up PR if needed.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-28T21:05:14Z

Review comments:
diff_hunk: @@ -1,28 +1,44 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
path: apiserver/Autoscaling.md
user: login: dentiny
body: For my own knowledge, could you please tell me what's the difference between "shell" vs "sh"?
created_at: 2025-05-10T22:45:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,28 +1,44 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the config map is successfully created, you should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, you need to first deploy a RayCluster with following command.
path: apiserver/Autoscaling.md
user: login: dentiny
body: Some of the curl request payloads are pretty big, do you think it good if we move then into a local json file?
- It makes readme doc smaller and easier to read (otherwise a large chunk of content is payload)
- Easier for users to make changes to payload and play with

For example, create a `doc` sub-directory, and put all the readme and payload json inside?
created_at: 2025-05-10T22:52:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,25 +114,45 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 }'
 ```
 
-## Validate that Ray cluster is deployed correctly
+This command performs two main operations:
 
-Run:
+1. Creates a compute template `default-template` that specifies resources to use when
+   scale-up (2 CPUs and 4 GiB memory).
 
-```shell
-kubectl get pods
-```
+2. Deploys a RayCluster (test-cluster) with:
+    - A head pod that manages the cluster
+    - A worker group configured to scale between 0 and 5 replicas
path: apiserver/Autoscaling.md
user: login: dentiny
body: it's unfortunately json doesn't support inline comments :(
created_at: 2025-05-10T22:53:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,25 +114,45 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 }'
 ```
 
-## Validate that Ray cluster is deployed correctly
+This command performs two main operations:
 
-Run:
+1. Creates a compute template `default-template` that specifies resources to use when
+   scale-up (2 CPUs and 4 GiB memory).
 
-```shell
-kubectl get pods
-```
+2. Deploys a RayCluster (test-cluster) with:
+    - A head pod that manages the cluster
+    - A worker group configured to scale between 0 and 5 replicas
 
-You should get something like this:
+The worker group uses the following autoscalerOptions to control scaling behavior:
 
-```shell
-test-cluster-head-pr25j             2/2     Running   0          2m49s
+- **`upscalingMode: "Default"`**: Default scaling behavior. Ray will scale up only as
+needed.
+- **`idleTimeoutSeconds: 30`** If a worker pod remains idle (i.e., not running any tasks)
+for 30 seconds, it will be automatically removed.
path: apiserver/Autoscaling.md
user: login: dentiny
body: curious does the cluster scales down to 0, when there's no task running?
IIRC, k8s k8s default HPA doesn't support (require extra HPA with keda or alike)
created_at: 2025-05-10T22:56:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,16 +203,28 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobs' \
 }'
 ```
 
-A worker Pod will be deleted after `idleTimeoutSeconds` (default 60s, we specified 30) seconds. Run:
+While actor is deleted, we do not need the worker anymore. The worker pod will be deleted
+after `idleTimeoutSeconds` (default 60, we specified 30) seconds.
 
-```shell
+List all pods to verify if the worker pod is deleted:
+
+```sh
 kubectl get pods
+
+# NAME                                READY   STATUS      RESTARTS   AGE
path: apiserver/Autoscaling.md
user: login: dentiny
body: curious why do we comment k8s access result?
created_at: 2025-05-10T23:01:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -69,36 +116,26 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
   }'
 ```
 
-Note, that before creating a cluster you need to install this [configmap] and
-create default template using the following command:
-
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "default-template",
-    "namespace": "default",
-    "cpu": 2,
-    "memory": 4
-  }'
-```
-
 To confirm that the cluster is created correctly, check created services using that following command:
 
-```shell
+```sh
 kubectl get service
+# NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                   AGE
+# test-cluster-head-svc    ClusterIP   None            <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP   7s
+# test-cluster-serve-svc   ClusterIP   10.96.161.26    <none>        8000/TCP                                                  7s
 ```
 
-that should return the following:
+You should see two services created, one for head node to access dashboard and manage the
+cluster, the other for submit the serve request.
 
-```shell
-test-cluster-head-svc    ClusterIP   10.96.19.185    <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP
-test-cluster-serve-svc   ClusterIP   10.96.144.162   <none>        8000/TCP
-```
+Note that the 52365 port for head node service is for serve configuration.
path: apiserver/CreatingServe.md
user: login: dentiny
body: Do you mind explain a bit?
created_at: 2025-05-11T11:04:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,57 +1,94 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
path: apiserver/HACluster.md
user: login: dentiny
body: Just want to make sure ray's HA mode relies on redis, but not necessarily HA redis, right?
created_at: 2025-05-11T11:05:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -134,84 +171,132 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 }'
 ```
 
-Note that computeTemplate here has to be created using this [command](test/cluster//template/simple)
-
-Lets discuss the important pieces here:
-You need to specify annotation, that tells Ray that this is cluster with GCS fault tolerance
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
 
-```sh
-ray.io/ft-enabled: "true"
+```json
+"annotations" : {
+    "ray.io/ft-enabled": "true"
+}
 ```
 
-For the `headGroupSpec` you need the following. In the `rayStartParams` you need to add information about Redis
-password.
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
 
-```sh
+```json
 "redis-password:: "$REDIS_PASSWORD"
 "num-cpu": "0"
 ```
 
-Where the value of `REDIS_PASSWORD` comes from environment variable (below). Additionally `num-cpus:
-0` ensures that no application code runs on a head node.
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
+    },
+    "valuesFrom": {
+        "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+        }
+    }
+},
+```
+
+For the `workerGroupSpecs`, we set the `gcs_rpc_server_reconnect_timeout` environment
+variable, which controls the GCS heartbeat timeout (default 60 seconds). This controls how
+long after the head node dies do we kill the worker node. While it takes time to restart
+the head node, we want this values to be large enough to prevent the worker node being
+killed during the restarting period.
+
+```json
+"environment": {
+    "values": {
+        "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
+    }
+},
+```
+
+### Validate that RayCluster is deployed correctly
 
-The following environment variable have to be added here:
+Run following command to get list of pods running. You should see one head and worker node
+like below:
 
 ```sh
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     1/1     Running   0          2m36s
+# ha-cluster-small-wg-worker-22lbx    1/1     Running   0          2m36s
 ```
 
-For the `workerGroupSpecs` you might want to increase `gcs_rpc_server_reconnect_timeout` by specifying the following
-environment variable:
+### Create an Actor
+
+Before we try to trigger the restoration, we need to find a way to validate our GCS restore
+is working correctly. We will validate this by creating a detached actor. If it still
+exists and functions after the head node deletion and restoration, we can confirm that the
+GCS data is restored correctly.
+
+Run following command for creating a detached actor. Please change `ha-cluster-head` to
+your head node's name:
 
 ```sh
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
+kubectl exec -it ha-cluster-head -- python3 /home/ray/samples/detached_actor.py
path: apiserver/HACluster.md
user: login: dentiny
body: Local script?
created_at: 2025-05-11T11:09:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -186,8 +204,7 @@ This should return the list of the submissions, that looks as follows:
 ```json
 {
    "submissions":[
-      {
-         "entrypoint":"python /home/ray/samples/sample_code.py",
+      { "entrypoint":"python /home/ray/samples/sample_code.py",
path: apiserver/JobSubmission.md
user: login: dentiny
body: why this change?
created_at: 2025-05-11T11:12:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,68 +1,106 @@
-# Monitoring of the API server and created Ray clusters
+# Monitoring of the API server and created RayClusters
 
-In order to ensure a proper functioning of the API server and created Ray clusters, it is typically necessary to
-monitor the API server itself and created Ray clusters. This document describes how to instrument both API server
-and created clusters with Prometheus and Grafana
-
-## Monitoring of the API server
+In order to ensure a proper functioning of the API server and created RayClusters, it is
+typically necessary to monitor them. This document describes how to monitor both API
+server and created clusters with Prometheus and Grafana
path: apiserver/Monitoring.md
user: login: dentiny
body: nit: add a "." at the end of the sentence
created_at: 2025-05-11T11:13:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,28 +1,44 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
path: apiserver/Autoscaling.md
user: login: machichima
body: They represent the same thing, but `shell` doesn’t always trigger syntax highlighting (e.g. in neovim). `sh` is used more often and usually gets recognized correctly.


![image](https://github.com/user-attachments/assets/ef0b69be-6e35-419a-b0b8-04b3d7617d51)

created_at: 2025-05-11T12:50:15Z
author_association: COLLABORATOR
diff_hunk: @@ -1,28 +1,44 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the config map is successfully created, you should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, you need to first deploy a RayCluster with following command.
path: apiserver/Autoscaling.md
user: login: machichima
body: Sounds good! I also see some of the curl commands use the same json request, let me collect them into json files
created_at: 2025-05-11T12:54:08Z
author_association: COLLABORATOR
diff_hunk: @@ -98,25 +114,45 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 }'
 ```
 
-## Validate that Ray cluster is deployed correctly
+This command performs two main operations:
 
-Run:
+1. Creates a compute template `default-template` that specifies resources to use when
+   scale-up (2 CPUs and 4 GiB memory).
 
-```shell
-kubectl get pods
-```
+2. Deploys a RayCluster (test-cluster) with:
+    - A head pod that manages the cluster
+    - A worker group configured to scale between 0 and 5 replicas
 
-You should get something like this:
+The worker group uses the following autoscalerOptions to control scaling behavior:
 
-```shell
-test-cluster-head-pr25j             2/2     Running   0          2m49s
+- **`upscalingMode: "Default"`**: Default scaling behavior. Ray will scale up only as
+needed.
+- **`idleTimeoutSeconds: 30`** If a worker pod remains idle (i.e., not running any tasks)
+for 30 seconds, it will be automatically removed.
path: apiserver/Autoscaling.md
user: login: machichima
body: No, only the worker is terminated. The head node still exists.
created_at: 2025-05-11T13:10:21Z
author_association: COLLABORATOR
diff_hunk: @@ -165,16 +203,28 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobs' \
 }'
 ```
 
-A worker Pod will be deleted after `idleTimeoutSeconds` (default 60s, we specified 30) seconds. Run:
+While actor is deleted, we do not need the worker anymore. The worker pod will be deleted
+after `idleTimeoutSeconds` (default 60, we specified 30) seconds.
 
-```shell
+List all pods to verify if the worker pod is deleted:
+
+```sh
 kubectl get pods
+
+# NAME                                READY   STATUS      RESTARTS   AGE
path: apiserver/Autoscaling.md
user: login: machichima
body: I followed how Kuberay docs did (for [example](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/rayservice.html#step-4-verify-the-kubernetes-cluster-status))

They put the command's output as the comment right below itself
created_at: 2025-05-11T13:14:37Z
author_association: COLLABORATOR
diff_hunk: @@ -1,57 +1,94 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
path: apiserver/HACluster.md
user: login: machichima
body: Yes!
created_at: 2025-05-11T13:39:22Z
author_association: COLLABORATOR
diff_hunk: @@ -69,36 +116,26 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
   }'
 ```
 
-Note, that before creating a cluster you need to install this [configmap] and
-create default template using the following command:
-
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "default-template",
-    "namespace": "default",
-    "cpu": 2,
-    "memory": 4
-  }'
-```
-
 To confirm that the cluster is created correctly, check created services using that following command:
 
-```shell
+```sh
 kubectl get service
+# NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                   AGE
+# test-cluster-head-svc    ClusterIP   None            <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP   7s
+# test-cluster-serve-svc   ClusterIP   10.96.161.26    <none>        8000/TCP                                                  7s
 ```
 
-that should return the following:
+You should see two services created, one for head node to access dashboard and manage the
+cluster, the other for submit the serve request.
 
-```shell
-test-cluster-head-svc    ClusterIP   10.96.19.185    <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP
-test-cluster-serve-svc   ClusterIP   10.96.144.162   <none>        8000/TCP
-```
+Note that the 52365 port for head node service is for serve configuration.
path: apiserver/CreatingServe.md
user: login: machichima
body: Sorry I think this is not clear. I updated to this:

Note that we set the 52365 port for dashboard agent in the above curl command, which is used internally by Ray Serve.

created_at: 2025-05-11T13:40:15Z
author_association: COLLABORATOR
diff_hunk: @@ -134,84 +171,132 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 }'
 ```
 
-Note that computeTemplate here has to be created using this [command](test/cluster//template/simple)
-
-Lets discuss the important pieces here:
-You need to specify annotation, that tells Ray that this is cluster with GCS fault tolerance
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
 
-```sh
-ray.io/ft-enabled: "true"
+```json
+"annotations" : {
+    "ray.io/ft-enabled": "true"
+}
 ```
 
-For the `headGroupSpec` you need the following. In the `rayStartParams` you need to add information about Redis
-password.
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
 
-```sh
+```json
 "redis-password:: "$REDIS_PASSWORD"
 "num-cpu": "0"
 ```
 
-Where the value of `REDIS_PASSWORD` comes from environment variable (below). Additionally `num-cpus:
-0` ensures that no application code runs on a head node.
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
+    },
+    "valuesFrom": {
+        "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+        }
+    }
+},
+```
+
+For the `workerGroupSpecs`, we set the `gcs_rpc_server_reconnect_timeout` environment
+variable, which controls the GCS heartbeat timeout (default 60 seconds). This controls how
+long after the head node dies do we kill the worker node. While it takes time to restart
+the head node, we want this values to be large enough to prevent the worker node being
+killed during the restarting period.
+
+```json
+"environment": {
+    "values": {
+        "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
+    }
+},
+```
+
+### Validate that RayCluster is deployed correctly
 
-The following environment variable have to be added here:
+Run following command to get list of pods running. You should see one head and worker node
+like below:
 
 ```sh
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     1/1     Running   0          2m36s
+# ha-cluster-small-wg-worker-22lbx    1/1     Running   0          2m36s
 ```
 
-For the `workerGroupSpecs` you might want to increase `gcs_rpc_server_reconnect_timeout` by specifying the following
-environment variable:
+### Create an Actor
+
+Before we try to trigger the restoration, we need to find a way to validate our GCS restore
+is working correctly. We will validate this by creating a detached actor. If it still
+exists and functions after the head node deletion and restoration, we can confirm that the
+GCS data is restored correctly.
+
+Run following command for creating a detached actor. Please change `ha-cluster-head` to
+your head node's name:
 
 ```sh
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
+kubectl exec -it ha-cluster-head -- python3 /home/ray/samples/detached_actor.py
path: apiserver/HACluster.md
user: login: machichima
body: This script `detached_actor.py` is defined in config map and mounted to head node, not in local. Let me add some description here
created_at: 2025-05-11T13:42:24Z
author_association: COLLABORATOR
diff_hunk: @@ -186,8 +204,7 @@ This should return the list of the submissions, that looks as follows:
 ```json
 {
    "submissions":[
-      {
-         "entrypoint":"python /home/ray/samples/sample_code.py",
+      { "entrypoint":"python /home/ray/samples/sample_code.py",
path: apiserver/JobSubmission.md
user: login: machichima
body: Sorry I copied the result from my terminal, and it looks like it does not format well. Just fixed
created_at: 2025-05-11T13:48:50Z
author_association: COLLABORATOR
diff_hunk: @@ -1,28 +1,44 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
path: apiserver/Autoscaling.md
user: login: dentiny
body: Thanks!
created_at: 2025-05-12T04:42:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,7 +17,7 @@ You could build and start apiserver from scratch on your local environment in on
 make start-local-apiserver
 ```
 
-apiserver supports HTTP request, so you could easily check whether it's started successfully by issuing two simple curl requests.
+Apiserver supports HTTP request, so you could easily check whether it's started successfully by issuing two simple curl requests.
path: apiserver/README.md
user: login: dentiny
body: L4, I don't think we're going to provide grpc API in V2
created_at: 2025-05-12T05:27:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,7 +17,7 @@ You could build and start apiserver from scratch on your local environment in on
 make start-local-apiserver
 ```
 
-apiserver supports HTTP request, so you could easily check whether it's started successfully by issuing two simple curl requests.
+Apiserver supports HTTP request, so you could easily check whether it's started successfully by issuing two simple curl requests.
path: apiserver/README.md
user: login: machichima
body: I think V2 is in `apiserversdk/` and the one here is V1? 
Should we also take gRPC out from here?
created_at: 2025-05-15T14:53:16Z
author_association: COLLABORATOR
diff_hunk: @@ -1,28 +1,44 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the config map is successfully created, you should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, you need to first deploy a RayCluster with following command.
path: apiserver/Autoscaling.md
user: login: machichima
body: Fixed! I put them in `docs/api-example/` directory
created_at: 2025-05-15T14:56:54Z
author_association: COLLABORATOR
diff_hunk: @@ -1,122 +1,105 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+### IMPORTANT: Change your working directory to `apiserver/`
path: apiserver/Autoscaling.md
user: login: dentiny
body: If you think it's something worth noticing, do you think it better to 
```
> [!IMPORTANT]
> Change your working directory to `apiserver/`; All the following guidance require you to switch your working directory to the KubeRay
`apiserver`
```
created_at: 2025-05-18T23:21:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,104 +1,99 @@
 # Creating a cluster with RayServe support
 
-Up until rescently the only way to create a Ray cluster supporting RayServe was by using `Create ray
-service` APIs. Although it does work, quite often you want to create cluster supporting Ray serve so
-that you can experiment with serve APIs directly. Now it is possible by adding the following
-annotation to the cluster:
+Currently, there are two ways for creating a RayCluster with RayService support:
+
+1. Create a RayCluster first, then use [Create RayService](./HttpRequestSpec.md#Create-ray-service-in-a-given-namespace)
+   APIs to create a service.
+2. Directly create cluster with RayServe support (described in this document)
+
+For creating a cluster with RayService support, simply add the following annotation when
+creating the cluster:
 
 ```json
 "annotations" : {
     "ray.io/enable-serve-service": "true"
   },
 ```
 
-the complete curl command to creation such cluster is as follows:
+## Setup
 
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "test-cluster",
-    "namespace": "default",
-    "user": "boris",
-    "annotations" : {
-      "ray.io/enable-serve-service": "true"
-    },
-    "clusterSpec": {
-      "headGroupSpec": {
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "serviceType": "ClusterIP",
-        "rayStartParams": {
-           "dashboard-host": "0.0.0.0",
-           "metrics-export-port": "8080",
-           "dashboard-agent-listen-port": "52365"
-         },
-         "volumes": [
-           {
-             "name": "code-sample",
-             "mountPath": "/home/ray/samples",
-             "volumeType": "CONFIGMAP",
-             "source": "ray-job-code-sample",
-             "items": {"sample_code.py" : "sample_code.py"}
-           }
-         ]
-      },
-      "workerGroupSpec": [
-        {
-          "groupName": "small-wg",
-          "computeTemplate": "default-template",
-          "image": "rayproject/ray:2.9.0-py310",
-          "replicas": 1,
-          "minReplicas": 0,
-          "maxReplicas": 5,
-          "rayStartParams": {
-             "node-ip-address": "$MY_POD_IP"
-           },
-          "volumes": [
-            {
-              "name": "code-sample",
-              "mountPath": "/home/ray/samples",
-              "volumeType": "CONFIGMAP",
-              "source": "ray-job-code-sample",
-              "items": {"sample_code.py" : "sample_code.py"}
-            }
-          ]
-        }
-      ]
-    }
-  }'
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+This example walks through how to create a cluster with RayService support.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### IMPORTANT: Change your working directory to `apiserver/`
+
+All the following guidance require you to switch your working directory to the KubeRay
+`apiserver`:
+
+```sh
+cd apiserver/
 ```
 
-Note, that before creating a cluster you need to install this [configmap] and
-create default template using the following command:
+### Install ConfigMap
 
-```shell
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/job/code.yaml
+```
+
+Check if the config map is successfully created, you should see `ray-job-code-sample` in
+the list:
+
+```sh
+kubectl get configmaps
+# NAME                  DATA   AGE
+# ray-job-code-sample   1      13s
+```
+
+### Create RayCluster
+
+Use following command to create a compute template and a RayCluster with RayService support:
+
+```sh
+cur
path: apiserver/CreatingServe.md
user: login: dentiny
body: what is `cur`?
created_at: 2025-05-18T23:32:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,57 +1,94 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
path: apiserver/HACluster.md
user: login: dentiny
body: I prefer using the word "redis" instead of "HA redis" here, these are two different products from cloud vendor offering.
created_at: 2025-05-18T23:37:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,57 +1,94 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
path: apiserver/HACluster.md
user: login: dentiny
body: redis itself doesn't provide persistence as well.
created_at: 2025-05-18T23:37:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
path: apiserver/HACluster.md
user: login: dentiny
body: ```suggestion
single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
```
created_at: 2025-05-18T23:39:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
+"redis-password:: "$REDIS_PASSWORD"
+"num-cpu": "0"
+```
+
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
path: apiserver/HACluster.md
user: login: dentiny
body: just curious (I never used HA ray before), why `redis.redis.` instead of `redis.`?
created_at: 2025-05-18T23:40:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -134,7 +101,7 @@ This should return JSON similar to the one below
 {
    "entrypoint":"python /home/ray/samples/sample_code.py",
    "jobId":"02000000",
-   "submissionId":"raysubmit_KWZLwme56esG3Wcr",
+   "submissionId":"<submissionID>",
path: apiserver/JobSubmission.md
user: login: dentiny
body: I think leaving a submission id here is not bad because users could easily see what sub-id looks like; it's not sensitive information related to security as well.
No strong opinion though
created_at: 2025-05-18T23:43:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,68 +1,106 @@
-# Monitoring of the API server and created Ray clusters
+# Monitoring of the API server and created RayClusters
 
-In order to ensure a proper functioning of the API server and created Ray clusters, it is typically necessary to
-monitor the API server itself and created Ray clusters. This document describes how to instrument both API server
-and created clusters with Prometheus and Grafana
-
-## Monitoring of the API server
+In order to ensure a proper functioning of the API server and created RayClusters, it is
+typically necessary to monitor them. This document describes how to monitor both API
+server and created clusters with Prometheus and Grafana.
 
 Current implementation of the API server provides a flag `collectMetricsFlag` that defines whether to collect and
 expose Prometheus metrics, which is by default set to `true`. To disable metrics collection, this flag needs to
-be set to `false` and the API server image needs to be rebuild.
+be set to `false` and the API server image needs to be rebuilt.
 
-If this flag is enabled, an `http` port at `/metrics` endpoint provides API server metrics in Prometheus format.
+If this flag is enabled, a `http` port at `/metrics` endpoint provides API server metrics in Prometheus format.
 
-### Testing monitoring of the API server
+## Monitoring of the API server
+
+### Deploy KubeRay operator and API server
+
+Refer to [README](README.md) for setting up KubeRay operator and API server. This will
+set the flag `collectMetricsFlag` to `true` which enable the metrics collection.
+
+### IMPORTANT: Change your working directory to project root
path: apiserver/Monitoring.md
user: login: dentiny
body: same comment as https://github.com/ray-project/kuberay/pull/3564/files#r2094653342
created_at: 2025-05-18T23:44:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,104 +1,99 @@
 # Creating a cluster with RayServe support
 
-Up until rescently the only way to create a Ray cluster supporting RayServe was by using `Create ray
-service` APIs. Although it does work, quite often you want to create cluster supporting Ray serve so
-that you can experiment with serve APIs directly. Now it is possible by adding the following
-annotation to the cluster:
+Currently, there are two ways for creating a RayCluster with RayService support:
+
+1. Create a RayCluster first, then use [Create RayService](./HttpRequestSpec.md#Create-ray-service-in-a-given-namespace)
+   APIs to create a service.
+2. Directly create cluster with RayServe support (described in this document)
+
+For creating a cluster with RayService support, simply add the following annotation when
+creating the cluster:
 
 ```json
 "annotations" : {
     "ray.io/enable-serve-service": "true"
   },
 ```
 
-the complete curl command to creation such cluster is as follows:
+## Setup
 
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "test-cluster",
-    "namespace": "default",
-    "user": "boris",
-    "annotations" : {
-      "ray.io/enable-serve-service": "true"
-    },
-    "clusterSpec": {
-      "headGroupSpec": {
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "serviceType": "ClusterIP",
-        "rayStartParams": {
-           "dashboard-host": "0.0.0.0",
-           "metrics-export-port": "8080",
-           "dashboard-agent-listen-port": "52365"
-         },
-         "volumes": [
-           {
-             "name": "code-sample",
-             "mountPath": "/home/ray/samples",
-             "volumeType": "CONFIGMAP",
-             "source": "ray-job-code-sample",
-             "items": {"sample_code.py" : "sample_code.py"}
-           }
-         ]
-      },
-      "workerGroupSpec": [
-        {
-          "groupName": "small-wg",
-          "computeTemplate": "default-template",
-          "image": "rayproject/ray:2.9.0-py310",
-          "replicas": 1,
-          "minReplicas": 0,
-          "maxReplicas": 5,
-          "rayStartParams": {
-             "node-ip-address": "$MY_POD_IP"
-           },
-          "volumes": [
-            {
-              "name": "code-sample",
-              "mountPath": "/home/ray/samples",
-              "volumeType": "CONFIGMAP",
-              "source": "ray-job-code-sample",
-              "items": {"sample_code.py" : "sample_code.py"}
-            }
-          ]
-        }
-      ]
-    }
-  }'
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+This example walks through how to create a cluster with RayService support.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### IMPORTANT: Change your working directory to `apiserver/`
+
+All the following guidance require you to switch your working directory to the KubeRay
+`apiserver`:
+
+```sh
+cd apiserver/
 ```
 
-Note, that before creating a cluster you need to install this [configmap] and
-create default template using the following command:
+### Install ConfigMap
 
-```shell
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/job/code.yaml
+```
+
+Check if the config map is successfully created, you should see `ray-job-code-sample` in
+the list:
+
+```sh
+kubectl get configmaps
+# NAME                  DATA   AGE
+# ray-job-code-sample   1      13s
+```
+
+### Create RayCluster
+
+Use following command to create a compute template and a RayCluster with RayService support:
+
+```sh
+cur
path: apiserver/CreatingServe.md
user: login: machichima
body: Sorry this is added in accident
created_at: 2025-05-19T14:04:51Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
+"redis-password:: "$REDIS_PASSWORD"
+"num-cpu": "0"
+```
+
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
path: apiserver/HACluster.md
user: login: machichima
body: I think the first `redis` is the service name, while the second is the namespace (which also called `redis`)
created_at: 2025-05-19T14:13:33Z
author_association: COLLABORATOR
diff_hunk: @@ -134,7 +101,7 @@ This should return JSON similar to the one below
 {
    "entrypoint":"python /home/ray/samples/sample_code.py",
    "jobId":"02000000",
-   "submissionId":"raysubmit_KWZLwme56esG3Wcr",
+   "submissionId":"<submissionID>",
path: apiserver/JobSubmission.md
user: login: machichima
body: No problem!
created_at: 2025-05-19T14:15:10Z
author_association: COLLABORATOR
diff_hunk: @@ -1,122 +1,99 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+> [!IMPORTANT]
+> All the following guidance require you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/cluster/cluster/detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the config map is successfully created, you should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, you need to first deploy a RayCluster with following command.
 
-```shell
+```sh
+# Create compute tempalte
path: apiserver/Autoscaling.md
user: login: rueian
body: ```suggestion
# Create compute template
```
created_at: 2025-05-19T19:01:37Z
author_association: COLLABORATOR
diff_hunk: @@ -1,122 +1,99 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+> [!IMPORTANT]
+> All the following guidance require you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/cluster/cluster/detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the config map is successfully created, you should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, you need to first deploy a RayCluster with following command.
path: apiserver/Autoscaling.md
user: login: rueian
body: ```suggestion
Before running the example, you need to first deploy a RayCluster with the following command.
```
created_at: 2025-05-19T19:01:47Z
author_association: COLLABORATOR
diff_hunk: @@ -1,122 +1,99 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+> [!IMPORTANT]
+> All the following guidance require you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/cluster/cluster/detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the config map is successfully created, you should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, you need to first deploy a RayCluster with following command.
 
-```shell
+```sh
+# Create compute tempalte
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
 --header 'Content-Type: application/json' \
---data '{
-  "name": "default-template",
-  "namespace": "default",
-  "cpu": 2,
-  "memory": 4
-}'
+--data  @docs/api-example/compute_template.json
+
+# Create RayCluster
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 --header 'Content-Type: application/json' \
---data '{
-  "name": "test-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "clusterSpec": {
-    "enableInTreeAutoscaling": true,
-    "autoscalerOptions": {
-        "upscalingMode": "Default",
-        "idleTimeoutSeconds": 30,
-        "cpu": "500m",
-        "memory": "512Mi"
-    },
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0"
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-            "detached_actor.py": "detached_actor.py",
-            "terminate_detached_actor.py": "terminate_detached_actor.py"
-           }
-         }
-       ]
-    },
-    "workerGroupSpec": [
-      {
-        "groupName": "small-wg",
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "replicas": 0,
-        "minReplicas": 0,
-        "maxReplicas": 5,
-        "rayStartParams": {
-           "node-ip-address": "$MY_POD_IP"
-         },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-example",
-            "items": {
-                "detached_actor.py": "detached_actor.py",
-                "terminate_detached_actor.py": "terminate_detached_actor.py"
-            }
-          }
-        ]
-      }
-    ]
-  }
-}'
+--data @docs/api-example/autoscaling_clusters.json
 ```
 
-## Validate that Ray cluster is deployed correctly
+This command performs two main operations:
 
-Run:
+1. Creates a compute template `default-template` that specifies resources to use when
+   scale-up (2 CPUs and 4 GiB memory).
 
-```shell
-kubectl get pods
-```
+2. Deploys a RayCluster (test-cluster) with:
+    - A head pod that manages the cluster
+    - A worker group configured to scale between 0 and 5 replicas
+
+The worker group uses the following autoscalerOptions to control scaling behavior:
+
+- **`upscalingMode: "Default"`**: Default scaling behavior. Ray will scale up only as
+needed.
+- **`idleTimeoutSeconds: 30`** If a worker pod remains idle (i.e., not running any tasks)
+for 30 seconds, it will be automatically removed.
+- **`cpu: "500m"`, `memory: "512Mi"`**: Defines the **minimum resource unit** Ray uses to
+assess scaling needs.  If no worker pod has at least this much free capacity, Ray will
+trigger a scale-up and launch a new worker pod.
 
-You should get something like this:
+> **Note:** These values **do not determine the actual size** of the worker pod. The
+> pod size comes from the `computeTemplate` (in this case, 2 CPUs and 4 GiB memory).
 
-```shell
-test-cluster-head-pr25j             2/2     Running   0          2m49s
+### Validate that RayCluster is deployed correctly
+
+Run following command to get list of pods running. You should see something like below:
path: apiserver/Autoscaling.md
user: login: rueian
body: ```suggestion
Run the following command to get a list of pods running. You should see something like below:
```
created_at: 2025-05-19T19:02:26Z
author_association: COLLABORATOR
diff_hunk: @@ -1,122 +1,99 @@
 # Creating Autoscaling clusters using API server
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains setting up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the API server and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+> [!IMPORTANT]
+> All the following guidance require you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/cluster/cluster/detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the config map is successfully created, you should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, you need to first deploy a RayCluster with following command.
 
-```shell
+```sh
+# Create compute tempalte
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
 --header 'Content-Type: application/json' \
---data '{
-  "name": "default-template",
-  "namespace": "default",
-  "cpu": 2,
-  "memory": 4
-}'
+--data  @docs/api-example/compute_template.json
+
+# Create RayCluster
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 --header 'Content-Type: application/json' \
---data '{
-  "name": "test-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "clusterSpec": {
-    "enableInTreeAutoscaling": true,
-    "autoscalerOptions": {
-        "upscalingMode": "Default",
-        "idleTimeoutSeconds": 30,
-        "cpu": "500m",
-        "memory": "512Mi"
-    },
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0"
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-            "detached_actor.py": "detached_actor.py",
-            "terminate_detached_actor.py": "terminate_detached_actor.py"
-           }
-         }
-       ]
-    },
-    "workerGroupSpec": [
-      {
-        "groupName": "small-wg",
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "replicas": 0,
-        "minReplicas": 0,
-        "maxReplicas": 5,
-        "rayStartParams": {
-           "node-ip-address": "$MY_POD_IP"
-         },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-example",
-            "items": {
-                "detached_actor.py": "detached_actor.py",
-                "terminate_detached_actor.py": "terminate_detached_actor.py"
-            }
-          }
-        ]
-      }
-    ]
-  }
-}'
+--data @docs/api-example/autoscaling_clusters.json
 ```
 
-## Validate that Ray cluster is deployed correctly
+This command performs two main operations:
 
-Run:
+1. Creates a compute template `default-template` that specifies resources to use when
+   scale-up (2 CPUs and 4 GiB memory).
 
-```shell
-kubectl get pods
-```
+2. Deploys a RayCluster (test-cluster) with:
+    - A head pod that manages the cluster
+    - A worker group configured to scale between 0 and 5 replicas
+
+The worker group uses the following autoscalerOptions to control scaling behavior:
+
+- **`upscalingMode: "Default"`**: Default scaling behavior. Ray will scale up only as
+needed.
+- **`idleTimeoutSeconds: 30`** If a worker pod remains idle (i.e., not running any tasks)
+for 30 seconds, it will be automatically removed.
+- **`cpu: "500m"`, `memory: "512Mi"`**: Defines the **minimum resource unit** Ray uses to
+assess scaling needs.  If no worker pod has at least this much free capacity, Ray will
+trigger a scale-up and launch a new worker pod.
 
-You should get something like this:
+> **Note:** These values **do not determine the actual size** of the worker pod. The
+> pod size comes from the `computeTemplate` (in this case, 2 CPUs and 4 GiB memory).
 
-```shell
-test-cluster-head-pr25j             2/2     Running   0          2m49s
+### Validate that RayCluster is deployed correctly
+
+Run following command to get list of pods running. You should see something like below:
+
+```sh
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# kuberay-operator-545586d46c-f9grr   1/1     Running   0          49m
+# test-cluster-head                   2/2     Running   0          3m1s
 ```
 
-Note that only head pod is running and it has 2 containers
+Note that there is no worker for `test-cluster` as we set its initial replicas to 0. You
+will only see head pod with 2 containers for `test-cluster`.
 
-## Trigger RayCluster scale-up
+### Trigger RayCluster scale-up
 
-Create a detached actor:
+Create a detached actor to trigger scale-up with following command:
path: apiserver/Autoscaling.md
user: login: rueian
body: ```suggestion
Create a detached actor to trigger scale-up with the following command:
```
created_at: 2025-05-19T19:02:42Z
author_association: COLLABORATOR
diff_hunk: @@ -165,16 +144,28 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/jobs' \
 }'
 ```
 
-A worker Pod will be deleted after `idleTimeoutSeconds` (default 60s, we specified 30) seconds. Run:
+While actor is deleted, we do not need the worker anymore. The worker pod will be deleted
path: apiserver/Autoscaling.md
user: login: rueian
body: ```suggestion
While the actor is deleted, we do not need the worker anymore. The worker pod will be deleted
```
created_at: 2025-05-19T19:03:25Z
author_association: COLLABORATOR
diff_hunk: @@ -1,104 +1,92 @@
 # Creating a cluster with RayServe support
 
-Up until rescently the only way to create a Ray cluster supporting RayServe was by using `Create ray
-service` APIs. Although it does work, quite often you want to create cluster supporting Ray serve so
-that you can experiment with serve APIs directly. Now it is possible by adding the following
-annotation to the cluster:
+Currently, there are two ways for creating a RayCluster with RayService support:
+
+1. Create a RayCluster first, then use [Create RayService](./HttpRequestSpec.md#Create-ray-service-in-a-given-namespace)
+   APIs to create a service.
+2. Directly create cluster with RayServe support (described in this document)
path: apiserver/CreatingServe.md
user: login: rueian
body: ```suggestion
2. Directly create a cluster with RayServe support (described in this document)
```
created_at: 2025-05-19T19:04:04Z
author_association: COLLABORATOR
diff_hunk: @@ -1,104 +1,92 @@
 # Creating a cluster with RayServe support
 
-Up until rescently the only way to create a Ray cluster supporting RayServe was by using `Create ray
-service` APIs. Although it does work, quite often you want to create cluster supporting Ray serve so
-that you can experiment with serve APIs directly. Now it is possible by adding the following
-annotation to the cluster:
+Currently, there are two ways for creating a RayCluster with RayService support:
+
+1. Create a RayCluster first, then use [Create RayService](./HttpRequestSpec.md#Create-ray-service-in-a-given-namespace)
+   APIs to create a service.
+2. Directly create cluster with RayServe support (described in this document)
+
+For creating a cluster with RayService support, simply add the following annotation when
+creating the cluster:
 
 ```json
 "annotations" : {
     "ray.io/enable-serve-service": "true"
   },
 ```
 
-the complete curl command to creation such cluster is as follows:
+## Setup
 
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "test-cluster",
-    "namespace": "default",
-    "user": "boris",
-    "annotations" : {
-      "ray.io/enable-serve-service": "true"
-    },
-    "clusterSpec": {
-      "headGroupSpec": {
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "serviceType": "ClusterIP",
-        "rayStartParams": {
-           "dashboard-host": "0.0.0.0",
-           "metrics-export-port": "8080",
-           "dashboard-agent-listen-port": "52365"
-         },
-         "volumes": [
-           {
-             "name": "code-sample",
-             "mountPath": "/home/ray/samples",
-             "volumeType": "CONFIGMAP",
-             "source": "ray-job-code-sample",
-             "items": {"sample_code.py" : "sample_code.py"}
-           }
-         ]
-      },
-      "workerGroupSpec": [
-        {
-          "groupName": "small-wg",
-          "computeTemplate": "default-template",
-          "image": "rayproject/ray:2.9.0-py310",
-          "replicas": 1,
-          "minReplicas": 0,
-          "maxReplicas": 5,
-          "rayStartParams": {
-             "node-ip-address": "$MY_POD_IP"
-           },
-          "volumes": [
-            {
-              "name": "code-sample",
-              "mountPath": "/home/ray/samples",
-              "volumeType": "CONFIGMAP",
-              "source": "ray-job-code-sample",
-              "items": {"sample_code.py" : "sample_code.py"}
-            }
-          ]
-        }
-      ]
-    }
-  }'
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+This example walks through how to create a cluster with RayService support.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Note, that before creating a cluster you need to install this [configmap] and
-create default template using the following command:
+> [!IMPORTANT]
+> All the following guidance require you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/job/code.yaml
+```
+
+Check if the config map is successfully created, you should see `ray-job-code-sample` in
+the list:
+
+```sh
+kubectl get configmaps
+# NAME                  DATA   AGE
+# ray-job-code-sample   1      13s
+```
+
+### Create RayCluster
+
+Use following command to create a compute template and a RayCluster with RayService support:
path: apiserver/CreatingServe.md
user: login: rueian
body: ```suggestion
Use the following command to create a compute template and a RayCluster with RayService support:
```
created_at: 2025-05-19T19:04:43Z
author_association: COLLABORATOR
diff_hunk: @@ -1,104 +1,92 @@
 # Creating a cluster with RayServe support
 
-Up until rescently the only way to create a Ray cluster supporting RayServe was by using `Create ray
-service` APIs. Although it does work, quite often you want to create cluster supporting Ray serve so
-that you can experiment with serve APIs directly. Now it is possible by adding the following
-annotation to the cluster:
+Currently, there are two ways for creating a RayCluster with RayService support:
+
+1. Create a RayCluster first, then use [Create RayService](./HttpRequestSpec.md#Create-ray-service-in-a-given-namespace)
+   APIs to create a service.
+2. Directly create cluster with RayServe support (described in this document)
+
+For creating a cluster with RayService support, simply add the following annotation when
+creating the cluster:
 
 ```json
 "annotations" : {
     "ray.io/enable-serve-service": "true"
   },
 ```
 
-the complete curl command to creation such cluster is as follows:
+## Setup
 
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "test-cluster",
-    "namespace": "default",
-    "user": "boris",
-    "annotations" : {
-      "ray.io/enable-serve-service": "true"
-    },
-    "clusterSpec": {
-      "headGroupSpec": {
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "serviceType": "ClusterIP",
-        "rayStartParams": {
-           "dashboard-host": "0.0.0.0",
-           "metrics-export-port": "8080",
-           "dashboard-agent-listen-port": "52365"
-         },
-         "volumes": [
-           {
-             "name": "code-sample",
-             "mountPath": "/home/ray/samples",
-             "volumeType": "CONFIGMAP",
-             "source": "ray-job-code-sample",
-             "items": {"sample_code.py" : "sample_code.py"}
-           }
-         ]
-      },
-      "workerGroupSpec": [
-        {
-          "groupName": "small-wg",
-          "computeTemplate": "default-template",
-          "image": "rayproject/ray:2.9.0-py310",
-          "replicas": 1,
-          "minReplicas": 0,
-          "maxReplicas": 5,
-          "rayStartParams": {
-             "node-ip-address": "$MY_POD_IP"
-           },
-          "volumes": [
-            {
-              "name": "code-sample",
-              "mountPath": "/home/ray/samples",
-              "volumeType": "CONFIGMAP",
-              "source": "ray-job-code-sample",
-              "items": {"sample_code.py" : "sample_code.py"}
-            }
-          ]
-        }
-      ]
-    }
-  }'
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+This example walks through how to create a cluster with RayService support.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Note, that before creating a cluster you need to install this [configmap] and
-create default template using the following command:
+> [!IMPORTANT]
+> All the following guidance require you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/job/code.yaml
+```
+
+Check if the config map is successfully created, you should see `ray-job-code-sample` in
+the list:
+
+```sh
+kubectl get configmaps
+# NAME                  DATA   AGE
+# ray-job-code-sample   1      13s
+```
+
+### Create RayCluster
+
+Use following command to create a compute template and a RayCluster with RayService support:
+
+```sh
+# Create compute tempalte
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+--header 'Content-Type: application/json' \
+--data  @docs/api-example/compute_template.json
+
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
   --header 'Content-Type: application/json' \
-  --data '{
-    "name": "default-template",
-    "namespace": "default",
-    "cpu": 2,
-    "memory": 4
-  }'
+  --data @docs/api-example/create_serving_clusters.json
 ```
 
 To confirm that the cluster is created correctly, check created services using that following command:
 
-```shell
+```sh
 kubectl get service
+# NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                   AGE
+# test-cluster-head-svc    ClusterIP   None            <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP   7s
+# test-cluster-serve-svc   ClusterIP   10.96.161.26    <none>        8000/TCP                                                  7s
 ```
 
-that should return the following:
+You should see two services created, one for head node to access dashboard and manage the
path: apiserver/CreatingServe.md
user: login: rueian
body: ```suggestion
You should see two services created, one for the head node to access the dashboard and manage the
```
created_at: 2025-05-19T19:05:14Z
author_association: COLLABORATOR
diff_hunk: @@ -1,104 +1,92 @@
 # Creating a cluster with RayServe support
 
-Up until rescently the only way to create a Ray cluster supporting RayServe was by using `Create ray
-service` APIs. Although it does work, quite often you want to create cluster supporting Ray serve so
-that you can experiment with serve APIs directly. Now it is possible by adding the following
-annotation to the cluster:
+Currently, there are two ways for creating a RayCluster with RayService support:
+
+1. Create a RayCluster first, then use [Create RayService](./HttpRequestSpec.md#Create-ray-service-in-a-given-namespace)
+   APIs to create a service.
+2. Directly create cluster with RayServe support (described in this document)
+
+For creating a cluster with RayService support, simply add the following annotation when
+creating the cluster:
 
 ```json
 "annotations" : {
     "ray.io/enable-serve-service": "true"
   },
 ```
 
-the complete curl command to creation such cluster is as follows:
+## Setup
 
-```shell
-curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "test-cluster",
-    "namespace": "default",
-    "user": "boris",
-    "annotations" : {
-      "ray.io/enable-serve-service": "true"
-    },
-    "clusterSpec": {
-      "headGroupSpec": {
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "serviceType": "ClusterIP",
-        "rayStartParams": {
-           "dashboard-host": "0.0.0.0",
-           "metrics-export-port": "8080",
-           "dashboard-agent-listen-port": "52365"
-         },
-         "volumes": [
-           {
-             "name": "code-sample",
-             "mountPath": "/home/ray/samples",
-             "volumeType": "CONFIGMAP",
-             "source": "ray-job-code-sample",
-             "items": {"sample_code.py" : "sample_code.py"}
-           }
-         ]
-      },
-      "workerGroupSpec": [
-        {
-          "groupName": "small-wg",
-          "computeTemplate": "default-template",
-          "image": "rayproject/ray:2.9.0-py310",
-          "replicas": 1,
-          "minReplicas": 0,
-          "maxReplicas": 5,
-          "rayStartParams": {
-             "node-ip-address": "$MY_POD_IP"
-           },
-          "volumes": [
-            {
-              "name": "code-sample",
-              "mountPath": "/home/ray/samples",
-              "volumeType": "CONFIGMAP",
-              "source": "ray-job-code-sample",
-              "items": {"sample_code.py" : "sample_code.py"}
-            }
-          ]
-        }
-      ]
-    }
-  }'
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+This example walks through how to create a cluster with RayService support.
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Note, that before creating a cluster you need to install this [configmap] and
-create default template using the following command:
+> [!IMPORTANT]
+> All the following guidance require you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
+Please install this [ConfigMap] which contains code for our example. Simply download
+this file and run:
+
+```sh
+kubectl apply -f test/job/code.yaml
+```
+
+Check if the config map is successfully created, you should see `ray-job-code-sample` in
+the list:
+
+```sh
+kubectl get configmaps
+# NAME                  DATA   AGE
+# ray-job-code-sample   1      13s
+```
+
+### Create RayCluster
+
+Use following command to create a compute template and a RayCluster with RayService support:
+
+```sh
+# Create compute tempalte
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+--header 'Content-Type: application/json' \
+--data  @docs/api-example/compute_template.json
+
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
   --header 'Content-Type: application/json' \
-  --data '{
-    "name": "default-template",
-    "namespace": "default",
-    "cpu": 2,
-    "memory": 4
-  }'
+  --data @docs/api-example/create_serving_clusters.json
 ```
 
 To confirm that the cluster is created correctly, check created services using that following command:
 
-```shell
+```sh
 kubectl get service
+# NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                   AGE
+# test-cluster-head-svc    ClusterIP   None            <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP   7s
+# test-cluster-serve-svc   ClusterIP   10.96.161.26    <none>        8000/TCP                                                  7s
 ```
 
-that should return the following:
+You should see two services created, one for head node to access dashboard and manage the
+cluster, the other for submit the serve request.
 
-```shell
-test-cluster-head-svc    ClusterIP   10.96.19.185    <none>        8265/TCP,52365/TCP,10001/TCP,8080/TCP,6379/TCP,8000/TCP
-test-cluster-serve-svc   ClusterIP   10.96.144.162   <none>        8000/TCP
-```
+Note that we set the 52365 port for dashboard agent in the above curl command, which is
path: apiserver/CreatingServe.md
user: login: rueian
body: ```suggestion
Note that we set the 52365 port for the dashboard agent in the above curl command, which is
```
created_at: 2025-05-19T19:05:29Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
One of the issues for long-running Ray application (e.g. RayServe) is that if the Ray head node
```
created_at: 2025-05-19T19:06:04Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
The RayCluster with high availability can also be created with the API server, which aims to
```
created_at: 2025-05-19T19:06:33Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
cluster-level metadata by storing all data in memory, which lacks fault tolerance. A
```
created_at: 2025-05-19T19:07:11Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
we should have a highly available Redis so that when GCS restarts, it can resume its
```
created_at: 2025-05-19T19:07:41Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
status by retrieving previous data from the Redis instance.
```
created_at: 2025-05-19T19:07:50Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
Check if the redis is successfully set up with the following command. You should see
```
created_at: 2025-05-19T19:08:36Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
Before creating your cluster, you need to create a secret in the
```
created_at: 2025-05-19T19:08:51Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
run the following command:
```
created_at: 2025-05-19T19:09:06Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
Please download the config map and deploy it with the following command:
```
created_at: 2025-05-19T19:09:16Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
Use the following command to create a compute template and a RayCluster:
```
created_at: 2025-05-19T19:09:29Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
`headGroupSpec`: We also added the following content in `rayStartParams` of `headGroupSpec`,
```
created_at: 2025-05-19T19:09:42Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
path: apiserver/HACluster.md
user: login: rueian
body: this doesn't look like a json.
created_at: 2025-05-19T19:11:16Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
+"redis-password:: "$REDIS_PASSWORD"
+"num-cpu": "0"
+```
+
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
     },
-    "workerGroupSpec": [
-      {
-        "groupName": "small-wg",
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "replicas": 1,
-        "minReplicas": 0,
-        "maxReplicas": 5,
-        "rayStartParams": {
-           "node-ip-address": "$MY_POD_IP",
-           "metrics-export-port": "8080"
-        },
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-example",
-            "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-          }
-        ]
-      }
-    ]
-  }
-}'
-```
-
-Note that computeTemplate here has to be created using this [command](test/cluster//template/simple)
-
-Lets discuss the important pieces here:
-You need to specify annotation, that tells Ray that this is cluster with GCS fault tolerance
+    "valuesFrom": {
+        "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+        }
+    }
+},
+```
 
-```sh
-ray.io/ft-enabled: "true"
+For the `workerGroupSpecs`, we set the `gcs_rpc_server_reconnect_timeout` environment
+variable, which controls the GCS heartbeat timeout (default 60 seconds). This controls how
+long after the head node dies do we kill the worker node. While it takes time to restart
+the head node, we want this values to be large enough to prevent the worker node being
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
the head node, we want these values to be large enough to prevent the worker node being
```
created_at: 2025-05-19T19:11:44Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
+"redis-password:: "$REDIS_PASSWORD"
+"num-cpu": "0"
+```
+
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
     },
-    "workerGroupSpec": [
-      {
-        "groupName": "small-wg",
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "replicas": 1,
-        "minReplicas": 0,
-        "maxReplicas": 5,
-        "rayStartParams": {
-           "node-ip-address": "$MY_POD_IP",
-           "metrics-export-port": "8080"
-        },
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-example",
-            "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-          }
-        ]
-      }
-    ]
-  }
-}'
-```
-
-Note that computeTemplate here has to be created using this [command](test/cluster//template/simple)
-
-Lets discuss the important pieces here:
-You need to specify annotation, that tells Ray that this is cluster with GCS fault tolerance
+    "valuesFrom": {
+        "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+        }
+    }
+},
+```
 
-```sh
-ray.io/ft-enabled: "true"
+For the `workerGroupSpecs`, we set the `gcs_rpc_server_reconnect_timeout` environment
+variable, which controls the GCS heartbeat timeout (default 60 seconds). This controls how
+long after the head node dies do we kill the worker node. While it takes time to restart
+the head node, we want this values to be large enough to prevent the worker node being
+killed during the restarting period.
+
+```json
+"environment": {
+    "values": {
+        "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
+    }
+},
 ```
 
-For the `headGroupSpec` you need the following. In the `rayStartParams` you need to add information about Redis
-password.
+### Validate that RayCluster is deployed correctly
+
+Run following command to get list of pods running. You should see one head and worker node
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
Run the following command to get a list of pods running. You should see one head and worker node
```
created_at: 2025-05-19T19:11:58Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
+"redis-password:: "$REDIS_PASSWORD"
+"num-cpu": "0"
+```
+
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
     },
-    "workerGroupSpec": [
-      {
-        "groupName": "small-wg",
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "replicas": 1,
-        "minReplicas": 0,
-        "maxReplicas": 5,
-        "rayStartParams": {
-           "node-ip-address": "$MY_POD_IP",
-           "metrics-export-port": "8080"
-        },
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-example",
-            "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-          }
-        ]
-      }
-    ]
-  }
-}'
-```
-
-Note that computeTemplate here has to be created using this [command](test/cluster//template/simple)
-
-Lets discuss the important pieces here:
-You need to specify annotation, that tells Ray that this is cluster with GCS fault tolerance
+    "valuesFrom": {
+        "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+        }
+    }
+},
+```
 
-```sh
-ray.io/ft-enabled: "true"
+For the `workerGroupSpecs`, we set the `gcs_rpc_server_reconnect_timeout` environment
+variable, which controls the GCS heartbeat timeout (default 60 seconds). This controls how
+long after the head node dies do we kill the worker node. While it takes time to restart
+the head node, we want this values to be large enough to prevent the worker node being
+killed during the restarting period.
+
+```json
+"environment": {
+    "values": {
+        "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
+    }
+},
 ```
 
-For the `headGroupSpec` you need the following. In the `rayStartParams` you need to add information about Redis
-password.
+### Validate that RayCluster is deployed correctly
+
+Run following command to get list of pods running. You should see one head and worker node
+like below:
 
 ```sh
-"redis-password:: "$REDIS_PASSWORD"
-"num-cpu": "0"
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     1/1     Running   0          2m36s
+# ha-cluster-small-wg-worker-22lbx    1/1     Running   0          2m36s
 ```
 
-Where the value of `REDIS_PASSWORD` comes from environment variable (below). Additionally `num-cpus:
-0` ensures that no application code runs on a head node.
+### Create an Actor
+
+Before we try to trigger the restoration, we need to find a way to validate our GCS restore
+is working correctly. We will validate this by creating a detached actor. If it still
+exists and functions after the head node deletion and restoration, we can confirm that the
+GCS data is restored correctly.
 
-The following environment variable have to be added here:
+Run following command for creating a detached actor. Please change `ha-cluster-head` to
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
Run the following command for creating a detached actor. Please change `ha-cluster-head` to
```
created_at: 2025-05-19T19:12:12Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
+"redis-password:: "$REDIS_PASSWORD"
+"num-cpu": "0"
+```
+
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
+
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
     },
-    "workerGroupSpec": [
-      {
-        "groupName": "small-wg",
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "replicas": 1,
-        "minReplicas": 0,
-        "maxReplicas": 5,
-        "rayStartParams": {
-           "node-ip-address": "$MY_POD_IP",
-           "metrics-export-port": "8080"
-        },
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-example",
-            "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-          }
-        ]
-      }
-    ]
-  }
-}'
-```
-
-Note that computeTemplate here has to be created using this [command](test/cluster//template/simple)
-
-Lets discuss the important pieces here:
-You need to specify annotation, that tells Ray that this is cluster with GCS fault tolerance
+    "valuesFrom": {
+        "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+        }
+    }
+},
+```
 
-```sh
-ray.io/ft-enabled: "true"
+For the `workerGroupSpecs`, we set the `gcs_rpc_server_reconnect_timeout` environment
+variable, which controls the GCS heartbeat timeout (default 60 seconds). This controls how
+long after the head node dies do we kill the worker node. While it takes time to restart
+the head node, we want this values to be large enough to prevent the worker node being
+killed during the restarting period.
+
+```json
+"environment": {
+    "values": {
+        "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
+    }
+},
 ```
 
-For the `headGroupSpec` you need the following. In the `rayStartParams` you need to add information about Redis
-password.
+### Validate that RayCluster is deployed correctly
+
+Run following command to get list of pods running. You should see one head and worker node
+like below:
 
 ```sh
-"redis-password:: "$REDIS_PASSWORD"
-"num-cpu": "0"
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     1/1     Running   0          2m36s
+# ha-cluster-small-wg-worker-22lbx    1/1     Running   0          2m36s
 ```
 
-Where the value of `REDIS_PASSWORD` comes from environment variable (below). Additionally `num-cpus:
-0` ensures that no application code runs on a head node.
+### Create an Actor
+
+Before we try to trigger the restoration, we need to find a way to validate our GCS restore
+is working correctly. We will validate this by creating a detached actor. If it still
+exists and functions after the head node deletion and restoration, we can confirm that the
+GCS data is restored correctly.
 
-The following environment variable have to be added here:
+Run following command for creating a detached actor. Please change `ha-cluster-head` to
+your head node's name. Note that the `detached_actor.py` file is defined in the
+[ConfigMap] we installed earlier and mounted to the head node:
 
 ```sh
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-```
-
-For the `workerGroupSpecs` you might want to increase `gcs_rpc_server_reconnect_timeout` by specifying the following
-environment variable:
+kubectl exec -it ha-cluster-head -- python3 /home/ray/samples/detached_actor.py
+```
+
+Then, open a new termianl and use port-forward to enable accessing to the Ray dashboard.
+The dashboard can be accessed through `http://localhost:8265`:
 
 ```sh
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
+kubectl port-forward pod/ha-cluster-head 8265:8265
 ```
 
-This environment variable allows to increase GCS heartbeat timeout, which is 60 sec by default. The reason for
-increasing it is because restart of the head node can take some time, and we want to make sure that the worker node
-will not be killed during this time.
+In the dashboard, You can see 2 nodes in the Cluster pane, which is head and worker:
+
+![hacluster-dashboard-cluster](img/hacluster-dashboard-cluster.png)
+
+If you go to the Actor pane, you can see the actor we created earlier:
 
-## Testing resulting cluster
+![hacluster-dashboard-actor](img/hacluster-dashboard-actor.png)
+
+### Trigger the GCS restore
+
+To trigger the restoration, we can simply delete the head node with:
+
+```sh
+kubectl delete pods ha-cluster-head
+```
 
-Once the cluster is created, we can validate that it is working correctly. To do this first create a detached actor.
-To do this, note the name of the head node and create a detached actor using the following command:
+If you list the pods now, you can see a new head node is recreated
 
 ```sh
-kubectl exec -it <head node pod name> -- python3 /home/ray/samples/detached_actor.py
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     0/1     Running   0          5s
+# ha-cluster-small-wg-worker-tpgqs    1/1     Running   0          9m19s
 ```
 
-Once this is done, open Ray dashboard (using port-forward). In the cluster tab you should see 2 nodes and in the
-Actor's pane you should see created actor.
+Note that only head node will be recreated, while the worker node stays as is.
path: apiserver/HACluster.md
user: login: rueian
body: ```suggestion
Note that only the head node will be recreated, while the worker node stays as is.
```
created_at: 2025-05-19T19:12:30Z
author_association: COLLABORATOR
diff_hunk: @@ -1,111 +1,78 @@
 # Job Submission using API server
 
-Ray provides very convinient and powerful [Job Submission APIs]. The issue is that it needs to
+Ray provides very convenient and powerful [Job Submission APIs]. The issue is that it needs to
 access cluster's dashboard Ingress, which currently does not have any security implementations.
 Alternatively you can use Ray cluster head service for this, but in this case your Ray job
-management code has to run in the same kubernetes cluster as Ray. Job submission implementation in
+management code has to run in the same Kubernetes cluster as Ray. Job submission implementation in
 the API server leverages already exposed URL of the API server to locate cluster and use its head
 service to implement Job management functionality. Because you can access API server running on the
-remote kubernetes cluster you can use these APIs for managing remote Ray clusters without exposing
+remote Kubernetes cluster you can use these APIs for managing remote Ray clusters without exposing
 them via Ingress.
 
 ## Using Job Submission APIs
 
-Note that job submission APIs will only work if you are running API server within kubernetes
+Note that job submission APIs will only work if you are running API server within Kubernetes
 cluster. Local Development option of the API server will not work.
 
-The first step is to deploy KubeRay operator and API server.
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
 
 ### Deploy KubeRay operator and API server
 
-Reffer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubeRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
-```
+### Install ConfigMap
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Note that this cluster is mounting a volume from a configmap. This config map should be created
+prior to cluster creation using [this YAML].
 
-### Deploy Ray cluster
+We will use this [ConfigMap] which contains code for our example. Please download the
+config map and deploy it with following command:
path: apiserver/JobSubmission.md
user: login: rueian
body: ```suggestion
config map and deploy it with the following command:
```
created_at: 2025-05-19T19:13:33Z
author_association: COLLABORATOR
diff_hunk: @@ -1,111 +1,78 @@
 # Job Submission using API server
 
-Ray provides very convinient and powerful [Job Submission APIs]. The issue is that it needs to
+Ray provides very convenient and powerful [Job Submission APIs]. The issue is that it needs to
 access cluster's dashboard Ingress, which currently does not have any security implementations.
 Alternatively you can use Ray cluster head service for this, but in this case your Ray job
-management code has to run in the same kubernetes cluster as Ray. Job submission implementation in
+management code has to run in the same Kubernetes cluster as Ray. Job submission implementation in
 the API server leverages already exposed URL of the API server to locate cluster and use its head
 service to implement Job management functionality. Because you can access API server running on the
-remote kubernetes cluster you can use these APIs for managing remote Ray clusters without exposing
+remote Kubernetes cluster you can use these APIs for managing remote Ray clusters without exposing
 them via Ingress.
 
 ## Using Job Submission APIs
 
-Note that job submission APIs will only work if you are running API server within kubernetes
+Note that job submission APIs will only work if you are running API server within Kubernetes
 cluster. Local Development option of the API server will not work.
 
-The first step is to deploy KubeRay operator and API server.
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
 
 ### Deploy KubeRay operator and API server
 
-Reffer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to [README](README.md) for setting up KubeRay operator and API server.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
-```
+### Install ConfigMap
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Note that this cluster is mounting a volume from a configmap. This config map should be created
+prior to cluster creation using [this YAML].
 
-### Deploy Ray cluster
+We will use this [ConfigMap] which contains code for our example. Please download the
+config map and deploy it with following command:
 
-Execute the following commands:
+```sh
+kubectl apply -f code_configmap.yaml
+```
 
-```shell
+### Deploy RayCluster
+
+Execute the following commands to deploy the RayCluster:
+
+```sh
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
-  --header 'Content-Type: application/json' \
-  --data '{
-    "name": "default-template",
-    "namespace": "default",
-    "cpu": 2,
-    "memory": 4
-  }'
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
 
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
   --header 'Content-Type: application/json' \
-  --data '{
-    "name": "test-cluster",
-    "namespace": "default",
-    "user": "boris",
-    "clusterSpec": {
-      "headGroupSpec": {
-        "computeTemplate": "default-template",
-        "image": "rayproject/ray:2.9.0-py310",
-        "serviceType": "NodePort",
-        "rayStartParams": {
-          "dashboard-host": "0.0.0.0",
-          "metrics-export-port": "8080"
-        },
-        "volumes": [
-          {
-            "name": "code-sample",
-            "mountPath": "/home/ray/samples",
-            "volumeType": "CONFIGMAP",
-            "source": "ray-job-code-sample",
-            "items": {"sample_code.py" : "sample_code.py"}
-          }
-        ]
-      },
-      "workerGroupSpec": [
-        {
-          "groupName": "small-wg",
-          "computeTemplate": "default-template",
-          "image": "rayproject/ray:2.9.0-py310",
-          "replicas": 1,
-          "minReplicas": 1,
-          "maxReplicas": 1,
-          "rayStartParams": {
-            "node-ip-address": "$MY_POD_IP"
-          },
-          "volumes": [
-            {
-              "name": "code-sample",
-              "mountPath": "/home/ray/samples",
-              "volumeType": "CONFIGMAP",
-              "source": "ray-job-code-sample",
-              "items": {"sample_code.py" : "sample_code.py"}
-            }
-          ]
-        }
-      ]
-    }
-  }'
+  --data @docs/api-example/jobsubmission_clusters.json
 ```
 
-Note that this cluster is mounting a volume from a configmap. This config map should be created
-prior to cluster creation using [this YAML].
+To check if the RayCluster setup correctly, list all pods with following command. You can
path: apiserver/JobSubmission.md
user: login: rueian
body: ```suggestion
To check if the RayCluster setup correctly, list all pods with the following command. You can
```
created_at: 2025-05-19T19:13:58Z
author_association: COLLABORATOR
diff_hunk: @@ -1,68 +1,104 @@
-# Monitoring of the API server and created Ray clusters
+# Monitoring of the API server and created RayClusters
 
-In order to ensure a proper functioning of the API server and created Ray clusters, it is typically necessary to
-monitor the API server itself and created Ray clusters. This document describes how to instrument both API server
-and created clusters with Prometheus and Grafana
-
-## Monitoring of the API server
+In order to ensure a proper functioning of the API server and created RayClusters, it is
+typically necessary to monitor them. This document describes how to monitor both API
+server and created clusters with Prometheus and Grafana.
 
 Current implementation of the API server provides a flag `collectMetricsFlag` that defines whether to collect and
 expose Prometheus metrics, which is by default set to `true`. To disable metrics collection, this flag needs to
-be set to `false` and the API server image needs to be rebuild.
+be set to `false` and the API server image needs to be rebuilt.
 
-If this flag is enabled, an `http` port at `/metrics` endpoint provides API server metrics in Prometheus format.
+If this flag is enabled, a `http` port at `/metrics` endpoint provides API server metrics in Prometheus format.
 
-### Testing monitoring of the API server
+## Monitoring of the API server
+
+### Deploy KubeRay operator and API server
+
+Refer to [README](README.md) for setting up KubeRay operator and API server. This will
+set the flag `collectMetricsFlag` to `true` which enable the metrics collection.
 
-On the kind server Install Kubernetes Prometheus Stack via Helm chart using the following
-[command](../install/prometheus/install.sh). The script additionally creates `ray-head-monitor` and
-`ray-workers-monitor` in the `prometheus-system` namespace, that we do not need. We can delete them using:
+> [!IMPORTANT]
+> All the following guidance require you to switch your working directory to the KubeRay project root
path: apiserver/Monitoring.md
user: login: rueian
body: ```suggestion
> All the following guidance requires you to switch your working directory to the KubeRay project root
```
created_at: 2025-05-19T19:15:37Z
author_association: COLLABORATOR
diff_hunk: @@ -1,217 +1,222 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single head node failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+```sh
+kubectl get configmaps -n redis
+
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/compute_template.json
+
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
---header 'Content-Type: application/json' \
---data '{
-  "name": "ha-cluster",
-  "namespace": "default",
-  "user": "boris",
-  "version": "2.9.0",
-  "environment": "DEV",
-  "annotations" : {
+    --header 'Content-Type: application/json' \
+    --data @docs/api-example/ha_clusters.json
+```
+
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
+
+```json
+"annotations" : {
     "ray.io/ft-enabled": "true"
-  },
-  "clusterSpec": {
-    "headGroupSpec": {
-      "computeTemplate": "default-template",
-      "image": "rayproject/ray:2.9.0-py310",
-      "serviceType": "NodePort",
-      "rayStartParams": {
-         "dashboard-host": "0.0.0.0",
-         "metrics-export-port": "8080",
-         "num-cpus": "0",
-         "redis-password": "$REDIS_PASSWORD"
-       },
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
-       "volumes": [
-         {
-           "name": "code-sample",
-           "mountPath": "/home/ray/samples",
-           "volumeType": "CONFIGMAP",
-           "source": "ray-example",
-           "items": {
-              "detached_actor.py" : "detached_actor.py",
-              "increment_counter.py" : "increment_counter.py"
-            }
-         }
-       ]
+}
+```
+
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
+
+```json
path: apiserver/HACluster.md
user: login: machichima
body: Thanks! I added {} around it. 

created_at: 2025-05-21T01:36:50Z
author_association: COLLABORATOR
diff_hunk: @@ -1,122 +1,99 @@
-# Creating Autoscaling clusters using API server
+# Creating Autoscaling clusters using APIServer
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains how to set up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the APIServer and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to the [README](README.md) for setting up the KubeRay operator and APIServer.
path: apiserver/Autoscaling.md
user: login: kevin85421
body: There are two sections in "Installation". Please explicitly ask users to follow "Install with Helm".

"Start a local apiserver" should only be used for development.
created_at: 2025-05-28T20:46:09Z
author_association: MEMBER
diff_hunk: @@ -1,122 +1,99 @@
-# Creating Autoscaling clusters using API server
+# Creating Autoscaling clusters using APIServer
 
-One of the fundamental features of Ray is autoscaling. This [document] describes how to set up
-autoscaling using Ray operator. Here we will describe how to set it up using API server.
+One of Ray's key features is autoscaling. This [document] explains how to set up autoscaling
+with the Ray operator. Here, we demonstrate how to configure it using the APIServer and
+run an example.
 
-## Deploy KubeRay operator and API server
+## Setup
 
-Refer to [readme](README.md) for setting up KubRay operator and API server.
+Refer to the [README](README.md) for setting up the KubeRay operator and APIServer.
 
-```shell
-make operator-image cluster load-operator-image deploy-operator
+## Example
+
+This example walks through how to trigger scale-up and scale-down for RayCluster.
+
+Before proceeding with the example, remove any running RayClusters to ensure a successful
+execution of the steps below.
+
+```sh
+kubectl delete raycluster --all
 ```
 
-Alternatively, you could build and deploy the Operator and API server from local repo for
-development purpose.
+> [!IMPORTANT]
+> All the following guidance requires you to switch your working directory to the KubeRay `apiserver`
+
+### Install ConfigMap
 
-```shell
-make operator-image cluster load-operator-image deploy-operator docker-image load-image deploy
+Install this [ConfigMap], which contains the code for our example. Simply download
+the file and run:
+
+```sh
+kubectl apply -f test/cluster/cluster/detachedactor.yaml
 ```
 
-Additionally install this [ConfigMap] containing code that we will use for testing.
+Check if the ConfigMap is successfully created. You should see `ray-example` in the list:
+
+```sh
+kubectl get configmaps
+# NAME               DATA   AGE
+# ray-example        2      8s
+```
 
-## Deploy Ray cluster
+### Deploy RayCluster
 
-Once they are set up, you first need to create a Ray cluster using the following commands:
+Before running the example, deploy a RayCluster with the following command:
 
-```shell
+```sh
+# Create compute template
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
path: apiserver/Autoscaling.md
user: login: kevin85421
body: Are you assuming users will install the KubeRay API server outside the Kubernetes cluster? The documentation should instruct users to install the KubeRay API server inside the cluster.

Then, you can ask users to either set up port forwarding or create a curl pod to submit requests.
created_at: 2025-05-28T20:50:59Z
author_association: MEMBER

PR comments:
user: login: machichima
created_at: 2025-05-10T05:46:57Z
body: Hi @dentiny , would you mind taking a look at this?
Thanks!
author_association: COLLABORATOR
user: login: kevin85421
created_at: 2025-05-19T17:46:50Z
body: @rueian can you also take a look at this PR? Thanks!
author_association: MEMBER
user: login: dentiny
created_at: 2025-05-21T08:19:20Z
body: We should get gpt / copilot to do proofreading first, it's easy to have spelling / wording issues slipped through
author_association: CONTRIBUTOR
user: login: machichima
created_at: 2025-05-21T12:57:46Z
body: > We should get gpt / copilot to do proofreading first, it's easy to have spelling / wording issues slipped through

Thanks for the suggestion! I just do the proof reading by copilot to fix the wording issues.
They should be fine now
author_association: COLLABORATOR
user: login: dentiny
created_at: 2025-05-23T12:00:24Z
body: cc @rueian / @kevin85421 for an another look :)
author_association: CONTRIBUTOR