Repository: ray-project/ray
Pull request #51988: [Test][KubeRay]add doc test to kuberay-gcs-ft
Author: login: popojk
Labels: name: kuberay
description: Issues for the Ray/Kuberay integration that are tracked on the Ray side
name: go
description: add ONLY when ready to merge, run all tests
name: community-contribution
description: Contributed by the community
name: community-backlog
Created: 2025-04-04T15:50:31Z | Merged: 2025-04-25T04:00:57Z

Implement doc tests described in [3157](https://github.com/ray-project/kuberay/issues/3157) for GCS fault tolerance in KubeRay.

## Related issue number

 [3157](https://github.com/ray-project/kuberay/issues/3157)

Doc URL: [https://anyscale-ray--51988.com.readthedocs.build/en/51988/cluster/kubernetes/user-guides/kuberay-gcs-ft.html](https://anyscale-ray--51988.com.readthedocs.build/en/51988/cluster/kubernetes/user-guides/kuberay-gcs-ft.html)
<!-- For example: "Closes #1234" -->

## Checks

- [X] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR.
- [X] I've run `scripts/format.sh` to lint the changes in this PR.
- [X] I've included any doc changes needed for https://docs.ray.io/en/master/.
    - [ ] I've added any new APIs to the API Reference. For example, if I added a
           method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [X] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(


Linked issue #1234: Ray fails to serialize self-reference objects
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **Ray installed from (source or binary)**: pip
- **Ray version**: 0.2.2
- **Python version**: 3.6.2

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem

Ray fails to serialize self-reference objects (for example, Graph objects in networkx).

I think it is because ray always tries to use pyarrow first and does not catch `pyarrow.lib.ArrowNotImplementedError`, see

https://github.com/ray-project/ray/blob/e0360eb4298371e308cd266dd512befaa457ce24/python/ray/worker.py#L285-L289

After catching `pyarrow.lib.ArrowNotImplementedError`, we **should not** use `use_dict=True` as a workaround, because it will cause endless loop. A correct approach may be:

```python
            except (pyarrow.SerializationCallbackError, pyarrow.lib.ArrowNotImplementedError) as e:
                try:
                    if isinstance(e, pyarrow.lib.ArrowNotImplementedError):
                        e.example_object = value
                        raise e  # redirect to use cloudpickle
```

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

```python
class Graph:
    def __init__(self):
        self.g = self

G = Graph()
ray.put(G)  # --> pyarrow.lib.ArrowNotImplementedError: This object exceeds the maximum recursion depth. It may contain itself recursively.

# another example

import networkx as nx
G = nx.Graph()
    
G.add_edges_from([(1, 2), (1, 3)])
G.add_node(1)
G.add_edge(1, 2)
G.add_node("spam")  # adds node "spam"
G.add_nodes_from("spam")  # adds 4 nodes: 's', 'p', 'a', 'm'
G.add_edge(3, 'm')
ray.put(G)  # --> pyarrow.lib.ArrowNotImplementedError: This object exceeds the maximum recursion depth. It may contain itself recursively.
```

@mitar 
Linked issue comments: user: login: robertnishihara
created_at: 2017-11-20T05:34:11Z
body: Can you try `ray.register_custom_serializer`? The following works for me.

```python
import ray
ray.init()

class Graph:
    def __init__(self):
        self.g = self

ray.register_custom_serializer(Graph, use_pickle=True)

G = Graph()
ray.put(G)
```

This is closely related to #319 and https://issues.apache.org/jira/browse/ARROW-1382.

A side comment. The original code worked for me in Python 2 because in Python 2 `Graph` is an old-style class and so we automatically fall back to Pickle anyway I think.
author_association: COLLABORATOR
user: login: mitar
created_at: 2017-11-20T06:02:44Z
body: Hm, so ideally we would like to serialize networkx graphs. Because they can be quite large, I am not sure if pickling is a good approach.
author_association: MEMBER
user: login: robertnishihara
created_at: 2017-11-20T06:32:14Z
body: Custom serializers/deserializers can be registered with the same approach. Not sure what the right one would be in this case, but just as a simple example, you could do something like

```python
import numpy as np
import ray

ray.init()

class Graph:
    def __init__(self, big_array):
        self.g = self
        self.big_array = big_array

def custom_graph_serializer(obj):
    return obj.big_array

def custom_graph_deserializer(serialized_obj):
    return Graph(serialized_obj)

ray.register_custom_serializer(Graph,
                               serializer=custom_graph_serializer,
                               deserializer=custom_graph_deserializer)

G = Graph(np.ones(100))
ray.put(G)
```
author_association: COLLABORATOR
user: login: edoakes
created_at: 2020-03-05T23:11:03Z
body: Stale - please open new issue if still relevant
author_association: COLLABORATOR

Diff:
diff --git a/ci/k8s/run-kuberay-doc-tests.sh b/ci/k8s/run-kuberay-doc-tests.sh
index 7997b12cf02a..538648bbb4d5 100644
--- a/ci/k8s/run-kuberay-doc-tests.sh
+++ b/ci/k8s/run-kuberay-doc-tests.sh
@@ -65,6 +65,7 @@ TESTS=(
   "getting-started/raycluster-quick-start.ipynb"
   "getting-started/rayjob-quick-start.ipynb"
   "getting-started/rayservice-quick-start.ipynb"
+  "user-guides/kuberay-gcs-ft.ipynb"
 )
 for test in "${TESTS[@]}"; do
   echo "Running test: ${test}"
diff --git a/doc/source/cluster/kubernetes/doc_sanitize.cfg b/doc/source/cluster/kubernetes/doc_sanitize.cfg
index 7c3a333eca1a..29035baa11fa 100644
--- a/doc/source/cluster/kubernetes/doc_sanitize.cfg
+++ b/doc/source/cluster/kubernetes/doc_sanitize.cfg
@@ -26,6 +26,18 @@ replace: RAYSERVICE-SAMPLE-WORKER-POD-NAME
 regex: rayservice-sample-raycluster-[a-z0-9]{5}-head-svc
 replace: RAYSERVICE-SAMPLE-K8S-SVC
 
+[raycluster-external-redis-head-name]
+regex: raycluster-external-redis-head-[a-z0-9]{5}
+replace: RAYCLUSTER-EXTERNAL-REDIS-HEAD-POD-NAME
+
+[raycluster-external-redis-worker-name]
+regex: raycluster-external-redis-small-group-worker-[a-z0-9]{5}
+replace: RAYCLUSTER-EXTERNAL-REDIS-WORKER-POD-NAME
+
+[raycluster-redis-name]
+regex: redis-[a-z0-9]{10}-[a-z0-9]{5}
+replace: RAYCLUSTER-REDIS-POD-NAME
+
 [rayjob-submit-name]
 regex: raysubmit_[a-zA-Z0-9]{16}
 replace: RAYJOB-SUBMIT-NAME
diff --git a/doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb b/doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
new file mode 100644
index 000000000000..0276a3e42ff5
--- /dev/null
+++ b/doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
@@ -0,0 +1,1111 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-ktbnh                    1/1     Running   0          3m4s\n",
+      "raycluster-external-redis-head-xrjff                 1/1     Running   0          2m41s\n",
+      "raycluster-external-redis-small-group-worker-dwt98   1/1     Running   0          2m41s\n",
+      "redis-6cf756c755-qljcv                               1/1     Running   0          2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "76aa215a-7a63-4b79-8d3c-f5b674ad8fdb",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      3m4s\n",
+      "ray-example        2      2m41s\n",
+      "redis-config       1      2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9d28f87-6e4e-4736-b3d3-b26888955d66",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.\n",
+    "There are two ConfigMaps in this example: `ray-example` and `redis-config`.\n",
+    "The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.\n",
+    "\n",
+    "* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    @ray.remote(num_cpus=1)\n",
+    "    class Counter:\n",
+    "        def __init__(self):\n",
+    "            self.value = 0\n",
+    "\n",
+    "        def increment(self):\n",
+    "            self.value += 1\n",
+    "            return self.value\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    Counter.options(name=\"counter_actor\", lifetime=\"detached\").remote()\n",
+    "    ```\n",
+    "\n",
+    "* `increment_counter.py` is a Python script that increments the counter.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    counter = ray.get_actor(\"counter_actor\")\n",
+    "    print(ray.get(counter.increment.remote()))\n",
+    "    ```\n",
+    "\n",
+    "### Step 5: Create a detached actor\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "978b889d-66af-408a-bd24-c36273edadb9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-18 02:51:25,359\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-18 02:51:25,361\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-18 02:51:25,557\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "2025-04-18 02:51:29,069\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-18 02:51:29,072\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-18 02:51:29,198\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "1\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 5.1: Create a detached actor with the name, `counter_actor`.\n",
+    "export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py\n",
+    "# Step 5.2: Increment the counter.\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a1817cd-4380-4c28-b867-a3f55cc20910",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "(kuberay-external-storage-namespace-example)=\n",
+    "### Step 6: Check the data in Redis\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0454a9bb-4908-4c69-95c9-b2d1c2eb42cc",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.1: Check the RayCluster's UID.\n",
+    "kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'\n",
+    "# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "d8e5d436-f9a4-44ef-86e4-635431caa34e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[1;39m[\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_external_storage_namespace\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"66a4e2af-7c89-43db-a79c-71d1d0d9d71d\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_REDIS_ADDRESS\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"redis:6379\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"REDIS_PASSWORD\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"secretKeyRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"key\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"password\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"redis-password-secret\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_CLUSTER_NAME\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.labels['ray.io/cluster']\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_CLOUD_INSTANCE_ID\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.name\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_NODE_TYPE_NAME\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.labels['ray.io/group']\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"KUBERAY_GEN_RAY_START_CMD\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ray start --head  --block  --dashboard-agent-listen-port=52365  --dashboard-host=0.0.0.0  --metrics-export-port=8080  --num-cpus=0  --redis-password=$REDIS_PASSWORD \"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_PORT\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"6379\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_ADDRESS\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"127.0.0.1:6379\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_USAGE_STATS_KUBERAY_IN_USE\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_USAGE_STATS_EXTRA_TAGS\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"kuberay_version=v1.3.0;kuberay_crd=RayCluster\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_DASHBOARD_ENABLE_K8S_DISK_USAGE\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "\u001b[1;39m]\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 6.2: Check the head Pod's environment variable `RAY_external_storage_namespace`.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "dcbba87e-0f9b-4997-ba71-08cb9c2a1fc6",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "RAY66a4e2af-7c89-43db-a79c-71d1d0d9d71d@ACTOR\n",
+      "RAY66a4e2af-7c89-43db-a79c-71d1d0d9d71d@JOB\n",
+      "RAY66a4e2af-7c89-43db-a79c-71d1d0d9d71d@ACTOR_TASK_SPEC\n",
+      "RAY66a4e2af-7c89-43db-a79c-71d1d0d9d71d@WORKERS\n",
+      "RAY66a4e2af-7c89-43db-a79c-71d1d0d9d71d@JobCounter\n",
+      "RAY66a4e2af-7c89-43db-a79c-71d1d0d9d71d@KV\n",
+      "RAY66a4e2af-7c89-43db-a79c-71d1d0d9d71d@NODE\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 6.3: Log into the Redis Pod.\n",
+    "# The password `5241590000000000` is defined in the `redis-config` ConfigMap.\n",
+    "# Step 6.4: Check the keys in Redis.\n",
+    "# Note: the schema changed in Ray 2.38.0. Previously we use a single HASH table,\n",
+    "# now we use multiple HASH tables with a common prefix.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- env REDISCLI_AUTH=\"5241590000000000\" redis-cli KEYS '*'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "76947751-45b0-4165-af36-d085356966ff",
+   "metadata": {
+    "editable": true,
+    "scrolled": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.5: Check the value of the key.\n",
+    "kubectl exec -i $REDIS_POD -- env REDISCLI_AUTH=\"5241590000000000\" redis-cli HGETALL RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\n",
+    "# Before Ray 2.38.0:\n",
+    "# HGETALL 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a67ff02-03d2-45df-ad5c-fbe3f72a3a1e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `gcsFaultToleranceOptions.externalStorageNamespace` option isn't set for the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster with the RayCluster's UID as the external storage namespace by default.\n",
+    "See [this section](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-external-storage-namespace) to learn more about the option.\n",
+    "\n",
+    "### Step 7: Kill the GCS process in the head Pod\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "297712df-81e6-40b8-80e4-cb43fd125dfd",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "export YOUR_WORKER_POD=$(kubectl get pods -l ray.io/group=small-group -o jsonpath='{.items[0].metadata.name}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "775d8da9-cea7-4f93-adec-653520af37d3",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[1;39m[\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_external_storage_namespace\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"66a4e2af-7c89-43db-a79c-71d1d0d9d71d\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_REDIS_ADDRESS\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"redis:6379\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"REDIS_PASSWORD\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"secretKeyRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"key\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"password\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"redis-password-secret\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_CLUSTER_NAME\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.labels['ray.io/cluster']\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_CLOUD_INSTANCE_ID\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.name\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_NODE_TYPE_NAME\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.labels['ray.io/group']\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"KUBERAY_GEN_RAY_START_CMD\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ray start --head  --block  --dashboard-agent-listen-port=52365  --dashboard-host=0.0.0.0  --metrics-export-port=8080  --num-cpus=0  --redis-password=$REDIS_PASSWORD \"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_PORT\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"6379\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_ADDRESS\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"127.0.0.1:6379\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_USAGE_STATS_KUBERAY_IN_USE\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_USAGE_STATS_EXTRA_TAGS\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"kuberay_version=v1.3.0;kuberay_crd=RayCluster\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_DASHBOARD_ENABLE_K8S_DISK_USAGE\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "\u001b[1;39m]\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 7.1: Check the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable in both the head Pod and worker Pod.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "d09b8a7c-1747-45c2-8afa-ad959b66129c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[1;39m[\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_gcs_rpc_server_reconnect_timeout_s\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"600\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"FQ_RAY_IP\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"raycluster-external-redis-head-svc.default.svc.cluster.local\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_IP\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"raycluster-external-redis-head-svc\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_CLUSTER_NAME\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.labels['ray.io/cluster']\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_CLOUD_INSTANCE_ID\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.name\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_NODE_TYPE_NAME\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"valueFrom\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"fieldRef\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "        \u001b[0m\u001b[34;1m\"apiVersion\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"v1\"\u001b[0m\u001b[1;39m,\n",
+      "        \u001b[0m\u001b[34;1m\"fieldPath\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"metadata.labels['ray.io/group']\"\u001b[0m\u001b[1;39m\n",
+      "      \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"KUBERAY_GEN_RAY_START_CMD\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ray start  --address=raycluster-external-redis-head-svc.default.svc.cluster.local:6379  --block  --dashboard-agent-listen-port=52365  --metrics-export-port=8080  --num-cpus=1 \"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_PORT\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"6379\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_ADDRESS\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"raycluster-external-redis-head-svc.default.svc.cluster.local:6379\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_USAGE_STATS_KUBERAY_IN_USE\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"RAY_DASHBOARD_ENABLE_K8S_DISK_USAGE\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1\"\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "\u001b[1;39m]\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "kubectl get pods $YOUR_WORKER_POD -o=jsonpath='{.spec.containers[0].env}' | jq"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "id": "4b56c6fa-fafa-4f4a-8ef0-3513bc49c9f9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 7.2: Kill the GCS process in the head Pod.\n",
+    "kubectl exec -i $HEAD_POD -- pkill gcs_server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "72a46dd5-9305-40c1-b378-456c260c8f36",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "sleep 60\n",
+    "kubectl wait --for=condition=ready pod/$HEAD_POD --timeout=500s\n",
+    "echo $HEAD_POD"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "e2ab3666-3204-4557-acba-c624604bbca2",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS      AGE\n",
+      "raycluster-external-redis-head-xrjff                 1/1     Running   1 (48s ago)   4m41s\n",
+      "raycluster-external-redis-small-group-worker-dwt98   1/1     Running   0             4m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 7.3: The head Pod fails and restarts after `RAY_gcs_rpc_server_reconnect_timeout_s` (60) seconds.\n",
+    "# In addition, the worker Pod isn't terminated by the new head after reconnecting because GCS fault\n",
+    "# tolerance is enabled.\n",
+    "kubectl get pods -l=ray.io/is-ray-node=yes"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dbea855d-a39f-4f3a-9e0d-165eb08858e7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable isn't set in the specifications for either the head Pod or the worker Pod within the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value **600** to the worker Pod and uses the default value **60** for the head Pod.\n",
+    "The timeout value for worker Pods must be longer than the timeout value for the head Pod so that the worker Pods don't terminate before the head Pod restarts from a failure.\n",
+    "\n",
+    "### Step 8: Access the detached actor again\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "id": "d9976f6b-a542-4f66-a45a-4e489aff3ad4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-18 02:53:25,356\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-18 02:53:25,359\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-18 02:53:25,488\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "2\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "615ac036-7b91-42fb-a9ae-efc2e9402d0f",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "```{admonition} The detached actor is always on the worker Pod in this example.\n",
+    "The head Pod's `rayStartParams` is set to `num-cpus: \"0\"`.Hence, no tasks or actors will be scheduled on the head Pod.\n",
+    "```\n",
+    "\n",
+    "With GCS fault tolerance enabled, you can still access the detached actor after the GCS process is dead and restarted.\n",
+    "Note that the fault tolerance doesn't persist the actor's state.\n",
+    "The reason why the result is 2 instead of 1 is that the detached actor is on the worker Pod which is always running.\n",
+    "On the other hand, if the head Pod hosts the detached actor, the `increment_counter.py` script yields a result of 1 in this step.\n",
+    "\n",
+    "### Step 9: Remove the key stored in Redis when deleting RayCluster\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f8b89045-8417-4b70-ba6d-4f38e64d3a43",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 9.1: Delete the RayCluster custom resource.\n",
+    "kubectl delete raycluster raycluster-external-redis"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "141bfbc4-f6f8-405b-a710-7c0a4bd6f64b",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "No resources found in default namespace.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 9.2: KubeRay operator deletes all Pods in the RayCluster.\n",
+    "# Step 9.3: KubeRay operator creates a Kubernetes Job to delete the Redis key after the head Pod is terminated.\n",
+    "# Step 9.4: Check whether the RayCluster has been deleted.\n",
+    "kubectl get raycluster"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3bfb69c2-7ee9-4e7a-a4af-0be328f82b25",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 9.5: Check Redis keys after the Kubernetes Job finishes.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- env REDISCLI_AUTH=\"5241590000000000\" redis-cli KEYS \"*\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5a3db991-73bb-408d-9970-4eb016c97586",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "In KubeRay v1.0.0, the KubeRay operator adds a Kubernetes finalizer to the RayCluster with GCS fault tolerance enabled to ensure Redis cleanup.\n",
+    "KubeRay only removes this finalizer after the Kubernetes Job successfully cleans up Redis.\n",
+    "\n",
+    "* In other words, if the Kubernetes Job fails, the RayCluster won't be deleted. In that case, you should remove the finalizer and cleanup Redis manually.\n",
+    "  ```shell\n",
+    "  kubectl patch rayclusters.ray.io raycluster-external-redis --type json --patch='[ { \"op\": \"remove\", \"path\": \"/metadata/finalizers\" } ]'\n",
+    "  ```\n",
+    "\n",
+    "Starting with KubeRay v1.1.0, KubeRay changes the Redis cleanup behavior from a mandatory to a best-effort basis.\n",
+    "KubeRay still removes the Kubernetes finalizer from the RayCluster if the Kubernetes Job fails, thereby unblocking the deletion of the RayCluster.\n",
+    "\n",
+    "Users can turn off this by setting the feature gate value `ENABLE_GCS_FT_REDIS_CLEANUP`.\n",
+    "Refer to the [KubeRay GCS fault tolerance configurations](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-redis-cleanup-gate) section for more details.\n",
+    "\n",
+    "### Step 10: Delete the Kubernetes cluster\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a2f96238-d408-43a0-93f8-062f43d796a5",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-output",
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind delete cluster"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "823518ab-e787-438e-9d47-5400ae68920a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "## KubeRay GCS fault tolerance configurations\n",
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) used in the quickstart example contains detailed comments about the configuration options.\n",
+    "***Read this section in conjunction with the YAML file.***\n",
+    "\n",
+    "```{admonition} These configurations require KubeRay 1.3.0+\n",
+    "The following section uses the new `gcsFaultToleranceOptions` field introduced in KubeRay 1.3.0. For the old GCS fault tolerance configurations, including the `ray.io/ft-enabled` annotation, please refer to [the old document](https://docs.ray.io/en/releases-2.42.1/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).\n",
+    "```\n",
+    "\n",
+    "### 1. Enable GCS fault tolerance\n",
+    "\n",
+    "* **`gcsFaultToleranceOptions`**: Add `gcsFaultToleranceOptions` field to the RayCluster custom resource to enable GCS fault tolerance.\n",
+    "\n",
+    "  ```yaml\n",
+    "    kind: RayCluster\n",
+    "    metadata:\n",
+    "    spec:\n",
+    "      gcsFaultToleranceOptions: # <- Add this field to enable GCS fault tolerance.\n",
+    "    ```\n",
+    "\n",
+    "### 2. Connect to an external Redis\n",
+    "\n",
+    "* **`redisAddress`**: Add `redisAddress` to the `gcsFaultToleranceOptions` field.\n",
+    "Use this option to specify the address for the Redis service, thus allowing the Ray head to connect to it.\n",
+    "In the [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the RayCluster custom resource uses the `redis` Kubernetes ClusterIP service name as the connection point to the Redis server. The ClusterIP service is also created by the YAML file.\n",
+    "    ```yaml\n",
+    "    kind: RayCluster\n",
+    "    metadata:\n",
+    "    spec:\n",
+    "      gcsFaultToleranceOptions:\n",
+    "        redisAddress: \"redis:6379\" # <- Add redis address here.\n",
+    "    ```\n",
+    "\n",
+    "* **`redisPassword`**: Add `redisPassword` to the `gcsFaultToleranceOptions` field.\n",
+    "Use this option to specify the password for the Redis service, thus allowing the Ray head to connect to it.\n",
+    "In the [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the RayCluster custom resource loads the password from a Kubernetes secret.\n",
+    "    ```yaml\n",
+    "    kind: RayCluster\n",
+    "    metadata:\n",
+    "    spec:\n",
+    "      gcsFaultToleranceOptions:\n",
+    "        redisAddress: \"redis:6379\"\n",
+    "        redisPassword: # <- Add redis password from a Kubernetes secret.\n",
+    "          valueFrom:\n",
+    "            secretKeyRef:\n",
+    "              name: redis-password-secret\n",
+    "              key: password\n",
+    "    ```\n",
+    "\n",
+    "\n",
+    "(kuberay-external-storage-namespace)=\n",
+    "### 3. Use an external storage namespace\n",
+    "\n",
+    "* **`externalStorageNamespace`** (**optional**): Add `externalStorageNamespace` to the `gcsFaultToleranceOptions` field.\n",
+    "KubeRay uses the value of this option to set the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster.\n",
+    "In most cases, ***you don't need to set `externalStorageNamespace`*** because KubeRay automatically sets it to the UID of RayCluster.\n",
+    "Only modify this option if you fully understand the behaviors of the GCS fault tolerance and RayService to avoid [this issue](https://docs.ray.io/en/latest/cluster/kubernetes/troubleshooting/rayservice-troubleshooting.html#kuberay-raysvc-issue10).\n",
+    "Refer to [this section](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-external-storage-namespace-example) in the earlier quickstart example for more details.\n",
+    "    ```yaml\n",
+    "    kind: RayCluster\n",
+    "    metadata:\n",
+    "    spec:\n",
+    "      gcsFaultToleranceOptions:\n",
+    "        externalStorageNamespace: \"my-raycluster-storage\" # <- Add this option to specify a storage namespace\n",
+    "    ```\n",
+    "\n",
+    "(kuberay-redis-cleanup-gate)=\n",
+    "### 4. Turn off Redis cleanup\n",
+    "\n",
+    "* `ENABLE_GCS_FT_REDIS_CLEANUP`: True by default. You can turn this feature off by setting the environment variable in the [KubeRay operator's Helm chart](https://github.com/ray-project/kuberay/blob/master/helm-chart/kuberay-operator/values.yaml).\n",
+    "\n",
+    "```{admonition} Key eviction setup on Redis\n",
+    "If you disable `ENABLE_GCS_FT_REDIS_CLEANUP` but want Redis to remove GCS metadata automatically,\n",
+    "set these two options in your `redis.conf` or in the command line options of your redis-server command [(example)](https://github.com/ray-project/ray/pull/40949#issuecomment-1799057691):\n",
+    "\n",
+    "* `maxmemory=<your_memory_limit>`\n",
+    "* `maxmemory-policy=allkeys-lru`\n",
+    "\n",
+    "These two options instruct Redis to delete the least recently used keys when it reaches the `maxmemory` limit.\n",
+    "See [Key eviction](https://redis.io/docs/reference/eviction/) from Redis for more information.\n",
+    "\n",
+    "Note that Redis does this eviction and it doesn't guarantee that Ray won't use the deleted keys.\n",
+    "```\n",
+    "\n",
+    "## Next steps\n",
+    "\n",
+    "* See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "* See [Ray Core GCS fault tolerance documentation](https://docs.ray.io/en/latest/ray-core/fault_tolerance/gcs.html#fault-tolerance-gcs) for more information.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Bash",
+   "language": "bash",
+   "name": "bash"
+  },
+  "language_info": {
+   "codemirror_mode": "shell",
+   "file_extension": ".sh",
+   "mimetype": "text/x-sh",
+   "name": "bash"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.md b/doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.md
deleted file mode 100644
index b77e63b549b5..000000000000
--- a/doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.md
+++ /dev/null
@@ -1,349 +0,0 @@
-(kuberay-gcs-ft)=
-# GCS fault tolerance in KubeRay
-
-Global Control Service (GCS) manages cluster-level metadata.
-By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.
-To make the GCS fault tolerant, you must have a high-availability Redis.
-This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.
-
-```{admonition} Fate-sharing without GCS fault tolerance
-Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing.
-If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds.
-If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod.
-The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as "unknown workers" by the new head Pod.
-This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases.
-Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability.
-See {ref}`Ray Serve end-to-end fault tolerance documentation <serve-e2e-ft-guide-gcs>` for more information.
-```
-
-```{seealso}
-If you need fault tolerance for Redis as well, see {ref}`Tuning Redis for a
-Persistent Fault Tolerant GCS <kuberay-gcs-persistent-ft>`.
-```
-
-## Use cases
-
-* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.
-See {ref}`Ray Serve end-to-end fault tolerance documentation <serve-e2e-ft-guide-gcs>` for more information.
-
-* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.
-
-## Prerequisites
-
-* Ray 2.0.0+
-* KubeRay 1.3.0+
-* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas
-
-## Quickstart
-
-### Step 1: Create a Kubernetes cluster with Kind
-
-```sh
-kind create cluster --image=kindest/node:v1.26.0
-```
-
-### Step 2: Install the KubeRay operator
-
-Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository.
-
-### Step 3: Install a RayCluster with GCS FT enabled
-
-```sh
-curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml
-kubectl apply -f ray-cluster.external-redis.yaml
-```
-
-### Step 4: Verify the Kubernetes cluster status
-
-```sh
-# Step 4.1: List all Pods in the `default` namespace.
-# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.
-kubectl get pods
-
-# Step 4.2: List all ConfigMaps in the `default` namespace.
-kubectl get configmaps
-
-# [Example output]
-# NAME                  DATA   AGE
-# ray-example           2      5m45s
-# redis-config          1      5m45s
-# ...
-```
-
-The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.
-There are two ConfigMaps in this example: `ray-example` and `redis-config`.
-The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.
-
-* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.
-    ```python
-    import ray
-
-    @ray.remote(num_cpus=1)
-    class Counter:
-        def __init__(self):
-            self.value = 0
-
-        def increment(self):
-            self.value += 1
-            return self.value
-
-    ray.init(namespace="default_namespace")
-    Counter.options(name="counter_actor", lifetime="detached").remote()
-    ```
-
-* `increment_counter.py` is a Python script that increments the counter.
-    ```python
-    import ray
-
-    ray.init(namespace="default_namespace")
-    counter = ray.get_actor("counter_actor")
-    print(ray.get(counter.increment.remote()))
-    ```
-
-### Step 5: Create a detached actor
-
-```sh
-# Step 4.1: Create a detached actor with the name, `counter_actor`.
-export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
-kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py
-
-# Step 4.2: Increment the counter.
-kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py
-
-# 2023-09-07 16:01:41,925 INFO worker.py:1313 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS
-# 2023-09-07 16:01:41,926 INFO worker.py:1431 -- Connecting to existing Ray cluster at address: 10.244.0.17:6379...
-# 2023-09-07 16:01:42,000 INFO worker.py:1612 -- Connected to Ray cluster. View the dashboard at http://10.244.0.17:8265
-# 1
-```
-
-(kuberay-external-storage-namespace-example)=
-### Step 6: Check the data in Redis
-
-```sh
-# Step 6.1: Check the RayCluster's UID.
-kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'
-# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752
-
-# Step 6.2: Check the head Pod's environment variable `RAY_external_storage_namespace`.
-kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq
-# [Example output]:
-# [
-#   {
-#     "name": "RAY_external_storage_namespace",
-#     "value": "864b004c-6305-42e3-ac46-adfa8eb6f752"
-#   },
-#   ...
-# ]
-
-# Step 6.3: Log into the Redis Pod.
-export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)
-# The password `5241590000000000` is defined in the `redis-config` ConfigMap.
-kubectl exec -it $REDIS_POD -- redis-cli -a "5241590000000000"
-
-# Step 6.4: Check the keys in Redis.
-# Note: the schema changed in Ray 2.38.0. Previously we use a single HASH table,
-# now we use multiple HASH tables with a common prefix.
-
-KEYS *
-# [Example output]:
-# 1) "RAY864b004c-6305-42e3-ac46-adfa8eb6f752@INTERNAL_CONFIG"
-# 2) "RAY864b004c-6305-42e3-ac46-adfa8eb6f752@KV"
-# 3) "RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE"
-# [Example output Before Ray 2.38.0]:
-# 2) "864b004c-6305-42e3-ac46-adfa8eb6f752"
-#
-
-# Step 6.5: Check the value of the key.
-HGETALL RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE
-# Before Ray 2.38.0:
-# HGETALL 864b004c-6305-42e3-ac46-adfa8eb6f752
-```
-
-In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `gcsFaultToleranceOptions.externalStorageNamespace` option isn't set for the RayCluster.
-Therefore, KubeRay automatically injects the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster with the RayCluster's UID as the external storage namespace by default.
-See [this section](kuberay-external-storage-namespace) to learn more about the option.
-
-### Step 7: Kill the GCS process in the head Pod
-
-```sh
-# Step 7.1: Check the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable in both the head Pod and worker Pod.
-kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq
-# [Expected result]:
-# No `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable is set. Hence, the Ray head uses its default value of `60`.
-kubectl get pods $YOUR_WORKER_POD -o=jsonpath='{.spec.containers[0].env}' | jq
-# [Expected result]:
-# KubeRay injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value `600` to the worker Pod.
-# [
-#   {
-#     "name": "RAY_gcs_rpc_server_reconnect_timeout_s",
-#     "value": "600"
-#   },
-#   ...
-# ]
-
-# Step 7.2: Kill the GCS process in the head Pod.
-kubectl exec -it $HEAD_POD -- pkill gcs_server
-
-# Step 7.3: The head Pod fails and restarts after `RAY_gcs_rpc_server_reconnect_timeout_s` (60) seconds.
-# In addition, the worker Pod isn't terminated by the new head after reconnecting because GCS fault
-# tolerance is enabled.
-kubectl get pods -l=ray.io/is-ray-node=yes
-# [Example output]:
-# NAME                                                 READY   STATUS    RESTARTS      AGE
-# raycluster-external-redis-head-xxxxx                 1/1     Running   1 (64s ago)   xxm
-# raycluster-external-redis-worker-small-group-yyyyy   1/1     Running   0             xxm
-```
-
-In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable isn't set in the specifications for either the head Pod or the worker Pod within the RayCluster.
-Therefore, KubeRay automatically injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value **600** to the worker Pod and uses the default value **60** for the head Pod.
-The timeout value for worker Pods must be longer than the timeout value for the head Pod so that the worker Pods don't terminate before the head Pod restarts from a failure.
-
-### Step 8: Access the detached actor again
-
-```sh
-kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py
-# 2023-09-07 17:31:17,793 INFO worker.py:1313 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS
-# 2023-09-07 17:31:17,793 INFO worker.py:1431 -- Connecting to existing Ray cluster at address: 10.244.0.21:6379...
-# 2023-09-07 17:31:17,875 INFO worker.py:1612 -- Connected to Ray cluster. View the dashboard at http://10.244.0.21:8265
-# 2
-```
-
-```{admonition} The detached actor is always on the worker Pod in this example.
-The head Pod's `rayStartParams` is set to `num-cpus: "0"`.
-Hence, no tasks or actors will be scheduled on the head Pod.
-```
-
-With GCS fault tolerance enabled, you can still access the detached actor after the GCS process is dead and restarted.
-Note that the fault tolerance doesn't persist the actor's state.
-The reason why the result is 2 instead of 1 is that the detached actor is on the worker Pod which is always running.
-On the other hand, if the head Pod hosts the detached actor, the `increment_counter.py` script yields a result of 1 in this step.
-
-### Step 9: Remove the key stored in Redis when deleting RayCluster
-
-```shell
-# Step 9.1: Delete the RayCluster custom resource.
-kubectl delete raycluster raycluster-external-redis
-
-# Step 9.2: KubeRay operator deletes all Pods in the RayCluster.
-# Step 9.3: KubeRay operator creates a Kubernetes Job to delete the Redis key after the head Pod is terminated.
-
-# Step 9.4: Check whether the RayCluster has been deleted.
-kubectl get raycluster
-# [Expected output]: No resources found in default namespace.
-
-# Step 9.5: Check Redis keys after the Kubernetes Job finishes.
-export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)
-kubectl exec -it $REDIS_POD -- redis-cli -a "5241590000000000"
-KEYS *
-# [Expected output]: (empty list or set)
-```
-
-In KubeRay v1.0.0, the KubeRay operator adds a Kubernetes finalizer to the RayCluster with GCS fault tolerance enabled to ensure Redis cleanup.
-KubeRay only removes this finalizer after the Kubernetes Job successfully cleans up Redis.
-
-* In other words, if the Kubernetes Job fails, the RayCluster won't be deleted. In that case, you should remove the finalizer and cleanup Redis manually.
-  ```shell
-  kubectl patch rayclusters.ray.io raycluster-external-redis --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
-  ```
-
-Starting with KubeRay v1.1.0, KubeRay changes the Redis cleanup behavior from a mandatory to a best-effort basis.
-KubeRay still removes the Kubernetes finalizer from the RayCluster if the Kubernetes Job fails, thereby unblocking the deletion of the RayCluster.
-
-Users can turn off this by setting the feature gate value `ENABLE_GCS_FT_REDIS_CLEANUP`.
-Refer to the [KubeRay GCS fault tolerance configurations](kuberay-redis-cleanup-gate) section for more details.
-
-### Step 10: Delete the Kubernetes cluster
-
-```sh
-kind delete cluster
-```
-
-## KubeRay GCS fault tolerance configurations
-
-The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) used in the quickstart example contains detailed comments about the configuration options.
-***Read this section in conjunction with the YAML file.***
-
-```{admonition} These configurations require KubeRay 1.3.0+
-The following section uses the new `gcsFaultToleranceOptions` field introduced in KubeRay 1.3.0.
-For the old GCS fault tolerance configurations, including the `ray.io/ft-enabled` annotation, please refer to [the old document](https://docs.ray.io/en/releases-2.42.1/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-```
-
-### 1. Enable GCS fault tolerance
-
-* **`gcsFaultToleranceOptions`**: Add `gcsFaultToleranceOptions` field to the RayCluster custom resource to enable GCS fault tolerance.
-    ```yaml
-    kind: RayCluster
-    metadata:
-    spec:
-      gcsFaultToleranceOptions: # <- Add this field to enable GCS fault tolerance.
-    ```
-
-### 2. Connect to an external Redis
-
-* **`redisAddress`**: Add `redisAddress` to the `gcsFaultToleranceOptions` field.
-Use this option to specify the address for the Redis service, thus allowing the Ray head to connect to it.
-In the [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the RayCluster custom resource uses the `redis` Kubernetes ClusterIP service name as the connection point to the Redis server. The ClusterIP service is also created by the YAML file.
-    ```yaml
-    kind: RayCluster
-    metadata:
-    spec:
-      gcsFaultToleranceOptions:
-        redisAddress: "redis:6379" # <- Add redis address here.
-    ```
-
-* **`redisPassword`**: Add `redisPassword` to the `gcsFaultToleranceOptions` field.
-Use this option to specify the password for the Redis service, thus allowing the Ray head to connect to it.
-In the [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the RayCluster custom resource loads the password from a Kubernetes secret.
-    ```yaml
-    kind: RayCluster
-    metadata:
-    spec:
-      gcsFaultToleranceOptions:
-        redisAddress: "redis:6379"
-        redisPassword: # <- Add redis password from a Kubernetes secret.
-          valueFrom:
-            secretKeyRef:
-              name: redis-password-secret
-              key: password
-    ```
-
-
-(kuberay-external-storage-namespace)=
-### 3. Use an external storage namespace
-
-* **`externalStorageNamespace`** (**optional**): Add `externalStorageNamespace` to the `gcsFaultToleranceOptions` field.
-KubeRay uses the value of this option to set the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster.
-In most cases, ***you don't need to set `externalStorageNamespace`*** because KubeRay automatically sets it to the UID of RayCluster.
-Only modify this option if you fully understand the behaviors of the GCS fault tolerance and RayService to avoid [this issue](kuberay-raysvc-issue10).
-Refer to [this section](kuberay-external-storage-namespace-example) in the earlier quickstart example for more details.
-    ```yaml
-    kind: RayCluster
-    metadata:
-    spec:
-      gcsFaultToleranceOptions:
-        externalStorageNamespace: "my-raycluster-storage" # <- Add this option to specify a storage namespace
-    ```
-
-(kuberay-redis-cleanup-gate)=
-### 4. Turn off Redis cleanup
-
-* `ENABLE_GCS_FT_REDIS_CLEANUP`: True by default. You can turn this feature off by setting the environment variable in the [KubeRay operator's Helm chart](https://github.com/ray-project/kuberay/blob/master/helm-chart/kuberay-operator/values.yaml).
-
-```{admonition} Key eviction setup on Redis
-If you disable `ENABLE_GCS_FT_REDIS_CLEANUP` but want Redis to remove GCS metadata automatically,
-set these two options in your `redis.conf` or in the command line options of your redis-server command [(example)](https://github.com/ray-project/ray/pull/40949#issuecomment-1799057691):
-
-* `maxmemory=<your_memory_limit>`
-* `maxmemory-policy=allkeys-lru`
-
-These two options instruct Redis to delete the least recently used keys when it reaches the `maxmemory` limit.
-See [Key eviction](https://redis.io/docs/reference/eviction/) from Redis for more information.
-
-Note that Redis does this eviction and it doesn't guarantee that
-Ray won't use the deleted keys.
-```
-
-## Next steps
-
-* See {ref}`Ray Serve end-to-end fault tolerance documentation <serve-e2e-ft-guide-gcs>` for more information.
-* See {ref}`Ray Core GCS fault tolerance documentation <fault-tolerance-gcs>` for more information.


Reviews:
user: login: MortalHappiness
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T17:05:30Z
user: login: MortalHappiness
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-15T09:33:30Z
user: login: MortalHappiness
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-22T15:55:03Z
user: login: kevin85421
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-25T03:38:42Z

Review comments:
diff_hunk: @@ -0,0 +1,760 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-27zhj                    1/1     Running   0          2m50s\n",
+      "raycluster-external-redis-head-28zqk                 1/1     Running   0          2m26s\n",
+      "raycluster-external-redis-small-group-worker-tn67c   1/1     Running   0          2m26s\n",
+      "redis-6cf756c755-n42cl                               1/1     Running   0          2m26s\n",
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      2m51s\n",
+      "ray-example        2      2m26s\n",
+      "redis-config       1      2m26s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods\n",
+    "\n",
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
path: doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
user: login: MortalHappiness
body: Split into 2 cells.
created_at: 2025-04-09T17:03:39Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,760 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-27zhj                    1/1     Running   0          2m50s\n",
+      "raycluster-external-redis-head-28zqk                 1/1     Running   0          2m26s\n",
+      "raycluster-external-redis-small-group-worker-tn67c   1/1     Running   0          2m26s\n",
+      "redis-6cf756c755-n42cl                               1/1     Running   0          2m26s\n",
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      2m51s\n",
+      "ray-example        2      2m26s\n",
+      "redis-config       1      2m26s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods\n",
+    "\n",
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9d28f87-6e4e-4736-b3d3-b26888955d66",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.\n",
+    "There are two ConfigMaps in this example: `ray-example` and `redis-config`.\n",
+    "The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.\n",
+    "\n",
+    "* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    @ray.remote(num_cpus=1)\n",
+    "    class Counter:\n",
+    "        def __init__(self):\n",
+    "            self.value = 0\n",
+    "\n",
+    "        def increment(self):\n",
+    "            self.value += 1\n",
+    "            return self.value\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    Counter.options(name=\"counter_actor\", lifetime=\"detached\").remote()\n",
+    "    ```\n",
+    "\n",
+    "* `increment_counter.py` is a Python script that increments the counter.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    counter = ray.get_actor(\"counter_actor\")\n",
+    "    print(ray.get(counter.increment.remote()))\n",
+    "    ```\n",
+    "\n",
+    "### Step 5: Create a detached actor\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "978b889d-66af-408a-bd24-c36273edadb9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-07 20:57:53,143\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-07 20:57:53,146\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...\n",
+      "2025-04-07 20:57:53,296\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.6:8265 \u001b[39m\u001b[22m\n",
+      "2025-04-07 20:57:56,528\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-07 20:57:56,531\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...\n",
+      "2025-04-07 20:57:56,686\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.6:8265 \u001b[39m\u001b[22m\n",
+      "1\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 5.1: Create a detached actor with the name, `counter_actor`.\n",
+    "export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py\n",
+    "# Step 5.2: Increment the counter.\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a1817cd-4380-4c28-b867-a3f55cc20910",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "(kuberay-external-storage-namespace-example)=\n",
+    "### Step 6: Check the data in Redis\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0454a9bb-4908-4c69-95c9-b2d1c2eb42cc",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.1: Check the RayCluster's UID.\n",
+    "kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'\n",
+    "# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752\n",
+    "\n",
+    "# Step 6.2: Check the head Pod's environment variable `RAY_external_storage_namespace`.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Example output]:\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_external_storage_namespace\",\n",
+    "#     \"value\": \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]\n",
+    "\n",
+    "# Step 6.3: Log into the Redis Pod.\n",
+    "# The password `5241590000000000` is defined in the `redis-config` ConfigMap.\n",
+    "# Step 6.4: Check the keys in Redis.\n",
+    "# Note: the schema changed in Ray 2.38.0. Previously we use a single HASH table,\n",
+    "# now we use multiple HASH tables with a common prefix.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" KEYS '*'\n",
+    "# [Example output]:\n",
+    "# 1) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@INTERNAL_CONFIG\"\n",
+    "# 2) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@KV\"\n",
+    "# 3) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\"\n",
+    "# [Example output Before Ray 2.38.0]:\n",
+    "# 2) \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#\n",
+    "\n",
+    "# Step 6.5: Check the value of the key.\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" HGETALL RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\n",
+    "# Before Ray 2.38.0:\n",
+    "# HGETALL 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
path: doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
user: login: MortalHappiness
body: Split into multiple cells.
created_at: 2025-04-09T17:04:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,760 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-27zhj                    1/1     Running   0          2m50s\n",
+      "raycluster-external-redis-head-28zqk                 1/1     Running   0          2m26s\n",
+      "raycluster-external-redis-small-group-worker-tn67c   1/1     Running   0          2m26s\n",
+      "redis-6cf756c755-n42cl                               1/1     Running   0          2m26s\n",
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      2m51s\n",
+      "ray-example        2      2m26s\n",
+      "redis-config       1      2m26s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods\n",
+    "\n",
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9d28f87-6e4e-4736-b3d3-b26888955d66",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.\n",
+    "There are two ConfigMaps in this example: `ray-example` and `redis-config`.\n",
+    "The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.\n",
+    "\n",
+    "* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    @ray.remote(num_cpus=1)\n",
+    "    class Counter:\n",
+    "        def __init__(self):\n",
+    "            self.value = 0\n",
+    "\n",
+    "        def increment(self):\n",
+    "            self.value += 1\n",
+    "            return self.value\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    Counter.options(name=\"counter_actor\", lifetime=\"detached\").remote()\n",
+    "    ```\n",
+    "\n",
+    "* `increment_counter.py` is a Python script that increments the counter.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    counter = ray.get_actor(\"counter_actor\")\n",
+    "    print(ray.get(counter.increment.remote()))\n",
+    "    ```\n",
+    "\n",
+    "### Step 5: Create a detached actor\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "978b889d-66af-408a-bd24-c36273edadb9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-07 20:57:53,143\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-07 20:57:53,146\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...\n",
+      "2025-04-07 20:57:53,296\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.6:8265 \u001b[39m\u001b[22m\n",
+      "2025-04-07 20:57:56,528\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-07 20:57:56,531\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...\n",
+      "2025-04-07 20:57:56,686\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.6:8265 \u001b[39m\u001b[22m\n",
+      "1\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 5.1: Create a detached actor with the name, `counter_actor`.\n",
+    "export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py\n",
+    "# Step 5.2: Increment the counter.\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a1817cd-4380-4c28-b867-a3f55cc20910",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "(kuberay-external-storage-namespace-example)=\n",
+    "### Step 6: Check the data in Redis\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0454a9bb-4908-4c69-95c9-b2d1c2eb42cc",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.1: Check the RayCluster's UID.\n",
+    "kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'\n",
+    "# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752\n",
+    "\n",
+    "# Step 6.2: Check the head Pod's environment variable `RAY_external_storage_namespace`.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Example output]:\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_external_storage_namespace\",\n",
+    "#     \"value\": \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]\n",
+    "\n",
+    "# Step 6.3: Log into the Redis Pod.\n",
+    "# The password `5241590000000000` is defined in the `redis-config` ConfigMap.\n",
+    "# Step 6.4: Check the keys in Redis.\n",
+    "# Note: the schema changed in Ray 2.38.0. Previously we use a single HASH table,\n",
+    "# now we use multiple HASH tables with a common prefix.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" KEYS '*'\n",
+    "# [Example output]:\n",
+    "# 1) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@INTERNAL_CONFIG\"\n",
+    "# 2) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@KV\"\n",
+    "# 3) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\"\n",
+    "# [Example output Before Ray 2.38.0]:\n",
+    "# 2) \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#\n",
+    "\n",
+    "# Step 6.5: Check the value of the key.\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" HGETALL RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\n",
+    "# Before Ray 2.38.0:\n",
+    "# HGETALL 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a67ff02-03d2-45df-ad5c-fbe3f72a3a1e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `gcsFaultToleranceOptions.externalStorageNamespace` option isn't set for the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster with the RayCluster's UID as the external storage namespace by default.\n",
+    "See [this section](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-external-storage-namespace) to learn more about the option.\n",
+    "\n",
+    "### Step 7: Kill the GCS process in the head Pod\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "297712df-81e6-40b8-80e4-cb43fd125dfd",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "export YOUR_WORKER_POD=$(kubectl get pods -l ray.io/group=small-group -o jsonpath='{.items[0].metadata.name}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "775d8da9-cea7-4f93-adec-653520af37d3",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 7.1: Check the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable in both the head Pod and worker Pod.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# No `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable is set. Hence, the Ray head uses its default value of `60`.\n",
+    "kubectl get pods $YOUR_WORKER_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# KubeRay injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value `600` to the worker Pod.\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_gcs_rpc_server_reconnect_timeout_s\",\n",
+    "#     \"value\": \"600\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]\n",
+    "\n",
+    "# Step 7.2: Kill the GCS process in the head Pod.\n",
+    "kubectl exec -i $HEAD_POD -- pkill gcs_server"
+   ]
path: doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
user: login: MortalHappiness
body: ditto
created_at: 2025-04-09T17:04:55Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,760 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-27zhj                    1/1     Running   0          2m50s\n",
+      "raycluster-external-redis-head-28zqk                 1/1     Running   0          2m26s\n",
+      "raycluster-external-redis-small-group-worker-tn67c   1/1     Running   0          2m26s\n",
+      "redis-6cf756c755-n42cl                               1/1     Running   0          2m26s\n",
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      2m51s\n",
+      "ray-example        2      2m26s\n",
+      "redis-config       1      2m26s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods\n",
+    "\n",
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9d28f87-6e4e-4736-b3d3-b26888955d66",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.\n",
+    "There are two ConfigMaps in this example: `ray-example` and `redis-config`.\n",
+    "The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.\n",
+    "\n",
+    "* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    @ray.remote(num_cpus=1)\n",
+    "    class Counter:\n",
+    "        def __init__(self):\n",
+    "            self.value = 0\n",
+    "\n",
+    "        def increment(self):\n",
+    "            self.value += 1\n",
+    "            return self.value\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    Counter.options(name=\"counter_actor\", lifetime=\"detached\").remote()\n",
+    "    ```\n",
+    "\n",
+    "* `increment_counter.py` is a Python script that increments the counter.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    counter = ray.get_actor(\"counter_actor\")\n",
+    "    print(ray.get(counter.increment.remote()))\n",
+    "    ```\n",
+    "\n",
+    "### Step 5: Create a detached actor\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "978b889d-66af-408a-bd24-c36273edadb9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-07 20:57:53,143\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-07 20:57:53,146\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...\n",
+      "2025-04-07 20:57:53,296\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.6:8265 \u001b[39m\u001b[22m\n",
+      "2025-04-07 20:57:56,528\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-07 20:57:56,531\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...\n",
+      "2025-04-07 20:57:56,686\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.6:8265 \u001b[39m\u001b[22m\n",
+      "1\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 5.1: Create a detached actor with the name, `counter_actor`.\n",
+    "export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py\n",
+    "# Step 5.2: Increment the counter.\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a1817cd-4380-4c28-b867-a3f55cc20910",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "(kuberay-external-storage-namespace-example)=\n",
+    "### Step 6: Check the data in Redis\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0454a9bb-4908-4c69-95c9-b2d1c2eb42cc",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.1: Check the RayCluster's UID.\n",
+    "kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'\n",
+    "# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752\n",
+    "\n",
+    "# Step 6.2: Check the head Pod's environment variable `RAY_external_storage_namespace`.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Example output]:\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_external_storage_namespace\",\n",
+    "#     \"value\": \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]\n",
+    "\n",
+    "# Step 6.3: Log into the Redis Pod.\n",
+    "# The password `5241590000000000` is defined in the `redis-config` ConfigMap.\n",
+    "# Step 6.4: Check the keys in Redis.\n",
+    "# Note: the schema changed in Ray 2.38.0. Previously we use a single HASH table,\n",
+    "# now we use multiple HASH tables with a common prefix.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" KEYS '*'\n",
+    "# [Example output]:\n",
+    "# 1) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@INTERNAL_CONFIG\"\n",
+    "# 2) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@KV\"\n",
+    "# 3) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\"\n",
+    "# [Example output Before Ray 2.38.0]:\n",
+    "# 2) \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#\n",
+    "\n",
+    "# Step 6.5: Check the value of the key.\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" HGETALL RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\n",
+    "# Before Ray 2.38.0:\n",
+    "# HGETALL 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a67ff02-03d2-45df-ad5c-fbe3f72a3a1e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `gcsFaultToleranceOptions.externalStorageNamespace` option isn't set for the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster with the RayCluster's UID as the external storage namespace by default.\n",
+    "See [this section](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-external-storage-namespace) to learn more about the option.\n",
+    "\n",
+    "### Step 7: Kill the GCS process in the head Pod\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "297712df-81e6-40b8-80e4-cb43fd125dfd",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "export YOUR_WORKER_POD=$(kubectl get pods -l ray.io/group=small-group -o jsonpath='{.items[0].metadata.name}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "775d8da9-cea7-4f93-adec-653520af37d3",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 7.1: Check the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable in both the head Pod and worker Pod.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# No `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable is set. Hence, the Ray head uses its default value of `60`.\n",
+    "kubectl get pods $YOUR_WORKER_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# KubeRay injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value `600` to the worker Pod.\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_gcs_rpc_server_reconnect_timeout_s\",\n",
+    "#     \"value\": \"600\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]\n",
+    "\n",
+    "# Step 7.2: Kill the GCS process in the head Pod.\n",
+    "kubectl exec -i $HEAD_POD -- pkill gcs_server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "72a46dd5-9305-40c1-b378-456c260c8f36",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "sleep 60\n",
+    "kubectl wait --for=condition=ready pod/$HEAD_POD --timeout=500s\n",
+    "echo $HEAD_POD"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "e2ab3666-3204-4557-acba-c624604bbca2",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS      AGE\n",
+      "raycluster-external-redis-head-28zqk                 1/1     Running   1 (43s ago)   4m22s\n",
+      "raycluster-external-redis-small-group-worker-tn67c   1/1     Running   0             4m22s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 7.3: The head Pod fails and restarts after `RAY_gcs_rpc_server_reconnect_timeout_s` (60) seconds.\n",
+    "# In addition, the worker Pod isn't terminated by the new head after reconnecting because GCS fault\n",
+    "# tolerance is enabled.\n",
+    "kubectl get pods -l=ray.io/is-ray-node=yes"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dbea855d-a39f-4f3a-9e0d-165eb08858e7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable isn't set in the specifications for either the head Pod or the worker Pod within the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value **600** to the worker Pod and uses the default value **60** for the head Pod.\n",
+    "The timeout value for worker Pods must be longer than the timeout value for the head Pod so that the worker Pods don't terminate before the head Pod restarts from a failure.\n",
+    "\n",
+    "### Step 8: Access the detached actor again\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "d9976f6b-a542-4f66-a45a-4e489aff3ad4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-07 20:59:48,540\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-07 20:59:48,543\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...\n",
+      "2025-04-07 20:59:48,681\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.6:8265 \u001b[39m\u001b[22m\n",
+      "2\n"
+     ]
+    }
+   ],
+   "source": [
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "615ac036-7b91-42fb-a9ae-efc2e9402d0f",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "```{admonition} The detached actor is always on the worker Pod in this example.\n",
+    "The head Pod's `rayStartParams` is set to `num-cpus: \"0\"`.Hence, no tasks or actors will be scheduled on the head Pod.\n",
+    "```\n",
+    "\n",
+    "With GCS fault tolerance enabled, you can still access the detached actor after the GCS process is dead and restarted.\n",
+    "Note that the fault tolerance doesn't persist the actor's state.\n",
+    "The reason why the result is 2 instead of 1 is that the detached actor is on the worker Pod which is always running.\n",
+    "On the other hand, if the head Pod hosts the detached actor, the `increment_counter.py` script yields a result of 1 in this step.\n",
+    "\n",
+    "### Step 9: Remove the key stored in Redis when deleting RayCluster\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f8b89045-8417-4b70-ba6d-4f38e64d3a43",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 9.1: Delete the RayCluster custom resource.\n",
+    "kubectl delete raycluster raycluster-external-redis\n",
+    "\n",
+    "# Step 9.2: KubeRay operator deletes all Pods in the RayCluster.\n",
+    "# Step 9.3: KubeRay operator creates a Kubernetes Job to delete the Redis key after the head Pod is terminated.\n",
+    "\n",
+    "# Step 9.4: Check whether the RayCluster has been deleted.\n",
+    "kubectl get raycluster\n",
+    "# [Expected output]: No resources found in default namespace.\n",
+    "\n",
+    "# Step 9.5: Check Redis keys after the Kubernetes Job finishes.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" KEYS \"*\"\n",
+    "# [Expected output]: (empty list or set)"
+   ]
path: doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
user: login: MortalHappiness
body: ditto
created_at: 2025-04-09T17:05:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,900 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-vpcgj                    1/1     Running   0          3m9s\n",
+      "raycluster-external-redis-head-vrsbq                 1/1     Running   0          2m41s\n",
+      "raycluster-external-redis-small-group-worker-rx25r   1/1     Running   0          2m41s\n",
+      "redis-6cf756c755-czs8r                               1/1     Running   0          2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "76aa215a-7a63-4b79-8d3c-f5b674ad8fdb",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      3m10s\n",
+      "ray-example        2      2m41s\n",
+      "redis-config       1      2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9d28f87-6e4e-4736-b3d3-b26888955d66",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.\n",
+    "There are two ConfigMaps in this example: `ray-example` and `redis-config`.\n",
+    "The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.\n",
+    "\n",
+    "* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    @ray.remote(num_cpus=1)\n",
+    "    class Counter:\n",
+    "        def __init__(self):\n",
+    "            self.value = 0\n",
+    "\n",
+    "        def increment(self):\n",
+    "            self.value += 1\n",
+    "            return self.value\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    Counter.options(name=\"counter_actor\", lifetime=\"detached\").remote()\n",
+    "    ```\n",
+    "\n",
+    "* `increment_counter.py` is a Python script that increments the counter.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    counter = ray.get_actor(\"counter_actor\")\n",
+    "    print(ray.get(counter.increment.remote()))\n",
+    "    ```\n",
+    "\n",
+    "### Step 5: Create a detached actor\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "978b889d-66af-408a-bd24-c36273edadb9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-14 18:53:36,998\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:53:37,001\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:53:37,158\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "2025-04-14 18:53:40,273\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:53:40,276\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:53:40,607\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "1\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 5.1: Create a detached actor with the name, `counter_actor`.\n",
+    "export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py\n",
+    "# Step 5.2: Increment the counter.\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a1817cd-4380-4c28-b867-a3f55cc20910",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "(kuberay-external-storage-namespace-example)=\n",
+    "### Step 6: Check the data in Redis\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0454a9bb-4908-4c69-95c9-b2d1c2eb42cc",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.1: Check the RayCluster's UID.\n",
+    "kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'\n",
+    "# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d8e5d436-f9a4-44ef-86e4-635431caa34e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
path: doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
user: login: MortalHappiness
body: ```suggestion
    "tags": [
     "nbval-ignore-output"
    ]
```
Ignore `nbval` for this cell but don't remove output for this cell. Also removes the `# [Example output]` above. ditto for step 6.2, 6.3, 6.4, 6.5, 7.1, 9.4, 9.5.
created_at: 2025-04-15T09:27:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,900 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-vpcgj                    1/1     Running   0          3m9s\n",
+      "raycluster-external-redis-head-vrsbq                 1/1     Running   0          2m41s\n",
+      "raycluster-external-redis-small-group-worker-rx25r   1/1     Running   0          2m41s\n",
+      "redis-6cf756c755-czs8r                               1/1     Running   0          2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "76aa215a-7a63-4b79-8d3c-f5b674ad8fdb",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      3m10s\n",
+      "ray-example        2      2m41s\n",
+      "redis-config       1      2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9d28f87-6e4e-4736-b3d3-b26888955d66",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.\n",
+    "There are two ConfigMaps in this example: `ray-example` and `redis-config`.\n",
+    "The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.\n",
+    "\n",
+    "* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    @ray.remote(num_cpus=1)\n",
+    "    class Counter:\n",
+    "        def __init__(self):\n",
+    "            self.value = 0\n",
+    "\n",
+    "        def increment(self):\n",
+    "            self.value += 1\n",
+    "            return self.value\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    Counter.options(name=\"counter_actor\", lifetime=\"detached\").remote()\n",
+    "    ```\n",
+    "\n",
+    "* `increment_counter.py` is a Python script that increments the counter.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    counter = ray.get_actor(\"counter_actor\")\n",
+    "    print(ray.get(counter.increment.remote()))\n",
+    "    ```\n",
+    "\n",
+    "### Step 5: Create a detached actor\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "978b889d-66af-408a-bd24-c36273edadb9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-14 18:53:36,998\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:53:37,001\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:53:37,158\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "2025-04-14 18:53:40,273\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:53:40,276\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:53:40,607\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "1\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 5.1: Create a detached actor with the name, `counter_actor`.\n",
+    "export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py\n",
+    "# Step 5.2: Increment the counter.\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a1817cd-4380-4c28-b867-a3f55cc20910",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "(kuberay-external-storage-namespace-example)=\n",
+    "### Step 6: Check the data in Redis\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0454a9bb-4908-4c69-95c9-b2d1c2eb42cc",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.1: Check the RayCluster's UID.\n",
+    "kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'\n",
+    "# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d8e5d436-f9a4-44ef-86e4-635431caa34e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.2: Check the head Pod's environment variable `RAY_external_storage_namespace`.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Example output]:\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_external_storage_namespace\",\n",
+    "#     \"value\": \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dcbba87e-0f9b-4997-ba71-08cb9c2a1fc6",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.3: Log into the Redis Pod.\n",
+    "# The password `5241590000000000` is defined in the `redis-config` ConfigMap.\n",
+    "# Step 6.4: Check the keys in Redis.\n",
+    "# Note: the schema changed in Ray 2.38.0. Previously we use a single HASH table,\n",
+    "# now we use multiple HASH tables with a common prefix.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" KEYS '*'\n",
+    "# [Example output]:\n",
+    "# 1) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@INTERNAL_CONFIG\"\n",
+    "# 2) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@KV\"\n",
+    "# 3) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\"\n",
+    "# [Example output Before Ray 2.38.0]:\n",
+    "# 2) \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "76947751-45b0-4165-af36-d085356966ff",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.5: Check the value of the key.\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" HGETALL RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\n",
+    "# Before Ray 2.38.0:\n",
+    "# HGETALL 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a67ff02-03d2-45df-ad5c-fbe3f72a3a1e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `gcsFaultToleranceOptions.externalStorageNamespace` option isn't set for the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster with the RayCluster's UID as the external storage namespace by default.\n",
+    "See [this section](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-external-storage-namespace) to learn more about the option.\n",
+    "\n",
+    "### Step 7: Kill the GCS process in the head Pod\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "297712df-81e6-40b8-80e4-cb43fd125dfd",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "export YOUR_WORKER_POD=$(kubectl get pods -l ray.io/group=small-group -o jsonpath='{.items[0].metadata.name}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "775d8da9-cea7-4f93-adec-653520af37d3",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 7.1: Check the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable in both the head Pod and worker Pod.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# No `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable is set. Hence, the Ray head uses its default value of `60`.\n",
+    "kubectl get pods $YOUR_WORKER_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# KubeRay injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value `600` to the worker Pod.\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_gcs_rpc_server_reconnect_timeout_s\",\n",
+    "#     \"value\": \"600\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]"
path: doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
user: login: MortalHappiness
body: Split into 2 cells.
created_at: 2025-04-15T09:30:26Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,900 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-vpcgj                    1/1     Running   0          3m9s\n",
+      "raycluster-external-redis-head-vrsbq                 1/1     Running   0          2m41s\n",
+      "raycluster-external-redis-small-group-worker-rx25r   1/1     Running   0          2m41s\n",
+      "redis-6cf756c755-czs8r                               1/1     Running   0          2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "76aa215a-7a63-4b79-8d3c-f5b674ad8fdb",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      3m10s\n",
+      "ray-example        2      2m41s\n",
+      "redis-config       1      2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9d28f87-6e4e-4736-b3d3-b26888955d66",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.\n",
+    "There are two ConfigMaps in this example: `ray-example` and `redis-config`.\n",
+    "The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.\n",
+    "\n",
+    "* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    @ray.remote(num_cpus=1)\n",
+    "    class Counter:\n",
+    "        def __init__(self):\n",
+    "            self.value = 0\n",
+    "\n",
+    "        def increment(self):\n",
+    "            self.value += 1\n",
+    "            return self.value\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    Counter.options(name=\"counter_actor\", lifetime=\"detached\").remote()\n",
+    "    ```\n",
+    "\n",
+    "* `increment_counter.py` is a Python script that increments the counter.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    counter = ray.get_actor(\"counter_actor\")\n",
+    "    print(ray.get(counter.increment.remote()))\n",
+    "    ```\n",
+    "\n",
+    "### Step 5: Create a detached actor\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "978b889d-66af-408a-bd24-c36273edadb9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-14 18:53:36,998\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:53:37,001\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:53:37,158\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "2025-04-14 18:53:40,273\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:53:40,276\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:53:40,607\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "1\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 5.1: Create a detached actor with the name, `counter_actor`.\n",
+    "export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py\n",
+    "# Step 5.2: Increment the counter.\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a1817cd-4380-4c28-b867-a3f55cc20910",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "(kuberay-external-storage-namespace-example)=\n",
+    "### Step 6: Check the data in Redis\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0454a9bb-4908-4c69-95c9-b2d1c2eb42cc",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.1: Check the RayCluster's UID.\n",
+    "kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'\n",
+    "# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d8e5d436-f9a4-44ef-86e4-635431caa34e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.2: Check the head Pod's environment variable `RAY_external_storage_namespace`.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Example output]:\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_external_storage_namespace\",\n",
+    "#     \"value\": \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dcbba87e-0f9b-4997-ba71-08cb9c2a1fc6",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.3: Log into the Redis Pod.\n",
+    "# The password `5241590000000000` is defined in the `redis-config` ConfigMap.\n",
+    "# Step 6.4: Check the keys in Redis.\n",
+    "# Note: the schema changed in Ray 2.38.0. Previously we use a single HASH table,\n",
+    "# now we use multiple HASH tables with a common prefix.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" KEYS '*'\n",
+    "# [Example output]:\n",
+    "# 1) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@INTERNAL_CONFIG\"\n",
+    "# 2) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@KV\"\n",
+    "# 3) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\"\n",
+    "# [Example output Before Ray 2.38.0]:\n",
+    "# 2) \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "76947751-45b0-4165-af36-d085356966ff",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.5: Check the value of the key.\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" HGETALL RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\n",
+    "# Before Ray 2.38.0:\n",
+    "# HGETALL 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a67ff02-03d2-45df-ad5c-fbe3f72a3a1e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `gcsFaultToleranceOptions.externalStorageNamespace` option isn't set for the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster with the RayCluster's UID as the external storage namespace by default.\n",
+    "See [this section](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-external-storage-namespace) to learn more about the option.\n",
+    "\n",
+    "### Step 7: Kill the GCS process in the head Pod\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "297712df-81e6-40b8-80e4-cb43fd125dfd",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "export YOUR_WORKER_POD=$(kubectl get pods -l ray.io/group=small-group -o jsonpath='{.items[0].metadata.name}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "775d8da9-cea7-4f93-adec-653520af37d3",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 7.1: Check the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable in both the head Pod and worker Pod.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# No `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable is set. Hence, the Ray head uses its default value of `60`.\n",
+    "kubectl get pods $YOUR_WORKER_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# KubeRay injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value `600` to the worker Pod.\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_gcs_rpc_server_reconnect_timeout_s\",\n",
+    "#     \"value\": \"600\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "4b56c6fa-fafa-4f4a-8ef0-3513bc49c9f9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 7.2: Kill the GCS process in the head Pod.\n",
+    "kubectl exec -i $HEAD_POD -- pkill gcs_server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "72a46dd5-9305-40c1-b378-456c260c8f36",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "sleep 60\n",
+    "kubectl wait --for=condition=ready pod/$HEAD_POD --timeout=500s\n",
path: doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
user: login: MortalHappiness
body: ```suggestion
    "kubectl wait --for=condition=ready pod/$HEAD_POD --timeout=500s\n",
```

Don't need to sleep because we've already use `kubectl wait`.
created_at: 2025-04-15T09:31:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,900 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "81f97824-8e89-487e-8857-ef973f240cf4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "(kuberay-gcs-ft)=\n",
+    "# GCS fault tolerance in KubeRay\n",
+    "\n",
+    "Global Control Service (GCS) manages cluster-level metadata.\n",
+    "By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the entire Ray cluster to fail.\n",
+    "To make the GCS fault tolerant, you must have a high-availability Redis.\n",
+    "This way, in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular functions.\n",
+    "\n",
+    "```{admonition} Fate-sharing without GCS fault tolerance \n",
+    "Without GCS fault tolerance, the Ray cluster, the GCS process, and the Ray head Pod are fate-sharing. If the GCS process dies, the Ray head Pod dies as well after `RAY_gcs_rpc_server_reconnect_timeout_s` seconds. If the Ray head Pod is restarted according to the Pod's `restartPolicy`, worker Pods attempt to reconnect to the new head Pod. The worker Pods are terminated by the new head Pod; without GCS fault tolerance enabled, the cluster state is lost, and the worker Pods are perceived as \"unknown workers\" by the new head Pod. This is adequate for most Ray applications; however, it is not ideal for Ray Serve, especially if high availability is crucial for your use cases. Hence, we recommend enabling GCS fault tolerance on the RayService custom resource to ensure high availability. See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.  \n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{seealso}\n",
+    "If you need fault tolerance for Redis as well, see [Tuning Redis for a Persistent Fault Tolerant GCS](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-persistent-ft.html#kuberay-gcs-persistent-ft).\n",
+    "```\n",
+    "\n",
+    "\n",
+    "## Use cases\n",
+    "\n",
+    "* **Ray Serve**: The recommended configuration is enabling GCS fault tolerance on the RayService custom resource to ensure high availability.\n",
+    "See [Ray Serve end-to-end fault tolerance documentation](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html#serve-e2e-ft-guide-gcs) for more information.\n",
+    "\n",
+    "* **Other workloads**: GCS fault tolerance isn't recommended and the compatibility isn't guaranteed.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "* Ray 2.0.0+\n",
+    "* KubeRay 1.3.0+\n",
+    "* Redis: single shard Redis Cluster or Redis Sentinel, one or multiple replicas\n",
+    "\n",
+    "## Quickstart\n",
+    "\n",
+    "### Step 1: Create a Kubernetes cluster with Kind\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df8a0bba-fb2b-43ca-a144-cd3b38bfda20",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "kind create cluster --image=kindest/node:v1.26.0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39cdfeaa-f265-46ea-89c9-3456258af4b7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 2: Install the KubeRay operator\n",
+    "\n",
+    "Follow [this document](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e608c1-0d18-4e27-b03f-875e26f37bc7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "../scripts/doctest-utils.sh install_kuberay_operator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c1257c0-f41c-4b15-92e0-86e9182da271",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 3: Install a RayCluster with GCS FT enabled\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8aced7d5-7852-4b3c-a366-90f82df62cba",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.external-redis.yaml\n",
+    "kubectl apply -f ray-cluster.external-redis.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fca5a21b-5029-4c70-b014-40e596b4a97a",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# make sure raycluster-external-redis condition met\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=headgroup --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l ray.io/group=small-group --timeout=500s\n",
+    "kubectl wait --for=condition=ready pod -l app=redis --timeout=500s"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "850a6516-6084-46c4-a26f-c5c24fab1a1c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "### Step 4: Verify the Kubernetes cluster status\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "feec5b5c-f99e-4344-83d4-ccd3ad2032ae",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS   AGE\n",
+      "kuberay-operator-6bc45dd644-vpcgj                    1/1     Running   0          3m9s\n",
+      "raycluster-external-redis-head-vrsbq                 1/1     Running   0          2m41s\n",
+      "raycluster-external-redis-small-group-worker-rx25r   1/1     Running   0          2m41s\n",
+      "redis-6cf756c755-czs8r                               1/1     Running   0          2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.1: List all Pods in the `default` namespace.\n",
+    "# The expected output should be 4 Pods: 1 head, 1 worker, 1 KubeRay, and 1 Redis.\n",
+    "kubectl get pods"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "76aa215a-7a63-4b79-8d3c-f5b674ad8fdb",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME               DATA   AGE\n",
+      "kube-root-ca.crt   1      3m10s\n",
+      "ray-example        2      2m41s\n",
+      "redis-config       1      2m41s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 4.2: List all ConfigMaps in the `default` namespace.\n",
+    "kubectl get configmaps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9d28f87-6e4e-4736-b3d3-b26888955d66",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "The [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml) file defines Kubernetes resources for RayCluster, Redis, and ConfigMaps.\n",
+    "There are two ConfigMaps in this example: `ray-example` and `redis-config`.\n",
+    "The `ray-example` ConfigMap houses two Python scripts: `detached_actor.py` and `increment_counter.py`.\n",
+    "\n",
+    "* `detached_actor.py` is a Python script that creates a detached actor with the name, `counter_actor`.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    @ray.remote(num_cpus=1)\n",
+    "    class Counter:\n",
+    "        def __init__(self):\n",
+    "            self.value = 0\n",
+    "\n",
+    "        def increment(self):\n",
+    "            self.value += 1\n",
+    "            return self.value\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    Counter.options(name=\"counter_actor\", lifetime=\"detached\").remote()\n",
+    "    ```\n",
+    "\n",
+    "* `increment_counter.py` is a Python script that increments the counter.\n",
+    "    ```python\n",
+    "    import ray\n",
+    "\n",
+    "    ray.init(namespace=\"default_namespace\")\n",
+    "    counter = ray.get_actor(\"counter_actor\")\n",
+    "    print(ray.get(counter.increment.remote()))\n",
+    "    ```\n",
+    "\n",
+    "### Step 5: Create a detached actor\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "978b889d-66af-408a-bd24-c36273edadb9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-14 18:53:36,998\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:53:37,001\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:53:37,158\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "2025-04-14 18:53:40,273\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:53:40,276\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:53:40,607\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "1\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 5.1: Create a detached actor with the name, `counter_actor`.\n",
+    "export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py\n",
+    "# Step 5.2: Increment the counter.\n",
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a1817cd-4380-4c28-b867-a3f55cc20910",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "(kuberay-external-storage-namespace-example)=\n",
+    "### Step 6: Check the data in Redis\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0454a9bb-4908-4c69-95c9-b2d1c2eb42cc",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.1: Check the RayCluster's UID.\n",
+    "kubectl get rayclusters.ray.io raycluster-external-redis -o=jsonpath='{.metadata.uid}'\n",
+    "# [Example output]: 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d8e5d436-f9a4-44ef-86e4-635431caa34e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.2: Check the head Pod's environment variable `RAY_external_storage_namespace`.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Example output]:\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_external_storage_namespace\",\n",
+    "#     \"value\": \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dcbba87e-0f9b-4997-ba71-08cb9c2a1fc6",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.3: Log into the Redis Pod.\n",
+    "# The password `5241590000000000` is defined in the `redis-config` ConfigMap.\n",
+    "# Step 6.4: Check the keys in Redis.\n",
+    "# Note: the schema changed in Ray 2.38.0. Previously we use a single HASH table,\n",
+    "# now we use multiple HASH tables with a common prefix.\n",
+    "export REDIS_POD=$(kubectl get pods --selector=app=redis -o custom-columns=POD:metadata.name --no-headers)\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" KEYS '*'\n",
+    "# [Example output]:\n",
+    "# 1) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@INTERNAL_CONFIG\"\n",
+    "# 2) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@KV\"\n",
+    "# 3) \"RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\"\n",
+    "# [Example output Before Ray 2.38.0]:\n",
+    "# 2) \"864b004c-6305-42e3-ac46-adfa8eb6f752\"\n",
+    "#"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "76947751-45b0-4165-af36-d085356966ff",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 6.5: Check the value of the key.\n",
+    "kubectl exec -i $REDIS_POD -- redis-cli -a \"5241590000000000\" HGETALL RAY864b004c-6305-42e3-ac46-adfa8eb6f752@NODE\n",
+    "# Before Ray 2.38.0:\n",
+    "# HGETALL 864b004c-6305-42e3-ac46-adfa8eb6f752"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a67ff02-03d2-45df-ad5c-fbe3f72a3a1e",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `gcsFaultToleranceOptions.externalStorageNamespace` option isn't set for the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster with the RayCluster's UID as the external storage namespace by default.\n",
+    "See [this section](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-external-storage-namespace) to learn more about the option.\n",
+    "\n",
+    "### Step 7: Kill the GCS process in the head Pod\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "297712df-81e6-40b8-80e4-cb43fd125dfd",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "export YOUR_WORKER_POD=$(kubectl get pods -l ray.io/group=small-group -o jsonpath='{.items[0].metadata.name}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "775d8da9-cea7-4f93-adec-653520af37d3",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 7.1: Check the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable in both the head Pod and worker Pod.\n",
+    "kubectl get pods $HEAD_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# No `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable is set. Hence, the Ray head uses its default value of `60`.\n",
+    "kubectl get pods $YOUR_WORKER_POD -o=jsonpath='{.spec.containers[0].env}' | jq\n",
+    "# [Expected result]:\n",
+    "# KubeRay injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value `600` to the worker Pod.\n",
+    "# [\n",
+    "#   {\n",
+    "#     \"name\": \"RAY_gcs_rpc_server_reconnect_timeout_s\",\n",
+    "#     \"value\": \"600\"\n",
+    "#   },\n",
+    "#   ...\n",
+    "# ]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "4b56c6fa-fafa-4f4a-8ef0-3513bc49c9f9",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 7.2: Kill the GCS process in the head Pod.\n",
+    "kubectl exec -i $HEAD_POD -- pkill gcs_server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "72a46dd5-9305-40c1-b378-456c260c8f36",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove-cell",
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "sleep 60\n",
+    "kubectl wait --for=condition=ready pod/$HEAD_POD --timeout=500s\n",
+    "echo $HEAD_POD"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "e2ab3666-3204-4557-acba-c624604bbca2",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NAME                                                 READY   STATUS    RESTARTS      AGE\n",
+      "raycluster-external-redis-head-vrsbq                 1/1     Running   1 (42s ago)   4m36s\n",
+      "raycluster-external-redis-small-group-worker-rx25r   1/1     Running   0             4m36s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 7.3: The head Pod fails and restarts after `RAY_gcs_rpc_server_reconnect_timeout_s` (60) seconds.\n",
+    "# In addition, the worker Pod isn't terminated by the new head after reconnecting because GCS fault\n",
+    "# tolerance is enabled.\n",
+    "kubectl get pods -l=ray.io/is-ray-node=yes"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dbea855d-a39f-4f3a-9e0d-165eb08858e7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "In [ray-cluster.external-redis.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.external-redis.yaml), the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable isn't set in the specifications for either the head Pod or the worker Pod within the RayCluster.\n",
+    "Therefore, KubeRay automatically injects the `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable with the value **600** to the worker Pod and uses the default value **60** for the head Pod.\n",
+    "The timeout value for worker Pods must be longer than the timeout value for the head Pod so that the worker Pods don't terminate before the head Pod restarts from a failure.\n",
+    "\n",
+    "### Step 8: Access the detached actor again\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "d9976f6b-a542-4f66-a45a-4e489aff3ad4",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-04-14 18:55:32,084\tINFO worker.py:1514 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS\n",
+      "2025-04-14 18:55:32,087\tINFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.8:6379...\n",
+      "2025-04-14 18:55:32,243\tINFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m10.244.0.8:8265 \u001b[39m\u001b[22m\n",
+      "2\n"
+     ]
+    }
+   ],
+   "source": [
+    "kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/increment_counter.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "615ac036-7b91-42fb-a9ae-efc2e9402d0f",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "\n",
+    "```{admonition} The detached actor is always on the worker Pod in this example.\n",
+    "The head Pod's `rayStartParams` is set to `num-cpus: \"0\"`.Hence, no tasks or actors will be scheduled on the head Pod.\n",
+    "```\n",
+    "\n",
+    "With GCS fault tolerance enabled, you can still access the detached actor after the GCS process is dead and restarted.\n",
+    "Note that the fault tolerance doesn't persist the actor's state.\n",
+    "The reason why the result is 2 instead of 1 is that the detached actor is on the worker Pod which is always running.\n",
+    "On the other hand, if the head Pod hosts the detached actor, the `increment_counter.py` script yields a result of 1 in this step.\n",
+    "\n",
+    "### Step 9: Remove the key stored in Redis when deleting RayCluster\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f8b89045-8417-4b70-ba6d-4f38e64d3a43",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 9.1: Delete the RayCluster custom resource.\n",
+    "kubectl delete raycluster raycluster-external-redis"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "0d00cf78-e7bd-403f-a0e9-e9c9fc9268e7",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "nbval-ignore-output",
+     "remove-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Step 9.2: KubeRay operator deletes all Pods in the RayCluster.\n",
+    "# Step 9.3: KubeRay operator creates a Kubernetes Job to delete the Redis key after the head Pod is terminated."
+   ]
+  },
path: doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.ipynb
user: login: MortalHappiness
body: Merge these 2 cells.
created_at: 2025-04-15T09:32:18Z
author_association: MEMBER

PR comments:
user: login: popojk
created_at: 2025-04-08T08:42:58Z
body: hi @MortalHappiness , please help me to review this doctest PR. 

For your information, I ignore the test output of [this block](https://anyscale-ray--51988.com.readthedocs.build/en/51988/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#step-5-create-a-detached-actor) and [this block](https://anyscale-ray--51988.com.readthedocs.build/en/51988/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#step-8-access-the-detached-actor-again) because sometimes pod execution latency makes the count number not shown in log, making the doctest flaky. Please let me know if you have any suggestions, thanks!
author_association: CONTRIBUTOR
user: login: MortalHappiness
created_at: 2025-04-15T04:08:35Z
body: Hello. Please resolve conflicts. Thanks.
author_association: MEMBER
user: login: popojk
created_at: 2025-04-15T09:35:25Z
body: Hello @MortalHappiness , I've made bellow updates to this PR:
1. Split some cells you mentioned into multiple cells
2. resolve conflicts

Pleas let me know if you would like to adjust anything, thanks!
author_association: CONTRIBUTOR
user: login: MortalHappiness
created_at: 2025-04-15T09:38:28Z
body: @popojk I've left some review comments above. Thanks.
author_association: MEMBER
user: login: MortalHappiness
created_at: 2025-04-18T14:32:57Z
body: Ping me again when this is ready for another review. Thanks.
author_association: MEMBER
user: login: popojk
created_at: 2025-04-19T03:43:42Z
body: Hi @MortalHappiness , please review my updated PR. Let me mention a few points:
1. I removed the output of 6.5 and 9.5 as the outputs are empty.
2. In [this cell](https://github.com/ray-project/ray/pull/51988/files#diff-e1a3266afe6f74ff061f24be17a87edae73ecc023f33b1159d91368cf03e874dR769) I kept `sleep 60` because the ray-head pod doesn't restart immediately after executing `kubectl exec -i $HEAD_POD -- pkill gcs_server`. As a result, the `kubectl wait` command completes too early, causing the following doc tests to fail.

Thanks!
author_association: CONTRIBUTOR
user: login: popojk
created_at: 2025-04-22T08:49:44Z
body: Hi @MortalHappiness , just a friendly reminder that this PR has been updated based on your suggestions.
I've also addressed a few points in my previous reply. Please let me know if you have any further feedback. Thanks!

> Hi @MortalHappiness , please review my updated PR. Let me mention a few points:
> 
> 1. I removed the output of 6.5 and 9.5 as the outputs are empty.
> 2. In [this cell](https://github.com/ray-project/ray/pull/51988/files#diff-e1a3266afe6f74ff061f24be17a87edae73ecc023f33b1159d91368cf03e874dR769) I kept `sleep 60` because the ray-head pod doesn't restart immediately after executing `kubectl exec -i $HEAD_POD -- pkill gcs_server`. As a result, the `kubectl wait` command completes too early, causing the following doc tests to fail.
> 
> Thanks!


author_association: CONTRIBUTOR
user: login: MortalHappiness
created_at: 2025-04-22T15:56:32Z
body: LGTM. But as requested by @kevin85421, could you post a screenshot of running this test consecutively successful 10 times on localhost? Thanks.
author_association: MEMBER
user: login: popojk
created_at: 2025-04-24T08:35:37Z
body: > LGTM. But as requested by @kevin85421, could you post a screenshot of running this test consecutively successful 10 times on localhost? Thanks.

Hi @MortalHappiness @kevin85421 . Here you go.

<img width="1590" alt="image" src="https://github.com/user-attachments/assets/7f4397b9-1af9-4d6d-9ce0-dfebfa2f000b" />
<img width="1590" alt="image" src="https://github.com/user-attachments/assets/642edf02-ad6a-4128-aa72-9d6db237758c" />


author_association: CONTRIBUTOR
user: login: kevin85421
created_at: 2025-04-25T03:39:06Z
body: cc @edoakes @jjyao would you mind merging this PR? Thanks!
author_association: MEMBER