Repository: ai-dynamo/dynamo
Pull request #720: docs: R1 disaggregation guide
Author: login: GuanLuo
Labels: name: size/M
Created: 2025-04-17T01:35:20Z | Merged: 2025-04-22T23:20:33Z

#### Overview:

<!-- Describe your pull request here. Please read the text below the line, and make sure you follow the checklist.-->

#### Details:

<!-- Describe the changes made in this PR. -->

#### Where should the reviewer start?

<!-- call out specific files that should be looked at closely -->

#### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

- closes GitHub issue: #xxx



Diff:
diff --git a/examples/llm/configs/mutinode_disagg_r1.yaml b/examples/llm/configs/mutinode_disagg_r1.yaml
new file mode 100644
index 000000000000..e8d0291f0c16
--- /dev/null
+++ b/examples/llm/configs/mutinode_disagg_r1.yaml
@@ -0,0 +1,46 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+Common:
+  model: deepseek-ai/DeepSeek-R1
+  block-size: 64
+  max-model-len: 16384
+  kv-transfer-config: '{"kv_connector":"DynamoNixlConnector"}'
+  tensor-parallel-size: 16
+
+Frontend:
+  served_model_name: deepseek-ai/DeepSeek-R1
+  endpoint: dynamo.Processor.chat/completions
+  port: 8000
+
+Processor:
+  router: round-robin
+  common-configs: [model, block-size]
+
+VllmWorker:
+  remote-prefill: true
+  conditional-disagg: false
+  ServiceArgs:
+    workers: 1
+    resources:
+      gpu: 16
+  common-configs: [model, block-size, max-model-len, kv-transfer-config, tensor-parallel-size]
+
+PrefillWorker:
+  max-num-batched-tokens: 16384
+  ServiceArgs:
+    workers: 1
+    resources:
+      gpu: 16
+  common-configs: [model, block-size, max-model-len, kv-transfer-config, tensor-parallel-size]
diff --git a/examples/llm/multinode-examples.md b/examples/llm/multinode-examples.md
index 040ca1ee5112..f92e0f6f42f7 100644
--- a/examples/llm/multinode-examples.md
+++ b/examples/llm/multinode-examples.md
@@ -163,4 +163,54 @@ curl <node1-ip>:8000/v1/chat/completions \
     "stream": true,
     "max_tokens": 300
   }'
-```
\ No newline at end of file
+```
+
+##### Disaggregated Deployment
+
+In this example, we will be deploying two replicas of the model (one prefill worker
+and one decode worker). We will be using 4 H100x8 nodes and group every two of them
+into one Ray cluster in the same way as described in aggregated deployment.
+However, for etcd and nats server, we will only run them in
+one node and let's consider that node to be the head node of the whole deployment.
+
+Note that if you are starting etcd server directly instead of using `docker compose`,
+you should add additional arguments to be discoverable in other node.
+```bash
+etcd --advertise-client-urls http://<head-node-ip>:2379 --listen-client-urls http://<head-node-ip>:2379,http://127.0.0.1:2379
+```
+
+**Step 1**: On every two nodes, set up Ray cluster as described in
+[aggregated deployment](#aggregated-deployment). After that, you should have
+two independent Ray cluster, each has access to 16 GPUs.
+
+**Step 2** start the deployment by running different flavors of `dynamo serve`
+on one of the node for each Ray cluster, using the configuration file,
+`configs/mutinode_disagg_r1.yaml`.
+
+For decode, below command will be used and the node will be the entry point of
+the whole deployment. In other words, the ip of the node should be used to send
+requests to.
+```bash
+# if not head node
+export NATS_SERVER='nats://<nats-server-ip>:4222'
+export ETCD_ENDPOINTS='<etcd-endpoints-ip>:2379'
+
+cd $DYNAMO_HOME/examples/llm
+dynamo serve graphs.agg:Frontend -f ./configs/mutinode_disagg_r1.yaml
+```
+
+For prefill:
+```bash
+# if not head node
+export NATS_SERVER='nats://<nats-server-ip>:4222'
+export ETCD_ENDPOINTS='<etcd-endpoints-ip>:2379'
+
+cd $DYNAMO_HOME/examples/llm
+dynamo serve components.prefill_worker:PrefillWorker -f ./configs/mutinode_disagg_r1.yaml
+```
+
+### Client
+
+In another terminal, you can send the same curl request as described in
+[aggregated deployment](#aggregated-deployment), addressing to the ip of
+the decode node.
\ No newline at end of file


Reviews:
user: login: rmccorm4
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T17:40:21Z
user: login: GuanLuo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:24:35Z
user: login: rmccorm4
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T20:35:39Z

Review comments:
diff_hunk: @@ -0,0 +1,46 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
path: examples/llm/configs/mutinode_disagg_r1.yaml
user: login: rmccorm4
body: Just curious - have you tried on GB200 / Linux ARM machine?
created_at: 2025-04-21T17:40:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,46 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
path: examples/llm/configs/mutinode_disagg_r1.yaml
user: login: GuanLuo
body: Haven't tried that
created_at: 2025-04-21T19:24:35Z
author_association: CONTRIBUTOR

PR comments:
user: login: copy-pr-bot[bot]
created_at: 2025-04-17T01:35:23Z
body: This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities [here](https://docs.gha-runners.nvidia.com/cpr/vetters).

Contributors can view more details about this message [here](https://docs.gha-runners.nvidia.com/cpr/contributors).

author_association: NONE
user: login: tlipoca9
created_at: 2025-04-17T02:49:12Z
body: Has the issue #513 been resolved?
author_association: CONTRIBUTOR
user: login: GuanLuo
created_at: 2025-04-21T19:24:23Z
body: > Has the issue #513 been resolved?

@tlipoca9  The issue is not fully resolved that PP = 1 is still a current limitation. But you may deploy large model by splitting the model with purely TP, i.e. this example uses TP = 16.
author_association: CONTRIBUTOR