Repository: wandb/docs
Pull request #1246: Experiments: multi node logging
Author: login: ngrayluna
Created: 2025-04-10T03:35:03Z | Merged: 2025-04-18T20:40:53Z

Cleans up existing "Distributed logging" doc and adds a section on what and how to use public "Multi node" feature.

Jira ticket: https://wandb.atlassian.net/browse/DOCS-1373


Diff:
diff --git a/assets/images/track/multi_node_console_logs.png b/assets/images/track/multi_node_console_logs.png
new file mode 100644
index 0000000000..13683d1ae9
Binary files /dev/null and b/assets/images/track/multi_node_console_logs.png differ
diff --git a/assets/images/track/multi_node_system_metrics.png b/assets/images/track/multi_node_system_metrics.png
new file mode 100644
index 0000000000..aaaa2c27ff
Binary files /dev/null and b/assets/images/track/multi_node_system_metrics.png differ
diff --git a/content/guides/models/track/log/distributed-training.md b/content/guides/models/track/log/distributed-training.md
index 300178ad0c..6690efbea6 100644
--- a/content/guides/models/track/log/distributed-training.md
+++ b/content/guides/models/track/log/distributed-training.md
@@ -7,26 +7,31 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During a distributed training experiment, you train a model using multiple machines or clients in parallel. W&B can help you track distributed training experiments. Based on your use case, track distributed training experiments using one of the following approaches:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Track a single process**: Track a rank 0 process (also known as a "leader" or "coordinator") with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Track multiple processes**: For multiple processes, you can either:
+   * Track each process separately using one run per process. You can optionally group them together in the W&B App UI.
+   * Track all processes to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
 
-### Method 1: One process
+## Track a single process
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+This section describes how to track values and metrics available to your rank 0 process. Use this approach to track only metrics that are available from a single process. Typical metrics include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
 
-{{% alert %}}
-**Use this method to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
-{{% /alert %}}
+Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run. 
 
-Within our [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py), we check to see if the rank is 0. To do so, we first launch multiple processes with `torch.distributed.launch`. Next, we check the rank with the `--local_rank` command line argument. If the rank is set to 0, we set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function. Within our Python script, we use the following check:
+The following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrates one way to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the implementation may differ.
+
+The Python script:
+1. Starts multiple processes with `torch.distributed.launch`.
+1. Checks the rank with the `--local_rank` command line argument.
+1. If the rank is set to 0, sets up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function.
 
 ```python
 if __name__ == "__main__":
@@ -45,22 +50,30 @@ if __name__ == "__main__":
         train(args)
 ```
 
-Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system) of metrics tracked from a single process. The dashboard displays system metrics such as temperature and utilization, that were tracked for both GPUs.
+Explore an [example dashboard showing metrics tracked from a single process](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system).
+
+The dashboard displays system metrics for both GPUs, such as temperature and utilization.
 
-{{< img src="/images/track/distributed_training_method1.png" alt="" >}}
+{{< img src="/images/track/distributed_training_method1.png" alt="Workspace that shows four line plot panels. From left to right the plots are: GPU Utilization (%), GPU Temp (C), GPU Time Spent Accessing Memory (%), and GPU Memory Allocated (%)" >}}
 
 However, the loss values as a function epoch and batch size were only logged from a single GPU.
 
-{{< img src="/images/experiments/loss_function_single_gpu.png" alt="" >}}
+{{< img src="/images/experiments/loss_function_single_gpu.png" alt="Two line plot panels in an example workspace. The left plot shows the loss measured as a function of step. The right plot show the loss recorded at each batch." >}}
 
-### Method 2: Many processes
+## Track multiple processes
 
-In this method, we track each process in the job, calling `wandb.init()` and `wandb.log()` from each process separately. We suggest you call `wandb.finish()` at the end of training, to mark that the run has completed so that all processes exit properly.
+Track multiple processes with W&B with one of the following approaches:
+* [Tracking each process separately]({{< relref "distributed-training/#track-each-process-separately" >}}) by creating a run for each process.
+* [Tracking all processes to a single run]({{< relref "distributed-training/#track-all-processes-to-a-single-run" >}}).
 
-This method makes more information accessible for logging. However, note that multiple W&B Runs are reported in the W&B App UI. It might be difficult to keep track of W&B Runs across multiple experiments. To mitigate this, provide a value to the group parameter when you initialize W&B to keep track of which W&B Run belongs to a given experiment. For more information about how to keep track of training and evaluation W&B Runs in experiments, see [Group Runs]({{< relref "/guides/models/track/runs/grouping.md" >}}).
+### Track each process separately
+
+This section describes how to track each process separately by creating a run for each process. Within each run you log metrics, artifacts, and forth to their respective run. Call `wandb.finish()` at the end of training, to mark that the run has completed so that all processes exit properly.
+
+You might find it difficult to keep track of runs across multiple experiments. To mitigate this, provide a value to the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to keep track of which run belongs to a given experiment. For more information about how to keep track of training and evaluation W&B Runs in experiments, see [Group Runs]({{< relref "/guides/models/track/runs/grouping.md" >}}).
 
 {{% alert %}}
-**Use this method if you want to track metrics from individual processes**. Typical examples include the data and predictions on each node (for debugging data distribution) and metrics on individual batches outside of the main node. This method is not necessary to get system metrics from all nodes nor to get summary statistics available on the main node.
+**Use this approach if you want to track metrics from individual processes**. Typical examples include the data and predictions on each node (for debugging data distribution) and metrics on individual batches outside of the main node. This approach is not necessary to get system metrics from all nodes nor to get summary statistics available on the main node.
 {{% /alert %}}
 
 The following Python code snippet demonstrates how to set the group parameter when you initialize W&B:
@@ -85,60 +98,109 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
 
-There are two common issues you might encounter when using W&B and distributed training:
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
 
-1. **Hanging at the beginning of training** - A `wandb` process can hang if the `wandb` multiprocessing interferes with the multiprocessing from distributed training.
-2. **Hanging at the end of training** - A training job might hang if the `wandb` process does not know when it needs to exit. Call the `wandb.finish()` API at the end of your Python script to tell W&B that the Run finished. The wandb.finish() API will finish uploading data and will cause W&B to exit.
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `v0.19.5` or newer.
+{{% /alert  %}}
 
-We recommend using the `wandb service` to improve the reliability of your distributed jobs. Both of the preceding training issues are commonly found in versions of the W&B SDK where wandb service is unavailable.
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
 
-### Enable W&B Service
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) wit with the following:
 
-Depending on your version of the W&B SDK, you might already have W&B Service enabled by default.
+1. The `mode` parameter set to `"shared"` to enable shared mode.
+2. A unique label for [`x_label`](https://github.com/wandb/wandb/blob/main/wandb/sdk/wandb_settings.py#L638). You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+3. Set the [`x_primary`](https://github.com/wandb/wandb/blob/main/wandb/sdk/wandb_settings.py#L660) parameter to `True` to indicate that this is the primary node.
 
-#### W&B SDK 0.13.0 and above
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
 
-W&B Service is enabled by default for versions of the W&B SDK `0.13.0` and above.
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
 
-#### W&B SDK 0.12.5 and above
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+   * Set the `x_primary` parameter to `False` to indicate that this is a worker node.
+2. Pass the run ID used by the primary node to the `id` parameter.
+3. Optionally set [`x_update_finish_state`](https://github.com/wandb/wandb/blob/main/wandb/sdk/wandb_settings.py#L772) to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
 
-Modify your Python script to enable W&B Service for W&B SDK version 0.12.5 and above. Use the `wandb.require` method and pass the string `"service"` within your main function:
+{{% alert %}}
+Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+{{% /alert %}}
+
+The following sample code demonstrates the high level requirements for tracking multiple processes to a single run:
 
 ```python
-if __name__ == "__main__":
-    main()
+import wandb
+
+# Initialize a run in the primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label="rank_0", mode="shared", x_primary=True)
+)
+
+# Note the run ID of the primary node.
+# Each worker node needs this run ID.
+run_id = run.id
+
+# Initialize a run in a worker node using the run ID of the primary node
+run = wandb.init(
+	settings=wandb.Settings(x_label="rank_1", mode="shared", x_primary=False),
+	id=run_id,
+)
+
+# Initialize a run in a worker node using the run ID of the primary node
+run = wandb.init(
+	settings=wandb.Settings(x_label="rank_2", mode="shared", x_primary=False),
+	id=run_id,
+)
+```
 
+In a real world example, each worker node might be on a separate machine.
 
-def main():
-    wandb.require("service")
-    # rest-of-your-script-goes-here
-```
+{{% alert %}}
+See the [Distributed Training with Shared Mode](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA) report for an end-to-end example on how to train a model on a multi-node and multi-GPU Kubernetes cluster in GKE.
+{{% /alert %}}
 
-For optimal experience we do recommend you upgrade to the latest version.
+View console logs from multi node processes in the project that the run logs to:
 
-**W&B SDK 0.12.4 and below**
+1. Navigate to the project that contains the run.
+2. Click on the **Runs** tab in the left sidebar.
+3. Click on the run you want to view.
+4. Click on the **Logs** tab in the left sidebar.
+
+You can filter console logs based on the labels you provide for `x_label` in the UI search bar located at the top of the console log page. For example, the following image shows which options are available to filter the console log by if values  `rank0`, `rank1`, `rank2`, `rank3`, `rank4`, `rank5`, and `rank6` are provided to `x_label`.` 
+
+{{< img src="/images/track/multi_node_console_logs.png" alt="Console logs from multiple nodes with the x_label filter applied." >}}
+
+W&B aggregates system metrics from all nodes and displays them in the W&B App UI. For example, the following image shows a sample dashboard with system metrics from multiple nodes. Each node possesses a unique label (`rank_0`, `rank_1`, `rank_2`) that you specify in the `x_label` parameter.
+
+{{< img src="/images/track/multi_node_system_metrics.png" alt="Line plot panel with numerous semi linear lines depicting system metrics logged by a multi node process." >}}
 
-Set the `WANDB_START_METHOD` environment variable to `"thread"` to use multithreading instead if you use a W&B SDK version 0.12.4 and below.
+See [Line plots]({{< relref "/guides/models/app/features/panels/line-plot/" >}}) for information on how to customize line plot panels. 
 
-### Example use cases for multiprocessing
+## Example use cases
 
-The following code snippets demonstrate common methods for advanced distributed use cases.
+The following code snippets demonstrate common scenarios for advanced distributed use cases.
 
-#### Spawn process
+### Spawn process
 
-Use the `wandb.setup()`method in your main function if you initiate a W&B Run in a spawned process:
+Use the `wandb.setup()`method in your main function if you initiate a run in a spawned process:
 
 ```python
 import multiprocessing as mp
 
-
 def do_work(n):
     run = wandb.init(config=dict(n=n))
     run.log(dict(this=n * n))
 
-
 def main():
     wandb.setup()
     pool = mp.Pool(processes=4)
@@ -149,15 +211,14 @@ if __name__ == "__main__":
     main()
 ```
 
-#### Share a W&B Run
+### Share a run
 
-Pass a W&B Run object as an argument to share W&B Runs between processes:
+Pass a run object as an argument to share runs between processes:
 
 ```python
 def do_work(run):
     run.log(dict(this=1))
 
-
 def main():
     run = wandb.init()
     p = mp.Process(target=do_work, kwargs=dict(run=run))
@@ -169,7 +230,42 @@ if __name__ == "__main__":
     main()
 ```
 
+W&B can not guarantee the logging order. Synchronization should be done by the author of the script.
 
-{{% alert %}}
-Note that we can not guarantee the logging order. Synchronization should be done by the author of the script.
-{{% /alert %}}
\ No newline at end of file
+
+## Troubleshooting
+
+There are two common issues you might encounter when using W&B and distributed training:
+
+1. **Hanging at the beginning of training** - A `wandb` process can hang if the `wandb` multiprocessing interferes with the multiprocessing from distributed training.
+2. **Hanging at the end of training** - A training job might hang if the `wandb` process does not know when it needs to exit. Call the `wandb.finish()` API at the end of your Python script to tell W&B that the Run finished. The wandb.finish() API will finish uploading data and will cause W&B to exit.
+
+We recommend using the `wandb service` to improve the reliability of your distributed jobs. Both of the preceding training issues are commonly found in versions of the W&B SDK where wandb service is unavailable.
+
+### Enable W&B Service
+
+Depending on your version of the W&B SDK, you might already have W&B Service enabled by default.
+
+#### W&B SDK 0.13.0 and above
+
+W&B Service is enabled by default for versions of the W&B SDK `0.13.0` and above.
+
+#### W&B SDK 0.12.5 and above
+
+Modify your Python script to enable W&B Service for W&B SDK version 0.12.5 and above. Use the `wandb.require` method and pass the string `"service"` within your main function:
+
+```python
+if __name__ == "__main__":
+    main()
+
+
+def main():
+    wandb.require("service")
+    # rest-of-your-script-goes-here
+```
+
+For optimal experience we do recommend you upgrade to the latest version.
+
+**W&B SDK 0.12.4 and below**
+
+Set the `WANDB_START_METHOD` environment variable to `"thread"` to use multithreading instead if you use a W&B SDK version 0.12.4 and below.
\ No newline at end of file


Reviews:
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:15:03Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:17:38Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:18:15Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:21:39Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:23:05Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:25:22Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:25:55Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:29:46Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:56:04Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T20:59:32Z
user: login: noaleetz
body: see in-line 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T21:00:02Z
user: login: ngrayluna
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T02:53:13Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T03:52:09Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T15:59:22Z
user: login: noaleetz
body: looks good! commented on including the console log experience in this shared mode (it is specific to shared mode / distributed setup). 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T16:01:11Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T16:03:24Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T16:04:30Z
user: login: ngrayluna
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T23:51:42Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T01:19:46Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T01:21:14Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T01:22:38Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T01:24:54Z
user: login: noaleetz
body: x_primary is also defined in setting
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T01:26:33Z
user: login: mdlinville
body: Partial review.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T23:17:02Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T20:47:20Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T20:48:39Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T21:11:44Z
user: login: noaleetz
body: server release constraint
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T21:12:31Z
user: login: ngrayluna
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T23:19:33Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T16:33:51Z
user: login: ngrayluna
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T16:59:19Z
user: login: mdlinville
body: The rest of my feedback -- please ignore anything where I've introduced errors or it doesn't make sense to you. I'm not so familiar with this area of content.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T17:52:19Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T18:50:58Z
user: login: ngrayluna
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T20:28:39Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T20:53:29Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T20:54:46Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T20:55:37Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T20:56:32Z
user: login: noaleetz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T20:57:22Z
user: login: noaleetz
body: missing server release constraint 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T20:57:59Z
user: login: noaleetz
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T01:01:25Z
user: login: mdlinville
body: Some small things, which I leave up to you whether to change or not. This is a big improvement to this page.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T20:12:30Z
user: login: ngrayluna
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T20:19:04Z

Review comments:
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines, such as GPUs or TPUs, in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
```
typically machines might be a cluster with multiple gpus or devices. 
created_at: 2025-04-10T20:15:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: Can we add a banner [as discussed](https://weightsandbiases.slack.com/archives/C081QT67T35/p1743013847199419?thread_ts=1742587252.937509&cid=C081QT67T35) denoting that this in public preview? Similar to the banner used in rewind docs?
created_at: 2025-04-10T20:17:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
```
created_at: 2025-04-10T20:18:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique label for each node with the `x_label` parameter. For the primary node, for example, you might set `x_label` to "rank0" and and `x_primary` to `True`. For the worker nodes, for example, you might set `x_label` to "rank1", "rank2", etc. and `x_primary` to `False`.

In shared mode, if you do not specify `x_label`, a label will be auto-generated based on the hostname. 
```
created_at: 2025-04-10T20:21:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: Can we break down these 3 different settings in a bit more detail:
- x_label
- x_primary
- mode="shared"

it doesn't make it very clear why a user should be setting them. This is the main new feature set.
created_at: 2025-04-10T20:23:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
+
+Each process requires the run ID of the primary node. Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: I think the workflow we would advise is to randomly generated (i.e let W&B pick) the run ID for the first time you call wandb.init, and then make sure you attach it it to all other processes. 

If you forget to reset the env var in between experiments you will keep writing to the same run and will not be able to compare between runs. 
created_at: 2025-04-10T20:25:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
+
+Each process requires the run ID of the primary node. Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+
+The following sample code demonstrates the high level requirements for this approach:
+
+```python
+import os
+import wandb
+
+wandb_run_id = os.environ.get("WANDB_RUN_ID", "default_run_id")
+
+# user calls init in primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label=0, mode="shared"),
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
	settings=wandb.Settings(x_label=0, mode="shared"),
```
recommended labels would most likely be rank0, rank1, etc. not just the digit
created_at: 2025-04-10T20:25:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
+
+Each process requires the run ID of the primary node. Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+
+The following sample code demonstrates the high level requirements for this approach:
+
+```python
+import os
+import wandb
+
+wandb_run_id = os.environ.get("WANDB_RUN_ID", "default_run_id")
+
+# user calls init in primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label=0, mode="shared"),
+	id=wandb_run_id
+    x_primary=True,
+)
+
+# user calls init in worker node 1
+run = wandb.init(
+	settings=wandb.Settings(x_label=1, mode="shared"),
+	id=wandb_run_id,
+    x_primary=False,
+)
+
+# user calls init in workder node 2
+run = wandb.init(
+	settings=wandb.Settings(x_label=2, mode="shared"),
+	id=wandb_run_id,
+    x_primary=True,
+)
+```
+
+{{% alert %}}
+See this report for an end-to-end example on how to [train a model on a multi-node multi-GPU Kubernetes cluster in GKE](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA). 
+{{% /alert %}}
+
+You can view console logs that your multi process creates in the project that contains your run.
+
+1. Navigate to the project that contains the run.
+2. Click on the **Runs** tab in the left sidebar.
+3. Click on the run you want to view.
+4. Click on the **Logs** tab in the left sidebar.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
4. Click on the **Logs** tab in the left sidebar.
```
can we make sure to show the new filtering experience in the console logs? If you are in shared mode, or define custom labels via x_label - your console logs will have a filter bar so you can only filter in specific labels. This is one of the main values of the feature. Can we show a gif, for example like the one below?  

https://github.com/user-attachments/assets/b98943fd-68de-44e2-a893-f30b65571068

[Here](https://wandb.ai/wandb/demo-multiple-ranks/runs/jeq69ubu/logs) is an example logs page that I made public in "wandb" entity if helpful to create a gif or if we want to share it 


created_at: 2025-04-10T20:29:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
+
+Each process requires the run ID of the primary node. Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+
+The following sample code demonstrates the high level requirements for this approach:
+
+```python
+import os
+import wandb
+
+wandb_run_id = os.environ.get("WANDB_RUN_ID", "default_run_id")
+
+# user calls init in primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label=0, mode="shared"),
+	id=wandb_run_id
+    x_primary=True,
+)
+
+# user calls init in worker node 1
+run = wandb.init(
+	settings=wandb.Settings(x_label=1, mode="shared"),
+	id=wandb_run_id,
+    x_primary=False,
+)
+
+# user calls init in workder node 2
+run = wandb.init(
+	settings=wandb.Settings(x_label=2, mode="shared"),
+	id=wandb_run_id,
+    x_primary=True,
+)
+```
+
+{{% alert %}}
+See this report for an end-to-end example on how to [train a model on a multi-node multi-GPU Kubernetes cluster in GKE](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA). 
+{{% /alert %}}
+
+You can view console logs that your multi process creates in the project that contains your run.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: We show also call out that your system metrics can also be broken down and filtered by the label - [link](https://wandb.ai/wandb/demo-multiple-ranks/runs/jeq69ubu?nw=nwusernoaschwartzwandb&panelDisplayName=Disk+Utilization+%28GB%29&panelSectionName=System) to example panel
<img width="1281" alt="Screenshot 2025-04-10 at 1 53 01 PM" src="https://github.com/user-attachments/assets/ac88b7fc-0878-4a76-b241-651a8b3328e3" />
<img width="1282" alt="Screenshot 2025-04-10 at 1 55 02 PM" src="https://github.com/user-attachments/assets/d9897cdc-4351-47cf-9727-ecf58e4c4e61" />

For example - here I am filtering in only ranks 1-3 for Y values


created_at: 2025-04-10T20:56:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: An important piece to call out is that a label will allow you to break down your system metrics and console logs by specific label. 

This is a good resource to help add some of these details - https://www.notion.so/wandbai/How-does-a-user-use-wandb-s-Multi-Process-APIs-18de2f5c7ef38028bd25e8bfd5978b4d?pvs=4#18de2f5c7ef38075b546ebc191747a93

The other run setting to explain here would be - x_update_finish_state
created_at: 2025-04-10T20:59:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
+
+Each process requires the run ID of the primary node. Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+
+The following sample code demonstrates the high level requirements for this approach:
+
+```python
+import os
+import wandb
+
+wandb_run_id = os.environ.get("WANDB_RUN_ID", "default_run_id")
+
+# user calls init in primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label=0, mode="shared"),
+	id=wandb_run_id
+    x_primary=True,
+)
+
+# user calls init in worker node 1
+run = wandb.init(
+	settings=wandb.Settings(x_label=1, mode="shared"),
+	id=wandb_run_id,
+    x_primary=False,
+)
+
+# user calls init in workder node 2
+run = wandb.init(
+	settings=wandb.Settings(x_label=2, mode="shared"),
+	id=wandb_run_id,
+    x_primary=True,
+)
+```
+
+{{% alert %}}
+See this report for an end-to-end example on how to [train a model on a multi-node multi-GPU Kubernetes cluster in GKE](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA). 
+{{% /alert %}}
+
+You can view console logs that your multi process creates in the project that contains your run.
+
+1. Navigate to the project that contains the run.
+2. Click on the **Runs** tab in the left sidebar.
+3. Click on the run you want to view.
+4. Click on the **Logs** tab in the left sidebar.
path: content/guides/models/track/log/distributed-training.md
user: login: ngrayluna
body: Let's file a ticket to add info on filtering console logs and link that as a fast follow to this public preview PR. (Re: Filtering console logs should exist in the general docs (not hidden in Distributed logging).
created_at: 2025-04-11T02:53:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,89 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+{{% alert color="secondary"  %}}
+The option to track multiple processes to a single run is in public preview.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
The run settings used in the following workflow, include `mode="shared"` are in public preview. They are denoted with an "x_" prefix accordingly. Users can share feedback by opening a [Github issue](https://github.com/wandb/wandb/issues).  
```
created_at: 2025-04-11T03:52:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
+
+Each process requires the run ID of the primary node. Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+
+The following sample code demonstrates the high level requirements for this approach:
+
+```python
+import os
+import wandb
+
+wandb_run_id = os.environ.get("WANDB_RUN_ID", "default_run_id")
+
+# user calls init in primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label=0, mode="shared"),
+	id=wandb_run_id
+    x_primary=True,
+)
+
+# user calls init in worker node 1
+run = wandb.init(
+	settings=wandb.Settings(x_label=1, mode="shared"),
+	id=wandb_run_id,
+    x_primary=False,
+)
+
+# user calls init in workder node 2
+run = wandb.init(
+	settings=wandb.Settings(x_label=2, mode="shared"),
+	id=wandb_run_id,
+    x_primary=True,
+)
+```
+
+{{% alert %}}
+See this report for an end-to-end example on how to [train a model on a multi-node multi-GPU Kubernetes cluster in GKE](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA). 
+{{% /alert %}}
+
+You can view console logs that your multi process creates in the project that contains your run.
+
+1. Navigate to the project that contains the run.
+2. Click on the **Runs** tab in the left sidebar.
+3. Click on the run you want to view.
+4. Click on the **Logs** tab in the left sidebar.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: Hmmm... I think we definitely to include it in this version. These will be the docs we're linking from the multi-node console logs - the filtering is only applicable if you are in a distributed setup, specifically mode="shared". If you are not in this mode the filtering will not be applicable to you. 

For context - if you are in this shared mode we show a banner that notes that we can break your console logs out by label, and have a link to the docs to explain what exactly this means. 
created_at: 2025-04-11T15:59:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,89 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+{{% alert color="secondary"  %}}
+The option to track multiple processes to a single run is in public preview.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+{{% /alert  %}}
+
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
+
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: this is still missing details of what behavior happens when you indicate something is in primary mode. 
created_at: 2025-04-11T16:03:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,89 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+{{% alert color="secondary"  %}}
+The option to track multiple processes to a single run is in public preview.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+{{% /alert  %}}
+
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
+
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: https://www.notion.so/wandbai/How-does-a-user-use-wandb-s-Multi-Process-APIs-18de2f5c7ef38028bd25e8bfd5978b4d?pvs=4#18de2f5c7ef380008e16fff5c8b50895
created_at: 2025-04-11T16:04:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,59 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+In this approach you use a primary node and one or more work nodes. Within the primary node you initialize a W&B run. That primary node then spawns additional worker nodes. Each worker node logs to the same run ID as the primary node. The W&B App UI aggregates these metrics, enabling you to see data from all processes in one place.
+
+Use `wandb.Settings` to set both the `mode` parameter to `"shared"` and to specify a unique rank for each node with the `x_label` parameter. For the primary node, set `x_label` to 0 and and `x_primary` to `True`. For the worker nodes, set `x_label` to 1, 2, etc. and `x_primary` to `False`.
+
+Each process requires the run ID of the primary node. Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+
+The following sample code demonstrates the high level requirements for this approach:
+
+```python
+import os
+import wandb
+
+wandb_run_id = os.environ.get("WANDB_RUN_ID", "default_run_id")
+
+# user calls init in primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label=0, mode="shared"),
+	id=wandb_run_id
+    x_primary=True,
+)
+
+# user calls init in worker node 1
+run = wandb.init(
+	settings=wandb.Settings(x_label=1, mode="shared"),
+	id=wandb_run_id,
+    x_primary=False,
+)
+
+# user calls init in workder node 2
+run = wandb.init(
+	settings=wandb.Settings(x_label=2, mode="shared"),
+	id=wandb_run_id,
+    x_primary=True,
+)
+```
+
+{{% alert %}}
+See this report for an end-to-end example on how to [train a model on a multi-node multi-GPU Kubernetes cluster in GKE](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA). 
+{{% /alert %}}
+
+You can view console logs that your multi process creates in the project that contains your run.
+
+1. Navigate to the project that contains the run.
+2. Click on the **Runs** tab in the left sidebar.
+3. Click on the run you want to view.
+4. Click on the **Logs** tab in the left sidebar.
path: content/guides/models/track/log/distributed-training.md
user: login: ngrayluna
body: What I'm thinking is that we have a dedicated page about console logs in general. Currently there are no docs on console logs.

In it, we have a dedicated section that describes filtering logs if you have a multi-node process. And we'd link to that page for info on console logs.  From the UI, this just means having a different URL instead of the one pointing to this doc.

For now I added an image more context. For maintainability, we avoid using gifs as much as possible.
created_at: 2025-04-14T23:51:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,98 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+{{% /alert  %}}
+
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
+
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
+
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
+
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
+
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `False` to indicate that this is a worker node.
+3. Pass the run ID of the primary node to the `id` parameter.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
3. Pass the same run ID used for the primary node to the `id` parameter.
```
created_at: 2025-04-15T01:19:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,98 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+{{% /alert  %}}
+
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
+
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
+
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
+
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
+
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `False` to indicate that this is a worker node.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: x_primary is also defined in the settings - https://github.com/wandb/wandb/blob/07c45883c3287545b8618e5546f52f4d85a2109c/wandb/sdk/wandb_settings.py#L660
created_at: 2025-04-15T01:21:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,98 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+{{% /alert  %}}
+
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
+
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
+
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
+
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
+
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `False` to indicate that this is a worker node.
+3. Pass the run ID of the primary node to the `id` parameter.
+4. Optionally set `x_update_finish_state` to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
+
+{{% alert %}}
+Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+{{% /alert %}}
+
+The following sample code demonstrates the high level requirements for tracking multiple processes to a single run:
+
+```python
+import wandb
+
+# Initialize a run in the primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label="rank_0", mode="shared"),
+    x_primary=True,
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: x_primary is also a setting, not an arg to wandb.init - https://colab.research.google.com/drive/1GOOYcvAaHY8cgi1oHEUJqgd--oWl4QXO#scrollTo=gBPEe8nzzrXb&line=37&uniqifier=1
created_at: 2025-04-15T01:22:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,98 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+{{% /alert  %}}
+
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
+
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
+
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
+
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
+
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which process the data is coming from in logs and system metrics in the W&B App UI.
+2. Set the `x_primary` parameter to `False` to indicate that this is a worker node.
+3. Pass the run ID of the primary node to the `id` parameter.
+4. Optionally set `x_update_finish_state` to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
+
+{{% alert %}}
+Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+{{% /alert %}}
+
+The following sample code demonstrates the high level requirements for tracking multiple processes to a single run:
+
+```python
+import wandb
+
+# Initialize a run in the primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label="rank_0", mode="shared"),
+    x_primary=True,
+)
+
+# Note the run ID of the primary node.
+# Each worker node needs this run ID.
+run_id = run.id
+
+# Initialize a run in a worker node using the run ID of the primary node
+run = wandb.init(
+	settings=wandb.Settings(x_label="rank_1", mode="shared"),
+	id=run_id,
+    x_primary=False,
+)
+
+# Initialize a run in a worker node using the run ID of the primary node
+run = wandb.init(
+	settings=wandb.Settings(x_label="rank_2", mode="shared"),
+	id=run_id,
+    x_primary=True,
+)
+```
+
+{{% alert %}}
+See this report for an end-to-end example on how to [train a model on a multi-node multi-GPU Kubernetes cluster in GKE](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA). 
+{{% /alert %}}
+
+View console logs from multi node processes in the project that the run logs to:
+
+1. Navigate to the project that contains the run.
+2. Click on the **Runs** tab in the left sidebar.
+3. Click on the run you want to view.
+4. Click on the **Logs** tab in the left sidebar.
+
+You can filter console logs based on the labels you provide for `x_label` in the UI search bar located at the top of the console log page. For example, the following image shows which options are available to filter the console log by if values  `rank0`, `rank1`, `rank2`, `rank3`, `rank4`, `rank5`, and `rank6` are provided to `x_label`.` 
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: Note that if the user doesn't pass in a label when in shared mode, we will auto-generate a label using the hostname and a random hash
created_at: 2025-04-15T01:24:54Z
author_association: CONTRIBUTOR
path: assets/images/track/multi_node_system_metrics.png
user: login: mdlinville
body: Is this a fake user or is it showing some PII?
created_at: 2025-04-15T23:06:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
During a distributed training experiment, you train a model using multiple machines or clients in parallel. W&B can help you track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
```
created_at: 2025-04-15T23:07:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
* **Single process**: Track a rank 0 process (also known as a "leader" or "coordinator" process) with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
```
created_at: 2025-04-15T23:08:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using a different run for each process. Optionally group them together in the W&B App UI.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
   * Track each process separately using one run per process. You can optionally group them together in the W&B App UI.
```
created_at: 2025-04-15T23:09:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using a different run for each process. Optionally group them together in the W&B App UI.
+   * Track all process to a single run.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
   * Track all processes to a single run.
```
created_at: 2025-04-15T23:09:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using a different run for each process. Optionally group them together in the W&B App UI.
+   * Track all process to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Single process
 
-### Method 1: One process
+In this method you track only values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: I don't love language like "In this method". It's kind of indirect and the word "method" is fairly overloaded. If this was a how-to section, I would probably make it action oriented: "To track only values and metrics....", but this is more of a description of the situation.
created_at: 2025-04-15T23:12:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using a different run for each process. Optionally group them together in the W&B App UI.
+   * Track all process to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Single process
 
-### Method 1: One process
+In this method you track only values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+Metrics from other nodes or processes, such as loss values or inputs from training batches, are not tracked. System metrics, such as usage and memory, are logged for all GPUs since that information is available to all processes.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: "are not tracked" is a bit passive. Maybe "The experiment does not track"
created_at: 2025-04-15T23:12:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using a different run for each process. Optionally group them together in the W&B App UI.
+   * Track all process to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Single process
 
-### Method 1: One process
+In this method you track only values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+Metrics from other nodes or processes, such as loss values or inputs from training batches, are not tracked. System metrics, such as usage and memory, are logged for all GPUs since that information is available to all processes.
 
 {{% alert %}}
 **Use this method to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: Another "this method"
created_at: 2025-04-15T23:13:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using a different run for each process. Optionally group them together in the W&B App UI.
+   * Track all process to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Single process
 
-### Method 1: One process
+In this method you track only values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+Metrics from other nodes or processes, such as loss values or inputs from training batches, are not tracked. System metrics, such as usage and memory, are logged for all GPUs since that information is available to all processes.
 
 {{% alert %}}
 **Use this method to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
 {{% /alert %}}
 
-Within our [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py), we check to see if the rank is 0. To do so, we first launch multiple processes with `torch.distributed.launch`. Next, we check the rank with the `--local_rank` command line argument. If the rank is set to 0, we set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function. Within our Python script, we use the following check:
+As an example, the following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrate how to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: Worth using Prism, since youa re naming the example?
created_at: 2025-04-15T23:13:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using a different run for each process. Optionally group them together in the W&B App UI.
+   * Track all process to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Single process
 
-### Method 1: One process
+In this method you track only values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+Metrics from other nodes or processes, such as loss values or inputs from training batches, are not tracked. System metrics, such as usage and memory, are logged for all GPUs since that information is available to all processes.
 
 {{% alert %}}
 **Use this method to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
 {{% /alert %}}
 
-Within our [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py), we check to see if the rank is 0. To do so, we first launch multiple processes with `torch.distributed.launch`. Next, we check the rank with the `--local_rank` command line argument. If the rank is set to 0, we set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function. Within our Python script, we use the following check:
+As an example, the following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrate how to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
As an example, the following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrate how to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the implementation may differ.
```
created_at: 2025-04-15T23:14:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 (also known as a "leader" or "coordinator") process with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using a different run for each process. Optionally group them together in the W&B App UI.
+   * Track all process to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Single process
 
-### Method 1: One process
+In this method you track only values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+Metrics from other nodes or processes, such as loss values or inputs from training batches, are not tracked. System metrics, such as usage and memory, are logged for all GPUs since that information is available to all processes.
 
 {{% alert %}}
 **Use this method to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
 {{% /alert %}}
 
-Within our [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py), we check to see if the rank is 0. To do so, we first launch multiple processes with `torch.distributed.launch`. Next, we check the rank with the `--local_rank` command line argument. If the rank is set to 0, we set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function. Within our Python script, we use the following check:
+As an example, the following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrate how to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+
+The Python script first start multiple processes with `torch.distributed.launch`. Next, it checks the rank with the `--local_rank` command line argument. If the rank is set to 0, set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
The Python script first starts multiple processes with `torch.distributed.launch`. Next, it checks the rank with the `--local_rank` command line argument. If the rank is set to 0, set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function.
```
This would be more readable as a numbered list.
created_at: 2025-04-15T23:14:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -45,19 +51,21 @@ if __name__ == "__main__":
         train(args)
 ```
 
-Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system) of metrics tracked from a single process. The dashboard displays system metrics such as temperature and utilization, that were tracked for both GPUs.
+Explore this example dashboard in the W&B App UI to [view metrics tracked from a single process](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system). The dashboard displays system metrics such as temperature and utilization, that were tracked for both GPUs.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: I'd prefer for the link to come earlier, maybe the link text could be "an example dashboard".

The last sentence is a bit of a jumble. I'd wish for something like "The dashboard displays system metrics for both GPUs, such as temperature and utilization."
created_at: 2025-04-15T23:16:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -45,19 +51,21 @@ if __name__ == "__main__":
         train(args)
 ```
 
-Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system) of metrics tracked from a single process. The dashboard displays system metrics such as temperature and utilization, that were tracked for both GPUs.
+Explore this example dashboard in the W&B App UI to [view metrics tracked from a single process](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system). The dashboard displays system metrics such as temperature and utilization, that were tracked for both GPUs.
 
 {{< img src="/images/track/distributed_training_method1.png" alt="" >}}
 
 However, the loss values as a function epoch and batch size were only logged from a single GPU.
 
 {{< img src="/images/experiments/loss_function_single_gpu.png" alt="" >}}
 
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: Alt text :) 
created_at: 2025-04-15T23:16:47Z
author_association: CONTRIBUTOR
path: assets/images/track/multi_node_system_metrics.png
user: login: noaleetz
body: this a fake demo project
created_at: 2025-04-16T20:47:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,96 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: this is also has a server release constraints - https://weightsandbiases.slack.com/archives/C081QT67T35/p1744344727024289
created_at: 2025-04-16T20:48:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -122,11 +219,11 @@ For optimal experience we do recommend you upgrade to the latest version.
 
 Set the `WANDB_START_METHOD` environment variable to `"thread"` to use multithreading instead if you use a W&B SDK version 0.12.4 and below.
 
-### Example use cases for multiprocessing
+## Example use cases for multiprocessing
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: the 2 main use caes would be:
- 1 training job distributed on multiple nodes or GPUs - https://www.notion.so/wandbai/PRD-Distributed-Training-in-wandb-129e2f5c7ef38001801ec5298a70836e?pvs=4#129e2f5c7ef3802e9d58ec1e101ca653
- 1 main training process with async eval processes - https://www.notion.so/wandbai/Shared-Mode-for-training-eval-runs-customer-use-cases-160e2f5c7ef380399c9cce251250d2e1?pvs=4
created_at: 2025-04-16T21:11:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,96 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
path: content/guides/models/track/log/distributed-training.md
user: login: ngrayluna
body: Noted. Waiting on confirmation from thread.
created_at: 2025-04-16T23:19:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: @ngrayluna - FYI Let's make sure we are keeping support + documentation for the older grouping method (i.e where you spin up an individual run and define them in a group). 

Because the new workflow is in experimental, and for example has performance limitations (no more than 400 parallel writers has been stress tested) we want to make sure we keep this as an option. 
created_at: 2025-04-17T16:33:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During distributed training you train models using multiple machines or clients in parallel. You can use W&B to track distributed training experiments. Based on your use case, use one of the following methods to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
path: content/guides/models/track/log/distributed-training.md
user: login: ngrayluna
body: @noaleetz It is. Content was moved, edited to reflect how the new process fits in with the overall narrative. 
created_at: 2025-04-17T16:59:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During a distributed training experiment, you train a model using multiple machines or clients in parallel. W&B can help you track distributed training experiments. Based on your use case, use one of the following approaches to track distributed training experiments:
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
During a distributed training experiment, you train a model using multiple machines or clients in parallel. W&B can help you track distributed training experiments. Based on your use case, track distributed training experiments using of the following approaches:
```
Trying to avoid "your use case, use"
created_at: 2025-04-17T17:40:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During a distributed training experiment, you train a model using multiple machines or clients in parallel. W&B can help you track distributed training experiments. Based on your use case, use one of the following approaches to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 process (also known as a "leader" or "coordinator") with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using one run per process. You can optionally group them together in the W&B App UI.
+   * Track all processes to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Track a single process
 
-### Method 1: One process
+This section describes how to track values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+W&B does not track metrics from other nodes or processes, such as loss values or inputs from training batches. W&B does track system metrics, such as usage and memory, for all GPUs since that information is available to all processes.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
W&B tracks system metrics, such as usage and memory, for all GPUs since that information is available to all processes. W&B does not track metrics from other nodes or processes, such as loss values or inputs from training batches. 
```
Flipping these around to avoid the need for an intensifying "does".
created_at: 2025-04-17T17:41:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During a distributed training experiment, you train a model using multiple machines or clients in parallel. W&B can help you track distributed training experiments. Based on your use case, use one of the following approaches to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 process (also known as a "leader" or "coordinator") with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using one run per process. You can optionally group them together in the W&B App UI.
+   * Track all processes to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Track a single process
 
-### Method 1: One process
+This section describes how to track values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+W&B does not track metrics from other nodes or processes, such as loss values or inputs from training batches. W&B does track system metrics, such as usage and memory, for all GPUs since that information is available to all processes.
 
 {{% alert %}}
-**Use this method to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
+**Use this approach to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
**Use this approach to track only metrics that are available from a single process**. Typical metrics include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
```
created_at: 2025-04-17T17:42:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During a distributed training experiment, you train a model using multiple machines or clients in parallel. W&B can help you track distributed training experiments. Based on your use case, use one of the following approaches to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 process (also known as a "leader" or "coordinator") with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using one run per process. You can optionally group them together in the W&B App UI.
+   * Track all processes to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Track a single process
 
-### Method 1: One process
+This section describes how to track values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+W&B does not track metrics from other nodes or processes, such as loss values or inputs from training batches. W&B does track system metrics, such as usage and memory, for all GPUs since that information is available to all processes.
 
 {{% alert %}}
-**Use this method to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
+**Use this approach to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
 {{% /alert %}}
 
-Within our [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py), we check to see if the rank is 0. To do so, we first launch multiple processes with `torch.distributed.launch`. Next, we check the rank with the `--local_rank` command line argument. If the rank is set to 0, we set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function. Within our Python script, we use the following check:
+As an example, the following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrates how to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the implementation may differ.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
The following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrates one way to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the implementation may differ.
```
created_at: 2025-04-17T17:43:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,26 +7,32 @@ menu:
 title: Log distributed training experiments
 ---
 
-In distributed training, models are trained using multiple GPUs in parallel. W&B supports two patterns to track distributed training experiments:
+During a distributed training experiment, you train a model using multiple machines or clients in parallel. W&B can help you track distributed training experiments. Based on your use case, use one of the following approaches to track distributed training experiments:
 
-1. **One process**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) from a single process. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. In some cases, users funnel data over from other processes using a multiprocessing queue (or another communication primitive) to the main logging process.
-2. **Many processes**: Initialize W&B ([`wandb.init`]({{< relref "/ref//python/init.md" >}})) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) in every process. Each process is effectively a separate experiment. Use the `group` parameter when you initialize W&B (`wandb.init(group='group-name')`) to define a shared experiment and group the logged values together in the W&B App UI.
+* **Single process**: Track a rank 0 process (also known as a "leader" or "coordinator") with W&B. This is a common solution for logging distributed training experiments with the [PyTorch Distributed Data Parallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) Class. 
+* **Multiple processes**: For multiple processes, you can either:
+   * Track each process separately using one run per process. You can optionally group them together in the W&B App UI.
+   * Track all processes to a single run.
 
-The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
+<!-- The proceeding examples demonstrate how to track metrics with W&B using PyTorch DDP on two GPUs on a single machine. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the details of implementation may differ.
 
 {{% alert %}}
 Explore the code behind these examples in the W&B GitHub examples repository [here](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-ddp). Specifically, see the [`log-dpp.py`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) Python script for information on how to implement one process and many process methods.
-{{% /alert %}}
+{{% /alert %}} -->
+
+## Track a single process
 
-### Method 1: One process
+This section describes how to track values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-In this method we track only a rank 0 process. To implement this method, initialize W&B (`wandb.init)`, commence a W&B Run, and log metrics (`wandb.log`) within the rank 0 process. This method is simple and robust, however, this method does not log model metrics from other processes (for example, loss values or inputs from their batches). System metrics, such as usage and memory, are still logged for all GPUs since that information is available to all processes.
+W&B does not track metrics from other nodes or processes, such as loss values or inputs from training batches. W&B does track system metrics, such as usage and memory, for all GPUs since that information is available to all processes.
 
 {{% alert %}}
-**Use this method to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
+**Use this approach to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
 {{% /alert %}}
 
-Within our [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py), we check to see if the rank is 0. To do so, we first launch multiple processes with `torch.distributed.launch`. Next, we check the rank with the `--local_rank` command line argument. If the rank is set to 0, we set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function. Within our Python script, we use the following check:
+As an example, the following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrates how to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the implementation may differ.
+
+The Python script first starts multiple processes with `torch.distributed.launch`. Next, it checks the rank with the `--local_rank` command line argument. If the rank is set to 0, set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
The Python script:
1. Starts multiple processes with `torch.distributed.launch`.
1. Checks the rank with the `--local_rank` command line argument.
1. If the rank is set to 0, sets up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function.
```
created_at: 2025-04-17T17:44:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -45,22 +51,24 @@ if __name__ == "__main__":
         train(args)
 ```
 
-Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system) of metrics tracked from a single process. The dashboard displays system metrics such as temperature and utilization, that were tracked for both GPUs.
+Explore this example dashboard in the W&B App UI to [view metrics tracked from a single process](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system). The dashboard displays system metrics for both GPUs, such as temperature and utilization.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
Explore an [example dashboard showing metrics tracked from a single process](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system).

The dashboard displays system metrics for both GPUs, such as temperature and utilization.
```
Splitting this up helps with the "However" sentence below and with the interspersed images.
created_at: 2025-04-17T17:45:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -45,22 +51,24 @@ if __name__ == "__main__":
         train(args)
 ```
 
-Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system) of metrics tracked from a single process. The dashboard displays system metrics such as temperature and utilization, that were tracked for both GPUs.
+Explore this example dashboard in the W&B App UI to [view metrics tracked from a single process](https://wandb.ai/ayush-thakur/DDP/runs/1s56u3hc/system). The dashboard displays system metrics for both GPUs, such as temperature and utilization.
 
-{{< img src="/images/track/distributed_training_method1.png" alt="" >}}
+{{< img src="/images/track/distributed_training_method1.png" alt="Workspace that shows four line plot panels. From left to right the plots are: GPU Utilization (%), GPU Temp (C), GPU Time Spent Accessing Memory (%), and GPU Memory Allocated (%)" >}}
 
 However, the loss values as a function epoch and batch size were only logged from a single GPU.
 
-{{< img src="/images/experiments/loss_function_single_gpu.png" alt="" >}}
+{{< img src="/images/experiments/loss_function_single_gpu.png" alt="Two line plot panels in an example workspace. The left plot shows the loss measured as a function of step. The right plot show the loss recorded at each batch." >}}
 
-### Method 2: Many processes
+## Track multiple processes
 
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: Ideally you'd have a sentence here explaining what the subsections are about, or you would skip this H2 and promote the ones below from H3 to H2. Skimming the structure of the rest of the page, I bet the promotion option would be fine.
created_at: 2025-04-17T17:48:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,7 +93,96 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
+
+
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
+
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+{{% /alert  %}}
+
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
+
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) wit with the following:
+
+1. The `mode` parameter set to `"shared"` to enable shared mode.
+2. A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+3. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
+
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
+
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
+
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+   * Set the `x_primary` parameter to `False` to indicate that this is a worker node.
+2. Pass the run ID used by the primary node to the `id` parameter.
+3. Optionally set `x_update_finish_state` to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
+
+{{% alert %}}
+Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+{{% /alert %}}
+
+The following sample code demonstrates the high level requirements for tracking multiple processes to a single run:
+
+```python
+import wandb
+
+# Initialize a run in the primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label="rank_0", mode="shared", x_primary=True)
+)
+
+# Note the run ID of the primary node.
+# Each worker node needs this run ID.
+run_id = run.id
+
+# Initialize a run in a worker node using the run ID of the primary node
+run = wandb.init(
+	settings=wandb.Settings(x_label="rank_1", mode="shared", x_primary=False),
+	id=run_id,
+)
+
+# Initialize a run in a worker node using the run ID of the primary node
+run = wandb.init(
+	settings=wandb.Settings(x_label="rank_2", mode="shared", x_primary=True),
+	id=run_id,
+)
+```
+
+In a real world example, each worker node might be on a separate machine.
+
+{{% alert %}}
+See the [Distributed Training with Shared Mode](https://wandb.ai/dimaduev/simple-cnn-ddp/reports/Distributed-Training-with-Shared-Mode--VmlldzoxMTI0NTE1NA) report for an end-to-end example on how to train a model on a multi-node and multi-GPU Kubernetes cluster in GKE.
+{{% /alert %}}
+
+View console logs from multi node processes in the project that the run logs to:
+
+1. Navigate to the project that contains the run.
+2. Click on the **Runs** tab in the left sidebar.
+3. Click on the run you want to view.
+4. Click on the **Logs** tab in the left sidebar.
+
+You can filter console logs based on the labels you provide for `x_label` in the UI search bar located at the top of the console log page. For example, the following image shows which options are available to filter the console log by if values  `rank0`, `rank1`, `rank2`, `rank3`, `rank4`, `rank5`, and `rank6` are provided to `x_label`.` 
+
+{{< img src="/images/track/multi_node_console_logs.png" alt="Console logs from multiple nodes with the x_label filter applied." >}}
+
+W&B aggregates system metrics from all nodes and displays them in the W&B App UI. For example, the following image shows a sample dashboard with system metrics from multiple nodes. Each node possesses a unique label (`rank_0`, `rank_1`, `rank_2`) that you specify in the `x_label` parameter.
+
+{{< img src="/images/track/multi_node_system_metrics.png" alt="Line plot panel with numerous semi linear lines depicting system metrics logged by a multi node process." >}}
+
+See [Line plots]({{< relref "/guides/models/app/features/panels/line-plot/" >}}) for information on how to customize line plot panels. 
+
+## Avoid common distributed training issues with W&B Service
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: This might be better as "Troubleshooting" or "Antipatterns" or something more terse. It seems like it should go after the Examples below.
created_at: 2025-04-17T17:48:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -122,11 +219,11 @@ For optimal experience we do recommend you upgrade to the latest version.
 
 Set the `WANDB_START_METHOD` environment variable to `"thread"` to use multithreading instead if you use a W&B SDK version 0.12.4 and below.
 
-### Example use cases for multiprocessing
+## Example use cases for multiprocessing
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: ```suggestion
## Example use cases
```
created_at: 2025-04-17T17:49:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -149,9 +246,9 @@ if __name__ == "__main__":
     main()
 ```
 
-#### Share a W&B Run
+### Share a run
 
-Pass a W&B Run object as an argument to share W&B Runs between processes:
+Pass a W&B run object as an argument to share runs between processes:
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: You capitalize Run up in line 228 but not here. This doesn't feel to me like a use case exactly.
created_at: 2025-04-17T17:51:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -122,11 +219,11 @@ For optimal experience we do recommend you upgrade to the latest version.
 
 Set the `WANDB_START_METHOD` environment variable to `"thread"` to use multithreading instead if you use a W&B SDK version 0.12.4 and below.
 
-### Example use cases for multiprocessing
+## Example use cases for multiprocessing
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: should this be resolved?
created_at: 2025-04-17T18:50:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -149,9 +246,9 @@ if __name__ == "__main__":
     main()
 ```
 
-#### Share a W&B Run
+### Share a run
 
-Pass a W&B Run object as an argument to share W&B Runs between processes:
+Pass a W&B run object as an argument to share runs between processes:
path: content/guides/models/track/log/distributed-training.md
user: login: ngrayluna
body: Oops. Both should be lower case. Fixed.
created_at: 2025-04-17T20:28:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,60 +102,109 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
 
-There are two common issues you might encounter when using W&B and distributed training:
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
 
-1. **Hanging at the beginning of training** - A `wandb` process can hang if the `wandb` multiprocessing interferes with the multiprocessing from distributed training.
-2. **Hanging at the end of training** - A training job might hang if the `wandb` process does not know when it needs to exit. Call the `wandb.finish()` API at the end of your Python script to tell W&B that the Run finished. The wandb.finish() API will finish uploading data and will cause W&B to exit.
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.5`.
+{{% /alert  %}}
 
-We recommend using the `wandb service` to improve the reliability of your distributed jobs. Both of the preceding training issues are commonly found in versions of the W&B SDK where wandb service is unavailable.
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
 
-### Enable W&B Service
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) wit with the following:
 
-Depending on your version of the W&B SDK, you might already have W&B Service enabled by default.
+1. The `mode` parameter set to `"shared"` to enable shared mode.
+2. A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+3. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
 
-#### W&B SDK 0.13.0 and above
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
 
-W&B Service is enabled by default for versions of the W&B SDK `0.13.0` and above.
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
 
-#### W&B SDK 0.12.5 and above
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+   * Set the `x_primary` parameter to `False` to indicate that this is a worker node.
+2. Pass the run ID used by the primary node to the `id` parameter.
+3. Optionally set `x_update_finish_state` to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
 
-Modify your Python script to enable W&B Service for W&B SDK version 0.12.5 and above. Use the `wandb.require` method and pass the string `"service"` within your main function:
+{{% alert %}}
+Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
+{{% /alert %}}
+
+The following sample code demonstrates the high level requirements for tracking multiple processes to a single run:
 
 ```python
-if __name__ == "__main__":
-    main()
+import wandb
+
+# Initialize a run in the primary node
+run = wandb.init(
+    entity="entity",
+    project="project",
+	settings=wandb.Settings(x_label="rank_0", mode="shared", x_primary=True)
+)
+
+# Note the run ID of the primary node.
+# Each worker node needs this run ID.
+run_id = run.id
+
+# Initialize a run in a worker node using the run ID of the primary node
+run = wandb.init(
+	settings=wandb.Settings(x_label="rank_1", mode="shared", x_primary=False),
+	id=run_id,
+)
+
+# Initialize a run in a worker node using the run ID of the primary node
+run = wandb.init(
+	settings=wandb.Settings(x_label="rank_2", mode="shared", x_primary=True),
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: why is this set to true? not a primary node
created_at: 2025-04-17T20:53:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,60 +102,109 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
 
-There are two common issues you might encounter when using W&B and distributed training:
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
 
-1. **Hanging at the beginning of training** - A `wandb` process can hang if the `wandb` multiprocessing interferes with the multiprocessing from distributed training.
-2. **Hanging at the end of training** - A training job might hang if the `wandb` process does not know when it needs to exit. Call the `wandb.finish()` API at the end of your Python script to tell W&B that the Run finished. The wandb.finish() API will finish uploading data and will cause W&B to exit.
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.5`.
+{{% /alert  %}}
 
-We recommend using the `wandb service` to improve the reliability of your distributed jobs. Both of the preceding training issues are commonly found in versions of the W&B SDK where wandb service is unavailable.
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
 
-### Enable W&B Service
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) wit with the following:
 
-Depending on your version of the W&B SDK, you might already have W&B Service enabled by default.
+1. The `mode` parameter set to `"shared"` to enable shared mode.
+2. A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+3. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
 
-#### W&B SDK 0.13.0 and above
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
 
-W&B Service is enabled by default for versions of the W&B SDK `0.13.0` and above.
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
 
-#### W&B SDK 0.12.5 and above
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+   * Set the `x_primary` parameter to `False` to indicate that this is a worker node.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
   * Set the `[x_primary](https://github.com/wandb/wandb/blob/6cf0e1d5e1da665dd0ce4691ae77fba85a56c5c9/wandb/sdk/wandb_settings.py#L660)` parameter to `False` to indicate that this is a worker node.
```
created_at: 2025-04-17T20:54:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,60 +102,109 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
 
-There are two common issues you might encounter when using W&B and distributed training:
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
 
-1. **Hanging at the beginning of training** - A `wandb` process can hang if the `wandb` multiprocessing interferes with the multiprocessing from distributed training.
-2. **Hanging at the end of training** - A training job might hang if the `wandb` process does not know when it needs to exit. Call the `wandb.finish()` API at the end of your Python script to tell W&B that the Run finished. The wandb.finish() API will finish uploading data and will cause W&B to exit.
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.5`.
+{{% /alert  %}}
 
-We recommend using the `wandb service` to improve the reliability of your distributed jobs. Both of the preceding training issues are commonly found in versions of the W&B SDK where wandb service is unavailable.
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
 
-### Enable W&B Service
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) wit with the following:
 
-Depending on your version of the W&B SDK, you might already have W&B Service enabled by default.
+1. The `mode` parameter set to `"shared"` to enable shared mode.
+2. A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+3. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
 
-#### W&B SDK 0.13.0 and above
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
 
-W&B Service is enabled by default for versions of the W&B SDK `0.13.0` and above.
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
 
-#### W&B SDK 0.12.5 and above
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
   * A unique label for `[x_label](https://github.com/wandb/wandb/blob/6cf0e1d5e1da665dd0ce4691ae77fba85a56c5c9/wandb/sdk/wandb_settings.py#L638)`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
```
created_at: 2025-04-17T20:55:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,60 +102,109 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
 
-There are two common issues you might encounter when using W&B and distributed training:
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
 
-1. **Hanging at the beginning of training** - A `wandb` process can hang if the `wandb` multiprocessing interferes with the multiprocessing from distributed training.
-2. **Hanging at the end of training** - A training job might hang if the `wandb` process does not know when it needs to exit. Call the `wandb.finish()` API at the end of your Python script to tell W&B that the Run finished. The wandb.finish() API will finish uploading data and will cause W&B to exit.
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.5`.
+{{% /alert  %}}
 
-We recommend using the `wandb service` to improve the reliability of your distributed jobs. Both of the preceding training issues are commonly found in versions of the W&B SDK where wandb service is unavailable.
+In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
 
-### Enable W&B Service
+Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) wit with the following:
 
-Depending on your version of the W&B SDK, you might already have W&B Service enabled by default.
+1. The `mode` parameter set to `"shared"` to enable shared mode.
+2. A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+3. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
 
-#### W&B SDK 0.13.0 and above
+Make note of the run ID of the primary node. Each worker node needs the run ID of the primary node.
 
-W&B Service is enabled by default for versions of the W&B SDK `0.13.0` and above.
+{{% alert %}}
+`x_primary=True` distinguishes a primary node from worker nodes. Primary nodes are the only nodes that upload files shared across nodes such as configuration files, telemetry and more. Worker nodes do not upload these files.
+{{% /alert %}}
 
-#### W&B SDK 0.12.5 and above
+For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}) and provide the following:
+1. A `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) with:
+   * The `mode` parameter set to `"shared"` to enable shared mode.
+   * A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
+   * Set the `x_primary` parameter to `False` to indicate that this is a worker node.
+2. Pass the run ID used by the primary node to the `id` parameter.
+3. Optionally set `x_update_finish_state` to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: ```suggestion
3. Optionally set `[x_update_finish_state](https://github.com/wandb/wandb/blob/6cf0e1d5e1da665dd0ce4691ae77fba85a56c5c9/wandb/sdk/wandb_settings.py#L766)` to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
```
created_at: 2025-04-17T20:56:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,60 +102,109 @@ Explore the W&B App UI to view an [example dashboard](https://wandb.ai/ayush-tha
 
 The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see two experiments. One labeled 'null' and a second (bound by a yellow box) called 'DPP'. If you expand the group (select the Group dropdown) you will see the W&B Runs that are associated to that experiment.
 
-### Use W&B Service to avoid common distributed training issues
+### Track all processes to a single run
 
-There are two common issues you might encounter when using W&B and distributed training:
+{{% alert color="secondary"  %}}
+Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
+{{% /alert %}}
 
-1. **Hanging at the beginning of training** - A `wandb` process can hang if the `wandb` multiprocessing interferes with the multiprocessing from distributed training.
-2. **Hanging at the end of training** - A training job might hang if the `wandb` process does not know when it needs to exit. Call the `wandb.finish()` API at the end of your Python script to tell W&B that the Run finished. The wandb.finish() API will finish uploading data and will cause W&B to exit.
+{{% alert %}}
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.5`.
path: content/guides/models/track/log/distributed-training.md
user: login: noaleetz
body: missing server release requirement
created_at: 2025-04-17T20:57:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,15 +24,18 @@ Explore the code behind these examples in the W&B GitHub examples repository [he
 
 This section describes how to track values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-W&B does not track metrics from other nodes or processes, such as loss values or inputs from training batches. W&B does track system metrics, such as usage and memory, for all GPUs since that information is available to all processes.
+W&B tracks system metrics, such as usage and memory, for all GPUs since that information is available to all processes. W&B does not track metrics from other nodes or processes, such as loss values or inputs, from training batches. 
 
 {{% alert %}}
-**Use this approach to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
+**Use this approach to track only metrics that are available from a single process**. Typical metrics include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
 {{% /alert %}}
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: The para before the alert is difficult to parse, and I feel that the first sentence is a run on. I think what you are trying to say is that  no action is needed to get system metrics for the process its node, but that processes on other nodes are opaque.

The details inside the alert partially duplicate the details about the types of metrics. I am not sure that the detail about typical metrics belongs in an alert.
created_at: 2025-04-18T20:07:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,15 +24,18 @@ Explore the code behind these examples in the W&B GitHub examples repository [he
 
 This section describes how to track values and metrics available to your rank 0 process. Within the rank 0 process, initialize a W&B run with [`wandb.init`]({{< relref "/ref//python/init.md" >}}) and log experiments ([`wandb.log`]({{< relref "/ref//python/log.md" >}})) to that run.
 
-W&B does not track metrics from other nodes or processes, such as loss values or inputs from training batches. W&B does track system metrics, such as usage and memory, for all GPUs since that information is available to all processes.
+W&B tracks system metrics, such as usage and memory, for all GPUs since that information is available to all processes. W&B does not track metrics from other nodes or processes, such as loss values or inputs, from training batches. 
 
 {{% alert %}}
-**Use this approach to only track metrics available from a single process**. Typical examples include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
+**Use this approach to track only metrics that are available from a single process**. Typical metrics include GPU/CPU utilization, behavior on a shared validation set, gradients and parameters, and loss values on representative data examples.
 {{% /alert %}}
 
-As an example, the following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrates how to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the implementation may differ.
+The following [sample Python script (`log-ddp.py`)](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py) demonstrates one way to track metrics on two GPUs on a single machine using PyTorch DDP. [PyTorch DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) (`DistributedDataParallel` in`torch.nn`) is a popular library for distributed training. The basic principles apply to any distributed training setup, but the implementation may differ.
 
-The Python script first starts multiple processes with `torch.distributed.launch`. Next, it checks the rank with the `--local_rank` command line argument. If the rank is set to 0, set up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function.
+The Python script:
+1. Starts multiple processes with `torch.distributed.launch`.
+1. Checks the rank with the `--local_rank` command line argument.
+1. If the rank is set to 0, sets up `wandb` logging conditionally in the [`train()`](https://github.com/wandb/examples/blob/master/examples/pytorch/pytorch-ddp/log-ddp.py#L24) function.
 
 ```python
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: I still think this might be better to use the Prism shortcode since you explicitly name the script here. You can grep around for some examples.
created_at: 2025-04-18T20:08:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,22 +104,21 @@ The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see
 
 ### Track all processes to a single run
 
-
 {{% alert color="secondary"  %}}
 Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
 {{% /alert %}}
 
 {{% alert %}}
-To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.5`.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: This might be better as "v0.19.5 or newer" or something like that rather than the math shorthand.
created_at: 2025-04-18T20:10:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,22 +104,21 @@ The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see
 
 ### Track all processes to a single run
 
-
 {{% alert color="secondary"  %}}
 Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
 {{% /alert %}}
 
 {{% alert %}}
-To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.5`.
 {{% /alert  %}}
 
 In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
 
 Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) wit with the following:
 
 1. The `mode` parameter set to `"shared"` to enable shared mode.
-2. A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
-3. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
+2. A unique label for [`x_label`](https://github.com/wandb/wandb/blob/6cf0e1d5e1da665dd0ce4691ae77fba85a56c5c9/wandb/sdk/wandb_settings.py#L638). You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: Try to avoid linking to a SHA. Can you just link to `main`? Also a few lines down.
created_at: 2025-04-18T20:10:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -124,7 +132,7 @@ For each worker node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/
    * A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
    * Set the `x_primary` parameter to `False` to indicate that this is a worker node.
 2. Pass the run ID used by the primary node to the `id` parameter.
-3. Optionally set `x_update_finish_state` to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
+3. Optionally set [`x_update_finish_state`](https://github.com/wandb/wandb/blob/6cf0e1d5e1da665dd0ce4691ae77fba85a56c5c9/wandb/sdk/wandb_settings.py#L766) to `False`. This prevents non-primary nodes from updating the [run's state]({{< relref "/guides/models/track/runs/#run-states" >}}) to `finished` prematurely, ensuring the run state remains consistent and managed by the primary node.
 
 {{% alert %}}
 Consider using an environment variable to set the run ID of the primary node that you can then define in each worker node's machine.
path: content/guides/models/track/log/distributed-training.md
user: login: mdlinville
body: Does this need to be an alert? This page has a lot of alerts.
created_at: 2025-04-18T20:11:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,22 +104,21 @@ The preceding image demonstrates the W&B App UI dashboard. On the sidebar we see
 
 ### Track all processes to a single run
 
-
 {{% alert color="secondary"  %}}
 Parameters prefixed by `x_` (such as `x_label`) are in public preview. Create a GitHub issue at [https://github.com/wandb/wandb](https://github.com/wandb/wandb) to provide feedback.
 {{% /alert %}}
 
 {{% alert %}}
-To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.0`.
+To track multiple processes to a single run, you must have W&B Python SDK version `>=0.19.5`.
 {{% /alert  %}}
 
 In this approach you use a primary node and one or more worker nodes. Within the primary node you initialize a W&B run. For each worker node, initialize a run using the run ID used by the primary node. During training each worker node logs to the same run ID as the primary node. W&B aggregates metrics from all nodes and displays them in the W&B App UI.
 
 Within the primary node, initialize a W&B run with [`wandb.init`]({{< relref "/ref/python/init.md" >}}). Pass in a `wandb.Settings` object to the `settings` parameter (`wandb.init(settings=wandb.Settings()`) wit with the following:
 
 1. The `mode` parameter set to `"shared"` to enable shared mode.
-2. A unique label for `x_label`. You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
-3. Set the `x_primary` parameter to `True` to indicate that this is the primary node.
+2. A unique label for [`x_label`](https://github.com/wandb/wandb/blob/6cf0e1d5e1da665dd0ce4691ae77fba85a56c5c9/wandb/sdk/wandb_settings.py#L638). You use the value you specify for `x_label` to identify which node the data is coming from in logs and system metrics in the W&B App UI. If left unspecified, W&B creates a label for you using the hostname and a random hash.
path: content/guides/models/track/log/distributed-training.md
user: login: ngrayluna
body: ooops. Good catch
created_at: 2025-04-18T20:19:04Z
author_association: CONTRIBUTOR

PR comments:
user: login: cloudflare-workers-and-pages[bot]
created_at: 2025-04-10T03:40:11Z
body: ## Deploying docs with &nbsp;<a href="https://pages.dev"><img alt="Cloudflare Pages" src="https://user-images.githubusercontent.com/23264/106598434-9e719e00-654f-11eb-9e59-6167043cfa01.png" width="16"></a> &nbsp;Cloudflare Pages

<table><tr><td><strong>Latest commit:</strong> </td><td>
<code>649dcea</code>
</td></tr>
<tr><td><strong>Status:</strong></td><td>&nbsp;✅&nbsp; Deploy successful!</td></tr>
<tr><td><strong>Preview URL:</strong></td><td>
<a href='https://f63c3744.docodile.pages.dev'>https://f63c3744.docodile.pages.dev</a>
</td></tr>
<tr><td><strong>Branch Preview URL:</strong></td><td>
<a href='https://multi-node-runs.docodile.pages.dev'>https://multi-node-runs.docodile.pages.dev</a>
</td></tr>
</table>

[View logs](https://dash.cloudflare.com/832d6d41f129452d5f4797d566bd6cc8/pages/view/docs/f63c3744-1326-4d35-abd4-b08062577fde)

author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-11T02:55:54Z
body: Images automagically compressed by [Calibre](https://calibreapp.com)'s [image-actions](https://github.com/marketplace/actions/image-actions) ✨

Compression reduced images by <strong>44.6%</strong>, saving <strong>235.95 KB</strong>.

| Filename | Before | After | Improvement | Visual comparison |
| -------- | ------ | ----- | ----------- | ----------------- |
| <code>assets/images/track/multi_node_system_metrics.png</code> | 529.03 KB | 293.08 KB | -44.6% | [View diff](/wandb/docs/pull/1246/commits/50fe22eca88d924cb1605218e6228f439a207223?short_path=7e06abf#diff-7e06abf2e70d18ccd3814b8f1feb7faf) |


437 images did not require optimisation.

author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-14T23:53:30Z
body: Images automagically compressed by [Calibre](https://calibreapp.com)'s [image-actions](https://github.com/marketplace/actions/image-actions) ✨

Compression reduced images by <strong>46.2%</strong>, saving <strong>263.42 KB</strong>.

| Filename | Before | After | Improvement | Visual comparison |
| -------- | ------ | ----- | ----------- | ----------------- |
| <code>assets/images/track/multi_node_console_logs.png</code> | 569.74 KB | 306.32 KB | -46.2% | [View diff](/wandb/docs/pull/1246/commits/7f46c1dc6bc47abe616f801cceb63f72af9d0248?short_path=a519b87#diff-a519b879508b65639216cf8eea1c3eeb) |


438 images did not require optimisation.

author_association: CONTRIBUTOR