Repository: deepspeedai/DeepSpeed
Pull request: Fix --enable_each_rank_log when used with PDSH multi-node runner
This PR addresses fixes https://github.com/microsoft/DeepSpeed/issues/6859 by threading this argument into the deepspeed launcher command build by PDSHRunner.

Problem statement:
[BUG] deepspeed --enable_each_rank_log does not work in multi-node (PDSH) runs
**Describe the bug**

In a single-node training run, the command `deepspeed --enable_each_rank_log logdir <training command here>` will cause each rank to write its stderr/stdout to a unique file in logdir/

However, in a multinode training run using the default launcher (PDSH) e.g. `deepspeed --hostfile ./hostfile --enable_each_rank_log logdir` the arg is not passed through to the local launcher script per node.

**To Reproduce**
Run a deepspeed training run with --enable_each_rank_log, and with a hostfile that specifies multiple nodes. Observe that each rank's stdout/stderr is not directed to a file.

**Expected behavior**
Each rank should log to its own file.

**Launcher context**
deepspeed launcher + PDSH

Hints from issue discussion:
The issue appears to me that in the PDSHRunner e.g. https://github.com/microsoft/DeepSpeed/blob/91829476a8fd4d0d9268c03c1d56795d20a51c12/deepspeed/launcher/multinode_runner.py#L99 we are reconstructing launcher args from the runner args, but several of the runner's args are not handled here (although they are supported by the launcher). One of those is --enable_each_rank_log

Commit message: fix

Patch to review:
diff --git a/deepspeed/launcher/multinode_runner.py b/deepspeed/launcher/multinode_runner.py
index ce58deadc281..f584763e2e7d 100644
--- a/deepspeed/launcher/multinode_runner.py
+++ b/deepspeed/launcher/multinode_runner.py
@@ -104,6 +104,8 @@ def get_cmd(self, environment, active_resources):
             deepspeed_launch.append("--no_local_rank")
         if self.args.save_pid:
             deepspeed_launch += ["--save_pid", f"{os.getpid()}"]
+        if self.args.enable_each_rank_log:
+            deepspeed_launch.append(f"--enable_each_rank_log={args.enable_each_rank_log}")
         if self.args.elastic_training:
             deepspeed_launch.append("--enable_elastic_training")
             deepspeed_launch.append(f"--max_elastic_nodes={self.args.max_elastic_nodes}")


Code review comments:
diff_hunk: @@ -104,6 +104,8 @@ def get_cmd(self, environment, active_resources):
             deepspeed_launch.append("--no_local_rank")
         if self.args.save_pid:
             deepspeed_launch += ["--save_pid", f"{os.getpid()}"]
+        if self.args.enable_each_rank_log:
+            deepspeed_launch.append(f"--enable_each_rank_log={args.enable_each_rank_log}")
path: deepspeed/launcher/multinode_runner.py
text: @author:
```suggestion
            deepspeed_launch.append(f"--enable_each_rank_log={self.args.enable_each_rank_log}")
```