Help with tee, redirects configuration

Vijay_Santhosh_Vammi: <p>Hello,</p>
<p>In my use case, instead of running torchrun, we need to invoke the module as a python function call. My set up looks like this:</p>
<p>In the below strip down, func is exactly same as any other train function which gets called my invoking torchrun  script.py </p>
<p>Instead of sending the script args, the wrapper function reads from a config file and calls the main function.</p>
<pre data-code-wrap="python"><code class="lang-python">def training_subprocess():
   # get the parameters from some conf file like json or yaml
   parameters = load_parameters()
   func(**parameters)
</code></pre>
<hr>
<p>Torch run configuration and set up:</p>
<p>Since we have to call torchrun programmatically, I created a dataclass/pydantic object which captures the config: This is inspired from <a href="https://github.com/pytorch/pytorch/blob/e0d4c43ad104fd72fa2ae412de563e91ebfde913/torch/distributed/run.py#L391" rel="noopener nofollow ugc">args from torch</a>.</p>
<pre data-code-wrap="python"><code class="lang-python">class StartMethod(str, Enum):
    spawn = "spawn"
    fork = "fork"
    forkserver = "forkserver"

class TorchConfig(BaseModel):
    nnodes: str = Field(default="1:1")
    nproc_per_node: int = Field(default=1)

    rdzv_backend: str | None = Field(default="static")
    rdzv_endpoint: str | None = Field(default="")
    rdzv_id: str | None = Field(default="none")
    rdzv_conf: str | None = Field(default="")

    max_restarts: int | None = Field(default=None)
    monitor_interval: float | None = Field(default=0.1)
    start_method: str | None = Field(default=StartMethod.spawn)
    role: str | None = Field(default="default")
    log_dir: str | None = Field(default="torch_logs")
    redirects: str | None = Field(default="0")
    tee: str | None = Field(default="0")
    master_addr: str | None = Field(default="localhost")
    master_port: str | None = Field(default="29500")
    training_script: str = Field(default="dummy_training_script")
    training_script_args: str = Field(default="")
    logs_specs: str | None = None
    local_addr: str | None = Field(default=None)

    # Optional fields
    local_ranks_filter: str | None = Field(default="0")
    node_rank: int | None = Field(default=0)
    standalone: bool | None = Field(default=None)
    module: bool | None = Field(default=False)
    no_python: bool | None = Field(default=False)
    run_path: bool | None = Field(default=False)
</code></pre>
<p>Using the function in: <a href="https://github.com/pytorch/pytorch/blob/e0d4c43ad104fd72fa2ae412de563e91ebfde913/torch/distributed/run.py#L747" rel="noopener nofollow ugc">config_from_args</a></p>
<p>I can convert this into a lauch_config by:</p>
<pre data-code-wrap="python"><code class="lang-python">launch_config, _, _ = config_from_args(**torch_config)
</code></pre>
<p>I can then call, [elastic_launch]</p>
<pre data-code-wrap="python"><code class="lang-python">launcher = elastic_launch(
     launch_config,
   training_subprocess,
 )
</code></pre>
<p>This set up works as expected and is similar to running torchrun.</p>
<p>I want to redirect stdout and stderr from the workers to but am struggling to find the right configuration. The default parameters do not capture any output during the model training phase.</p>
<p>I played around with the <code>redirects</code> and <code>tee</code> parameters but most of the time I get errors of <code>"io.UnsupportedOperation: fileno"</code>.</p>
<p>Could you please help in getting the right configuration values to stream both stdout and stderr? I think I can get them via the torch logs at the end of the run but it would be nice to see the progress as it happens.</p>
<p>Cheers,</p>