Simple "Hello World" to run on multiple PCs in local network

MartinPerry1: <p>I have a home local network with a Windws Intel-powered PC and MacBook with M2.</p>
<p>I want to test distributed training on CPU and train the network on both computers. However, I don’t understand how to run it.</p>
<p>There are multiple tutorials, each with slightly different information.<br>
For example, some use torch.distributed.launch, which complaints to replaced by torchrun.<br>
Als, it seems that all of them are using one or more GPUs, but none is CPU-only.</p>
<p>I have read <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.0.1+cu117 documentation</a>  and copied code from <strong>Basic Use Case</strong> with <em>ToyModel</em></p>
<pre><code class="lang-auto">class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 10)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))
</code></pre>
<p>Based on this <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp/README.md" class="inline-onebox" rel="noopener nofollow ugc">examples/README.md at main · pytorch/examples · GitHub</a>, I have used:</p>
<pre><code class="lang-auto">#modified to use only CPU
def demo_basic(local_world_size, local_rank):
    
    print(
        f"[{os.getpid()}] rank = {dist.get_rank()}, "
        + f"world_size = {dist.get_world_size()}, local_world_size={local_world_size}, local_rank={local_rank}"
    )

    model = ToyModel().to('cpu')
    ddp_model = DDP(model)

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to('cpu')
    loss_fn(outputs, labels).backward()
    optimizer.step()
</code></pre>
<pre><code class="lang-auto">def spmd_main(local_world_size, local_rank):
    # These are the parameters used to initialize the process group
    env_dict = {
        key: os.environ[key]
        for key in ("MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE")
    }
    print(f"[{os.getpid()}] Initializing process group with: {env_dict}")
    dist.init_process_group(backend="nccl")
    print(
        f"[{os.getpid()}] world_size = {dist.get_world_size()}, "
        + f"rank = {dist.get_rank()}, backend={dist.get_backend()}"
    )

    demo_basic(local_world_size, local_rank)

    # Tear down the process group
    dist.destroy_process_group()
</code></pre>
<p>And in main I have</p>
<pre><code class="lang-auto">if __name__=="__main__":
       
    parser = argparse.ArgumentParser()
    parser.add_argument("--local_rank", type=int, default=0)
    parser.add_argument("--local_world_size", type=int, default=1)
    args = parser.parse_args()
    spmd_main(args.local_world_size, args.local_rank)
</code></pre>
<p>However, I don’t know how to run this code, so it spawns on my host PC and also on Mac. I am not entirely familiar with networks and I have only very basic knowledge.</p>

H-Huang: <p>A few suggestions:</p>
<blockquote>
<p>I have a home local network with a Windws Intel-powered PC and MacBook with M2</p>
</blockquote>
<p>If you are using multiple hosts, then you will need to find the private IP address of one of the computers and make sure that the other host can access it (use <code>ping</code>). Then you should set MASTER_ADDR to the IP address and MASTER_PORT to an available port on that machine</p>
<blockquote>
<pre><code>dist.init_process_group(backend="nccl")
</code></pre>
</blockquote>
<p>This line should be updated to <code>dist.init_process_group(backend="gloo")</code> as nccl is only for cuda tensors</p>
<blockquote>
<p>However, I don’t know how to run this code, so it spawns on my host PC and also on Mac.</p>
</blockquote>
<p>If the code is all in a file, e.g. <code>script.py</code> then you would just need to run <code>python script.py</code> on each machine. Granted that you have all the dependencies (e.g. torch) installed on both machines.</p>
<p>All that being said, the simpler way to try out distributed training, especially if using CPUs is to just use 1 host (1 machine) and create multiple processes.</p>