Mp.spawn() doesn't parralize the DDP in HPC

ashimdahal: <p>Hi,</p>
<p>I was following this tutorial by PyTorch on DDP.<br>
And I encountered a logging error in the HPC. The code written is the same as the one in the github example here: <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multigpu.py" class="inline-onebox" rel="noopener nofollow ugc">examples/distributed/ddp-tutorial-series/multigpu.py at main · pytorch/examples · GitHub</a></p>
<p>The code was submitted to the HPC using sbatch and the following configuration</p>
<pre><code class="lang-auto">#!/bin/bash
#SBATCH --job-name=testing           # Job name
#SBATCH --output=output.log          # Output file name
#SBATCH --error=error.log            # Error file name
#SBATCH --partition=gpu              # Partition 
#SBATCH --gres=gpu:4                 # GPU resources, requesting 4 GPUs
#SBATCH --ntasks=1                   # Number of tasks or processes
#SBATCH --nodes=1                    # Number of nodes

# Load necessary modules or activate virtual environment
module load cuda-toolkit/11.6.2
module load python/3.8.6

# Actual command to be executed
python torch_testing.py
</code></pre>
<p>but the problem arises in the output.log where the mp.spawn() seems to run all the epochs in one gpu before moving into the other gpu</p>
<pre><code class="lang-auto">[GPU2] Epoch 0 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 1 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 2 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 3 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 4 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 5 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 6 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 7 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 8 | Batchsize: 64 | Steps: 8
[GPU2] Epoch 9 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 0 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 1 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 2 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 3 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 4 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 5 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 6 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 7 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 8 | Batchsize: 64 | Steps: 8
[GPU3] Epoch 9 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 0 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 1 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 2 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 3 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 4 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 5 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 6 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 7 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 8 | Batchsize: 64 | Steps: 8
[GPU1] Epoch 9 | Batchsize: 64 | Steps: 8
[GPU0] Epoch 0 | Batchsize: 64 | Steps: 8
Epoch 0 | Training checkpoint saved at checkpoint.pt
[GPU0] Epoch 1 | Batchsize: 64 | Steps: 8
[GPU0] Epoch 2 | Batchsize: 64 | Steps: 8
Epoch 2 | Training checkpoint saved at checkpoint.pt
[GPU0] Epoch 3 | Batchsize: 64 | Steps: 8
[GPU0] Epoch 4 | Batchsize: 64 | Steps: 8
Epoch 4 | Training checkpoint saved at checkpoint.pt
[GPU0] Epoch 5 | Batchsize: 64 | Steps: 8
[GPU0] Epoch 6 | Batchsize: 64 | Steps: 8
Epoch 6 | Training checkpoint saved at checkpoint.pt
[GPU0] Epoch 7 | Batchsize: 64 | Steps: 8
[GPU0] Epoch 8 | Batchsize: 64 | Steps: 8
Epoch 8 | Training checkpoint saved at checkpoint.pt
[GPU0] Epoch 9 | Batchsize: 64 | Steps: 8
</code></pre>
<p>I set the code from the <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multigpu.py" rel="noopener nofollow ugc">github</a> to save after every 2 epochs and run for a total of 10 epochs.</p>
<p>Any insight on how and why the mp.spawn() behaves this way or is there a way to fix this in the hpc would be greatly appreciated.</p>

ptrblck: <p>I would guess the logs are wrong and serialized for some reason and that <code>DDP</code> itself is still working. You could write out the system time to the log to check if your setup is creating these interleaved logs.</p>

ashimdahal: <p>would try and update.</p>

ashimdahal: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47ceafed3ca312000575422141c8ecc4c0648997.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/47ceafed3ca312000575422141c8ecc4c0648997" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47ceafed3ca312000575422141c8ecc4c0648997_2_445x500.jpeg" alt="image" data-base62-sha1="afeIZEbNQeX90c3rUKTg2B9cIV9" width="445" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47ceafed3ca312000575422141c8ecc4c0648997_2_445x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47ceafed3ca312000575422141c8ecc4c0648997_2_667x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/7/47ceafed3ca312000575422141c8ecc4c0648997.jpeg 2x" data-dominant-color="3B3850"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">745×837 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Update: Indeed the output.log seems to be serialized.</p>

ptrblck: <p>Thanks for checking. In this case you might want to sort the logs based on the timestamps and GPU IDs to check the real progress (or reduce the losses from all GPUs in case you want a single loss per epoch).</p>