PyTorch DDP consuming more power for uneven data distribution

Monzurul_Amin: <p>Hi,<br>
I am using a 2 node distributed setup(each having a single GPU respectively) to train a Neural Network (NN). I utilize PyTorch Data Distributed Parallel with Join Context Manager to achieve this. I am measuring power consumption varying data distribution on those two nodes. I am noticing more power consumption in the node which has a lesser part of the data, e.g.,for a scenario where node 1 training on 20% of the dataset and node2 is training on the rest 80% of data, I am getting higher power consumption in node1 once it finishes training its part. I know how Join Context manager works and it is intuitive why node1 is consuming more power. But there is nothing mentioned in documentation about power consumption. Is this<br>
PyTorch implementation-specific  or this is how any synchronous training work (in any framework - PyTorch, Tensorflow etc.) ?</p>

Monzurul_Amin: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , can you please look into this issue?</p>

ptrblck: <p>I’m unsure how your use case works exactly as it seems node1 might be idling for some time while node2 is still busy based on the data split? How are you measuring the power consumption and did you check if node1 performs any calculations or just waits and potentially spins?</p>

Monzurul_Amin: <p>Thanks for the prompt reply. Please find more details here:</p>
<p>Dataset: MNIST (only train)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ee29118e6c8ab642da8ecd5a078b26e83f5ccf2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0ee29118e6c8ab642da8ecd5a078b26e83f5ccf2" title="Screenshot_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0ee29118e6c8ab642da8ecd5a078b26e83f5ccf2_2_690x255.jpeg" alt="Screenshot_1" data-base62-sha1="27G6iSROFpwQNoBJcCQuuAYikee" width="690" height="255" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0ee29118e6c8ab642da8ecd5a078b26e83f5ccf2_2_690x255.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ee29118e6c8ab642da8ecd5a078b26e83f5ccf2.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ee29118e6c8ab642da8ecd5a078b26e83f5ccf2.jpeg 2x" data-dominant-color="282A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_1</span><span class="informations">906×336 58.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Model: Convolution networks with 2 convolution layers and a fully connected layer.<br>
Distributed backend: NCCL<br>
World size (number of processess participating in the job): 2<br>
Device: GPU<br>
Setup: two workers on two different machines. Each machine has one GPU respectively (NVIDIA Quadro RTX 8000).<br>
CUDA version: 11.7<br>
Batch size: 100<br>
Used ‘nn.parallel.DistributedDataParallel’ to distribute the model over GPU on those two machine instances.</p>
<p><strong>Join Context Manager (JCM):</strong><br>
Inside the join context manager wrapper, data is loaded into GPU, perform the forward and backward pass. According to the official documentation ([Distributed Training with Uneven Inputs Using the Join Context Manager — PyTorch Tutorials 2.0.1+cu117 documentation),<br>
“]([<a href="https://pytorch.org/tutorials/advanced/generic_join.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Training with Uneven Inputs Using the Join Context Manager — PyTorch Tutorials 2.2.0+cu121 documentation</a>](<a href="https://JCM" rel="noopener nofollow ugc">https://JCM</a> PyTorch)) Join is a context manager to be used around your per-rank training loop to facilitate training with uneven inputs. The context manager allows the ranks that exhaust their inputs early (i.e. join early) to shadow the collective communications performed by those that have not yet joined.”</p>
<p><strong>Power measurement:</strong> Collected power draw logs by using command ‘nvidia-smi’ in every 1s.<br>
<strong>Experiment result:</strong><br>
For simplicity, let’s start with the uniform scenario where worker1 at machine1 is training over 50% of MNIST dataset (1-30K examples) and worker2 at machine2 is also training the other 50% (30001-60K examples). In this case, total power consumption and peak power consumption in each worker is alike.<br>
Now, for scenario 2, worker1 has 10% of training data while worker2 has the reamining 90%. Here, for join context manager, worker1 waits after finish training its part and performing the shadow collective communication until worker2 finishes its training. According to my logs, total power draw in worker1 is 400W higher than worker2. Moreover, peak power draw in worker1 happens after finishing the training (while it is performing the shadow communication and waiting for worker2 to finish).<br>
I experimented a lot of rounds with different data distribution in worker1 and worker2 and received similar behaviour. In PyTorch documentation, I didn’t any details about this power consumption behaviour. Is it due to how PyTorch implemented JCM ? Or should I change my deployment in some way – to rephrase what is the preferred way for distributed training when data is distributed unevenly among the workers?<br>
This is my first post in PyTorch forum, so might miss details that you may need. Please let me know if you need any more details.</p>

Monzurul_Amin: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a> , I have shared details of my deployment. Can you please have a look?</p>

Conless: <p>Hi, <a class="mention" href="/u/monzurul_amin">@Monzurul_Amin</a> , I’m interested in this issue. Would you like to share you code so I can produce this problem?</p>