Ask about run my model with Parallel device

Hyeonuk_Woo: <p>Hi, I have a question about using multiple GPU devices.</p>
<p>I set my model to use multiple device as below.</p>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"]="0,1,2,3"
model = model().to(device)
model = nn.DataParallel(model.to(device))
(...)
def train():
      (....)
      for epoch in range(start_epoch, 10001):
           (....)
           for ..... in training_generator:
                (.......)
                output=model(user_input)
</code></pre>
<p>My model is 3D-CNN and my ‘user_input’ has (A,B,C,D,E) dimension. At here, if I set A =4 , does my model run with 4 devices parallely?  If I use 4 devices, does the number A have to be a multiple of 4 for efficient calculation?<br>
Thanks.</p>

ptrblck: <p>Yes, <code>nn.DataParallel</code> would split the input batch in <code>dim0</code> and transfer each chunk to the corresponding device.</p>
<aside class="quote no-group" data-username="Hyeonuk_Woo" data-post="1" data-topic="124156">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyeonuk_woo/48/39179_2.png" class="avatar"> Hyeonuk_Woo:</div>
<blockquote>
<p>If I use 4 devices, does the number A have to be a multiple of 4 for efficient calculation?</p>
</blockquote>
</aside>
<p>Also yes, but to get the most efficient approach you should use <code>DisttributedDataParallel</code> with a single process per GPU.</p>

Hyeonuk_Woo: <p>Frist of all, thankyou so much!</p>
<p>I have one more basic question.</p>
<p>If I use 4 devices and <code>dim0</code> of my input data is 8, in this case, each GPU handle two process simultaneously or sequentially?</p>
<p>Why single process per  GPU is recommended?</p>
<p>+) the word ‘single process’ means that batch size of the data assigned to single GPU is 1?</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="Hyeonuk_Woo" data-post="3" data-topic="124156">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyeonuk_woo/48/39179_2.png" class="avatar"> Hyeonuk_Woo:</div>
<blockquote>
<p>If I use 4 devices and <code>dim0</code> of my input data is 8, in this case, each GPU handle two process simultaneously or sequentially?</p>
<p>+) the word ‘single process’ means that batch size of the data assigned to single GPU is 1?</p>
</blockquote>
</aside>
<p>No, each device will get a batch containing 2 samples.</p>
<aside class="quote no-group" data-username="Hyeonuk_Woo" data-post="3" data-topic="124156">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyeonuk_woo/48/39179_2.png" class="avatar"> Hyeonuk_Woo:</div>
<blockquote>
<p>Why single process per GPU is recommended?</p>
</blockquote>
</aside>
<p>DDP should be faster as it reduces the communication overhead in <code>nn.DataParallel</code>.<br>
The details of the latter (including the scatter/gather calls) are described in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a>.</p>