Speed up model training

adywi: <p>Hello Torch users,</p>
<p>I’m currently implementing a 3D resnet18 on fMRI data of dimension [27, 75, 93, 81]. I couldn’t do one epoch in 48 hours on two GPUs A100.</p>
<p>I have already tried to transform my data directly into NumPy array in order to speed up the process.</p>
<p>I already use the following code to run the model on 2 GPUs:</p>
<pre><code class="lang-auto"> if torch.cuda.device_count() &gt; 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -&gt; [10, ...], [10, ...], [10, ...] on 3 GPUs
  model = nn.DataParallel(model)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model.to(device)
</code></pre>
<p>By the way, how can I check if the two GPUs are used?</p>
<p>I use this for my train loader:</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(train_set,
                              batch_size=64,
                              shuffle=True,
                              num_workers=0)
</code></pre>
<p>Any ideas or tricks to speed up the process?</p>

ptrblck: <aside class="quote no-group" data-username="adywi" data-post="1" data-topic="153655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adywi/48/50385_2.png" class="avatar"> adywi:</div>
<blockquote>
<p>I already use the following code to run the model on 2 GPUs:</p>
</blockquote>
</aside>
<p><code>nn.DataParallel</code> is not the recommended approach for data parallelism as it can create an imbalanced memory usage on the devices and is slower than <code>DistributedDataParallel</code> (DDP). Use DDP with a single process per GPU (even on a single node) for a faster performance.</p>
<aside class="quote no-group" data-username="adywi" data-post="1" data-topic="153655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adywi/48/50385_2.png" class="avatar"> adywi:</div>
<blockquote>
<p>By the way, how can I check if the two GPUs are used?</p>
</blockquote>
</aside>
<p><code>nvidia-smi</code> gives you the GPU util. as well as its memory usage and is often good enough as a quick test to see how many devices are used.</p>
<aside class="quote no-group" data-username="adywi" data-post="1" data-topic="153655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adywi/48/50385_2.png" class="avatar"> adywi:</div>
<blockquote>
<p>Any ideas or tricks to speed up the process?</p>
</blockquote>
</aside>
<p>Generally, check the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a> for some common issues and recommendations.</p>

adywi: <p>Thanks a lot for your response! I’ll try to implement the DDP. In my current script, do you think that <code>nn.DataParallel</code> accelerates the process or it may even slow it down?</p>

ptrblck: <p>I would guess the current approach could yield a speedup, but you would need to profile the use case on your system to check it.</p>