How to reduce the execution time of "forward pass" on GPU

username: dat_pham_thanh
cooked: <p>Hello,<br>
I already had a pre-train model. I only use it to extract the feature (only use “forward pass”).<br>
Now, I only load it into single GPU and use with   <code>torch.no_grad()</code><br>
My question is that: I have 2 GPUs on my computer and how to reduce the execution time with 2 GPUs.<br>
P/s: Input of my model have a size as <code>(8,3,256,128)</code></p>
<p>Thanks.</p>

username: mrshenli
cooked: <p><a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">DataParallel</a> and <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L33" rel="nofollow noopener">DistributedDataParallel</a> would help here.</p>

username: dat_pham_thanh
cooked: <p>Thanks for replaying!!</p>
<p>However, when I read the document of <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">DataParallel</a> and <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L33" rel="nofollow noopener">DistributedDataParallel </a>, I think it would not help me to reduce the execution time because I do not need the backward pass.</p>
<pre><code class="lang-auto">&gt; assert any((p.requires_grad for p in module.parameters())), (
&gt;             "DistributedDataParallel is not needed when a module "
&gt;             "doesn't have any parameter that requires a gradient."
&gt;         )
</code></pre>
<p>I will try with it and tell u the result.</p>

username: wangwwno1
cooked: <p>How to use DataParallel:<br>
model = DataParallel(model, dim=<em>your batch dim in input</em>, device_ids=[main_id, other_ids …], output_device=main_id)<br>
Note that main_id (GPU that store the original model parameter) should be the first in the list of device_ids.</p>
<p>You can use DataParallel since it’s easier to setup and test, but remember to:</p>
<ol>
<li>Set the batch dimension of DataParallel. The default is dim=0 but sometime you might want to apply another dimension.<br>
(e.g. my input size is (Time, Batch, Dim_Data), and the model require a fully time series. In this scenario I will apply dim=1 instead of dim=0, because If I choose dim=0, the time serie will be split into multiple fragments.)</li>
<li>DataParallel return a wrapped model, so use the wrapped model to forward instead of original one, and remember to…</li>
<li>…handle the state_dict of wrapped model before save the state_dict into files. DataParallel will append prefix “module.” to each key of the original state_dict.keys(), you have to remove the prefix before saving the state_dict.</li>
<li>Set the GPU with larger memory as output_device , and also pass model parameter and your input to this GPU. output_device need to store both data and model parameters, so larger GPU memory is favorable.</li>
</ol>

username: dat_pham_thanh
cooked: <p>Here is my code with DataParallel</p>
<pre><code class="lang-auto">import time
import torchvision.models as models
import torch
import torch.nn as nn
model = models.resnet50(num_classes=1000).to('cuda:0')

model = nn.DataParallel(model, device_ids=[0,1], output_device=0)
#####
batch_size = 8
image_w = 128
image_h = 128

#####
#warm up GPU
input = torch.randn(batch_size,3,image_w, image_h).to('cuda:0')
model.eval()

listTime = []
for i in range(20):
    with torch.no_grad():
        startTime = time.time()
        input = torch.randn(batch_size,3,image_w, image_h).to('cuda:0')
        out = model(input)
        esl = time.time() - startTime
        listTime.append(esl)
        print("Total time of loop {} :: {}".format(i, esl))

meanTime = torch.mean(torch.tensor(listTime[9:]))
print(meanTime)
</code></pre>
<p>I test with resnet50(). The size of input is (8,3,128,128).<br>
I run the forward() pass in 20 steps and choice the last 10 steps to find the mean of execution time.</p>
<p>Without Dataparallel, <code>meanTime = 0.0064s</code> (run with single GPU)<br>
and with Dataparalle, <code>meanTime = 0.0396s</code>(run with 2 GPUs)<br>
P/s: I have 2 GPUs as below image.<br>
Do you have any solution for my problem ?<br>
Thanks</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ef855adac77bba3401ad880399e3e69120deab8.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ef855adac77bba3401ad880399e3e69120deab8" title="nvida"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ef855adac77bba3401ad880399e3e69120deab8.png" alt="nvida" data-base62-sha1="bgBiqNW9i7ZtBck63zHEWumetPW" width="690" height="200" data-dominant-color="422037"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nvida</span><span class="informations">710×206 26.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: pritamdamania87
cooked: <p><a class="mention" href="/u/dat_pham_thanh">@dat_pham_thanh</a> Can you benchmark using at least 1000 iterations and also track throughput instead (images/s)? Mean time can be misleading since a single outlier could change the mean quite a bit.</p>

username: mrshenli
cooked: <p><a class="mention" href="/u/dat_pham_thanh">@dat_pham_thanh</a></p>
<p>DataParallel would replicate the model, scatter the input, and gather outputs in every iteration. So, if the input size is too small, the overhead of replicating the model might overshadow the benefits of parallelizing the computation. Besides what <a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> suggested above, could you please also try with large batch size?</p>

username: dat_pham_thanh
cooked: <p>Thank for your reply!!<br>
I think you are correct. I can not increase the batch size because it is fixed ( batch size always equals 8) for each iteration.<br>
I use the ONNX model to solve my problem!!</p>