[pytorch0.3.1] Forward pass takes 10x longer time for every 2nd batch inference

username: pramod.srinivasan
cooked: <p>Say, we have 30K records and the dataloader is feeding in batches of size 32 to a V100. The dataloader has 8 <code>num_workers</code>.</p>
<p>The setup is as follows:</p>
<pre><code class="lang-auto">#model init
#dataset init
for (id_batch, img_batch) in data_loader:
     probs = mysterious_function(img_batch)
</code></pre>
<p>The forward pass happens inside the <code>mysterious_function</code> :</p>
<pre><code class="lang-auto">def mysterious_function(image_batch):
    image_batch_gpu_var = Variable(torch.squeeze(image_batch), requires_grad=False).type(self.dtype_float).cuda(device=self.gpu_id)
    prob_predict_gpu = self.seg_net(image_batch_gpu_var)['probs']
    prob_predict_res = prob_predict_gpu.data.cpu().numpy()
    return prob_predict_res
</code></pre>
<p>The time tracker computes the execution time of the decorated function and logs the result right to stdout after it finishes execution. Here is the output of the time tracker:</p>
<pre><code class="lang-auto"> mysterious_function running time on batch 0 0.141 seconds
 mysterious_function running time on batch 1 2.420 seconds
 mysterious_function running time on batch 2 0.106 seconds
 mysterious_function running time on batch 3 2.391 seconds
 mysterious_function running time on batch 4 0.113 seconds
 mysterious_function running time on batch 5 2.341 seconds
 mysterious_function running time on batch 6 0.117 seconds
 mysterious_function running time on batch 7 2.393 seconds
 mysterious_function running time on batch 8 0.122 seconds
 mysterious_function running time on batch 9 2.257 seconds
</code></pre>
<p>Curious to know if this had anything to do with the dataloader read/decode/resizing from NFS, I ran the same forward pass twice on the <strong>same batch</strong> of data.</p>
<pre><code class="lang-auto">def mysterious_function(image_batch):
    cp1 = time()
    image_batch_gpu_var = Variable(torch.squeeze(image_batch), requires_grad=False).type(self.dtype_float).cuda(device=self.gpu_id)
    cp2 = time()
    print('moving image batch of shape {} to GPU {}'.format(image_batch_var.shape, cp2 - cp1)
    prob_predict_gpu = self.seg_net(image_batch_gpu_var)['probs'] 
    cp3 = time()
    print('model forward pass 1 out of 2 {}'.format(image_batch_var.shape, cp3 - cp2))
    prob_predict_gpu = self.seg_net(image_batch_gpu_var)['probs']
    cp4 = time()
    print('model forward pass 2 out of 2 {}'.format(image_batch_var.shape, cp4 - cp3))
    #note time taken
    prob_predict_res = prob_predict_gpu.data.cpu().numpy()
    return prob_predict_res
</code></pre>
<pre><code class="lang-auto">[] mysterious function running time 1.303
moving image batch of shape torch.Size([32, 3, 224, 224]) to GPU 0.0034639835357666016 seconds
model forward pass 1 of 2 takes 0.12161993980407715 seconds
model forward pass 2 of 2 takes 1.1894159317016602 seconds
[] mysterious function running time 1.317
moving image batch of shape torch.Size([32,  3, 224, 224]) to GPU 0.003448963165283203 seconds
model forward pass 1 of 2 takes 0.1061561107635498 seconds
model forward pass 2 of 2 takes 1.1635644435882568 seconds
mysterious function running time 1.275
moving image batch of shape torch.Size([32,  3, 224, 224]) to GPU 0.0034742355346679688 seconds
model forward pass 1 of 2 takes 0.10687828063964844 seconds
model forward pass 2 of 2 takes 0.8193213939666748 seconds
[] mysterious function running time 0.932
moving image batch of shape torch.Size([32, 3, 224, 224]) to GPU 0.0034782886505126953 seconds
model forward pass 1 of 2 takes 0.09704208374023438 seconds
model forward pass 2 of 2 takes 1.1900320053100586 seconds
[] mysterious function running time 1.293
moving image batch of shape torch.Size([32, 3, 224, 224]) to GPU 0.003448486328125 seconds
segnet forward pass 1 of 2 takes 0.13462281227111816 seconds
segnet forward pass 2 of 2 takes 1.1501150131225586 seconds
[] mysterious function running time 1.291
moving image batch of shape torch.Size([32, 3, 224, 224]) to GPU 0.00835871696472168 seconds
segnet forward pass 1 of 2 takes 0.09815263748168945 seconds
segnet forward pass 2 of 2 takes 1.0340092182159424 seconds
[] mysterious function running time 1.143
moving image batch of shape torch.Size([32, 3, 224, 224]) to GPU 0.0038101673126220703 seconds
segnet forward pass 1 of 2 takes 0.1068427562713623 seconds
segnet forward pass 2 of 2 takes 1.1246507167816162 seconds
[] mysterious function running time 1.238
moving image batch of shape torch.Size([32, 3, 224, 224]) to GPU 0.003797292709350586 seconds
segnet forward pass 1 of 2 takes 0.10389494895935059 seconds
segnet forward pass 2 of 2 takes 1.2108190059661865 seconds
</code></pre>
<p>As you can see, the 2nd forward pass takes almost 10x more time than the first forward pass on the same batch.</p>
<p>So, back to one forward pass and I tried to free up cuda memory after the first forward pass, but it did not help:</p>
<pre><code class="lang-auto">@time_tracker(_log)
def mysterious_function(image_batch):
    image_batch_gpu_var = Variable(torch.squeeze(image_batch), requires_grad=False).type(self.dtype_float).cuda(device=self.gpu_id)
    prob_predict_gpu = self.seg_net(image_batch_gpu_var)['probs'] 
    torch.cuda.empty_cache()
    prob_predict_res = prob_predict_gpu.data.cpu().numpy()
    return prob_predict_res
</code></pre>
<pre><code class="lang-auto">mysterious function running time on batch 0 0.729 seconds
mysterious function running time on batch 1 2.190seconds
mysterious function running time on batch 2 0.171seconds
mysterious function running time on batch 3 1.912seconds
mysterious function running time on batch 4 0.152seconds
mysterious function running time on batch 5 2.329seconds
</code></pre>
<p>Something seems to be wrong with the odd batch.</p>

username: rasbt
cooked: <p>Make sure you run your “inference” section in a</p>
<pre><code>with torch.set_grad_enabled(False):
    # your "inference" code here
</code></pre>
<p>Otherwise, it will be building on the autograd graph in the background and consuming more and more memory.</p>

username: pramod.srinivasan
cooked: <p>Should have probably mentioned this earlier, I am using pytorch 0.3.1, is there an equivalent?</p>
<p>Edit:</p>
<p>From the <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 0.4.0 Migration Guide | PyTorch</a> :</p>
<blockquote>
<p>The  <code>volatile</code>  flag is now deprecated and has no effect. Previously, any computation that involves a  <code>Variable</code>  with  <code>volatile=True</code>  wouldn’t be tracked by  <code>autograd</code> . This has now been replaced by a <a href="https://pytorch.org/docs/0.4.0/torch.html#locally-disabling-gradient-computation" rel="noopener nofollow ugc">set of more flexible context managers</a> including  <code>torch.no_grad()</code> ,  <code>torch.set_grad_enabled(grad_mode)</code> , and others</p>
</blockquote>
<p>Maybe I didn’t understand this fully, but how would setting <code>volatile</code> to <code>True</code> for the batch Variable disable the autograph build?</p>

username: rasbt
cooked: <p>I think there’s some way to do it in PyTorch 0.3, but I don’t remember exactly since it has been quite some time ago since I used 0.3, sorry <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=6" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>It may have been the volatile=True setting you mentioned. In any case, due to this and various other improvements, I really recommend switching to PyTorch 1.0 if you can.</p>
<p>EDIT: if you don’t want to make any big code modification to your 0.3 code, you could also call .backward() on the model after each forward pass during inference. It will probably slow down the inference (because you have an addition backprop step) but only by a constant, because backward() will free the graph after each call (so you don’t let the graph grow and grow). However, i really recommend upgrading and using the context managers.</p>

username: pramod.srinivasan
cooked: <p>Setting volatile to True, and <code>torch.cuda.empty_cache</code> did not help. The issue still persists.</p>

username: lugiavn
cooked: <p>Instead of 2, can you do forward the same batch 10 times and check<br>
Your model might be doing something fishy, what is it forward function like?</p>

username: pramod.srinivasan
cooked: <p>I finally figured a way around this issue. The problem was that we had a group of worker threads launched in the CPU to collect the detached outputs from the GPU and perform some post-processing before saving them to disk.</p>
<p>Let me explain the complete setup:</p>
<ul>
<li>Launch a set of worker threads which collect images from an asynchronous message queue and feed them to a <code>torch.multiprocessing</code> queue <code>P</code>.</li>
<li>Launch a process task which dequeues items from <code>P</code> and prepares a dataloader and feeds batches of these items to the GPU for inference.</li>
<li>Append items from each batch into a list <code>&lt;item_id, prob&gt;</code> into a <code>torch.multiprocessing</code> queue <code>Q</code></li>
<li>Launch of set of postprocessing worker threads which collect the items from the <code>Q</code>, do some manipulation in CPU and write them to disk.</li>
</ul>
<p>With the postprocessing worker threads, my inference rate was around 25 items per second on V100 (10x longer for every batch), when I removed these worker threads, I found that the inference rate increased to 160 records per second. Please note that I manually did the postprocessing before dequeuing the next dataset of items from <code>P</code>- this takes approx 25% of the total time to process a dataset and is essentially idle time for the GPU but I could not improve beyond that.</p>
<p>Story wasnt over yet <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> ! After removing the postprocesing workers, I was able to get a 6x improvement in inference time with a high GPU memory and usage as shown below. However, do you notice anything odd in the graph below?<br>
￼<br>
￼￼ <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/444635b860c59d4b16f918d17beb2d2786daa5ae.png" data-download-href="https://discuss.pytorch.org/uploads/default/444635b860c59d4b16f918d17beb2d2786daa5ae" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/444635b860c59d4b16f918d17beb2d2786daa5ae_2_586x500.png" alt="image" data-base62-sha1="9JYTiq6ldHL5CoI9M2HinAAWIKG" width="586" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/444635b860c59d4b16f918d17beb2d2786daa5ae_2_586x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/444635b860c59d4b16f918d17beb2d2786daa5ae_2_879x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/444635b860c59d4b16f918d17beb2d2786daa5ae_2_1172x1000.png 2x" data-dominant-color="F7F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1440×1228 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The CPU usage keeps increasing over time and before we know the system is bound to crash, once again for no fault of the GPU. Fortunately, this can be resolved by a small trick, by replacing the <code>prob_predict_gpu</code> from <code>prob_predict_gpu.data</code> to <code>prob_predict_gpu.detach</code>.</p>
<p><code>.data()</code>  is an old api and should not be used anymore. The detach() method constructs a new view on a tensor that is declared not to need gradients  and therefore the subgraph involving the view would not be tracked (nice example here <a href="http://www.bnikolic.co.uk/blog/pytorch-detach.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch .detach() method | B. Nikolic Software and Computing Blog</a>)</p>
<pre><code class="lang-auto">def mysterious_function(image_batch):
    image_batch_gpu_var = Variable(torch.squeeze(image_batch), requires_grad=False).type(self.dtype_float).cuda(device=self.gpu_id)
    prob_predict_gpu = self.seg_net(image_batch_gpu_var)['probs'] 
    torch.cuda.empty_cache()
    prob_predict_res = prob_predict_gpu.detach.cpu().numpy() #prob_predict_gpu.data.cpu().numpy()
    return prob_predict_res
</code></pre>
<p>The final CPU and GPU utilization looked like this:</p>
<p>￼<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5d11a44b19f0636f1c3de053c3b486fe0fbee5f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/5d11a44b19f0636f1c3de053c3b486fe0fbee5f4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5d11a44b19f0636f1c3de053c3b486fe0fbee5f4_2_598x500.png" alt="image" data-base62-sha1="dhkcwLlxND6ANL6FqWHJdXvNUCo" width="598" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5d11a44b19f0636f1c3de053c3b486fe0fbee5f4_2_598x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5d11a44b19f0636f1c3de053c3b486fe0fbee5f4_2_897x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5d11a44b19f0636f1c3de053c3b486fe0fbee5f4_2_1196x1000.png 2x" data-dominant-color="E2E8F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1440×1204 140 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Lisa_Koenigsberg
cooked: <p>deleted since I started I new topic</p>