Trainer Class Predict Func == Decreasing Prediction Time

username: semaegrii
cooked: <p>Hi everyone,</p>
<ol>
<li>I have a dataloader with 100 images.<br>
If I set batch size =100 in “InferenceDataset” func,<strong>Total prediction time==23,6 s</strong> one image =236 ms</li>
</ol>
<p>If I set batch size=2 in  “InferenceDataset” func, <strong>Total prediction time==36,7 s</strong><br>
<strong>one image =367 ms</strong></p>
<p>I want to improve prediction time, what is your recommendation for this process?<br>
the code is :</p>
<pre><code class="lang-auto">inference_dataset = InferenceDataset(path=images_path, image_size=(256, 256))
test_dataloader = DataLoader(dataset=inference_dataset,batch_size=100, pin_memory=True)
start=time.time()
predictions = trainer.predict(model=model, dataloaders=test_dataloader)[0]                 
end=time.time()
print(f'Total prediction time: {round((end-start)*1000,2)} ms ')

</code></pre>

username: ptrblck
cooked: <p>The inference time should not increase if you lower the batch size, so something looks wrong.<br>
Are you using your GPU? If so, note that CUDA kernels are executed asynchronously so you would need to synchronize the code before starting and stopping host timers via <code>torch.cuda.synchronize()</code>.</p>

username: semaegrii
cooked: <p>First of all, thank you for your prompt response. I understood what you said. Using torch.cuda.synchronize() doesn’t help in speeding up. My GPU looks like active.==&gt;</p>
<pre><code class="lang-auto">import torch
print(torch.cuda.is_available())
print(torch.cuda.current_device())
print(torch.cuda.device(0))
print(torch.cuda.device_count())
print(torch.cuda.get_device_name(0)) 
</code></pre>
<p>True<br>
0<br>
&lt;torch.cuda.device object at 0x0000013E3BE594B0&gt;<br>
1<br>
NVIDIA RTX A4000</p>

username: ptrblck
cooked: <p>I didn’t mean to claim synchronizations would speed up your code, but should be used if you want to profile the actual GPU execution.<br>
I.e. you should add a synchronization via <code>torch.cuda.synchronize()</code> before using host timers via e.g. <code>time.perf_counter()</code>.</p>

username: semaegrii
cooked: <p>I am ı little bit confused. Here it says “batch size affects inference time”</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/docs/setfit/how_to/batch_sizes">
  <header class="source">

      <a href="https://huggingface.co/docs/setfit/how_to/batch_sizes" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/docs/setfit.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/docs/setfit/how_to/batch_sizes" target="_blank" rel="noopener nofollow ugc">Batch sizes for Inference</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Is the situation different with Torch?</p>

username: ptrblck
cooked: <p>No, it’s the same in PyTorch.</p>