What does torch.backends.cudnn.benchmark do?

John_Zhang: <p>whats the difference when setting it True or False?</p>

albanD: <p>This flag allows you to enable the inbuilt cudnn auto-tuner to find the best algorithm to use for your hardware.</p>

fmassa: <p>It enables benchmark mode in cudnn.<br>
benchmark mode is good whenever your input sizes for your network do not vary. This way, cudnn will look for the optimal set of algorithms for that particular configuration (which takes some time). This usually leads to faster runtime.<br>
But if your input sizes changes at each iteration, then cudnn will benchmark every time a new size appears, possibly leading to worse runtime performances.</p>

yxchng: <p>I tried using it with torchvision’s resnet101 but it gives worse performance. Is it normal? <a class="mention" href="/u/fmassa">@fmassa</a> <a class="mention" href="/u/alband">@albanD</a></p>

fmassa: <p>It depends on the task. If your input size is changing a lot, then it might hurt runtime, if not, it should be much faster.</p>

mjchen611: <p>whats the difference when setting cudnn.enabled True or False?</p>

tstandley: <p>Does it work to turn it on for training (where I have a constant input size) and turn it off for validation (where my input size isn’t constant)? Do I just set the constant before doing my validation?</p>

albanD: <p>Yes, it will work to change the value!</p>

Alpha: <p>I have a question.</p>
<p>e.g. I have a <strong>net1</strong> whose input sizes don’t vary. And I have other nn.Module named <strong>net_Loss</strong>  whose input sizes vary.  I only need to optimeze <strong>net1’s</strong> parameters.  So should I use the  cudnn.benchmark   <em>True or False</em>?<br>
Thank you !</p>

salihkaragoz: <p>Hello,<br>
IMHO, if your networks connect each other and if net1 is first network, you should use cudnn.benchmark = True.<br>
and also you ve mentioned I only need net1’s parameters. I think you should use.</p>

DeepLearner17: <p>Hello,<br>
<a class="mention" href="/u/alband">@albanD</a><br>
<code>When we set cudnn.benchmark=True</code></p>
<ol>
<li>
<p>How can l get access to the whole family of algorithms that potentially can be executed? (display them)</p>
</li>
<li>
<p>How can l print the cudnn algorithm run at each iteration ?</p>
</li>
</ol>
<p>Thank you</p>

albanD: <ol>
<li>I’m afraid you can’t. That would depend on the operation you’re performing. For example, for forward pass of convolution, you can find the list <a href="https://github.com/pytorch/pytorch/blob/1848cad10802db9fa0aa066d9de195958120d863/aten/src/ATen/native/cudnn/Conv.cpp#L486-L494">here</a>.</li>
<li>You would have to add this print directly in the cpp code linked above.</li>
</ol>

Diego: <p>Can you show an example on how to use this line? Can you place it anywhere in the code? Or before the forward pass?</p>

ruotianluo: <p>Can I have part of the inference to be benchmarked.</p>
<p>Like this:</p>
<pre><code class="lang-auto">x = func1(x)
x = x[:20]
torch.backends.cudnn.benchmark = True
x = func2(x)
torch.backends.cudnn.benchmark = False
</code></pre>

PabloRR100: <p>What if you are training different networks in the same script exposed to the same input sizes?<br>
I am training a single ResNet44 and after that an ensemble of 3 ResNets 18 on CIFAR10.</p>
<p>How do I manage the <code>cudnn.benchmark = True</code>?<br>
First at the beggining of the ResNet44 training and then once again after the beggining of the first ResNet18?</p>
<p>Thanks!</p>

kuzand: <aside class="quote no-group" data-username="fmassa" data-post="3" data-topic="5936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/48/58_2.png" class="avatar"> fmassa:</div>
<blockquote>
<p>benchmark mode is good whenever your input sizes for your network do not vary.</p>
</blockquote>
</aside>
<p>Hi. Can you please clarify what do you mean by “input size”? Is it the image size, like 224? Since the input size of network is always fixed and the images are resized to the same size before inputing them to network, in which cases it can vary?</p>

fabianjul: <p>When you have a fully convolutional network</p>

Oli: <p>I find that <code>torch.backends.cudnn.benchmark</code> increases the speed for my YOLOv3 model by a lot, like 30-40%. Furthermore, it lowers the memory footprint after it completes the benchmark.</p>
<p>It even works when my input images vary in size between each batch, neat! I was thinking about having the network optimize on a few smaller <code>torch.randn(...)</code> to benchmark on, and then start the training. I hope that this could allow me to increase the batch size since the memory footprint is lower after the bechmark. What do you guys thing?</p>

Shubhankar: <p>Is this still the only way to do it?</p>

albanD: <p>Hi,</p>
<p>I don’t think this has changed I’m afraid. But there might have been changes to this code so not sure.</p>