Conv2D is very slow on trained weights (vs random weights)

username: pgmmpk
cooked: <p>Super weird.</p>
<p>Doing <code>torch.nn.functional.conv2d(x, weight)</code> <strong>sometimes</strong> takes 30x times longer (using CPU). There are no NaNs and all data seem to be reasonable.</p>
<p>Anyone seen something like this?</p>
<p>See my gist for the minimal working example: <a href="https://gist.github.com/pgmmpk/b64901e3bd77ed58c00be83bd170d982" rel="nofollow noopener">https://gist.github.com/pgmmpk/b64901e3bd77ed58c00be83bd170d982</a></p>
<p>I see this in torch 0.4.1 and 1.0.1.post2 (did not try others). Installed from pip.</p>
<p>-Mike</p>

username: albanD
cooked: <p>Hi,</p>
<p>The problem is most likely that you’re hitting denormal numbers (float numbers too close to zero are very expensive to work with).<br>
Does adding a <a href="https://pytorch.org/docs/stable/torch.html?highlight=denormal#torch.set_flush_denormal" rel="nofollow noopener"><code>torch.set_flush_denormal(True)</code></a> before the op solves the issue?<br>
This will reduce the precision of the operations for very very small numbers but should remove the slowdown.</p>
<p>EDIT: cross post from github <a href="https://github.com/pytorch/pytorch/issues/19651" rel="nofollow noopener">here</a>.</p>

username: pgmmpk
cooked: <p>Problem solved. For the sake of people who might see this post, let me summarize the problem and solution:</p>
<p>Problem is that floating-point operations on CPU can become very slow if numbers are <a href="https://en.wikipedia.org/wiki/Denormal_number" rel="nofollow noopener">“denormal” or “subnormal”</a>. This means that values are very very small (smaller than 1.e-32). This is very unusual.</p>
<p>Workaround is to force CPU to treat these numbers as zeroes. At the beginning of your code add <code>torch.set_flush_denormal(True)</code> (this may not work on some older Intel CPUs though).</p>
<p>Alternative workaround is to “manually” remove denormals from your weights:</p>
<pre><code class="lang-auto">weight = ...
mask = (weight.abs() &lt; 1.e-32).float()
weight = weight * mask
</code></pre>
<p>Better solution is to find out why your training ended up with such weird weights, and fix it!</p>
<p>Thanks to <a class="mention" href="/u/alband">@albanD</a> and others for the awesome support.</p>

username: addisonklinke
cooked: <p>Have <a class="mention" href="/u/pgmmpk">@pgmmpk</a> or others noticed any common reasons that cause trainings to end up with denormal weights? I have been running some quick, proof-of-concept experiments with an opensource framework on Github, and their training code results in multiple denormal weights. Not having written the code myself, it would be helpful to have some pointers where to start looking</p>

username: green0765
cooked: <p>Notice that setting torch.set_flush_denormal(True) may cause accuracy regression. I suppose it’s because the denormal numbers are too many in the Conv’s weight.<br>
So the final solution has to be reduce or avoid denoraml weights in the training. Does anyone have best practise of it?<br>
None more observasion is when updating the pytorch from 1.13 to 2.0. The denormal weight of same recipe increased. Any clue?</p>