AdaptiveAvgPool2d causes some data to contain Inf

MartinPerry: <p>I have network that has somewhere inside <code>AdaptiveAvgPool2d</code> block.<br>
For <em><strong>some</strong></em> inputs, however, the network crash and result contains NaN. I tracked down the problem probably to the <code>AdaptiveAvgPool2d</code> .</p>
<p><code>avg_pool = nn.AdaptiveAvgPool2d((2, 2))</code></p>
<pre><code class="lang-auto">print(x.shape)
print("IsInf x", torch.isinf(x).any())
print("IsNan x", torch.isnan(x).any())
x0 = self.avg_pool(x)
print(x0.shape)
print("IsInf after avg pool", torch.isinf(x0).any())
print("IsNan after avg pool", torch.isnan(x0).any())
</code></pre>
<p>Which outputs:</p>
<pre><code class="lang-auto">torch.Size([20, 32, 256, 256])
IsInf x tensor(False, device='cuda:0')
IsNan x tensor(False, device='cuda:0')
torch.Size([20, 32, 2, 2])
IsInf after avg pool tensor(True, device='cuda:0')
IsNan after avg pool tensor(False, device='cuda:0')
</code></pre>
<p>What can cause this behaviour?</p>

ptrblck: <p>Since you have already isolated it to a single tensor, store it and analyze its values to see how the <code>Inf</code> output is calculated.</p>

MartinPerry: <p>I have isolated single channel from all batched that is 256x256.<br>
Based on stats:</p>
<blockquote>
<p>min_val = -63.97, max_val = 13.19</p>
</blockquote>
<p>Most of the values are something like <code>Xe-01</code></p>
<p>Why, or how, can <code>AdaptiveAvgPool2d</code> end with <code>Inf</code>? I dont know, how it is exactly calculated, but average value from such “small” numbers should not be <code>Inf</code>. If some division with 0 occurs, I would expect <code>NaN</code> result, not <code>Inf</code>.</p>

ptrblck: <aside class="quote no-group" data-username="MartinPerry" data-post="3" data-topic="177420">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/martinperry/48/29609_2.png" class="avatar"> MartinPerry:</div>
<blockquote>
<p>If some division with 0 occurs, I would expect <code>NaN</code> result, not <code>Inf</code>.</p>
</blockquote>
</aside>
<p>Dividing my <code>0</code> should create Infs:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.randn(2, 2) / torch.tensor(0.)
# tensor([[-inf, -inf],
#         [inf, inf]])
</code></pre>
<p>but I also don’t know how this result should be calculated assuming you are indeed not dividing by <code>0</code>.<br>
Are you able to store the tensor and manually reproduce the output by running the pooling layer alone?</p>

MartinPerry: <p>I have totally forgot, I have autocast enabled, so it may cause the problems.<br>
Since the input tensor is large (256x256) and output is only 2x2, the cumulations maybe breaks float16 precission.</p>
<p>As for 0 division, you are right. I have mistaken it with sqrt operation of negative numbers.</p>

ptrblck: <p>Ah OK, in this case I assume the input is already in <code>float16</code> coming from another layer and uses this <code>dtype</code> as its internal compute type.<br>
We might need to check if the internal compute type should be in a larger floating point precision (<code>float32</code> in this case).</p>

MartinPerry: <p>In most of the cases, for smaller sizes, <code>float16</code> would be fine. But this is rather large … is there some way, how to force single module compute precision to be <code>float32</code> during mixed precision in general?</p>

ptrblck: <p>Not directly, as you would need to change the backend. <a class="mention" href="/u/eqy">@eqy</a> is proposing a fix using a wider <code>acctype</code> during the accumulation. For now you could cast the inputs manually back to <code>torch.float32</code> to avoid these issues.</p>