How to set 'nan' in Tensor to 0

Chen-Wei_Xie: <p>Hi, all</p>
<p>How to set ‘nan’ in Tensor to 0? Now I have a extremely inefficient method:</p>
<pre><code class="lang-python">my_tensor_np = my_tensor.cpu().numpy()
my_tensor_np [np.isnan(my_tensor_np )] = 0
my_tensor.copy_(torch.from_numpy(my_tensor_np ).cuda())
</code></pre>
<p>But copy tensor between gpu and cpu takes lots of time, so I need a more efficient way.</p>
<p>Anyone can help me? Thanks a lot!</p>

iamalbert: <p>NaN means a value which is undefined or unrepresentable. In most cases it makes no sense to simply set NaNs to zero.</p>

Chen-Wei_Xie: <p>Thank you, iamalbert</p>
<p>A paper I recently read use this trick but implemented in Theano. I want to re-implement their algorithm in PyTorch.</p>
<p>I think np.isnan is a useful function, but torch doesn’t implement it, is there a efficient solution?</p>
<p>Thank you!</p>

iamalbert: <p>It’s simple, <code>a != a</code> will give you a ByteTensor, indicating the positions of NaNs</p>
<pre><code class="lang-auto">&gt;&gt;&gt; b

    nan     nan -0.8395
    nan     nan     nan
-1.7921     nan  0.1864
[torch.FloatTensor of size 3x3]

&gt;&gt;&gt; b != b

 1  1  0
 1  1  1
 0  1  0
[torch.ByteTensor of size 3x3]
</code></pre>
<p>you can use <code>b[b != b] = 0</code> to set all NaNs to zero.</p>

Ben: <p>Seems not work now.</p>
<p>I tried below code in v0.20:</p>
<pre><code>a = torch.Tensor([1,2,3,4,5])
b = 0.0
c = a / b
c != c
</code></pre>
<p>got all 0s.</p>
<p>Is there any function like np.nan_to_num?</p>

colesbury: <p><a class="mention" href="/u/ben">@Ben</a>, that’s because <code>c</code> in your example is all <code>Inf</code>, not <code>NaN</code>. Albert’s suggestion works:</p>
<pre><code class="lang-python">a = torch.Tensor([float('NaN'), 1, float('NaN'), 2, 3])
print(a)
a[a != a] = 0
print(a)
</code></pre>

chauhankaranraj: <p>Hi <a class="mention" href="/u/chen-wei_xie">@Chen-Wei_Xie</a>,</p>
<p>torch.isnan() was implemented earlier this year. So now you can set the NaN’s to 0’s using <code>my_tensor[torch.isnan(my_tensor)] = 0</code></p>
<p>Cheers</p>

egilm-ragulpr: <p>See discussion in <a href="https://discuss.pytorch.org/t/relu-turns-nan-into-zeros/4146/3">ReLU turns nan into zeros</a>. As of pytorch 4.1 ReLU can’t be used for this anymore though. I’ve asked about it there.</p>

justusschock: <p>For completness I’ll copy my answer:</p>
<aside class="quote no-group" data-username="justusschock" data-post="5" data-topic="4146" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/justusschock/48/4637_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/relu-turns-nan-into-zeros/4146/5">ReLU turns nan into zeros</a></div>
<blockquote>
<p>since <code>NaN != NaN</code> you could do <code>my_tensor[my_tensor!=my_tensor] = 0</code></p>
</blockquote>
</aside>

egilm-ragulpr: <p>Thank you, I’ll continue the discussion here. This works both forwards/backwards on CPU:</p>
<pre><code class="lang-auto">import torch

model = torch.nn.Linear(10,10)
x = torch.ones(10,10).detach()
x[0,0] = x[0,0]+float('NaN')

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
model.zero_grad()
y = model(x)
y[y!=y] = 0 
loss = y.sum()
loss.backward()
optimizer.step()
</code></pre>
<p>Could someone verify if it works on GPU?</p>
<p>Also, anyone has insights on the computational cost vs the ol’ ReLU-hack?</p>

Tomato3351: <p>wonderful!thanks very much.</p>

mjstevens777: <p>Modifying tensors in-place can cause issues with backprop. Here is my solution, since this is still ranked highly on google:</p>
<pre><code class="lang-auto">safe_tensor = torch.where(torch.isnan(my_tensor), torch.zeros_like(my_tensor), my_tensor)
</code></pre>

Nikronic: <p>Hi,</p>
<p>From version 1.8.1, <a href="https://pytorch.org/docs/stable/generated/torch.nan_to_num.html" rel="noopener nofollow ugc">torch.nan_to_num — PyTorch 1.8.1 documentation</a> is now available. It replaces <code>NaN</code> , positive infinity, and negative infinity values in <code>input</code> with the values specified by <code>nan</code> , <code>posinf</code> , and <code>neginf</code> , respectively.</p>