NaN in input tensors

justusschock: <p>Hi,</p>
<p>I wonder how PyTorch deals with NaN-Values in the inputs? Are convolutions of NaN again NaN? And What is ReLU(NaN)?</p>
<p>Is there a recommended way to deal with NaN values (other then setting NaNs to a constant value e.g. zero)?</p>

ptrblck: <p>As far as I know the <code>nan</code> values will be propagated by mathematical operations.<br>
<code>ReLU</code> seems to zero out the <code>nan</code>s:</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 10, 10)
x[0, 0, 0, 0] = torch.log(torch.tensor([-1.]))

m = nn.Conv2d(3, 6, 3, 1, 1)
output = m(x)
print(output[0, 0, 0:3, 0:3])
&gt; tensor([[    nan,     nan,  0.5057],
        [    nan,     nan, -0.4137],
        [-0.8564,  0.1068, -1.0790]])

r = nn.ReLU()
output = r(output)
print(output[0, 0, 0:3, 0:3])
&gt; tensor([[ 0.0000,  0.0000,  0.5057],
        [ 0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.1068,  0.0000]])
</code></pre>
<p>I would try to look for the origin of the <code>nan</code> values, because it seems to be wrong in most cases.</p>

justusschock: <p>Thanks a lot. Unfortunately they aren’t really avoidable, because they are part of my dataset.</p>
<p>Would you recommend setting them to zero or would you set it to values outside the usual data range like 1e6?</p>

ptrblck: <p>Do you have a dataset consisting of features, i.e. are the <code>nan</code> values indicate a missing feature?<br>
If so, you could try to set them to the median value of this particular feature or to a categorical value indicating a missing feature, e.g. <code>-1</code>.</p>
<p>But I think it depends on the data you have.<br>
Could you explain what kind of data you have and why the <code>NaN</code>s are there?</p>

hughperkins: <p>The conv layers cant magically handle nan values using some kind of interpolation/semi-supervised learning/etc. So you need yourself to figure out how to convert the nan values into an actual concrete value that will make sense to the model you have written. As Patrick says, using the median/average value might be a quick way to get something working. Otherwise you can look at more complex techniques, which you could find by googling eg ‘handling missing data’, which throws up eg <a href="https://en.wikipedia.org/wiki/Missing_data" rel="nofollow noopener">https://en.wikipedia.org/wiki/Missing_data</a>  (I imagine a ton of Kaggle competition write-ups would give some insight into how to handle missing data ?)</p>

Pablo_Jeken: <p>I have the same issue, so maybe his topic is also related to mine. I am handling data sets that describe certain chemicals on the ocean surface. You can see this as a map with valid values on the sea surface and a mask (or nans) on the land surface.</p>
<p>Ideally, I would like to ignore nans by using the same padding techniques as at the image borders and completely remove full nan fields after the flattening. Hereby, it is worth mentioning that the nans are at the same position for all images and features. Any hints on how I can do this? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>NaN values are tricky to handle, as you cannot simply mask them e.g. by multiplying with zeros.<br>
If you are using a mask for your targets I would recommend to use a specific class index (e.g. <code>nb_classes+1</code>) and try to deal with these values instead.</p>

sbelharbi: <p>in pytorch <code>1.7.1</code>, <code>relu</code> does not seem to  behave like this.</p>
<pre><code class="lang-bash">torch version: 1.7.1
tensor([[    nan,     nan, -0.2346],
        [    nan,     nan,  1.3086],
        [-0.0514, -0.6495, -0.5092]], grad_fn=&lt;SliceBackward&gt;)
tensor([[   nan,    nan, 0.0000],
        [   nan,    nan, 1.3086],
        [0.0000, 0.0000, 0.0000]], grad_fn=&lt;SliceBackward&gt;)
</code></pre>
<p>code:</p>
<pre><code class="lang-python">import random

import torch
import numpy as np


# repro.
seed = 0
torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

print('torch version: {}'.format(torch.__version__))

x = torch.randn(1, 3, 10, 10)
x[0, 0, 0, 0] = torch.log(torch.tensor([-1.]))

m = torch.nn.Conv2d(3, 6, 3, 1, 1)
output = m(x)
print(output[0, 0, 0:3, 0:3])


r = torch.nn.ReLU()
output = r(output)
print(output[0, 0, 0:3, 0:3])
</code></pre>
<p>seems to have been <a href="https://discuss.pytorch.org/t/how-to-set-nan-in-tensor-to-0/3918/8">changed</a> in earlier version.<br>
thanks</p>

ehtram_darvishi: <p>hello<br>
I also have dataset that has null values<br>
But I don’t know how much to replace.<br>
My data is null because they don’t exist in some situations.<br>
What do you think I should do?<br>
Many thanks</p>

ptrblck: <p>I don’t know which approach would work best and would refer to <a href="https://discuss.pytorch.org/t/nan-in-input-tensors/19109/5">this post</a>.</p>

ehtram_darvishi: <p>Thank you so much.<br>
I realized I had to stop using Nan because the model couldn’t work with it. <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

SalCometa: <p>Hi Pablo,<br>
How did you solve it?</p>