VAE (Ecoder as well) is outputting nan

Flock1: <p>I am working with VAE and I don’t know why but during the training process, I am getting the output of VAE as well as that of the encoder as <code>nan</code>. I am reading a CSV file with rows as my data. Here’s my code:</p>
<p>My data loader:</p>
<pre><code class="lang-auto">class data_gen(torch.utils.data.Dataset):
    def __init__(self, files):
        
        self.data = files
        
    def __getitem__(self, i):

        tmp = self.data[i]
        tmp = np.reshape(tmp,(1025))
        tmp = torch.from_numpy(tmp).float()

        return tmp

    def __len__(self): 
        
        return len(self.data)
</code></pre>
<p>Here’s how I am normalising it:</p>
<pre><code class="lang-auto">def norm(data):
    norm_data = []
    for j in data:
        temp = float(2.0*(j-np.amin(data))/(np.amax(data)-np.amin(data)) - 1.0)
        norm_data.append(temp)
    return norm_data
</code></pre>
<p>VAE:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.lin1 = nn.Linear(in_features = 1025, out_features = 500)
        self.lin2 = nn.Linear(in_features=500, out_features=100)
        self.lin3 = nn.Linear(in_features=100, out_features=10) 
        self.fc_mu = nn.Linear(in_features=10, out_features=latent_dims)
        self.fc_logvar = nn.Linear(in_features=10, out_features=latent_dims)
            
    def forward(self, x):

        x = F.relu(self.lin1(x))
        x = F.relu(self.lin2(x))
        x = torch.tanh(self.lin3(x))
        x_mu = self.fc_mu(x)
        x_logvar = self.fc_logvar(x)
        return x_mu, x_logvar


class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()
        c = capacity
        self.fc = nn.Linear(in_features=latent_dims, out_features=10)
        self.lin3 = nn.Linear(in_features=10, out_features=100)
        self.lin2 = nn.Linear(in_features=100, out_features=500)
        self.lin1 = nn.Linear(in_features=500, out_features=1025)

        
    def forward(self, x):
        x = self.fc(x)
        x = F.relu(self.lin3(x))
        x = torch.tanh(self.lin2(x))
        x = torch.tanh(self.lin1(x)) # last layer before output is sigmoid, since we are using BCE as reconstruction loss
        return x
</code></pre>
<p>I am using <code>tanh</code> because I have some negative values. The weird thing is, it sometimes gives proper values. I have normalized by input data, my libraries are up to date and I have tried almost all the solutions provided. Even if I just run the inference, I still get <code>nan</code>. So I don’t know if the learning rate is the issue. What should I do?</p>

ptrblck: <p>Could you check, if your normalization is creating invalid values via <code>torch.isfinite(input).all()</code>?</p>

Flock1: <p>I get this as well when I run the normalizing code on my data</p>
<pre><code class="lang-auto">  0%|          | 10/63100 [00:00&lt;10:32, 99.74it/s]&lt;ipython-input-17-a47b51071277&gt;:4: RuntimeWarning: invalid value encountered in double_scalars
  temp = float(2.0*(j-np.amin(data))/(np.amax(data)-np.amin(data)) - 1.0)
</code></pre>
<p>However, it ends normalizing the data though.</p>
<p>And you pointed to something here. Some of the output for</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="124248">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.isfinite(input).all()</code>?</p>
</blockquote>
</aside>
<p>are <code>false</code>. What am I doing wrong in <code>norm</code>?</p>

ptrblck: <p>I guess you might encounter specific <code>data</code> arrays with constant values, so that <code>amin</code> and <code>amax</code> would return the same value and divide by 0:</p>
<pre><code class="lang-python">data = np.array([0.1, 0.1])
print(np.amin(data) == np.amax(data))
&gt; True
1 / (np.amin(data) - np.amax(data))
&gt; inf
</code></pre>

Flock1: <p>Yup, I was getting true for that. What a catch. WOW!</p>