Nan Values appears after some training on mel spectrograms but when printing to see where they turn nan, they don't appear ¿?

Alex_Chung: <p>I have been working on a model for voice cloning, but I can’t make it train because I get Nan values suddenly when training. Please help!!! However, when I print the L1 Loss which is the one that turns Nan because of the input, it runs more epochs! But the progress spectrogram doesn’t look good.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6ad17bef5ef818dbb23b061f6adca1967e0cc97e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6ad17bef5ef818dbb23b061f6adca1967e0cc97e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6ad17bef5ef818dbb23b061f6adca1967e0cc97e_2_655x500.jpeg" alt="image" data-base62-sha1="feXqD6qjNDjXEErEl9vFhze35OC" width="655" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6ad17bef5ef818dbb23b061f6adca1967e0cc97e_2_655x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6ad17bef5ef818dbb23b061f6adca1967e0cc97e.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6ad17bef5ef818dbb23b061f6adca1967e0cc97e.jpeg 2x" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">844×644 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
These are the Layers that cause the Nan Values:</p>
<pre><code class="lang-auto">class DecConvBlock(nn.Module):
    def __init__(self, 
                 input_size,
                 in_hidden_size,
                 dec_kernel_size,
                 gen_kernel_size,
                 **kwargs):
        super(DecConvBlock,self).__init__(**kwargs)

        self.dec_conv = ConvModule(
                    input_size,
                    in_hidden_size,
                    dec_kernel_size)
        self.gen_conv = ConvModule(
                    input_size,
                    in_hidden_size,
                    gen_kernel_size)
        
    def forward(self, x):
        print("Input")
        print(x)
        y = self.dec_conv(x)
        print("After first conv")
        print(y)
        y = y + self.gen_conv(y)
        print("After second conv")
        print(y)
        return x + y
</code></pre>
<pre><code class="lang-auto">class ConvModule(nn.Module):
    def __init__(self, input_size, filter_size, kernel_size, **kwargs):
        super(ConvModule,self).__init__(**kwargs)
        
        self.seq = nn.Sequential(
                    ConvNorm(input_size,filter_size,kernel_size),
                    nn.BatchNorm1d(filter_size),
                    nn.LeakyReLU(),
                    ConvNorm(filter_size,input_size,kernel_size)
                    )
    def forward(self,x):
        return self.seq(x)
</code></pre>
<pre><code class="lang-auto">class ConvNorm(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size=1, stride=1,
                 padding='same', dilation=1, groups=1, bias=True, initializer_range=0.02, padding_mode='zeros'):
        super(ConvNorm, self).__init__()
       
        self.conv = nn.Conv1d(in_channels, out_channels,
                              kernel_size=kernel_size, stride=stride,
                              padding=padding, dilation=dilation,
                              groups=groups,
                              bias=bias, padding_mode=padding_mode)

        nn.init.trunc_normal_(
            self.conv.weight, std=initializer_range)

    def forward(self, signal):
        conv_signal = self.conv(signal)
        return conv_signal
</code></pre>

hassamwazir: <p>Hello Alex,</p>
<p>I have a similar problem. I’m not sure if we are getting it for the same reason though. I have a model that classifies sounds. It calculates the log Melspectrogram of an audio waveform inside the model using torchAudio and the result is fed to a bunch of CNN layers. After a couple of epochs, I started getting NaNs. But previously, when I was performing the same operations outside the model, i.e., in the dataset class, and using the log Melspectrograms as input to the model, it trained well.</p>
<p>Currently, I’ve found a workaround. If I move everything to the CPU, my training resumes without any NaNs. I do not know the cause of this, but just want to share it here in case it helps someone.</p>

Alex_Chung: <p>What I found in that month to overcome this was that I needed to add a normalization layer after each layer in another module. I don’t know if this was the solution, though, because I didn’t get the results I was expecting.</p>