Model returns NaN after first iteration

francisco: <p>Hello everyone</p>
<p>I’m testing how suitable the models made available by torchvision are at, among other things, analyzing both images and audio (In regards to the audio, I first extract MFCC features from the audio clip, and turn said MFCC features into an image, as I saw some people doing it, and saying that apparently it’s  somewhat common practice).</p>
<p>However, after the first iteration, some of the models weights turn into NaN and subsequently return NaN as predictions. (These weights were checked using this code:</p>
<pre><code class="lang-auto">for a in [6,8]:
    print(a, model_a.features[a].weight)
</code></pre>
<p>)</p>
<p>I’m currently using a pretrained AlexNet (despite eventually wanting to attempt this pipeline with other models such as VGGs, GoogleNets, etc.). Up until now, I was using an LR value of 0.001, but as a result of finding a topic here in the forums in which someone suggested that a cause for exploding gradients was a high LR value, I have since lowered it to 0.000001 which does not solve the problem.</p>
<p>Help would be greatly appreciated!</p>
<p>Thank you to everyone who reads this</p>

JuanFMontesinos: <p>The most probable situation is you have a NaN in your input or in some operation which is generating a NaN. I would focus on checking the correctness of your input data.</p>

francisco: <p>Hello Juan!</p>
<p>First of all, thank you for spending your time reading and replying, I truly appreciate it.</p>
<p>After reading your response I changed my program to stop predicting and to print instead if the extracted features contain any NaN.</p>
<p>I have 2 folders ‘train’ and ‘val’, each with 2 folders inside, one for each of the classes (in my case ‘nv’ and ‘v’ which stand for not violent and violent, as I’m doing using the models to determine if a certain scene is or not violent) and each of these folders have 50 videos, so 200 videos total.</p>
<p>This whole situation occured with the videos in the ‘train’ folder, and since the videos were randomly selected, I tested all 100 videos, from the ‘nv’ and ‘v’ folders, extracting their features using the following code:</p>
<pre><code class="lang-auto">def audio_feature_extractor(video_name):
    y, sr = librosa.load(video_name) # load file

    mfccs = librosa.feature.mfcc(y=y,sr=sr,n_mfcc=20,norm='ortho') # extract mfccs

    mfccs = mfccs / np.linalg.norm(mfccs) # normalize array (20,x)

    quadmesh = librosa.display.specshow(mfccs) # convert array to quadmesh

    fig = quadmesh.get_figure() # get figure from quadmesh

    inputs = get_img_from_fig(fig) # get img from figure (3,224,224)

    inputs = inputs/255

    return inputs
</code></pre>
<p>and then I checked the resulting tensor to see if it contained any NaN’s with the following code:</p>
<pre><code class="lang-auto"># AUDIO ANALYSIS
audio_features = audio_feature_extractor(video_name)

print(torch.isnan(audio_features).any())
</code></pre>
<p>Unfortunately, every single clip came out without any NaN’s, so this does not appear to be the problem here.</p>
<p>If you have any clue about other issues that might cause this trouble, I’d love to test them, since I’ve been stagnated with this problem for the last 2 weeks <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>Kind Regards,</p>
<p>Francisco</p>

JuanFMontesinos: <p>Yes but for example you are normalizing <code>np.linalg.norm(mfccs)</code>.<br>
If the tensor is near zero you would be in troubles.</p>
<p>Anyway a good thing you can do is check for NaNs in the loss (it’s a cheap operation).<br>
If there is a NaN you can follow back the error and revise the input or the layers and try to find the reasoin reproducing that<br>
There are very few resons why you would get a NaN.<br>
These are:<br>
A NaN in the weights (propbably due to loss=NaN which was backpropagated)<br>
A operator over an empty tensor.<br>
A normalization step over.</p>

francisco: <blockquote>
<p>Anyway a good thing you can do is check for NaNs in the loss (it’s a cheap operation).<br>
If there is a NaN you can follow back the error and revise the input or the layers and try to find the reasoin reproducing that</p>
</blockquote>
<p>Can you tell me how I’d be able to do that? So far, I’ve only found suggestions on using something called <code>autograd.set_detect_anomaly(True)</code> which I used as follows:</p>
<pre><code class="lang-auto">with torch.set_grad_enabled(phase == 'train'):
    with torch.autograd.set_detect_anomaly(True):
        audio_output = model_a(audio_features) # prediction
                                
        #audio_output = torch.nan_to_num(model_a(audio_features), neginf=0.0, posinf=1.0) # make prediction

        if (isinstance(audio_output,GoogLeNetOutputs)): # extract tensor from GoogLeNetOutputs
            audio_output = audio_output.logits

            preds =  torch.argmax(audio_output, 1) # get prediction
            # print(f"preds: {preds}, audio_output: {audio_output}, labels: {labels}\n")
            a_loss = criterion(audio_output, labels) # calculate loss
            print(f"A Loss: {a_loss}")
            #a_loss = a1_loss.float() # convert from long to float

            if phase == 'train':
                print('\n--- IF ---\n')
                a_loss.backward()

                for name, param in model_a.named_parameters():
                    print(name, torch.isfinite(param.grad).all()) # print grads

                model_a_optimizer.step()

                print('\n--- STEP ---\n')
                for name, param in model_a.named_parameters():
                    print(name, torch.isfinite(param.grad).all()) # print grads
                            
                #statistics
                print('a: ', a_loss, a_loss.item(), audio_features.size(0))
                a_running_loss += a_loss.item() * audio_features.size(0)
                a_running_corrects += torch.sum(preds == labels.data)
                                
                audio_dataset_sizes[phase] = audio_dataset_sizes[phase]+1
</code></pre>
<p>Should I be using <code>set_detect_anomaly</code> or should I use something else?</p>
<blockquote>
<p>There are very few resons why you would get a NaN.</p>
</blockquote>
<p>I assume you mean get a NaN in the loss component, correct?</p>