HELP --> RuntimeWarning: invalid value encountered in double_scalars

hfdp: <p>Hello,</p>
<p>I get this error: <strong>RuntimeWarning: invalid value encountered in double_scalars</strong><br>
related to this line of code: <strong>FILE.write("\nEpoch “+str(epoch)+”: Training Loss = " +str(np.sum(trg_loss).max()/len(trg_loss)))</strong></p>
<p>The output looks as follows with NaN:  Epoch : 0<br>
Epoch 0: Training Loss = nan</p>
<p>The entire code is posted below:</p>
<pre><code class="lang-auto">&gt; 
&gt; model = MyModel()
&gt; 
&gt; print(model)
&gt; 
&gt; ## name of text file to save training stats
&gt; TRAINING_LOG = "log.txt"
&gt; FILE = open(TRAINING_LOG, 'w')
&gt; 
&gt; 
&gt; #%% Here we train and evaluate the model
&gt;         
&gt; saved_models_path = os.getcwd() + '/models/'
&gt; if(not os.path.isdir(saved_models_path)):
&gt;     os.makedirs(saved_models_path)
&gt; 
&gt; 
&gt; # Check if CUDA is available
&gt; use_cuda = torch.cuda.is_available()
&gt; # print("\n\n USE CUDA: ", use_cuda)
&gt; # print("\n\n")
&gt; 
&gt; 
&gt; ## Setting cuda as False for stability. Please comment out if using GPU
&gt; use_cuda = False
&gt; 
&gt; device = torch.device("cuda:0" if use_cuda else "cpu")
&gt; # cudnn.benchmark = True
&gt; 
&gt; 
&gt; 
&gt; 
&gt; # Set training parameters
&gt; params = {'batch_size': 64,
&gt;           'shuffle': True,
&gt;           'num_workers': 10}
&gt; max_epochs = 100
&gt; core_num = 20
&gt; tune_every = 10 # test and validate the model every x epochs
&gt; 
&gt; data_library = {}
&gt; prior = '100MHz'
&gt; target = '250MHz'
&gt; 
&gt; ## Amount of dataset to use for the experiment
&gt; data_frac = 1  ## What fraction of data to use for training (please set to 1 for final experiment ) 
&gt; 
&gt; 
&gt; 
&gt; # Load all the data from the txt file
&gt; file_IDs = open('ID_list.txt','r').read().split('\n')
&gt; file_IDs = file_IDs[:-1] # remove last line
&gt; file_IDs = file_IDs[:int(len(file_IDs)*data_frac)]
&gt; complete_dataset = Dataset(file_IDs)
&gt; 
&gt; 
&gt; # create your optimizer
&gt; optimizer = optim.SGD(model.parameters(), lr=0.005, momentum = 0.1)
&gt; scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.95)
&gt; 
&gt; #%% Here we train the network
&gt; 
&gt; # Divide the dataset into the training and validation set
&gt; lengths = [int(np.ceil(len(complete_dataset)*0.8)), int(np.floor(len(complete_dataset)*0.1)), int(np.floor(len(complete_dataset)*0.1))]
&gt; training_set, validation_set, evaluation_set = torch.utils.data.random_split(complete_dataset, lengths)
&gt; training_generator = data.DataLoader(training_set, **params)
&gt; validation_generator = data.DataLoader(validation_set, **params)
&gt; evaluation_generator = data.DataLoader(evaluation_set, **params)
&gt; 
&gt; 
&gt; # instantiate the model to make it a double tensor
&gt; forward_model = model.double()
&gt; 
&gt; # # check if model works for random instance of the data 
&gt; # t100 ,t250 = next(iter(training_generator))
&gt; # one_prediction = forward_model(t100)
&gt; 
&gt; 
&gt; loss_function = nn.MSELoss()
&gt; 
&gt; 
&gt; for param in forward_model.parameters():
&gt;     param.requires_grad = True
&gt; 
&gt; def train(low_res, high_res):
&gt; # Loop over epochs
&gt;     global forward_model, optimizer
&gt;     low_res, high_res = low_res.to(device), high_res.to(device)
&gt;     optimizer.zero_grad()
&gt;     prediction_training = forward_model(low_res)
&gt;     loss = loss_function(prediction_training, high_res) #Here compute tha value that estimates how far the outut is from the target
&gt;     #print("\nTraining Loss: ", loss)
&gt;     loss.backward() #The whole graph is differentiatted wrt the loss and all ensors in the graph that have 'required_gard = true' will have their .grad tensot accumulaed ith the gradient. 
&gt;     optimizer.step()
&gt;     return loss
&gt; 
&gt; 
&gt; def val(low_res, high_res):
&gt;     global forward_model, optimizer
&gt;     low_res, high_res = low_res.to(device), high_res.to(device)
&gt;     optimizer.zero_grad()
&gt;     prediction_training = forward_model(low_res)
&gt;     loss = loss_function(prediction_training, high_res) #Here compute tha value that estimates how far the outut is from the target
&gt;     #print("\nValidation Loss: ", loss)
&gt;     #scheduler.step()
&gt;     return loss
&gt; 
&gt; 
&gt; 
&gt; for epoch in range(max_epochs):
&gt;     print('\nEpoch : ' + str(epoch))   
&gt;     FILE.write('\nEpoch : ' + str(epoch))
&gt; 
&gt;    	# Training
&gt;     trg_loss = [] 
&gt;     forward_model.train()
&gt;     with torch.set_grad_enabled(True):
&gt;         for data_library['100MHz'], data_library['250MHz'] in training_generator:
&gt;             if(core_num&gt;1):
&gt;                 Parallel(n_jobs=core_num)
&gt;                 #print(data_library['20MHz'].shape)
&gt;                 delayed(train)(data_library[prior],data_library[target])
&gt;             else:
&gt;                 new_loss = train(data_library[prior],data_library[target])
&gt;                 trg_loss.append(new_loss)
&gt;             # Parallel(n_jobs=core_num)
&gt;             # print(data_library['20MHz'].shape)
&gt;             # delayed(train)(data_library[prior],data_library[target])
&gt;     #print("\nTraining Loss: ", sum(trg_loss).detach().numpy().max()/len(trg_loss))
&gt;     FILE.write("\nEpoch "+str(epoch)+": Training Loss = " +str(np.sum(trg_loss).max()/len(trg_loss)))
&gt;            
&gt;             
&gt;     if epoch % tune_every == 0 and epoch &gt; 0:
&gt;    	# validation
&gt;         val_loss = []
&gt;         forward_model.eval()
&gt;         with torch.set_grad_enabled(False):
&gt;             for data_library['100MHz'], data_library['250MHz'] in validation_generator:
&gt;                 if(core_num&gt;1):
&gt;                     Parallel(n_jobs=core_num)
&gt;                     #print(data_library['20MHz'].shape)
&gt;                     delayed(train)(data_library[prior],data_library[target])
&gt;                 else:
&gt;                     new_loss = val(data_library[prior],data_library[target])
&gt;                     val_loss.append(new_loss)
&gt; 
&gt;         #print("\nValidation Loss: ", sum(val_loss).numpy().max()/len(val_loss))
&gt;         FILE.write("\nEpoch "+str(epoch)+": Validation Loss = " +str(np.sum(val_loss).max()/len(val_loss)))
&gt;             # HYPERPARAMETER TUNING GOES IN HERE
&gt; 
&gt; FILE.close()
</code></pre>

albanD: <p>Are you sure it is not numpy that through this warning?<br>
Also I guess this is because you have nan values. Fixing the nan values will most likely make the warning go away.</p>

hfdp: <aside class="quote no-group" data-username="hfdp" data-post="1" data-topic="85034">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hfdp/48/25422_2.png" class="avatar"> hfdp:</div>
<blockquote>
<p>FILE.write("\nEpoch “+str(epoch)+”: Training Loss = " +str(np.sum(trg_loss).max()/len(trg_loss)))</p>
</blockquote>
</aside>
<p>Then would you suggest it wuthout the numpy? aka:</p>
<p><strong>FILE.write("\nEpoch “+str(epoch)+”: Training Loss = " +str(sum(trg_loss).max()/len(trg_loss)))</strong></p>
<p>The NaN also needs to be fixed, but no clue where this comes from yet. Any way to check it quickly?</p>

albanD: <p>Yes, you actually do the same sum on the commented line just above without the np <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>If it happens during the forward, you can add a few prints in your code to find out where it appears.<br>
If it is during the backward, you can enable the <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly">anomaly detection</a> to know where it appears in the backward.</p>