NaN Output From Loss Function

chaytanchief: <p>After a few passes through my network, the loss seems to explode exponentially until it reaches inf and then NaN the rest of the way through. I know I’m not the first to have these problems, so here is what I’ve already tried</p>
<ol>
<li>My input doesn’t contain any NaNs, I replaced them with the average of the df column</li>
<li>I have tried NL1Loss and MSELoss and both have this problem</li>
<li>My learning rate is 1E-5</li>
<li>I have used anomaly detection, and sure enough it returns</li>
</ol>
<blockquote>
<p>Function ‘AddmmBackward’ returned nan values in its 2th output.</p>
</blockquote>
<p>which it just tells me about a problem I already knew I had. Still don’t know how to fix it.</p>
<p>So here is some code from the model as it is right now:</p>
<pre><code class="lang-auto">class Crypto_Net(nn.Module):
	def __init__(self, n_input, n_hidden, n_output):
		super().__init__()
		# Define the architecture
		self.net = nn.Sequential(
			# Layers 1 and 2 with 95 inputs, 20 outputs
			nn.Linear(n_input, n_hidden),
			# Activation function
			nn.LeakyReLU(),
			# Layers 2 and 3 with 20 inputs 5 outputs
			nn.Linear(n_hidden, n_output)
			)

	def forward(self, volumes_sample):
		x = self.net(volumes_sample)
		return x

def train(p, model, criterion, optimizer, train_loader):

	with autograd.detect_anomaly():
		# epoch is one full pass through dataset
		for epoch in range(p['epochs']):
			for i, (sample, target) in enumerate(train_loader):
				# Forward pass for each batch of volumes stored in train_loader
				model_output = model(sample)
				loss = criterion(model_output, target)
				# Backpropagation of error
				optimizer.zero_grad()
				# computes new grad
				loss.backward(retain_graph=True)
				# update weights
				optimizer.step()
	#			log.debug('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
	#				.format(
	#					epoch+1, p['epochs'], 
	#					i+1, len(train_loader), loss.item()
	#			))

				#import pdb; pdb.set_trace()


</code></pre>
<p>Thanks in advance for any help!<br>
Also here’s the full code on gihub: <a href="https://github.com/lollyi/crypto_net" class="inline-onebox" rel="noopener nofollow ugc">GitHub - chaytanc/crypto_net: This repository has a feed forward neural network and a whole lot of data processing on cryptocurrency stock prices. It uses Pytorch.</a></p>

ptrblck: <p>If you are using the CPU, could you check your numpy version, as we recently had an issue with an older numpy, which apparently created NaN outputs at one point?</p>

chaytanchief: <p>I have version 1.18.2</p>

chaytanchief: <p>Also yes I am using the CPU</p>

ptrblck: <p>Thanks for the information!<br>
Could you post an executable code snippet using random inputs so that we can reproduce and debug this issue, please?</p>
<p>EDIT: Also, which PyTorch version are you using?</p>

chaytanchief: <p>1.4.0</p>
<p>Also I can try to rearrange the code to get a snippet that recreates the problem, but will you download the dataset or should I make a smaller fake version as well?</p>
<p>Edit: I see that you said random inputs – I will try that</p>

chaytanchief: <p>Sorry to waste your time, I found the problem. Lesson to anyone else who reads this thread, write a test for the function that gets the nans out of your input because they’re probably still there!</p>