Embedding returns nan issue

username: my_torch
cooked: <p>Hi, I created a model with 2 embeddings and an lstm module like below.<br>
My input is of the shape <strong>(batch_size, window_size, feature_len)</strong>, so a 3-dimension matrix.</p>
<pre><code class="lang-auto">class model (nn.Module):
    def __init__(self, some parameters):
          self.embed1 = nn.Embedding(len1, 10)
          self.embed2 = nn.Embedding(len2, 10)
    def forward(self, vec_seq):
          # vec_seq.size() --&gt; (batch_size, window_size, feature_len)
          # vec_seq.type() --&gt; torch.cuda.FloatTensor
          var1 = autograd.Variable(vec_seq[:,:,0].type(torch.cuda.LongTensor))
          var2 = autograd.Variable(vec_seq[:,:,1].type(torch.cuda.LongTensor))
          var1_emb = self.embed1(var1)
          var2_emb = self.embed2(var2)
          var_embed = torch.cat((var1_emb, var2_emb,
                                         autograd.Variable(vec_seq[:,:,2].contiguous().view(batch_size, window_size, -1)),
                                        autograd.Variable(vec_seq[:,:,3].contiguous().view(batch_size, window_size, -1)), 2)
          #  following some LSTM operations.
</code></pre>
<p>However, when I debug my program, I found all the values of <code>var1_embed</code> and <code>var2_embed</code> are <code>nan</code>, which is quite weird. In some cases, they are not all <code>nan</code>, instead part of the embedding is <code>nan</code> and the remaining is a float. like the image below.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5c2267789dcc3f8431f5df8af417ced05247ca59.png" alt="ip_var" data-base62-sha1="d93Dy6RSVMfRgzNl0zpBxLTYzyx" width="441" height="122"><br>
The corresponding embedding is like below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/08691433ade858469b8ce51dee1e18904b01f615.png" data-download-href="https://discuss.pytorch.org/uploads/default/08691433ade858469b8ce51dee1e18904b01f615" title="ipembed"><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/08691433ade858469b8ce51dee1e18904b01f615.png" alt="ipembed" data-base62-sha1="1coWUWwOMYZkV9aMtruNNKiV1Ih" width="690" height="435" data-dominant-color="37122B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ipembed</span><span class="informations">692×437 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I used <code>DataLoader</code> to load the data with following statement.<br>
<code>train_dataloader = DataLoader(train_data,batch_size=opt.batch_size,shuffle=True,num_workers=opt.num_workers)</code>.<br>
where  <code>batch_size=4</code> and <code>num_workers=4</code>.<br>
What’s leading to this error?</p>
<p>Thanks.</p>

username: chenyuntc
cooked: <p>Hard to say. It would be better if you could provide a minimal scripts that I  reproduce your problem directly in my laptop.</p>

username: chenyuntc
cooked: <p>Currently, I’ll guess something is wrong with your lstm module.</p>

username: my_torch
cooked: <p>Dear Yun,</p>
<p>Thanks for your help!</p>
<p>I finally found out where the error came from. When I did normalization on my data, some value divides 0 and got <strong>np.nan</strong> and when more and more data were fed to the model, the <strong>nan</strong> error propagates through the network.</p>
<p>Thanks very much for your help!</p>

username: 3801e223979e98646718
cooked: <p>So the problem was not about Embedding layer? Also, i cannot find normalization part in your code attached</p>

username: joseph280
cooked: <p>This answer didn’t help me but I found the problem I had.</p>
<p>My db was so small compare to the dimensions of the tutorial I was following and my embeddings were way to big for my data available, so eventually the NaN propagates through the network. Making my embedding nets smaller (smaller number of factors / columns in the matrix) solved the NaN problem for me.</p>