Word2Vec embedding gives strange gradients to Linear Layers

Dumiy: <p>Hello,</p>
<p>Sorry if I dont give all the info from the start, this is my first post on forums.</p>
<p>I’ve been using as a feature extractor from text Word2Vec from gensim to have word embedding and at training with a pytorch model the validation result are very alterning from epoch to epoch, I think due to the gradients being very concentrated in the last layer and close to 0 to the other layers. I attached also a plot from the layers gradients. The thing that bothered me is that scikit-learn MLPRegressor gives more stable results out of the same features and replicated in pytorch, giving different results.</p>
<p>The model looks like this, using L1Loss and Adam optimizer</p>
<pre><code class="lang-auto">class Net(nn.Module):
    Preformatted textdef __init__(self):
          super(Net, self).__init__()
        self.h0 = torch.randn(1, 1, 150).cuda()
        self.c0 = torch.randn(1, 1, 150).cuda()
        self.lstm = nn.LSTM(200,150,1)
        for name, param in self.lstm.named_parameters():
            if 'bias' in name:
                pass
            elif 'weight' in name:
                nn.init.xavier_normal_(param,gain=nn.init.calculate_gain('tanh'))
        self.fc1 = nn.Linear(150,500)
        nn.init.xavier_normal_(self.fc1.weight,gain=nn.init.calculate_gain('tanh'))
        self.fc2 = nn.Linear(500,300)
        nn.init.xavier_normal_(self.fc2.weight,gain=nn.init.calculate_gain('tanh'))
        self.fc3 = nn.Linear(300, 2)
        nn.init.xavier_normal_(self.fc3.weight,gain=nn.init.calculate_gain('tanh'))
        #self.bc1 = nn.BatchNorm1d(150)
        #self.bc3 = nn.LayerNorm(300)
        self.f = nn.Tanh()
    def forward(self, x):
        x = x.unsqueeze(1)
        x,(self.h0,self.c0) = (self.lstm(x,(self.h0,self.c0)))
        x = self.f(x)
        #print(x.shape)
        x = x.flatten(1)
        x = self.f(self.fc1(x))
        x = self.f(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f007a18e68829745c3a6d129fbefe9d387a6aba5.png" alt="image" data-base62-sha1="yfoWqEYS0CmlLSnzBuQIO3HKCvH" width="427" height="351"></p>
<p>I want to know why this happens and why I can’t achieve better results then scikit-learn MLPRegressor</p>

Abhilash_Srivastava: <p>This could be due to the vanishing gradient problem. Can you try using just one fc layer instead of 3?<br>
Also please format your code for clear reading.</p>

Dumiy: <p>Thank you for the response, I made the changes and kept only one fc and I attached the gradients which kinda look the same and the validation result are worse. I looked out for the vanishing gradients on LSTM and dont seem to find anything<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44896900d1e0824d478af1d2fd7d56441b93ec9a.png" alt="image" data-base62-sha1="9MiROhtRGXL8JPgwzArBfxVThXk" width="459" height="370"></p>
<p>I tried also increasing the batchsize and to do a gradient update on 3 batches instead at every batch and the gradient resulting acumulating gradiens looking like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55903de961a28d925bbb27920a09c3e606129bca.png" alt="image" data-base62-sha1="ccVCxQWvp262IUAm1hoyYQjADQ6" width="474" height="356"></p>

Abhilash_Srivastava: <ol>
<li>Try using ReLU instead of Tanh for activation.</li>
<li>Can you share the rest of code for training and validation?</li>
</ol>

Dumiy: <p>Sure the loss is L1Loss and optimizer is Adam and here are the training and  validation</p>
<pre><code class="lang-auto">def test(loss):
    network.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        val = network(torch.Tensor(wordvec_val).cuda())
        loss_value = loss(val, torch.Tensor(test_labels).long().cuda())
        print(loss_value.item())
        val = val.cpu().detach().numpy()
        print(" ")
        print("Validation MAE : scikit-learn ",mean_absolute_error(val,test_labels))
        print(" ")
        test_val.append(mean_absolute_error(val,test_labels))
def train(epoch,loss):
  network.train()
  for batch_idx, (data, target) in enumerate(train_loader):
        
    optimizer.zero_grad()
    
    output = network(data.cuda())
    
    loss_value = loss(output, target.long().cuda())
    
    loss_value.backward(retain_graph=True)
    
    plot_grad_flow(network.named_parameters())
    
    optimizer.step()
    
    if batch_idx % log_interval == 0:
      print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
        epoch, batch_idx * len(data), len(train_loader.dataset),
        100. * batch_idx / len(train_loader), loss_value.item()))
</code></pre>

Abhilash_Srivastava: <p>The train and validation code looks fine to me.</p>
<ol>
<li>Why are you using <code>loss_value.backward(retain_graph=True)</code> and not just <code>loss_value.backward()</code>?</li>
<li>Any particular reason for using L1 loss instead of any other loss (eg: CrossEntropyLoss)?</li>
</ol>