MaLstm not training

username: steveoni
cooked: <p>i tried training on qoura qeustion pair using MaLSTM</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a233480bfd9c7611ac97563d937e5cb9b0b9f74.png" data-download-href="https://discuss.pytorch.org/uploads/default/8a233480bfd9c7611ac97563d937e5cb9b0b9f74" title="f1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a233480bfd9c7611ac97563d937e5cb9b0b9f74_2_690x433.png" alt="f1" data-base62-sha1="jI1jGo0OJv9SH3pNZKG5vXXv9Qw" width="690" height="433" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a233480bfd9c7611ac97563d937e5cb9b0b9f74_2_690x433.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a233480bfd9c7611ac97563d937e5cb9b0b9f74_2_1035x649.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a233480bfd9c7611ac97563d937e5cb9b0b9f74.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">f1</span><span class="informations">1200×754 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57065121740c2109d2273ae6371665ffd5c49b8c.png" data-download-href="https://discuss.pytorch.org/uploads/default/57065121740c2109d2273ae6371665ffd5c49b8c" title="f2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57065121740c2109d2273ae6371665ffd5c49b8c_2_690x353.png" alt="f2" data-base62-sha1="cpR4zV4rfyOiBuGhHXR0Ig5A9ly" width="690" height="353" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57065121740c2109d2273ae6371665ffd5c49b8c_2_690x353.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57065121740c2109d2273ae6371665ffd5c49b8c_2_1035x529.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/57065121740c2109d2273ae6371665ffd5c49b8c.png 2x" data-dominant-color="F3F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">f2</span><span class="informations">1200×614 97.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/831deb587cf2d1f6d37053bed8b7c9484ba23522.png" data-download-href="https://discuss.pytorch.org/uploads/default/831deb587cf2d1f6d37053bed8b7c9484ba23522" title="f4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/831deb587cf2d1f6d37053bed8b7c9484ba23522_2_690x436.png" alt="f4" data-base62-sha1="iHUEkqbF8MVQGzJzQgZmPMlaFnI" width="690" height="436" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/831deb587cf2d1f6d37053bed8b7c9484ba23522_2_690x436.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/831deb587cf2d1f6d37053bed8b7c9484ba23522_2_1035x654.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/3/831deb587cf2d1f6d37053bed8b7c9484ba23522.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">f4</span><span class="informations">1200×760 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb5d829928cd59fed043a7ad352f61b09ab76cd9.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb5d829928cd59fed043a7ad352f61b09ab76cd9" title="f5"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb5d829928cd59fed043a7ad352f61b09ab76cd9_2_690x387.png" alt="f5" data-base62-sha1="xA8y6GE3gDzJ68CKEyWDMIuiWnf" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb5d829928cd59fed043a7ad352f61b09ab76cd9_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb5d829928cd59fed043a7ad352f61b09ab76cd9_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb5d829928cd59fed043a7ad352f61b09ab76cd9.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">f5</span><span class="informations">1200×674 197 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I use MSELoss has described in the paper, but it seems it is not learning anything</p><aside class="onebox pdf" data-onebox-src="https://hal.science/hal-02271338/file/Manhattan_Siamese_LSTM_for_Question_Retrieval_in_Community_Question_Answering__1_.pdf">
  <header class="source">

      <a href="https://hal.science/hal-02271338/file/Manhattan_Siamese_LSTM_for_Question_Retrieval_in_Community_Question_Answering__1_.pdf" target="_blank" rel="noopener nofollow ugc">hal.science</a>
  </header>

  <article class="onebox-body">
    <a href="https://hal.science/hal-02271338/file/Manhattan_Siamese_LSTM_for_Question_Retrieval_in_Community_Question_Answering__1_.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://hal.science/hal-02271338/file/Manhattan_Siamese_LSTM_for_Question_Retrieval_in_Community_Question_Answering__1_.pdf" target="_blank" rel="noopener nofollow ugc">Manhattan_Siamese_LSTM_for_Question_Retrieval_in_Community_Question_Answering__1_.pdf</a></h3>

  <p class="filesize">649.53 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>Could you post the code by wrapping it into three backticks ``` please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
This would make debugging a lot easier and also would enable the forum search to index your code.</p>

username: steveoni
cooked: <p>Thanks for your quick reply</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
  
    def __init__(self, vocab_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):

      super(LSTM, self).__init__()
   

      self.n_layers = n_layers
      self.hidden_dim = hidden_dim
    
      # embedding and LSTM layers
      self.embedding = nn.Embedding(vocab_size, embedding_dim)
      self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, 
                          dropout=drop_prob, batch_first=True)
    
    def forward(self,x,hidden):


      embeds = self.embedding(x)
      lstm_out, hidden = self.lstm(embeds,hidden)

      return hidden[0]


    def init_hidden(self, batch_size):


      ''' Initializes hidden state '''
      # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
      # initialized to zero, for hidden state and cell state of LSTM
      weight = next(self.parameters()).data
      
      if (True):
          hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),
                weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())
      else:
          hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                    weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())
      
      return hidden
</code></pre>
<p>and then create the MaLSTM here</p>
<pre><code class="lang-auto">class MaLstm(nn.Module):

  def __init__(self, vocab_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):

    super(MaLstm, self).__init__()

    self.lstm1 = LSTM( vocab_size,  embedding_dim, hidden_dim, n_layers, drop_prob=0.5)
    self.lstm2 = LSTM( vocab_size,  embedding_dim, hidden_dim, n_layers, drop_prob=0.5)

    
  def forward(self,x,b):

    # x,b = x[0], x[1]
    batch_size = x.size(0)

    hidden1 = self.lstm1.init_hidden(batch_size)
    hidden2 = self.lstm2.init_hidden(batch_size)

    lstm1_h = self.lstm1(x, hidden1)
    lstm2_h = self.lstm2(b, hidden2)

    
    lstm1_h = lstm1_h.permute(1,2,0).contiguous().view(batch_size,-1)
    lstm2_h = lstm2_h.permute(1,2,0).contiguous().view(batch_size,-1)

    similarity = manhathan_dis(lstm1_h,lstm2_h)

    return similarity
</code></pre>
<pre><code class="lang-auto">vocab_size = word.n_words
embedding_dem = 400
hidden_dim = 256
n_layer = 10
net = MaLstm(vocab_size,embedding_dem, hidden_dim, n_layer)
lr = 0.001

criterion = nn.MSELoss()
optimizer = torch.optim.Adadelta(net.parameters(), lr=lr)

epochs = 20

counter = 0
print_every = 100
clip = 5

net.cuda()

net.train()

train_on_gpu = True

for e in range(epochs):
    # initialize hidden state

    # batch loop
    for inputs, labels in data.train_dl:
        counter += 1

        if(train_on_gpu):
            inp, labels = inputs.cuda(), labels.cuda()
      

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output = net(inp1)

        # calculate the loss and perform backprop
        loss = criterion(output, labels)
        loss.backward()
        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        optimizer.step()

        # loss stats
        if counter % print_every == 0:
            # Get validation loss
           
            val_losses = []
            net.eval()
            for inputs, labels in data.valid_dl:

                if(train_on_gpu):
                    inp, labels = inputs.cuda(),labels.cuda()

                output, val_h = net(inp)
                val_loss = criterion(output, labels)

                val_losses.append(val_loss.item())

            net.train()
            print("Epoch: {}/{}...".format(e+1, epochs),
                  "Step: {}...".format(counter),
                  "Loss: {:.6f}...".format(loss.item()),
                  "Val Loss: {:.6f}".format(np.mean(val_losses)))
</code></pre>
<p>output</p>
<pre><code class="lang-auto">Epoch: 2/20... Step: 100... Loss: 0.450000... Val Loss: 0.380000
Epoch: 3/20... Step: 200... Loss: 0.150000... Val Loss: 0.380000
Epoch: 4/20... Step: 300... Loss: 0.400000... Val Loss: 0.380000
Epoch: 5/20... Step: 400... Loss: 0.350000... Val Loss: 0.380000
Epoch: 6/20... Step: 500... Loss: 0.550000... Val Loss: 0.380000
Epoch: 7/20... Step: 600... Loss: 0.450000... Val Loss: 0.380000
Epoch: 8/20... Step: 700... Loss: 0.350000... Val Loss: 0.380000
Epoch: 9/20... Step: 800... Loss: 0.400000... Val Loss: 0.380000
Epoch: 10/20... Step: 900... Loss: 0.400000... Val Loss: 0.380000
Epoch: 12/20... Step: 1000... Loss: 0.450000... Val Loss: 0.380000
Epoch: 13/20... Step: 1100... Loss: 0.400000... Val Loss: 0.380000
Epoch: 14/20... Step: 1200... Loss: 0.400000... Val Loss: 0.380000
Epoch: 15/20... Step: 1300... Loss: 0.350000... Val Loss: 0.380000
Epoch: 16/20... Step: 1400... Loss: 0.450000... Val Loss: 0.380000
Epoch: 17/20... Step: 1500... Loss: 0.400000... Val Loss: 0.380000
Epoch: 18/20... Step: 1600... Loss: 0.300000... Val Loss: 0.380000
Epoch: 19/20... Step: 1700... Loss: 0.450000... Val Loss: 0.380000
Epoch: 20/20... Step: 1800... Loss: 0.200000... Val Loss: 0.380000
</code></pre>
<p>thanks a lot</p>

username: steveoni
cooked: <p>i forgot to add the manhathan distance</p>
<pre><code class="lang-auto">def manhathan_dis(f,s):
  return torch.exp(-torch.sum(torch.abs(f-s), dim=1))
</code></pre>

username: ptrblck
cooked: <p>The training loss seems to go down, while the validation loss is constant.<br>
Could you check the model output during training and validation and compare the training and validation data distribution?</p>

username: steveoni
cooked: <p>ok i will do that now.</p>
<p>But did i get the implementation right?</p>
<p>and when u say distribution; You mean i should check the mean and the std if they are (near 0 and 1)?</p>

username: steveoni
cooked: <p>I’ve checked the output they are all zero</p>
<pre><code class="lang-auto">train output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
train output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
train output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
train output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
train output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
train output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
validation output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
validation output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
validation output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
validation output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
validation output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
validation output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
validation output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
validation output tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
</code></pre>

username: steveoni
cooked: <p>is it because for every <code>model.forward(x)</code> i’m always doing <code>init_hidden(batch_size)</code></p>
<pre><code class="lang-auto"> def forward(self,x,b):

    # x,b = x[0], x[1]
    batch_size = x.size(0)

    hidden1 = self.lstm1.init_hidden(batch_size)
    hidden2 = self.lstm2.init_hidden(batch_size)

    lstm1_h = self.lstm1(x, hidden1)
    lstm2_h = self.lstm2(b, hidden2)
</code></pre>

username: ptrblck
cooked: <p>If you are using the same workflow during training, I would assume it should be fine to use it during validation.</p>