Repository: OpenNMT/OpenNMT-py

Issue:
small bug with truncated decoding and tgt feature concat.
I'm running https://github.com/Henry-E/opennmt-gen and getting an error with matrix multiplication.
` File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/blas.py", line 26, in forward
    matrix1, matrix2, out=output)` 
The full stack trace is below

The problem is in a multiplication inside the decoder RNN. It's trying to multiply matrix 1 by matrix 2. At a guess it's something to do with the extra dimensions of the features not being accounted for by the decoder. I'm not super familiar with the equations being used for the features though so it's hard to tell where the model is being initialised incorrectly. 
```
ipdb> matrix1.size()
torch.Size([5, 2400])
ipdb> matrix2.size()
torch.Size([600, 2400])
```

The model
```
NMTModel (
  (encoder): MeanEncoder (
    (embeddings): Embeddings (
      (make_embedding): Sequential (
        (emb_luts): Elementwise (
          (0): Embedding(10818, 600, padding_idx=1)
          (1): Embedding(704, 600, padding_idx=1)
          (2): Embedding(40, 600, padding_idx=1)
          (3): Embedding(4, 600, padding_idx=1)
        )
      )
    )
  )
  (decoder): InputFeedRNNDecoder (
    (embeddings): Embeddings (
      (make_embedding): Sequential (
        (emb_luts): Elementwise (
          (0): Embedding(10818, 600, padding_idx=1)
        )
      )
    )
    (dropout): Dropout (p = 0.3)
    (rnn): StackedLSTM (
      (dropout): Dropout (p = 0.3)
      (layers): ModuleList (
        (0): LSTMCell(1200, 600)
        (1): LSTMCell(600, 600)
      )
    )
    (attn): GlobalAttention (
      (linear_out): Linear (1200 -> 600)
      (sm): Softmax ()
      (tanh): Tanh ()
    )
    (copy_attn): GlobalAttention (
      (linear_out): Linear (1200 -> 600)
      (sm): Softmax ()
      (tanh): Tanh ()
    )
  )
  (generator): CopyGenerator (
    (linear): Linear (600 -> 10818)
    (linear_copy): Linear (600 -> 1)
  )
)
```



Stack trace

```
Traceback (most recent call last):
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/ipdb/__main__.py", line 198, in main
    pdb._runscript(mainpyfile)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/pdb.py", line 1548, in _runscript
    self.run(statement)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 1, in <module>
    from __future__ import division
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 284, in main
    train_model(model, train, valid, fields, optim)
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 144, in train_model
    train_stats = trainer.train(epoch, report_func)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Trainer.py", line 119, in train
    self.model(src, tgt, src_lengths, dec_state)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Models.py", line 421, in forward
    else dec_state)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Models.py", line 175, in forward
    self._run_forward_pass(input, context, state)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Models.py", line 337, in _run_forward_pass
    rnn_output, hidden = self.rnn(emb_t, hidden)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/henrye/downloads/OpenNMT-py/onmt/modules/StackedRNN.py", line 24, in forward
    h_1_i, c_1_i = layer(input, (h_0[i], c_0[i]))
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 608, in forward
    self.bias_ih, self.bias_hh,
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 26, in LSTMCell
    hgates = F.linear(hidden[0], w_hh)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py", line 555, in linear
    output = input.matmul(weight.t())
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py", line 560, in matmul
    return torch.matmul(self, other)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/functional.py", line 173, in matmul
    return torch.mm(tensor1, tensor2)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py", line 579, in mm
    return Addmm.apply(output, self, matrix, 0, 1, True)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/blas.py", line 26, in forward
    matrix1, matrix2, out=output)
RuntimeError: size mismatch at /opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generic/THCTensorMathBlas.cu:243
```


Discussion:
Looking into this. 
Can you try with `-feat_merge mlp`
Using `-feat_merge mlp` it appears to be able to run through the decoder and encoder steps ok. It runs into another error afterwards but I can look into that separately. 
oh this must be @bpopeters tgt_side feature code breaking. Can you print `tgt_size` after then line
`https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/Trainer.py#L101`

and then print 

`tgt.size()`  at line 111? 


@bpopeters or @JianyuZhan I think the other issue above is also a tgt_side feature bug. It seems like the embedding size on the tgt side is getting set based on src_feature sizes. 
Still using `-feat_merge mlp` it returns
```
ipdb> target_size
394
```
and
```
ipdb> tgt.size()
torch.Size([394, 5, 1])
```
The new error that pops up is in the second iteration of the loop in the training function.
```
ipdb> tgt.size()
torch.Size([100, 5, 1])
ipdb> p j 
100
```

```
Traceback (most recent call last):
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/ipdb/__main__.py", line 198, in main
    pdb._runscript(mainpyfile)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/pdb.py", line 1548, in _runscript
    self.run(statement)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 289, in <module>
    main()
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 285, in main
    train_model(model, train, valid, fields, optim)
  File "/home/henrye/downloads/OpenNMT-py/train.py", line 144, in train_model
    train_stats = trainer.train(epoch, report_func)
  File "/home/henrye/downloads/OpenNMT-py/onmt/Trainer.py", line 114, in train
    tgt = tgt[j: j + trunc_size]
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py", line 76, in __getitem__
    return Index.apply(self, key)
  File "/home/henrye/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/tensor.py", line 16, in forward
    result = i.index(ctx.index)
ValueError: result of slicing is an empty tensor
```



Oh I see the issue. The variable names overlap. Can change the name of the outer variable? just try calling it tgt_outer and then tgt = tgt_outer[j:j+trunc_size]

Sorry on my phone. Otherwise would do.
Yep that fixes it. Seemingly no more errors popping up. Using concatenation vs a multilayer perceptron for the features is more or less pretty similar right?
yeah, I think `mlp` is actually what you want, but we changed the default. 

Can you send a PR to fix the trunc issue? That is a bug. 
Also `concat` should work as well, but I think it is somehow being applied to tgt side as well. 
I'll have a look at this in a couple hours as well.

On Tue, Sep 26, 2017 at 4:57 PM, srush <notifications@github.com> wrote:

> Reopened #302 <https://github.com/OpenNMT/OpenNMT-py/issues/302>.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/OpenNMT/OpenNMT-py/issues/302#event-1265930300>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AJvP7wPRi6EqRb_r2yY337Fox588TqPQks5smRDugaJpZM4Pj_u3>
> .
>


Resolving patch:
diff --git a/onmt/Trainer.py b/onmt/Trainer.py
--- a/onmt/Trainer.py
+++ b/onmt/Trainer.py
@@ -106,12 +106,12 @@ def train(self, epoch, report_func=None):
             _, src_lengths = batch.src
 
             src = onmt.IO.make_features(batch, 'src')
-            tgt = onmt.IO.make_features(batch, 'tgt')
+            tgt_outer = onmt.IO.make_features(batch, 'tgt')
             report_stats.n_src_words += src_lengths.sum()
 
             for j in range(0, target_size-1, trunc_size):
                 # 1. Create truncated target.
-                tgt = tgt[j: j + trunc_size]
+                tgt = tgt_outer[j: j + trunc_size]
 
                 # 2. F-prop all but generator.
                 self.model.zero_grad()


Test patch: