Repository: OpenNMT/OpenNMT-py

Issue:
Ensemble translate does not support copy_attn
Hi,
I tried to average two checkpoints but got the following error:

> Traceback (most recent call last):
  File "../OpenNMT-py/translate.py", line 36, in <module>
    main(opt)
  File "../OpenNMT-py/translate.py", line 24, in main
    attn_debug=opt.attn_debug)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 222, in translate
    batch_data = self.translate_batch(batch, data, fast=self.fast)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 326, in translate_batch
    return self._translate_batch(batch, data)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 619, in _translate_batch
    src_map)
TypeError: forward() takes 2 positional arguments but 4 were given

The command option is -model models/baseline_step_140000.pt models/baseline_step_160000.pt 
How to solve this problem?


Discussion:
Not sure what you're trying to accomplish.
Ensemble.py: will ensemble 2 checkpoints from 2 runs (or N)
averaging (seems what you're trying to do) will average checkpoints in time of a run.
But you need to average before translationg.
you trace seems to show you're running a translation directly. won't work.
@vince62s 
Sorry. I just want to use multiple checkpoints for translation by following this  [FAQ ](http://opennmt.net/OpenNMT-py/FAQ.html#how-can-i-ensemble-models-at-inference). 
Is this the correct way to average checkpoints at inference?
Give your full command line
Here is the full command line:
```
python ../OpenNMT-py/translate.py \
-batch_size 20 \
-beam_size 5 \
-model models/cnndm_step_140000.pt models/cnndm_step_160000.pt \
-src data/cnndm/val.txt.src \
-output testout/cnndm.out \
-min_length 9 \
-max_length 25 \
-stepwise_penalty \
-coverage_penalty summary \
-beta 5 \
-length_penalty wu \
-alpha 0.9 \
-block_ngram_repeat 3 \
-ignore_when_blocking "." "</t>" "<t>" \
-replace_unk
``` 
okay, is the same command line with only one model working fine ?
> okay, is the same command line with only one model working fine ?

yes
@Waino was your code supposed to support copy_attn ?
can you please pull and run it again, I don't understand the error.
https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/translate/translator.py#L654-L656
it takes 3 arguments so I don't get why the error says 2 vs 4
thanks.
did you modify the code ?
The error is still the same  at inference after I run the git pull command. You can try the example of text summarization in the document if possible, in  case I missed something.
please repost the error trace lines have changed.
> Traceback (most recent call last):
  File "../OpenNMT-py/translate.py", line 36, in <module>
    main(opt)
  File "../OpenNMT-py/translate.py", line 24, in main
    attn_debug=opt.attn_debug)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 223, in translate
    batch_data = self.translate_batch(batch, data, fast=self.fast)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 327, in translate_batch
    return self._translate_batch(batch, data)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 656, in _translate_batch
    src_map)
TypeError: forward() takes 2 positional arguments but 4 were given

yes that can't work as is.
if you can try the following patch, in here:
https://github.com/OpenNMT/OpenNMT-py/blob/49185121e46c4f65d68101590bad231a8dd73e4f/onmt/decoders/ensemble.py#L92-L101

def forward(self, hidden, attn, src_map):
        """
        Compute a distribution over the target dictionary
        by averaging distributions from models in the ensemble.
        All models in the ensemble must share a target vocabulary.
        """
        distributions = [model_generator(hidden[i], attn[i], src_map[i])
                         for i, model_generator
                         in enumerate(self.model_generators)]
        return torch.stack(distributions).mean(0)

let me know if it works
but it won't in the non copy mode
It raised another error:

> Traceback (most recent call last):
  File "../OpenNMT-py/translate.py", line 36, in <module>
    main(opt)
  File "../OpenNMT-py/translate.py", line 24, in main
    attn_debug=opt.attn_debug)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 223, in translate
    batch_data = self.translate_batch(batch, data, fast=self.fast)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 327, in translate_batch
    return self._translate_batch(batch, data)
  File "/data/comp/OpenNMT-py/onmt/translate/translator.py", line 656, in _translate_batch
    src_map)
  File "/data/comp/OpenNMT-py/onmt/decoders/ensemble.py", line 98, in forward
    distributions = [model_generator(hidden[i], attn[i], src_map[i]) for i, model_generator in enumerate(self.model_generators)]
  File "/data/comp/OpenNMT-py/onmt/decoders/ensemble.py", line 98, in <listcomp>
    distributions = [model_generator(hidden[i], attn[i], src_map[i]) for i, model_generator in enumerate(self.model_generators)]
  File "/home/server3/.conda/envs/master/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/comp/OpenNMT-py/onmt/modules/copy_generator.py", line 86, in forward
    batch_by_tlen_, slen = attn.size()
ValueError: not enough values to unpack (expected 2, got 1)


can you make sure your pull was after this fix:
https://github.com/OpenNMT/OpenNMT-py/commit/513c06899d9cbc95c827083ba8e2b2ed52461f42
> can you make sure your pull was after this fix:
> [513c068](https://github.com/OpenNMT/OpenNMT-py/commit/513c06899d9cbc95c827083ba8e2b2ed52461f42)

Yes. I just pulled the latest version yesterday. Manual inspection also confirms it. And I think the above ValueError is not related to this file.
I will have to dig then.
what model did you train ? transformer or RNN ?
The model is a pointer-generator(i.e., RNN) and the commands are described in [the example](http://opennmt.net/OpenNMT-py/Summarization.html). 

in my patch, change 
distributions = [model_generator(hidden[i], attn[i], src_map[i])
to
distributions = [model_generator(hidden[i], attn, src_map)

it should work.

It really works.
Thank you for your time and effort in fixing this issue!

Resolving patch:
diff --git a/onmt/decoders/ensemble.py b/onmt/decoders/ensemble.py
--- a/onmt/decoders/ensemble.py
+++ b/onmt/decoders/ensemble.py
@@ -89,15 +89,21 @@ def __init__(self, model_generators):
         self.model_generators = tuple(model_generators)
         super(EnsembleGenerator, self).__init__()
 
-    def forward(self, hidden):
+    def forward(self, hidden, attn=None, src_map=None):
         """
         Compute a distribution over the target dictionary
         by averaging distributions from models in the ensemble.
         All models in the ensemble must share a target vocabulary.
         """
-        distributions = [model_generator(hidden[i])
-                         for i, model_generator
-                         in enumerate(self.model_generators)]
+        if attn:
+            distributions = [model_generator(hidden[i], attn, src_map)
+                             for i, model_generator
+                             in enumerate(self.model_generators)]
+        else:
+            distributions = [model_generator(hidden[i])
+                             for i, model_generator
+                             in enumerate(self.model_generators)]
+
         return torch.stack(distributions).mean(0)
 
 
diff --git a/onmt/translate/translator.py b/onmt/translate/translator.py
--- a/onmt/translate/translator.py
+++ b/onmt/translate/translator.py
@@ -387,7 +387,13 @@ def _fast_translate_batch(self,
         # Tile states and memory beam_size times.
         self.model.decoder.map_state(
             lambda state, dim: tile(state, beam_size, dim=dim))
-        memory_bank = tile(memory_bank, beam_size, dim=1)
+        if isinstance(memory_bank, tuple):
+            memory_bank = tuple(tile(x, beam_size, dim=1) for x in memory_bank)
+            mb_device = memory_bank[0].device
+        else:
+            memory_bank = tile(memory_bank, beam_size, dim=1)
+            mb_device = memory_bank.device
+
         memory_lengths = tile(src_lengths, beam_size)
         src_map = (tile(batch.src_map, beam_size, dim=1)
                    if data.data_type == 'text' and self.copy_attn else None)
@@ -399,18 +405,18 @@ def _fast_translate_batch(self,
             batch_size * beam_size,
             step=beam_size,
             dtype=torch.long,
-            device=memory_bank.device)
+            device=mb_device)
         alive_seq = torch.full(
             [batch_size * beam_size, 1],
             start_token,
             dtype=torch.long,
-            device=memory_bank.device)
+            device=mb_device)
         alive_attn = None
 
         # Give full probability to the first beam on the first step.
         topk_log_probs = (
             torch.tensor([0.0] + [float("-inf")] * (beam_size - 1),
-                         device=memory_bank.device).repeat(batch_size))
+                         device=mb_device).repeat(batch_size))
 
         # Structure that holds finished hypotheses.
         hypotheses = [[] for _ in range(batch_size)]  # noqa: F812
@@ -548,7 +554,12 @@ def _fast_translate_batch(self,
                               -1, alive_attn.size(-1))
 
             # Reorder states.
-            memory_bank = memory_bank.index_select(1, select_indices)
+            if isinstance(memory_bank, tuple):
+                memory_bank = tuple(x.index_select(1, select_indices)
+                                    for x in memory_bank)
+            else:
+                memory_bank = memory_bank.index_select(1, select_indices)
+
             memory_lengths = memory_lengths.index_select(0, select_indices)
             self.model.decoder.map_state(
                 lambda state, dim: state.index_select(dim, select_indices))


Test patch: