Repository: facebookresearch/fairseq

Issue:
Off by one / out of bounds in sequence generation
[Seeing this with commit 7e86e30cc5b97db30403b738de2dbbb55f06a92b from Feb 12 – apologies if already fixed; I saw no obvious fix in commits since that, and I suspect my model would not load or work on HEAD.]

With for example interactive.py, and I think also with generate.py, I see this error for a fconv_wmt_en_ro model and very long input sentence with some beam sizes (essentially whenever the generated sentence does not properly terminate):

```
Traceback (most recent call last):
  File "interactive.py", line 78, in <module>
    main()
  File "interactive.py", line 60, in main
    translations = translator.generate(Variable(src_tokens.view(1, -1)))
  File "/home/sliedes/projects/fairseq-py/fairseq/sequence_generator.py", line 92, in generate
    return self._generate(src_tokens, beam_size, maxlen)
  File "/home/sliedes/projects/fairseq-py/fairseq/sequence_generator.py", line 227, in _generate
    probs, avg_attn_scores = self._decode(tokens[:, :step+1], encoder_outs)
  File "/home/sliedes/projects/fairseq-py/fairseq/sequence_generator.py", line 333, in _decode
    decoder_out, attn = model.decoder(tokens, encoder_out)
  File "/home/sliedes/torch-venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sliedes/projects/fairseq-py/fairseq/models/fconv.py", line 186, in forward
    positions = self.embed_positions(input_tokens)
  File "/home/sliedes/torch-venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sliedes/projects/fairseq-py/fairseq/modules/learned_positional_embedding.py", line 38, in forward
    return super().forward(positions)
  File "/home/sliedes/torch-venv/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 103, in forward
    self.scale_grad_by_freq, self.sparse
  File "/home/sliedes/torch-venv/lib/python3.6/site-packages/torch/nn/_functions/thnn/sparse.py", line 59, in forward
    output = torch.index_select(weight, 0, indices.view(-1))
RuntimeError: index out of range at /pytorch/torch/lib/TH/generic/THTensorMath.c:277
```

This always happens at the last iteration of this loop in SequenceGenerator._generate():

`for step in range(maxlen + 1):  # one extra step for EOS marker`

This patch fixes the problem for me; however I do not know if it is the right fix:

```
diff --git a/fairseq/sequence_generator.py b/fairseq/sequence_generator.py
index b9d5fd8..da08bdd 100644
--- a/fairseq/sequence_generator.py
+++ b/fairseq/sequence_generator.py
@@ -217,7 +217,7 @@ class SequenceGenerator(object):
             return num_finished
 
         reorder_state = None
-        for step in range(maxlen + 1):  # one extra step for EOS marker
+        for step in range(maxlen):  # one extra step for EOS marker
             # reorder decoder internal states based on the prev choice of beams
             if reorder_state is not None:
                 for model in self.models:
@@ -290,7 +290,7 @@ class SequenceGenerator(object):
 
             # finalize all active hypotheses once we hit maxlen
             # finalize_hypos will take care of adding the EOS markers
-            if step == maxlen:
+            if step == maxlen - 1:
                 num_remaining_sent -= finalize_hypos(step, active_bbsz_idx, active_scores)
                 assert num_remaining_sent == 0
                 break
```



Discussion:
Thanks for reporting. When using learned positional embeddings there's a maximum supported length, based on the number of embeddings that were learned (defaults to ~1000). This error is happening because it's trying to produce sequences longer than this max length, because of the extra EOS. Your fix is reasonable, although generation will still have a max length roughly equal to the number of learned positional embeddings.

> I suspect my model would not load or work on HEAD

We generally keep backwards-compatibility with old model checkpoints, although the recent changes to the incremental generation API (https://github.com/facebookresearch/fairseq-py/commit/9438019ff066116534c94e7f0a154aff6c51409e) might require some small code changes if you've added any new model types.

Resolving patch:
diff --git a/fairseq/sequence_generator.py b/fairseq/sequence_generator.py
--- a/fairseq/sequence_generator.py
+++ b/fairseq/sequence_generator.py
@@ -37,6 +37,7 @@ def __init__(self, models, beam_size=1, minlen=1, maxlen=None,
         self.beam_size = beam_size
         self.minlen = minlen
         max_decoder_len = min([m.max_decoder_positions() for m in self.models])
+        max_decoder_len -= 1  # we define maxlen not including the EOS marker
         self.maxlen = max_decoder_len if maxlen is None else min(maxlen, max_decoder_len)
         self.stop_early = stop_early
         self.normalize_scores = normalize_scores


Test patch: