Repository: OpenNMT/OpenNMT-py

Issue:
Fast translate - multiple end token bug
Hi @guillaumekln ,

Me again. Thank you very much for the fast response on the last issue! I ran my tests on the fast translate, there is still a small bug. Again the top-1 is perfectly fine, but the top-n predictions (n>1) can potentially be repeated outputs just with a different log prob.

Example: 
BEST HYP:
**[-14.9730] ['▁Herr', '▁Präsident', ',', '▁das', '▁Parlament', '▁hat', '▁seine', '▁Meinung', '▁geändert', ‘.’]**
[-15.6614] ['▁Herr', '▁Präsident', ',', '▁das', '▁Parlament', '▁hat', '▁seine', '▁Entscheidung', '▁getroffen', ‘.’]
[-16.3986] ['▁Herr', '▁Präsident', ',', '▁das', '▁Parlament', '▁hat', '▁seine', '▁Ansicht', 'en', '▁vertreten', '.']
**[-17.5395] ['▁Herr', '▁Präsident', ',', '▁das', '▁Parlament', '▁hat', '▁seine', '▁Meinung', '▁geändert', ‘.’]**
[-17.9110] ['▁Herr', '▁Präsident', ',', '▁das', '▁Parlament', '▁hat', '▁seine', '▁Entscheidung', '▁getroffen', '.']

Top-1 and top-4 are exactly the same prediction, but with a different probability (also top-2 and top-5). 

What happens is that, the `_fast_translate_batch` function keeps adding tokens to the finished hypotheses, e.g. (*3* as end token):

Top-1 [5, 5, 8, 5, 6, 10, 8, 7, 5, 6, 10, 8, 7, 5, 6, 10, 8, 7, 8, 5, 5, **3**], 
Top-4 [5, 5, 8, 5, 6, 10, 8, 7, 5, 6, 10, 8, 7, 5, 6, 10, 8, 7, 8, 5, 5, **3**, 5, **3**] 

In case a second end token is predicted and the sum of the log probs falls into the top-n, the twice finished sentences end up in the output. 

The translator, later on, converts it back to the original vocabulary until it finds the first end token, therefore, we get twice the same phrase with different probabilities in the output. 

So, either you keep track of the hypotheses that are done and you don’t add any new token to them or a quick fix would be to add a check for the number of end_token before adding the hypotheses (line 490, translator.py):

```
# Store finished hypotheses for this batch.
for j in finished_hyp:
    if (predictions[i, j, 1:] == end_token).sum() <= 1:  # force to have exactly 1 end token
        hypotheses[b].append((
            topk_scores[i, j],
            predictions[i, j, 1:],  # Ignore start_token.
            attention[:, i, j, :memory_lengths[i]]
            if attention is not None else None))
```

Thank you so much for all the work you do, OpenNMT is a great library!

Edit: just realised that my quick fix did not work, as well as I thought... what if the **max_length** is reached without predicting any end token? changed `(predictions[i, j, 1:] == end_token).sum() == 1` to `(predictions[i, j, 1:] == end_token).sum() <= 1`


Discussion:
Hi,

This is indeed a known behavior. I think the best fix is to penalize beams that reached `
`. This way they will no longer be advanced. I'm sending a patch.

Resolving patch:
diff --git a/onmt/translate/translator.py b/onmt/translate/translator.py
--- a/onmt/translate/translator.py
+++ b/onmt/translate/translator.py
@@ -446,13 +446,16 @@ def _fast_translate_batch(self,
                     alive_attn = alive_attn.index_select(1, select_indices)
                     alive_attn = torch.cat([alive_attn, current_attn], 0)
 
-            is_finished = topk_ids.to('cpu').eq(end_token)
+            is_finished = topk_ids.eq(end_token)
             if step + 1 == max_length:
                 is_finished.fill_(1)
-            top_beam_finished |= is_finished[:, 0].eq(1)
 
             # Save finished hypotheses.
             if is_finished.any():
+                # Penalize beams that finished.
+                topk_log_probs.masked_fill_(is_finished, -1e10)
+                is_finished = is_finished.to('cpu')
+                top_beam_finished |= is_finished[:, 0].eq(1)
                 predictions = alive_seq.view(-1, beam_size, alive_seq.size(-1))
                 attention = (
                     alive_attn.view(


Test patch: