Repository: d2l-ai/d2l-en

Issue:
PT implementation of Transformer has very bad translation results
http://preview.d2l.ai.s3-website-us-west-2.amazonaws.com/d2l-en/master/chapter_attention-mechanisms/transformer.html

PyTorch:

go . => <unk> !, bleu 0.000
i lost . => je l'ai vu ., bleu 0.000
so long . => <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk>, bleu 0.000
i'm home . => <unk> à la maison ., bleu 0.000
he's calm . => sois <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk>, bleu 0.000

MXNet:

go . => va !, bleu 1.000
i lost . => j'ai perdu ., bleu 1.000
so long . => <unk> !, bleu 0.000
i'm home . => je suis chez <unk> chez vous en <unk> <unk> <unk>, bleu 0.376
he's calm . => il <unk> calme ., bleu 0.658

Though the translation looks good in other sections:
http://preview.d2l.ai.s3-website-us-west-2.amazonaws.com/d2l-en/master/chapter_recurrent-modern/seq2seq.html
http://preview.d2l.ai.s3-website-us-west-2.amazonaws.com/d2l-en/master/chapter_attention-mechanisms/seq2seq-attention.html


Discussion:
Hmm, I can't see the root cause here.
If you check the stable version that seems to be doing very well. http://d2l.ai/chapter_attention-mechanisms/transformer.html#training

Could it be just this run or the change of hyperparams?  I'll try to investigate more.
In the stable version, both mx & pt perform poorly because there are issues in the initial implementations. Now they're fixed, but the pt version still didn't perform as well as the mx version:

http://preview.d2l.ai.s3-website-us-west-2.amazonaws.com/d2l-en/master/chapter_attention-mechanisms/transformer.html

MXNet:
go . => va !, bleu 1.000
i lost . => j'ai perdu ., bleu 1.000
i'm home . => je suis chez moi ., bleu 1.000
he's calm . => il est malade ., bleu 0.658

PyTorch:
go . => c'est le <unk> !, bleu 0.000
i lost . => je suis paresseuse ., bleu 0.000
i'm home . => je suis chez moi ., bleu 1.000
he's calm . => je suis calme ., bleu 0.537
One hint is that 
mx: loss 0.030
pt: loss 0.016

These are quite different
I checked again and PyTorch is actually performing better than MX in this case if you look closely bleu score for the 1st example is perfect in PT unlike MX.

MX
```
go . => va le chercher !, bleu 0.000
i lost . => j'ai perdu ., bleu 1.000
i'm home . => je suis chez moi ., bleu 1.000
he's calm . => il est bon ., bleu 0.658
```

PT
```
go . => va !, bleu 1.000
i lost . => j'ai perdu ., bleu 1.000
i'm home . => je suis <unk> ., bleu 0.512
he's calm . => il est mouillé ., bleu 0.658
```

The loss difference is not a lot if you are hinting towards overfitting in PT.
Thanks. Could you run experiments for multiple runs to see if mx and pt results are comparable?
Run 1: `loss 0.014, 5796.9 tokens/sec on cpu`
```
go . => entrez !, bleu 0.000
i lost . => j'ai perdu ., bleu 1.000
i'm home . => j'en suis chez moi ., bleu 0.832
he's calm . => il est calme ., bleu 1.000
```

Run 2: `loss 0.014, 5826.1 tokens/sec on cpu`
```
go . => <unk> !, bleu 0.000
i lost . => je suis perdu ., bleu 0.537
i'm home . => je suis chez moi ., bleu 1.000
he's calm . => je suis calme ., bleu 0.537
```

Run 3: `loss 0.016, 5568.3 tokens/sec on cpu`
```
go . => <unk> !, bleu 0.000
i lost . => je sais perdu ., bleu 0.537
i'm home . => je suis chez moi ., bleu 1.000
he's calm . => il est paresseux ., bleu 0.658
```

Run 4: `loss 0.018, 6071.4 tokens/sec on cpu`
```
go . => vas-y !, bleu 0.000
i lost . => je <unk> ., bleu 0.000
i'm home . => je suis partie ., bleu 0.512
he's calm . => <unk> ., bleu 0.000
```

Run 5: `loss 0.017, 4642.7 tokens/sec on cpu`
```
go . => va !, bleu 1.000
i lost . => j'ai perdu ., bleu 1.000
i'm home . => je suis chez <unk> ., bleu 0.752
he's calm . => je suis calme !, bleu 0.000
```

I guess there is indeed some issue with PT tranformer chapter since seq2seq has comparable results to mx but the transformer results are not very consistent. Thanks for flagging this @astonzhang , I'll try to find and fix the issue. @ypandya Would be nice if you can also take a look. Thanks!
Just one more example of inconsistent performance

![Screen Shot 2020-10-22 at 12 59 49 PM](https://user-images.githubusercontent.com/22279212/96923708-85445600-1466-11eb-8751-1c456b3bab99.png)
![Screen Shot 2020-10-22 at 12 59 43 PM](https://user-images.githubusercontent.com/22279212/96923714-86758300-1466-11eb-8e8c-61b4a91b6d89.png)

First, *torch.repeat* should be used instead of *torch.repeat_interleave*. Someone pointed this out in the discussion of https://d2l.ai/chapter_attention-mechanisms/transformer.html
Second, pytorch version predicts differently with the same input. I think it's wired. Mxnet version doesn't have this issue.

Resolving patch:
diff --git a/d2l/mxnet.py b/d2l/mxnet.py
--- a/d2l/mxnet.py
+++ b/d2l/mxnet.py
@@ -991,7 +991,7 @@ def translate(engs, fras, model, src_vocab, tgt_vocab, num_steps, device):
             f'{eng} => {translation}, bleu {bleu(translation, fra, k=2):.3f}')
 
 
-# Defined in file: ./chapter_attention-mechanisms/attention.md
+# Defined in file: ./chapter_attention-mechanisms/attention-functions.md
 def masked_softmax(X, valid_len):
     """Perform softmax by filtering out some elements."""
     # X: 3-D tensor, valid_len: 1-D or 2-D tensor
@@ -1009,7 +1009,7 @@ def masked_softmax(X, valid_len):
         return npx.softmax(X).reshape(shape)
 
 
-# Defined in file: ./chapter_attention-mechanisms/attention.md
+# Defined in file: ./chapter_attention-mechanisms/attention-functions.md
 class DotProductAttention(nn.Block):
     def __init__(self, dropout, **kwargs):
         super(DotProductAttention, self).__init__(**kwargs)
@@ -1027,7 +1027,7 @@ def forward(self, query, key, value, valid_len=None):
         return npx.batch_dot(attention_weights, value)
 
 
-# Defined in file: ./chapter_attention-mechanisms/attention.md
+# Defined in file: ./chapter_attention-mechanisms/attention-functions.md
 class MLPAttention(nn.Block):
     def __init__(self, units, dropout, **kwargs):
         super(MLPAttention, self).__init__(**kwargs)
diff --git a/d2l/torch.py b/d2l/torch.py
--- a/d2l/torch.py
+++ b/d2l/torch.py
@@ -1022,6 +1022,8 @@ def xavier_init_weights(m):
 def predict_s2s_ch9(model, src_sentence, src_vocab, tgt_vocab, num_steps,
                     device):
     """Predict sequences (defined in Chapter 9)."""
+    # Set model to eval mode for inference
+    model.eval()
     src_tokens = src_vocab[src_sentence.lower().split(' ')] + [
         src_vocab['<eos>']]
     enc_valid_len = torch.tensor([len(src_tokens)], device=device)
@@ -1077,7 +1079,7 @@ def translate(engs, fras, model, src_vocab, tgt_vocab, num_steps, device):
             f'{eng} => {translation}, bleu {bleu(translation, fra, k=2):.3f}')
 
 
-# Defined in file: ./chapter_attention-mechanisms/attention.md
+# Defined in file: ./chapter_attention-mechanisms/attention-functions.md
 def masked_softmax(X, valid_len):
     """Perform softmax by filtering out some elements."""
     # X: 3-D tensor, valid_len: 1-D or 2-D tensor
@@ -1095,7 +1097,7 @@ def masked_softmax(X, valid_len):
         return nn.functional.softmax(X.reshape(shape), dim=-1)
 
 
-# Defined in file: ./chapter_attention-mechanisms/attention.md
+# Defined in file: ./chapter_attention-mechanisms/attention-functions.md
 class DotProductAttention(nn.Module):
     def __init__(self, dropout, **kwargs):
         super(DotProductAttention, self).__init__(**kwargs)
@@ -1113,7 +1115,7 @@ def forward(self, query, key, value, valid_len=None):
         return torch.bmm(attention_weights, value)
 
 
-# Defined in file: ./chapter_attention-mechanisms/attention.md
+# Defined in file: ./chapter_attention-mechanisms/attention-functions.md
 class MLPAttention(nn.Module):
     def __init__(self, key_size, query_size, units, dropout, **kwargs):
         super(MLPAttention, self).__init__(**kwargs)


Test patch: