Repository: huggingface/transformers (Python)
Date: 2020-11-15 21:55:06+00:00
Task type: Bug Fix

Issue:
`disable_ngram_loss` doesn't work correctly in ProphetNetForConditionalGeneration
When I am using ProphetNet with `disable_ngram_loss=True` I am getting loss that is greater than with `disable_ngram_loss=False`. It seems to me that this is the problem of setting `fill_(self.padding_idx)` in `_compute_loss` instead of -100 so that ngram part is omitted in loss calculation 

Also I think that `loss = F.nll_loss(lprobs, expend_targets.view(-1), reduction="sum")` reduce should be set to `mean` so that model loss is comparable between models working on the same task (like `mbart`). Can somebody tell me if it's a good point or should I leave it as it is? I am planning to add PR with this changes.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.5.0 from source
- Platform: macOS Catalina
- Python version: 3.7.5
- PyTorch version (GPU?): 1.6.0

### Who can help

I can't figure out whom to tag.

## Information

Model I am using (Bert, XLNet ...): ProphetNetForConditionalGeneration

## To reproduce

```
from transformers import XLMProphetNetTokenizer, XLMProphetNetForConditionalGeneration
tokenizer = XLMProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased')
model = XLMProphetNetForConditionalGeneration.from_pretrained('microsoft/xprophetnet-large-wiki100-cased')
inputs = tokenizer('Hi my name is', return_tensors='pt').input_ids
targets = tokenizer('Hi my name is John', return_tensors='pt').input_ids

model_loss = model(input_ids=inputs, labels=targets, return_dict=True).loss

model.disable_ngram_loss = True
model_disable_loss = model(input_ids=inputs, labels=targets, return_dict=True).loss

from torch.nn import CrossEntropyLoss

loss_fct = CrossEntropyLoss(reduction='sum')
logits = model(input_ids=inputs, labels=targets, return_dict=True).logits
loss_cross_entropy = loss_fct(logits.view(-1, model.config.vocab_size), targets.view(-1))
```
the problem is `model_loss < model_disable_loss` and `model_disable_loss != loss_cross_entropy` which it should be I think.

Note: 
`CrossEntropyLoss(reduction='sum')` is used to match implementation in `_compute_loss` (`loss = F.nll_loss(lprobs, expend_targets.view(-1), reduction="sum")`) but other models use default reduction which makes outputs incomparable (at least directly)

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
when `model.disable_ngram_loss=True` `CrossEntropyLoss` should be equal to `model(input_ids=inputs, labels=targets, return_dict=True).loss`


Issue discussion:

Solution patch:
diff --git a/src/transformers/models/prophetnet/modeling_prophetnet.py b/src/transformers/models/prophetnet/modeling_prophetnet.py
--- a/src/transformers/models/prophetnet/modeling_prophetnet.py
+++ b/src/transformers/models/prophetnet/modeling_prophetnet.py
@@ -1793,8 +1793,8 @@ def forward(
                 encoder_attentions=outputs.encoder_attentions,
             )
 
-    def _compute_loss(self, logits, labels):
-        expend_targets = labels.new_zeros(self.config.ngram, labels.size(0), labels.size(1)).fill_(self.padding_idx)
+    def _compute_loss(self, logits, labels, ignore_index=-100):
+        expend_targets = labels.new_zeros(self.config.ngram, labels.size(0), labels.size(1)).fill_(ignore_index)
 
         for i in range(self.config.ngram):
             if i > 0 and self.disable_ngram_loss:
@@ -1807,13 +1807,13 @@ def _compute_loss(self, logits, labels):
             dtype=torch.float32,
         )
 
-        loss = F.nll_loss(lprobs, expend_targets.view(-1), reduction="sum")
+        loss = F.nll_loss(lprobs, expend_targets.view(-1), reduction="mean")
 
         if self.config.eps > 0.0:
             smooth_loss = -lprobs.sum(dim=-1, keepdim=True)
-            non_pad_mask = expend_targets.ne(self.padding_idx).view(-1)
-            smooth_loss = smooth_loss[non_pad_mask]
-            smooth_loss = smooth_loss.sum()
+            non_masked_tokens = expend_targets.ne(ignore_index).view(-1)
+            smooth_loss = smooth_loss[non_masked_tokens]
+            smooth_loss = smooth_loss.mean()
 
             eps_i = self.config.eps / lprobs.size(-1)
             loss = (1.0 - self.config.eps) * loss + eps_i * smooth_loss
@@ -2010,8 +2010,8 @@ def forward(
                 cross_attentions=outputs.cross_attentions,
             )
 
-    def _compute_loss(self, logits, labels):
-        expend_targets = labels.new_zeros(self.config.ngram, labels.size(0), labels.size(1)).fill_(self.padding_idx)
+    def _compute_loss(self, logits, labels, ignore_index=-100):
+        expend_targets = labels.new_zeros(self.config.ngram, labels.size(0), labels.size(1)).fill_(ignore_index)
 
         for i in range(self.config.ngram):
             if i > 0 and self.disable_ngram_loss:
@@ -2024,13 +2024,13 @@ def _compute_loss(self, logits, labels):
             dtype=torch.float32,
         )
 
-        loss = F.nll_loss(lprobs, expend_targets.view(-1), reduction="sum")
+        loss = F.nll_loss(lprobs, expend_targets.view(-1), reduction="mean")
 
         if self.config.eps > 0.0:
             smooth_loss = -lprobs.sum(dim=-1, keepdim=True)
-            non_pad_mask = expend_targets.ne(self.padding_idx).view(-1)
-            smooth_loss = smooth_loss[non_pad_mask]
-            smooth_loss = smooth_loss.sum()
+            non_masked_tokens = expend_targets.ne(ignore_index).view(-1)
+            smooth_loss = smooth_loss[non_masked_tokens]
+            smooth_loss = smooth_loss.mean()
 
             eps_i = self.config.eps / lprobs.size(-1)
             loss = (1.0 - self.config.eps) * loss + eps_i * smooth_loss


Test patch:
diff --git a/tests/test_modeling_prophetnet.py b/tests/test_modeling_prophetnet.py
--- a/tests/test_modeling_prophetnet.py
+++ b/tests/test_modeling_prophetnet.py
@@ -417,7 +417,7 @@ def check_fast_integration(
                 decoder_attention_mask=decoder_attention_mask,
                 labels=lm_labels,
             )
-        self.parent.assertTrue(torch.allclose(result.loss, torch.tensor(128.2925, device=torch_device), atol=1e-3))
+        self.parent.assertTrue(torch.allclose(result.loss, torch.tensor(4.5819, device=torch_device), atol=1e-3))
 
         expected_logit_slice = torch.tensor(
             [-0.1565, 0.0418, 0.1207, 0.0030, 0.0665, 0.0467, 0.0412], device=torch_device