Repository: huggingface/transformers

Issue:
BART cannot accept -100 as ignored label
## Environment info

- `transformers` version: 4.0.1
- Platform: Linux
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

 Bart: @patrickvonplaten

## Information

I'm using ``BartForConditionalGeneration`` to do some natural language generation tasks. By the [doc](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration) I should be able to set -100 for some tokens to ignore. However, it would raise an out of index error.

## To reproduce

```python
from transformers import BartForConditionalGeneration, AutoTokenizer

b = BartForConditionalGeneration.from_pretrained("facebook/bart-base")
t = AutoTokenizer.from_pretrained("facebook/bart-base")

s1 = "hello hello hello hello world"
inputs = t(s1, return_tensors="pt")
label = inputs["input_ids"].clone()
label[0, 2:3] = -100

outputs = b(**inputs, labels=label)
```

Then it raise the following error:

> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 1032, in forward
>     return_dict=return_dict,
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 915, in forward
>     return_dict=return_dict,
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 564, in forward
>     x = self.embed_tokens(input_ids) * self.embed_scale
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
>     self.norm_type, self.scale_grad_by_freq, self.sparse)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
>     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
> IndexError: index out of range in self

Without giving -100 in the label, it can return the output correctly.

## Expected behavior

Should return the output correctly.


Discussion:

Resolving patch:
diff --git a/src/transformers/models/bart/modeling_bart.py b/src/transformers/models/bart/modeling_bart.py
--- a/src/transformers/models/bart/modeling_bart.py
+++ b/src/transformers/models/bart/modeling_bart.py
@@ -70,6 +70,11 @@ def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int):
     index_of_eos = (input_ids.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1)
     prev_output_tokens[:, 0] = input_ids.gather(1, index_of_eos).squeeze()
     prev_output_tokens[:, 1:] = input_ids[:, :-1]
+
+    assert pad_token_id is not None, "self.model.config.pad_token_id has to be defined."
+    # replace possible -100 values in labels by `pad_token_id`
+    prev_output_tokens.masked_fill_(prev_output_tokens == -100, pad_token_id)
+
     return prev_output_tokens
 
 


Test patch: