Repository: huggingface/transformers

Issue:
`DataCollatorForLanguageModeling` modifies `input_ids` via `labels` variable
The cloning step was removed in https://github.com/huggingface/transformers/pull/8308 at https://github.com/huggingface/transformers/pull/8308/files#diff-046566f2b40a246c7d533457cd7f6f07830516da845b904086f36b3cfe0d5965L201 so now the code that sets padded labels to `-100` is operating on the `input_ids` tensor directly.

I suspect the code then fails when trying to look up the embedding for `-100` .

cc @sgugger 

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.5.1
- Platform: Linux-5.4.72-x86_64-with
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Use `DataCollatorForLanguageModeling` with `Trainer`  and a tokenizer with `pad_token`


```
  File "/home/lulu/r/buganart/dialog/.build/pip_packages/bin/finetune", line 33, in <module>
    sys.exit(load_entry_point('dialog', 'console_scripts', 'finetune')())
  File "/home/lulu/r/buganart/dialog/dialog/finetune.py", line 139, in main
    trainer.train()
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/transformers/trainer.py", line 775, in train
    tr_loss += self.training_step(model, inputs)
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/transformers/trainer.py", line 1112, in training_step
    loss = self.compute_loss(model, inputs)
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/transformers/trainer.py", line 1136, in compute_loss
    outputs = model(**inputs)
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/transformers/modeling_gpt2.py", line 774, in forward
    transformer_outputs = self.transformer(
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/transformers/modeling_gpt2.py", line 612, in forward
    inputs_embeds = self.wte(input_ids)
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 124, in forward
    return F.embedding(
  File "/nix/store/0jdyxgmg88y6sbjm3xkqdn06f493ahf2-python3-3.8.6-env/lib/python3.8/site-packages/torch/nn/functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self 
```
My script is here https://github.com/buganart/dialog/blob/master/dialog/finetune.py .


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->



Discussion:
Ah yes, only the detach was supposed to be removed but I guess I went a bit too far with my mouse, sorry about that. Will fix right now, thanks for flagging!

Resolving patch:
diff --git a/src/transformers/data/data_collator.py b/src/transformers/data/data_collator.py
--- a/src/transformers/data/data_collator.py
+++ b/src/transformers/data/data_collator.py
@@ -261,7 +261,7 @@ def __call__(
                 batch["input_ids"], special_tokens_mask=special_tokens_mask
             )
         else:
-            labels = batch["input_ids"]
+            labels = batch["input_ids"].clone()
             if self.tokenizer.pad_token_id is not None:
                 labels[labels == self.tokenizer.pad_token_id] = -100
             batch["labels"] = labels


Test patch: