Repository: huggingface/transformers

Issue:
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.FloatTensor instead (while checking arguments for embedding)
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Linux-4.19.0-10-amd64-x86_64-with-debian-10.5
- Python version: 3.6.10
- PyTorch version (GPU?): 1.3.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger

 -->

## Information

Model I am using (GPT2-large) for fine-tuning on custom data:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

Trace:

>   File "gpt_language_generation.py", line 209, in <module>
>     main()
>   File "gpt_language_generation.py", line 136, in main
>     trainer.train(model_path=None)
>   File "<conda_env>/lib/python3.6/site-packages/transformers/trainer.py", line 708, in train
>     tr_loss += self.training_step(model, inputs)
>   File "<conda_env>/lib/python3.6/site-packages/transformers/trainer.py", line 995, in training_step
>     outputs = model(**inputs)
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "<conda_env>/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 731, in forward
>     return_dict=return_dict,
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "<conda_env>/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 593, in forward
>     inputs_embeds = self.wte(input_ids)
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
>     self.norm_type, self.scale_grad_by_freq, self.sparse)
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
>     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
model_class, tokenizer_class = GPT2LMHeadModel, GPT2Tokenizer
model = model_class.from_pretrained("gpt2-large")
tokenizer = tokenizer_class.from_pretrained("gpt2-large")
special_tokens_dict = {'bos_token': '<BOS>', 'eos_token': '<EOS>', 'pad_token': '<PAD>'}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))
input_text = ["a cat is sitting on the mat"]*100
train_dataset = tokenizer(input_text,add_special_tokens=True, truncation=True, max_length=64)
train_dataset = train_dataset["input_ids"]
eval_dataset = tokenizer(input_text,add_special_tokens=True, truncation=True, max_length=64)
eval_dataset = eval_dataset["input_ids"]
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer,mlm=False)
training_args = TrainingArguments(output_dir='./gpt_model/')
training_args.do_train = True
training_args.do_eval = True
training_args.per_device_train_batch_size = 32
trainer = Trainer(
           model=model,
            args=training_args,
            data_collator=data_collator,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            prediction_loss_only=True)

trainer.train(model_path=None) // The error occurs here
trainer.save_model()
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Expect the training to continue without an error.




Discussion:
@patrickvonplaten , I did find out that the problem arises from `data collator` (`DataCollatorForLanguageModeling`). The returned tensors (tensor of indices to vocab) are not Long, which  is creating the problem.

Resolving patch:
diff --git a/src/transformers/data/data_collator.py b/src/transformers/data/data_collator.py
--- a/src/transformers/data/data_collator.py
+++ b/src/transformers/data/data_collator.py
@@ -149,7 +149,7 @@ def _tensorize_batch(
     ) -> torch.Tensor:
         # In order to accept both lists of lists and lists of Tensors
         if isinstance(examples[0], (list, tuple)):
-            examples = [torch.Tensor(e) for e in examples]
+            examples = [torch.tensor(e, dtype=torch.long) for e in examples]
         length_of_first = examples[0].size(0)
         are_tensors_same_length = all(x.size(0) == length_of_first for x in examples)
         if are_tensors_same_length:


Test patch: