Cannot use TransfoXLLMHeadModel with Trainer class because it returns a non scalar loss
Author: gstranger
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.5.1
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: Yes through Trainer class
- Using distributed or parallel set-up in script?: No

### Who can help  
@TevenLeScao

## Information

Model I am using: TransfoXLLMHeadModel

The problem arises when using:
- [ ] the official example scripts: (give details below)
- [x] my own modified scripts: (give details below)

The tasks I am working on is:
- [x] my own task or dataset: (give details below)

I am using a set of music data encoded as a language modeling problem. I have a Pytorch Dataset that returns a dictionary with the keys `input_ids`and `labels` from its `__getitem__` method which are 1D tensors that contain the example sequence to train on and predict. 

## To reproduce

Steps to reproduce the behavior:

1. Create a Pytorch dataset whose `__getitem__` method returns a dictionary with `input_ids` and `labels` with 1D Tensors
```python
class ExampleDataset(Dataset):
    def __getitem__(self, index):
        sample = self.encodings[index]
        return {'input_ids': torch.tensor(sample.ids), 'labels': torch.tensor(sample.ids), 'mems': None}

example_dataset = ExampleDataset()
example_dataset[0]
# { "input_ids": torch.tensor(0, 1,3 5, ... 330, 330), "labels": torch.tensor(0, 1, 3, 5, .. 330, 330) }
#  len: 512                                                 len: 512
# '330' is pad token
```
2. Instantiate the needed config and model
```python
from transformers import TransfoXLConfig, TransfoXLLMHeadModel
configuration = TransfoXLConfig(
    dropatt=0.1,
    vocab_size=len(tokenizer.get_vocab()), # Current size of vocab
    mem_len=512,                           # WordLevel tokenizers.Tokenizer
    d_inner=2048,                   
    n_layer=12,
    d_embed=512,
    n_head=8,
    d_head=64,
    cutoffs=[]
)
test_conf = TransfoXLConfig(vocab_size=len(tokenizer.get_vocab()))
model = TransfoXLLMHeadModel(configuration)

model.resize_token_embeddings(len(tokenizer.get_vocab()))
```
3. Instatiate TrainingArguments and Trainer, begin training
```python
train_args = TrainingArguments(
    overwrite_output_dir=True, # Change this to continue training, ie load from checkpoint
    output_dir = 'example-train',
    do_train = True,
    num_train_epochs=2,
    per_device_train_batch_size=1,
    )

trainer = Trainer(
    model=model,
    args=train_args,
    train_dataset=example_dataset,
    )

trainer.train()
```

```error
/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py:445: UserWarning: This overload of nonzero is deprecated:
	nonzero()
Consider using one of the following signatures instead:
	nonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)
  indices_i = mask_i.nonzero().squeeze()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-3435b262f1ae> in <module>
----> 1 trainer.train()

/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in train(self, model_path, trial)
    773                         tr_loss += self.training_step(model, inputs)
    774                 else:
--> 775                     tr_loss += self.training_step(model, inputs)
    776                 self._total_flos += self.floating_point_ops(inputs)
    777 

/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in training_step(self, model, inputs)
   1124                 scaled_loss.backward()
   1125         else:
-> 1126             loss.backward()
   1127 
   1128         return loss.detach()

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    183                 products. Defaults to ``False``.
    184         """
--> 185         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    186 
    187     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    119         grad_tensors = list(grad_tensors)
    120 
--> 121     grad_tensors = _make_grads(tensors, grad_tensors)
    122     if retain_graph is None:
    123         retain_graph = create_graph

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in _make_grads(outputs, grads)
     45             if out.requires_grad:
     46                 if out.numel() != 1:
---> 47                     raise RuntimeError("grad can be implicitly created only for scalar outputs")
     48                 new_grads.append(torch.ones_like(out, memory_format=torch.preserve_format))
     49             else:

RuntimeError: grad can be implicitly created only for scalar outputs
```

## Expected behavior

The model should be able to successfully complete training.


Comments:
According to the error here, this seems to be because the ouptut of the `TransfoXLLMHeadModel` is not a scalar output. Taking a look at this model's loss output, named `losses`, it is an array of size `[bsz, tgt_len - 1]`.

Maybe @TevenLeScao or @sgugger can chime in here at what the best procedure would be here, from a quick look the loss needs to be reduced. It seems this should be happening in the `Trainer` itself, but I'll let @sgugger decide.

`TransfoXLLMHeadModel` is not compatible with `Trainer` as it does not output a loss. The model should be fixed to output one loss and not the losses, like all the other ones (which would be a breaking change).

I see thank you for your replies. So to make this model compatible, I would need to create a custom `Trainer` class which overrides the `training_step` method and reduces the `losses` output to a scalar? How should I reduce the set? Would it be simpler to just train with a different causal language model from the library?

I think it would be easier to use another model, in all honesty.
If you really want this one, you can use a subclass of `Trainer` and override the `compute_loss` function. There is an example of this in the [documentation](https://huggingface.co/transformers/main_classes/trainer.html). I think taking the mean would be a proper reduction.

Thank you for your help. I've changed the title to better reflect the issue. You can close this ticket is you'd prefer this flagged a different way. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.