Train Transformer XL language modeling with padding 
Author: StefanHeng
Pull request: no
Labels: name: bug

### System Info

```shell
`transformers` version `4.19.2`, `python` version `3.9.12`
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

I'm using `DataCollatorForLanguageModeling` **and padding** to train Transformer XL with LM head from scratch. 

I'm not sure if the error is intended, i.e. do HuggingFace support training with padding tokens? 

I'm not sure because I had an error with padding, but the forward pass for LM head seems to consider padding? 

```python
        if labels is not None:
            losses = softmax_output.view(bsz, tgt_len - 1)
            # Avoids from incorporating padding (-100) tokens into loss value
            loss = losses[losses != 0].mean()
```

I don't find much support for this. I checked [this issue](https://github.com/huggingface/transformers/issues/586). 

And I don't think it's relevant in the notebook [language_modeling_from_scratch](https://github.com/huggingface/notebooks/blob/main/examples/language_modeling_from_scratch.ipynb) since all texts are concatenated. 


Also, I'm using `n_clusters` == 0. 

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


```
Traceback (most recent call last):
  File "/Users/stefanhg/Documents/UMich/Research/Music with NLP/Symbolic-Music-Generation/musicnlp/trainer/train.py", line 407, in <module>
    train_xl()
  File "/Users/stefanhg/Documents/UMich/Research/Music with NLP/Symbolic-Music-Generation/musicnlp/trainer/train.py", line 406, in train_xl
    trainer.train(ignore_keys_for_eval=ignore_keys_for_eval)
  File "/Users/stefanhg/opt/anaconda3/envs/music-nlp/lib/python3.9/site-packages/transformers/trainer.py", line 1317, in train
    return inner_training_loop(
  File "/Users/stefanhg/opt/anaconda3/envs/music-nlp/lib/python3.9/site-packages/transformers/trainer.py", line 1554, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/Users/stefanhg/opt/anaconda3/envs/music-nlp/lib/python3.9/site-packages/transformers/trainer.py", line 2183, in training_step
    loss = self.compute_loss(model, inputs)
  File "/Users/stefanhg/Documents/UMich/Research/Music with NLP/Symbolic-Music-Generation/musicnlp/util/train/train_util_wrap.py", line 81, in compute_loss
    outputs = model(**inputs)
  File "/Users/stefanhg/opt/anaconda3/envs/music-nlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/stefanhg/Documents/UMich/Research/Music with NLP/Symbolic-Music-Generation/musicnlp/models/transformer_xl.py", line 189, in forward
    softmax_output = self.crit(pred_hid, labels)
  File "/Users/stefanhg/opt/anaconda3/envs/music-nlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/stefanhg/opt/anaconda3/envs/music-nlp/lib/python3.9/site-packages/transformers/models/transfo_xl/modeling_transfo_xl_utilities.py", line 112, in forward
    out = -nn.functional.log_softmax(logit, dim=-1).gather(1, labels.unsqueeze(1)).squeeze(1)
RuntimeError: index -100 is out of bounds for dimension 1 with size 462
```
Here's my stack trace. 

### Expected behavior

```shell
I hope the `ProjectedAdaptiveLogSoftmax` implementation can consider support ignoring padding tokens.
```

Looks like just filtering out the labels in adaptive softmax fixes it (and for this I need to ignore reshaping the `losses` in LM head forward
```python
        if self.n_clusters == 0:
            logit = self._compute_logit(hidden, self.out_layers[0].weight, self.out_layers[0].bias, self.out_projs[0])
            if labels is not None:
                # ========================== Begin of modified ==========================
                out = -nn.functional.log_softmax(logit, dim=-1).gather(1, labels[labels != -100].unsqueeze(1)).squeeze(1)
                # ========================== End of modified ==========================
            else:
                out = nn.functional.log_softmax(logit, dim=-1)
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.