Transformer XL div_val != 1 does not work with fp16
Author: StefanHeng
Pull request: no

### System Info

python version `3.10.4`, 

Package versions 
torch                   1.12.0+cu116
torchaudio              0.12.0+cu116
torchvision             0.13.0+cu116
transformers            4.22.2


### Who can help?

@patrickvonplaten
@thomwolf

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Here's a MWE to reproduce the bug. 

```python
        import torch
        from transformers import (
            TransfoXLConfig, TransfoXLTokenizer, TransfoXLLMHeadModel, Trainer, DataCollatorForLanguageModeling
        )
        from transformers.training_args import TrainingArguments
        import datasets

        config = TransfoXLConfig.from_pretrained('transfo-xl-wt103')
        config.d_model = 128
        config.n_head = 8
        config.n_layer = 4

        tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
        tokenizer.model_max_length = 16
        tokenizer.add_special_tokens(dict(pad_token='[PAD]'))

        model = TransfoXLLMHeadModel(config)
        dataset = datasets.Dataset.from_dict(dict(text=['Hello world', 'XL blah']))
        # mic(dataset)
        dataset = dataset.map(lambda x: tokenizer(x['text'], return_tensors='pt', padding='max_length'), batched=True)

        train_args = TrainingArguments(
            output_dir='./debug',
            fp16=torch.cuda.is_available(),
            num_train_epochs=100,
            per_device_train_batch_size=2
        )
        # mic(train_args)
        trainer = Trainer(
            model=model, train_dataset=dataset, args=train_args,
            data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
        )
        trainer.train()
```

Here's the stack trace I got 
```bash
Traceback (most recent call last):
  File "/home/stefanhg/Music-with-NLP/Symbolic-Music-Generation/test-lang.py", line 992, in <module>
    check_xl_fp16()
  File "/home/stefanhg/Music-with-NLP/Symbolic-Music-Generation/test-lang.py", line 991, in check_xl_fp16
    trainer.train()
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/transformers/trainer.py", line 1521, in train
    return inner_training_loop(
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/transformers/trainer.py", line 1763, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/transformers/trainer.py", line 2499, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/transformers/trainer.py", line 2531, in compute_loss
    outputs = model(**inputs)
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/transformers/models/transfo_xl/modeling_transfo_xl.py", line 1094, in forward
    transformer_outputs = self.transformer(
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/transformers/models/transfo_xl/modeling_transfo_xl.py", line 929, in forward
    word_emb = self.word_emb(input_ids)
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/stefanhg/miniconda3/envs/music-nlp/lib/python3.10/site-packages/transformers/models/transfo_xl/modeling_transfo_xl.py", line 451, in forward
    emb_flat.index_copy_(0, indices_i, emb_i)
RuntimeError: index_copy_(): self and source expected to have the same dtype, but got (self) Float and (source) Half
```

### Expected behavior

In short, something wrong with adaptive softmax, I assume type cast for fp16 not working properly

Comments:
According to git blame, @thomwolf added Transformer Xl. Can you help? 

I don't think TransformerXL supports FP16 as this is an old model with very specific code for the softmax layer. This won't be an issue we will fix ourselves given that Transformer-XL is not very used anymore, but if someone wants to make a PR, we'll review!

I see. I will think about make a PR. Thank you!