Repository: huggingface/transformers

Issue:
run_language_modeling, RuntimeError: expected scalar type Half but found Float
# 🐛 Bug

## Information

CUDA: 10.1
Python 3.6.8 (default, Oct  7 2019, 12:59:55)
Installed transformers from source.
Trying to train **gpt2-medium**

Command Line:
```
export TRAIN_FILE=dataset/wiki.train.raw
export TEST_FILE=dataset/wiki.test.raw

python3 examples/run_language_modeling.py --fp16 \
    --per_gpu_eval_batch_size 1 \
    --output_dir=output \
    --model_type=gpt2-medium \
    --model_name_or_path=gpt2-medium \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --overwrite_output_dir

```


```
Traceback (most recent call last):
  File "examples/run_language_modeling.py", line 284, in <module>
    main()
  File "examples/run_language_modeling.py", line 254, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 314, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 388, in _training_step
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 385, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 616, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 500, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 237, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 196, in forward
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 149, in _attn
    w = torch.where(mask, w, self.masked_bias)
RuntimeError: expected scalar type Half but found Float
```

run_language_modeling, RuntimeError: expected scalar type Half but found Float
# 🐛 Bug

## Information

CUDA: 10.1
Python 3.6.8 (default, Oct  7 2019, 12:59:55)
Installed transformers from source.
Trying to train **gpt2-medium**

Command Line:
```
export TRAIN_FILE=dataset/wiki.train.raw
export TEST_FILE=dataset/wiki.test.raw

python3 examples/run_language_modeling.py --fp16 \
    --per_gpu_eval_batch_size 1 \
    --output_dir=output \
    --model_type=gpt2-medium \
    --model_name_or_path=gpt2-medium \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --overwrite_output_dir

```


```
Traceback (most recent call last):
  File "examples/run_language_modeling.py", line 284, in <module>
    main()
  File "examples/run_language_modeling.py", line 254, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 314, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 388, in _training_step
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 385, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 616, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 500, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 237, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 196, in forward
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 149, in _attn
    w = torch.where(mask, w, self.masked_bias)
RuntimeError: expected scalar type Half but found Float
```



Discussion:
What's your PyTorch version? Can you post the output of `transformers-cli env`

This does not seem specific to `examples/run_language_modeling.py` so I'm gonna ping @patrickvonplaten on this
@julien-c thanks for the prompt response.
`- `transformers` version: 2.8.0
- Platform: Linux-4.14.138+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.3.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no`
Hi @ankit-chadha, 

I will take a look next week :-). Maybe @sshleifer - do you something from the top of your head? This issue is also related: https://github.com/huggingface/transformers/issues/3676.
I have tried to replicate in torch 1.4 and mask, masked_bias are float16, and there is no error.

Would it be possible to upgrade torch to 1.4 and see if the problem persists @ankit-chadha ?

@sshleifer @patrickvonplaten 

`- transformers version: 2.8.0
- Platform: Linux-4.14.138+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.0 (False)`

````
raceback (most recent call last):                                                                                                                                           | 0/372 [00:00<?, ?it/s]
  File "examples/run_language_modeling.py", line 284, in <module>
    main()
  File "examples/run_language_modeling.py", line 254, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 314, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 388, in _training_step
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 616, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 500, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 237, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 196, in forward
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 149, in _attn
    w = torch.where(mask, w, self.masked_bias)
RuntimeError: expected scalar type Half but found Float
```

Could you provide a small subset of your data so that I can reproduce?
@sshleifer  - I am using the wiki.train.raw default dataset. 
What's your PyTorch version? Can you post the output of `transformers-cli env`

This does not seem specific to `examples/run_language_modeling.py` so I'm gonna ping @patrickvonplaten on this
@julien-c thanks for the prompt response.
`- `transformers` version: 2.8.0
- Platform: Linux-4.14.138+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.3.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no`
Hi @ankit-chadha, 

I will take a look next week :-). Maybe @sshleifer - do you something from the top of your head? This issue is also related: https://github.com/huggingface/transformers/issues/3676.
I have tried to replicate in torch 1.4 and mask, masked_bias are float16, and there is no error.

Would it be possible to upgrade torch to 1.4 and see if the problem persists @ankit-chadha ?

@sshleifer @patrickvonplaten 

`- transformers version: 2.8.0
- Platform: Linux-4.14.138+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.0 (False)`

````
raceback (most recent call last):                                                                                                                                           | 0/372 [00:00<?, ?it/s]
  File "examples/run_language_modeling.py", line 284, in <module>
    main()
  File "examples/run_language_modeling.py", line 254, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 314, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 388, in _training_step
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 616, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 500, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 237, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 196, in forward
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 149, in _attn
    w = torch.where(mask, w, self.masked_bias)
RuntimeError: expected scalar type Half but found Float
```

Could you provide a small subset of your data so that I can reproduce?
@sshleifer  - I am using the wiki.train.raw default dataset. 

Resolving patch:
diff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py
--- a/src/transformers/modeling_gpt2.py
+++ b/src/transformers/modeling_gpt2.py
@@ -145,7 +145,7 @@ def _attn(self, q, k, v, attention_mask=None, head_mask=None):
             w = w / (v.size(-1) ** 0.5)
         nd, ns = w.size(-2), w.size(-1)
         mask = self.bias[:, :, ns - nd : ns, :ns]
-        w = torch.where(mask, w, self.masked_bias)
+        w = torch.where(mask, w, self.masked_bias.to(w.dtype))
 
         if attention_mask is not None:
             # Apply the attention mask


Test patch: