run_glue.py fails with RoBERTa but succeeds with other models
Author: yonatanbitton
Pull request: no

## Environment info
I'm following this instructions: https://github.com/huggingface/transformers/tree/master/examples, meaning I installed the library from source.
     
- `transformers` version: 4.1.0.dev0
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.7.0
- Using GPU in script?: Yes, Tesla K80
- Using distributed or parallel set-up in script?: running `CUDA_VISIBLE_DEVICES=0 python run_glue.py` 

 -->

## The problem

I'm running the official `run_glue.py` code, with the command and arguments given here: https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-pytorch-version

When I use BERT - it **succeeds**. 
For example, BERT: 
```CUDA_VISIBLE_DEVICES=0 python run_glue.py --task_name cola --output_dir results/normal/bert/cola/ --cache_dir cache/normal/bert --model_name_or_path bert-base-cased --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --do_predict --overwrite_output_dir```

And I receive score that makes sense:
```
[d]$ cat /transformers/examples/text-classification/results/normal/bert/cola/eval_results_cola.txt
eval_loss = 0.518086314201355
eval_matthews_correlation = 0.572739655014278
epoch = 3.0
```

When I use RoBERTA, it **fails** with a stacktrace: 
```CUDA_VISIBLE_DEVICES=0 python run_glue.py --task_name cola --output_dir results/normal/roberta/cola/ --cache_dir cache/normal/roberta --model_name_or_path roberta-base --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --do_predict --overwrite_output_dir```

The error message: 
```python
[INFO|trainer.py:674] 2020-12-04 20:23:30,937 >>   Total optimization steps = 804
  0%|                                                   | 0/804 [00:00<?, ?it/s]/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [33,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [33,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
Traceback (most recent call last):
  File "/transformers/examples/text-classification/run_yonatan.py", line 464, in <module>
    main()
  File "/transformers/examples/text-classification/run_yonatan.py", line 399, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/transformers/src/transformers/trainer.py", line 767, in train
    tr_loss += self.training_step(model, inputs)
  File "/transformers/src/transformers/trainer.py", line 1096, in training_step
    loss = self.compute_loss(model, inputs)
  File "/transformers/src/transformers/trainer.py", line 1120, in compute_loss
    outputs = model(**inputs)
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 1029, in forward
    return_dict=return_dict,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 717, in forward
    return_dict=return_dict,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 450, in forward
    output_attentions,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 368, in forward
    output_attentions=output_attentions,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 302, in forward
    output_attentions,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 184, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
  0%|                                                   | 0/804 [00:00<?, ?it/s]
```

I've searched for related solutions but didn't find any relevant solution (https://github.com/huggingface/transformers/issues?q=CUBLAS_STATUS_ALLOC_FAILED).

What am I missing? 

Thanks

Comments:
When **importing** transformers (instead of using the source) the problem does not occur.

Pinging the Trainer master @sgugger!

This looks like a problem on the CUDA initialization in your enviromnent. The command runs fine on my side. 

> When **importing** transformers (instead of using the source) the problem does not occur.

What do you mean exactly by this?

> This looks like a problem on the CUDA initialization in your enviromnent. The command runs fine on my side.
> 
> > When **importing** transformers (instead of using the source) the problem does not occur.
> 
> What do you mean exactly by this?

The note here: https://github.com/huggingface/transformers/tree/master/examples#important-note
suggests to install the library from source. When I do it (with git clone) it doesn't work - I receive the error described here.

On the other hand, when I use the 'transformers' from 'pip install transformers', it does work.

I'm not sure if this specific difference causes the error only in my environment or not. 

This issue has been stale for 1 month.