Parallell compute failing for finetuning GPT2 using GPU
Author: aclifton314
Pull request: no

## ? Questions & Help
**SYSTEM**
OS: Ubuntu 18.04
Python version: 3.6.8
Torch version: 1.0.0
Transformers version: 2.1.1

I am trying to finetune GPT2 using a GPU and am getting an error. I run the `run_lm_finetuning.py` example script with the following parameters:

```python
python run_lm_finetuning.py \
--output_dir=/home/user/gpt2_finetuned_isr/model \
--model_type=gpt2 \
--model_name_or_path=gpt2 \
--do_train \
--train_data_file=/path/to/train/data/train_data.txt \
--do_eval \
--eval_data_file=/path/to/test/data/test_data.txt
```

Here is the error:
```python
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html
11/14/2019 11:01:48 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
11/14/2019 11:01:49 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /home/user/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
11/14/2019 11:01:49 - INFO - transformers.configuration_utils -   Model config {
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

11/14/2019 11:01:50 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /home/user/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
11/14/2019 11:01:50 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /home/user/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
11/14/2019 11:01:50 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at /home/user/.cache/torch/transformers/4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
/home/user/.local/lib/python3.6/site-packages/torch/cuda/__init__.py:117: UserWarning:
    Found GPU0 Quadro 4000 which is of cuda capability 2.0.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
11/14/2019 11:02:10 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=1024, cache_dir='', config_name='', device=device(type='cuda'), do_eval=True, do_lower_case=False, do_train=True, eval_all_checkpoints=False, eval_data_file='/home/user/gpt2_finetuned_isr/finetune_gpt2_test.txt', evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, local_rank=-1, logging_steps=50, max_grad_norm=1.0, max_steps=-1, mlm=False, mlm_probability=0.15, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_train_epochs=1.0, output_dir='/home/user/gpt2_finetuned_isr/model', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=50, save_total_limit=None, seed=42, server_ip='', server_port='', tokenizer_name='', train_data_file='/home/user/gpt2_finetuned_isr/finetune_gpt2_train.txt', warmup_steps=0, weight_decay=0.0)
11/14/2019 11:02:10 - INFO - __main__ -   Creating features from dataset file at /home/user/gpt2_finetuned_isr
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (1003332 > 1024). Running this sequence through the model will result in indexing errors
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.

BTW, I get TONS of these tokenizer warnings... I won't paste them here as they say the same thing...

11/14/2019 11:02:16 - INFO - __main__ -   Saving features into cached file /home/user/gpt2_finetuned_isr/cached_lm_1024_finetune_gpt2_train.txt
11/14/2019 11:02:16 - INFO - __main__ -   ***** Running training *****
11/14/2019 11:02:16 - INFO - __main__ -     Num examples = 979
11/14/2019 11:02:16 - INFO - __main__ -     Num Epochs = 1
11/14/2019 11:02:16 - INFO - __main__ -     Instantaneous batch size per GPU = 4
11/14/2019 11:02:16 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
11/14/2019 11:02:16 - INFO - __main__ -     Gradient Accumulation steps = 1
11/14/2019 11:02:16 - INFO - __main__ -     Total optimization steps = 245
Epoch:   0%|                                                                                                                                                                                | 0/1 [00:00<?, ?it/sTraceback (most recent call last):                                                                                                                                                         | 0/245 [00:00<?, ?it/s]
  File "run_lm_finetuning.py", line 545, in <module>
    main()
  File "run_lm_finetuning.py", line 497, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 228, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 533, in forward
    head_mask=head_mask)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 385, in forward
    position_ids = torch.arange(past_length, input_ids.size(-1) + past_length, dtype=torch.long, device=input_ids.device)
RuntimeError: parallel_for failed: no kernel image is available for execution on the device
Epoch:   0%|                                                                                                                                                                                | 0/1 [00:00<?, ?it/s]
Iteration:   0%|                                                                                                                                                                          | 0/245 [00:00<?, ?it/s]
```

I see the pytorch warning about the GPU, but it still seems available to use:
```python
Python 3.6.8 (default, Oct  7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.current_device()
/home/user/.local/lib/python3.6/site-packages/torch/cuda/__init__.py:117: UserWarning:
    Found GPU0 Quadro 4000 which is of cuda capability 2.0.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
0
>>> torch.cuda.device(0)
<torch.cuda.device object at 0x7fb833751d30>
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name(0)
'Quadro 4000'
>>> torch.cuda.is_available()
True
```

Any guidance is much appreciated!




Comments: