`run_language_modeling` fails with community model (BioClinicalBERT)
Author: xhluca
Pull request: no
Labels: name: wontfix

name: Ex: LM (Finetuning)

name: Core: Modeling

name: Should Fix

# 🐛 Bug

The `run_language_modeling.py` fails when using community models.

## Information

Model I am using (Bert, XLNet ...): BioClinical_BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. download wikitext-2 raw
2. run exactly the same 
3.

This code command line yields the following error:
```
python run_language_modeling.py \
    --output_dir=output \
    --model_type=BioClinicalBERT \
    --model_name_or_path=emilyalsentzer/Bio_ClinicalBERT \
    --output_dir=/kaggle/working/model \
    --do_train \
    --line_by_line \
    --train_data_file=wikitext-2-raw/wiki.train.raw \
    --do_eval \
    --eval_data_file=wikitext-2-raw/wiki.valid.raw \
    --num_train_epochs=4 \
    --mlm
```

the following error will appear:
```
  File "run_language_modeling.py", line 782, in <module>
    main()
  File "run_language_modeling.py", line 732, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_language_modeling.py", line 333, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 987, in forward
    encoder_attention_mask=encoder_attention_mask,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 790, in forward
    encoder_attention_mask=encoder_extended_attention_mask,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 407, in forward
    hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 368, in forward
    self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 314, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 234, in forward
    attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/THC/THCBlas.cu:368
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
transformers v2.6.0
     
- `transformers` version:
- Platform: Kaggle
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
Seems like scibert is also broken. When I run

```
python run_language_modeling.py \
    --output_dir=output \
    --model_type=scibert \
    --model_name_or_path=allenai/scibert_scivocab_cased \
    --output_dir=/kaggle/working/model \
    --do_train \
    --line_by_line \
    --train_data_file=wikitext-2-raw/wiki.train.raw \
    --do_eval \
    --eval_data_file=wikitext-2-raw/wiki.valid.raw \
    --num_train_epochs=4 \
    --mlm
```

I get this:
```
Epoch:   0%|                                              | 0/4 [00:00<?, ?it/s]
Iteration:   0%|                                       | 0/5942 [00:00<?, ?it/s]
Iteration:   0%|                               | 1/5942 [00:00<33:48,  2.93it/s]
Iteration:   0%|                               | 2/5942 [00:00<29:32,  3.35it/s]
Iteration:   0%|                               | 3/5942 [00:00<25:44,  3.84it/s]
Iteration:   0%|                               | 4/5942 [00:00<24:19,  4.07it/s]
Iteration:   0%|                               | 5/5942 [00:01<22:20,  4.43it/s]Traceback (most recent call last):
  File "run_language_modeling.py", line 782, in <module>
    main()
  File "run_language_modeling.py", line 732, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_language_modeling.py", line 345, in train
    loss.backward()
  File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered
```

This is because the tokenizers you mention do not have a `tokenizer_config.json` file on the S3. There should be one limiting the maximum length to 512 tokens.

Here it is set to 1e12 because it doesn't detect a maximum length in the configuration.

cc @julien-c 

Yes, in that case you would need to pass a `max_block` arg to the script.

Let us know if it fixes your issue.

I see, thanks a lot! Here it would be `max_block=512` right (or whatever is the maximum length supported by that model)?

Yes, that's right!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I haven't had the chance to test `max_block=512`, but I hope this solves the problem. If it still persists, I'll re-open this issue.