RobertaTokenizerFast unexpectedly quits when creating a TextDataset
Author: josiahdavis
Pull request: no
Labels: name: wontfix

# 🐛 Bug

When creating a `TextDataset` using `RobertaTokenizerFast` my program unexpectedly dies. (Not so with `RobertaTokenizer`).

## Information

Model I am using: RoBERTa

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: [language modelling](https://github.com/huggingface/transformers/blob/33d3072e1c54bcd235447b98c6dea1b4cb71234c/examples/run_lm_finetuning.py)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import AutoTokenizer, TextDataset

tokenizer = AutoTokenizer.from_pretrained("roberta-base", use_fast=True)

train_dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="/home/ubuntu/data/wikitext-103-raw/wiki.train.raw",
    block_size=-1,
    overwrite_cache=False,
)
print(train_dataset)
```
## Expected behavior

Creation of the training dataset, not having the process killed. eg:

```
<transformers.data.datasets.language_modeling.TextDataset object at 0x7f138a1fd2b0>
```

## Environment info

- `transformers` version: 2.11.0
- Platform: Linux-5.3.0-1030-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No



Comments:
@n1t0 

This seems to work for me, I guess it crashes because you don't have enough memory. Unfortunately `TextDataset` has not been optimized for fast tokenizers yet, so it does a lot more work than needed when using them. It's probably better to use python tokenizers for now with `TextDataset`.

Also, maybe the [huggingface/nlp](https://github.com/huggingface/nlp) library might be better suited here. cc @lhoestq 

You could try
```python
from transformers import AutoTokenizer
from nlp import load_dataset

tokenizer = AutoTokenizer.from_pretrained("roberta-base", use_fast=True)
dataset = load_dataset("text", data_files="path/to/wiki.train.raw", split="train")
tokenized_dataset = dataset.map(lambda ex: tokenizer(ex["text"]), batched=True)
print(tokenized_dataset[0]["input_ids"])
```
We're still working on making it as fast as we can, but at least you won't have any memory issues.

Re @n1t0 comment: "I guess it crashes because you don't have enough memory" this is correct. (I was hoping I could get away with 61.0 GiB, the standard for an AWS `p3.2xlarge`.)

Re @lhoestq your code ran without errors for me. Thanks! 

I did get a lot of the [`Token indices sequence length is longer than the specified maximum sequence length for this model (522 > 512). Running this sequence through the model will result in indexing errors`](https://github.com/huggingface/transformers/issues/1791) warnings which I wasn't getting before.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.