Repository: huggingface/transformers

Issue:
IndexError: index out of bounds when running run_mlm.py



## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-4.15.0-46-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.7
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
?

## Information

Model I am using (Bert, XLNet ...): neuralmind/bert-base-portuguese-cased



## To reproduce

Steps to reproduce the behavior:

I want to fine-tune a pretrained language model using [run_mlm.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). I have a corpus file (ful_corpus.csv) that contains one doc (raw text) per line. When I run the following command:

`python run_mlm.py --model_name_or_path "neuralmind/bert-base-portuguese-cased" --train_file ../data/full_corpus.csv --cache_dir /home/mwon/data-mwon/paperChega/src_classificador/data/hugingface  --output models/  --do_train`

it results in the error:

```
Traceback (most recent call last):
  File "run_mlm.py", line 449, in <module>
    main()
  File "run_mlm.py", line 384, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map
    for k, dataset in self.items()
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1260, in map
    update_data=update_data,
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1529, in _map_single
    writer.write_batch(batch)
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_writer.py", line 278, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1474, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 322, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 1058, in pyarrow.lib.Array.__getitem__
  File "pyarrow/array.pxi", line 540, in pyarrow.lib._normalize_index
IndexError: index out of bounds 
```



Discussion:
@sgugger 
It's very hard to help you without being able to reproduce the bug. Could you share a small version of your csv file that reproduces it?
Yes, no problem. I just tried with a sample created from the `head`of my `full_corpus.csv` file and got the same error. This is the head:

```
A tomada de posse já está marcada para esta quarta feira ao fim da tarde...
Lobo Xavier está infetado com Covid-19. Esteve no Conselho de Estado na terça-feira.
"Porque está descida é temporária. Se descessem agora, depois não poderiam explicar a necessidade de uma nova subida."
Em acumulação com o Banco de Portugal.
"EUA: Há muitas maneiras de isto acabar mal. A newsletter Novo Normal do ⁦⁩ no ECO. Um guia do que pode suceder nas eleições americanas (sentem-se, é melhor)"
Costa vai substituir presidente do Tribunal de Contas via
Como criar filhos felizes?
Uma economia a 90 por cento via
Apoio à Retoma Progressiva vai permitir suspender contratos via Falta saber qual o valor do salário e quem o paga.
O perigo de esperar que o Estado nos salve
```
The problem is that you are not passing a `max_seq_length` so the script uses the tokenizer `model_lax_length`, which is in turn excessively large (1000000000000000019884624838656). So this results in all your texts not even being able to produce one batch.

Just pass `--max_seq_length 512` or something else and you should be good.

Resolving patch:
diff --git a/examples/language-modeling/run_mlm.py b/examples/language-modeling/run_mlm.py
--- a/examples/language-modeling/run_mlm.py
+++ b/examples/language-modeling/run_mlm.py
@@ -338,6 +338,12 @@ def tokenize_function(examples):
 
         if data_args.max_seq_length is None:
             max_seq_length = tokenizer.model_max_length
+            if max_seq_length > 1024:
+                logger.warn(
+                    f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). "
+                    "Picking 1024 instead. You can change that default value by passing --max_seq_length xxx."
+                )
+                max_seq_length = 1024
         else:
             if data_args.max_seq_length > tokenizer.model_max_length:
                 logger.warn(


Test patch: