Repository: huggingface/transformers

Issue:
Issues Fine-tuning XLNET 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Google colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101
- Tensorflow version (GPU?): 2.3.0
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 Text Generation: @patrickvonplaten @TevenLeScao
 TransfoXL/XLNet: @TevenLeScao

 -->

## Information

Model I am using XLNET:

The problem arises when using:
* [ ] the official example scripts: The old version of transformers. The script is run_language_modeling.py 
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] Fine-tuning 

## To reproduce

Steps to reproduce the behavior:

1. !git clone https://github.com/huggingface/transformers
import os
os.chdir('/content/transformers')
!git checkout b1ff0b2ae7d368b7db3a8a8472a29cc195d278d8
!pip install .
!pip install -r ./examples/requirements.txt
os.chdir('/content/transformers/examples')
!pip install dict_to_obj

2. !python run_language_modeling.py \
    --output_dir='/content/drive/My Drive/finetuned_models/xlnet_large'\
    --model_type=xlnet \
    --model_name_or_path=xlnet-large-cased \
    --should_continue \
    --save_total_limit=5 \
    --num_train_epochs=1.0 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=500 \
    --save_steps=500 \
    --train_data_file='/content/drive/My Drive/finetuned_models/train.txt' \
    --do_eval \
    --eval_data_file='/content/drive/My Drive/finetuned_models/valid.txt' \
    --per_gpu_train_batch_size=2 \
    --per_gpu_eval_batch_size=2 \
    --block_size=128 \
    --gradient_accumulation_steps=5
3.

[INFO|modeling_utils.py:1065] 2020-11-17 20:59:57,425 >> All the weights of XLNetLMHeadModel were initialized from the model checkpoint at /content/drive/MyDrive/finetuned_models/xlnet_base/checkpoint-26500.
If your task is similar to the task the model of the checkpoint was trained on, you can already use XLNetLMHeadModel for predictions without further training.
100%|██████████| 311/311 [01:01<00:00,  5.09ba/s]
100%|██████████| 133/133 [00:25<00:00,  5.13ba/s]
  0%|          | 0/311 [00:00<?, ?ba/s]Traceback (most recent call last):
  File "run_clm.py", line 348, in <module>
    main()
  File "run_clm.py", line 300, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py", line 300, in map
    for k, dataset in self.items()
  File "/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py", line 300, in <dictcomp>
    for k, dataset in self.items()
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py", line 1256, in map
    update_data=update_data,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py", line 156, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py", line 1525, in _map_single
    writer.write_batch(batch)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_writer.py", line 278, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1474, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 322, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 1058, in pyarrow.lib.Array.__getitem__
  File "pyarrow/array.pxi", line 540, in pyarrow.lib._normalize_index
IndexError: index out of bounds
  0%|          | 0/311 [00:02<?, ?ba/s] 

## Expected behavior

<!-- I started fine-tuning the XLNET models using google colab. The colab notebook times out after 20 hours which is fine but when I try to continue training, I get the error I above.  I have looked at similar issue reports on this repo but I was still unable to get around this error. Please, do you know what I am doing wrong? And what I can do to fix it? 
Thanks. -->



Discussion:
It looks like your using causual language modeling (next word prediction) - _File "run_clm.py", line 300, in main_, but xlnet does not use that. It uses permutation language modeling.

Have you tried with the [new scripts](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#xlnet-and-permutation-language-modeling), I believe this will solve it 👍 

Resolving patch:
diff --git a/examples/language-modeling/run_plm.py b/examples/language-modeling/run_plm.py
--- a/examples/language-modeling/run_plm.py
+++ b/examples/language-modeling/run_plm.py
@@ -93,11 +93,11 @@ class DataTrainingArguments:
     overwrite_cache: bool = field(
         default=False, metadata={"help": "Overwrite the cached training and evaluation sets"}
     )
-    max_seq_length: Optional[int] = field(
-        default=None,
+    max_seq_length: int = field(
+        default=512,
         metadata={
             "help": "The maximum total input sequence length after tokenization. Sequences longer "
-            "than this will be truncated. Default to the max input length of the model."
+            "than this will be truncated."
         },
     )
     preprocessing_num_workers: Optional[int] = field(
@@ -286,15 +286,12 @@ def tokenize_function(examples):
             load_from_cache_file=not data_args.overwrite_cache,
         )
 
-        if data_args.max_seq_length is None:
-            max_seq_length = tokenizer.model_max_length
-        else:
-            if data_args.max_seq_length > tokenizer.model_max_length:
-                logger.warn(
-                    f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the"
-                    f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
-                )
-            max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)
+        if data_args.max_seq_length > tokenizer.model_max_length:
+            logger.warn(
+                f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the"
+                f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
+            )
+        max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)
 
         # Main data processing function that will concatenate all texts from our dataset and generate chunks of
         # max_seq_length.


Test patch: