Repository: huggingface/transformers

Issue:
Issues Fine-tuning XLNET 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Google colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101
- Tensorflow version (GPU?): 2.3.0
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 Text Generation: @patrickvonplaten @TevenLeScao
 TransfoXL/XLNet: @TevenLeScao

 -->

## Information

Model I am using XLNET:

The problem arises when using:
* [ ] the official example scripts: The old version of transformers. The script is run_language_modeling.py 
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] Fine-tuning 

## To reproduce

Steps to reproduce the behavior:

1. !git clone https://github.com/huggingface/transformers
import os
os.chdir('/content/transformers')
!git checkout b1ff0b2ae7d368b7db3a8a8472a29cc195d278d8
!pip install .
!pip install -r ./examples/requirements.txt
os.chdir('/content/transformers/examples')
!pip install dict_to_obj

2. !python run_language_modeling.py \
    --output_dir='/content/drive/My Drive/finetuned_models/xlnet_large'\
    --model_type=xlnet \
    --model_name_or_path=xlnet-large-cased \
    --should_continue \
    --save_total_limit=5 \
    --num_train_epochs=1.0 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=500 \
    --save_steps=500 \
    --train_data_file='/content/drive/My Drive/finetuned_models/train.txt' \
    --do_eval \
    --eval_data_file='/content/drive/My Drive/finetuned_models/valid.txt' \
    --per_gpu_train_batch_size=2 \
    --per_gpu_eval_batch_size=2 \
    --block_size=128 \
    --gradient_accumulation_steps=5
3.

[INFO|modeling_utils.py:1065] 2020-11-17 20:59:57,425 >> All the weights of XLNetLMHeadModel were initialized from the model checkpoint at /content/drive/MyDrive/finetuned_models/xlnet_base/checkpoint-26500.
If your task is similar to the task the model of the checkpoint was trained on, you can already use XLNetLMHeadModel for predictions without further training.
100%|██████████| 311/311 [01:01<00:00,  5.09ba/s]
100%|██████████| 133/133 [00:25<00:00,  5.13ba/s]
  0%|          | 0/311 [00:00<?, ?ba/s]Traceback (most recent call last):
  File "run_clm.py", line 348, in <module>
    main()
  File "run_clm.py", line 300, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py", line 300, in map
    for k, dataset in self.items()
  File "/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py", line 300, in <dictcomp>
    for k, dataset in self.items()
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py", line 1256, in map
    update_data=update_data,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py", line 156, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py", line 1525, in _map_single
    writer.write_batch(batch)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_writer.py", line 278, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1474, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 322, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 1058, in pyarrow.lib.Array.__getitem__
  File "pyarrow/array.pxi", line 540, in pyarrow.lib._normalize_index
IndexError: index out of bounds
  0%|          | 0/311 [00:02<?, ?ba/s] 

## Expected behavior

<!-- I started fine-tuning the XLNET models using google colab. The colab notebook times out after 20 hours which is fine but when I try to continue training, I get the error I above.  I have looked at similar issue reports on this repo but I was still unable to get around this error. Please, do you know what I am doing wrong? And what I can do to fix it? 
Thanks. -->



Discussion:
It looks like your using causual language modeling (next word prediction) - _File "run_clm.py", line 300, in main_, but xlnet does not use that. It uses permutation language modeling.

Have you tried with the [new scripts](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#xlnet-and-permutation-language-modeling), I believe this will solve it 👍 
Thank you, Tim. So I know XLNET and Transformer-XL are fine-tuned the same
way and tried the *run_plm.py* like you suggested and got the error below:

[INFO|tokenization_utils_base.py:1650] 2020-11-20 23:00:33,440 >> loading
file https://huggingface.co/transfo-xl-wt103/resolve/main/vocab.pkl from
cache at
/root/.cache/torch/transformers/6860d92833eb9d2a42cf185e974ca967fbf4cd58fa8d3d9298e56b9ef7ff8d5c.56c8ef92e693414ef2313bde4ba3679a404de1edbcd5a5780def3971f9706850
[INFO|modeling_utils.py:940] 2020-11-20 23:00:34,096 >> loading weights
file https://huggingface.co/transfo-xl-wt103/resolve/main/pytorch_model.bin
from cache at
/root/.cache/torch/transformers/891af5f0c8372327a961a768d4ee40b7ca95c428f9384c534e73b9b655c75468.923bd8e0844a782c35f009eddd08a3600739804fbe13bd234f592f36230ab8a9
Traceback (most recent call last): File "run_plm.py", line 382, in <module>
main() File "run_plm.py", line 244, in main cache_dir=model_args.cache_dir,
File
"/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py",
line 947, in from_pretrained model = cls(config, *model_args,
**model_kwargs) File
"/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlnet.py",
line 1294, in __init__ self.transformer = XLNetModel(config) File
"/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlnet.py",
line 940, in __init__ self.reuse_len = config.reuse_len AttributeError:
'TransfoXLConfig' object has no attribute 'reuse_len'

On Fri, Nov 20, 2020 at 1:52 PM Tim Isbister <notifications@github.com>
wrote:

> It looks like your using causual language modeling (next word prediction)
> - *run_clm.py*, but xlnet does not use that. It uses permutation language
> modeling.
>
> Have you tried with the new scripts
> <https://github.com/huggingface/transformers/tree/master/examples/language-modeling#xlnet-and-permutation-language-modeling>,
> I believe this will solve it 👍
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8674#issuecomment-731348230>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHYMJAF5X3H4GW7WFSAQEOLSQ23F3ANCNFSM4T4CBVKQ>
> .
>


Hmm strange, well actually I now tried running the provided example that I suggested to you. But also failing with `IndexError: index out of bounds` as you got in the first attempt. 

Edit: Looks like we have problems with loading the data, if I hardcoded a dataset to the `run_plm.py` 
`datasets = load_dataset('wikitext', 'wikitext-103-raw-v1’)` it works. 

[Provided example](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#xlnet-and-permutation-language-modeling): 

```
python run_plm.py \
    --model_name_or_path=xlnet-base-cased \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-plm
```

```
Traceback (most recent call last):
  File "run_plm.py", line 382, in <module>
    main()
  File "run_plm.py", line 321, in main
    tokenized_datasets = tokenized_datasets.map(
  File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/dataset_dict.py", line 286, in map
    {
  File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/dataset_dict.py", line 287, in <dictcomp>
    k: dataset.map(
  File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1243, in map
    return self._map_single(
  File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1528, in _map_single
    writer.write_batch(batch)
  File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_writer.py", line 278, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1474, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 322, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 1058, in pyarrow.lib.Array.__getitem__
  File "pyarrow/array.pxi", line 540, in pyarrow.lib._normalize_index
IndexError: index out of bounds
```
Thanks, Tim. Do you have any suggestions for how I can load my own train
and validation datasets? And how will it work with the following code
below. I am now using the current transformer version.
Thanks again, Tim.


python run_plm.py \
    --model_name_or_path=transfo-xl-wt103 \
    --train_file='/content/drive/My Drive/finetuned_models/train.txt' \
    --validation_file='/content/drive/My Drive/finetuned_models/valid.txt' \
    --save_total_limit=5 \
    --num_train_epochs=1.0 \
    --do_train \
    --do_eval \
    --per_gpu_train_batch_size=2 \
    --per_gpu_eval_batch_size=2 \

On Fri, Nov 20, 2020 at 7:23 PM Tim Isbister <notifications@github.com>
wrote:

> Hmm strange, well actually I now tried running the provided example that I
> suggested to you. But also failing with IndexError: index out of bounds
> as you got in the first attempt. I have the latest versions of all the
> libraries running on Ubuntu 18.04 LTS with Titan RTX.
>
> Provided example
> <https://github.com/huggingface/transformers/tree/master/examples/language-modeling#xlnet-and-permutation-language-modeling>
> :
>
> python run_plm.py \
>     --model_name_or_path=xlnet-base-cased \
>     --dataset_name wikitext \
>     --dataset_config_name wikitext-2-raw-v1 \
>     --do_train \
>     --do_eval \
>     --output_dir /tmp/test-plm
>
> Traceback (most recent call last):
>   File "run_plm.py", line 382, in <module>
>     main()
>   File "run_plm.py", line 321, in main
>     tokenized_datasets = tokenized_datasets.map(
>   File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/dataset_dict.py", line 286, in map
>     {
>   File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/dataset_dict.py", line 287, in <dictcomp>
>     k: dataset.map(
>   File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1243, in map
>     return self._map_single(
>   File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
>     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
>   File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/fingerprint.py", line 163, in wrapper
>     out = func(self, *args, **kwargs)
>   File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1528, in _map_single
>     writer.write_batch(batch)
>   File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_writer.py", line 278, in write_batch
>     pa_table = pa.Table.from_pydict(typed_sequence_examples)
>   File "pyarrow/table.pxi", line 1474, in pyarrow.lib.Table.from_pydict
>   File "pyarrow/array.pxi", line 322, in pyarrow.lib.asarray
>   File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
>   File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
>   File "/home/tim/anaconda3/envs/transformers/lib/python3.8/site-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
>     if trying_type and out[0].as_py() != self.data[0]:
>   File "pyarrow/array.pxi", line 1058, in pyarrow.lib.Array.__getitem__
>   File "pyarrow/array.pxi", line 540, in pyarrow.lib._normalize_index
> IndexError: index out of bounds
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8674#issuecomment-731472522>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHYMJABV347A2NH2SK6NU4LSQ4B6HANCNFSM4T4CBVKQ>
> .
>
Maybe @sgugger has an idea!
The problem is that the script uses the tokenizer max length when no `max_seq_length` is passed, and that the XLNet tokenizer has a ridiculously high maximum sequence length. I have suggested a fix in #8738.

While waiting for this PR to be merged, a temporary fix is to just add --max_seq_length 512 (or any value you'd like) to your command.
I tried using the max_seq_length argument and set it to 400 and got the
following error:

Traceback (most recent call last):
  File "run_plm.py", line 379, in <module>
    main()
  File "run_plm.py", line 244, in main
    cache_dir=model_args.cache_dir,
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py",
line 947, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlnet.py",
line 1294, in __init__
    self.transformer = XLNetModel(config)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlnet.py",
line 940, in __init__
    self.reuse_len = config.reuse_len
AttributeError: 'TransfoXLConfig' object has no attribute 'reuse_len'


On Mon, Nov 23, 2020 at 4:02 PM Lysandre Debut <notifications@github.com>
wrote:

> Closed #8674 <https://github.com/huggingface/transformers/issues/8674>
> via #8738 <https://github.com/huggingface/transformers/pull/8738>.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8674#event-4029627504>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHYMJADWOOMGPRQUSIUPRBDSRLEXRANCNFSM4T4CBVKQ>
> .
>
The script only works for XLNet models, you will need to tweak it for other models
I see. Thank you, Sylvain.

On Tue, Nov 24, 2020 at 10:46 AM Sylvain Gugger <notifications@github.com>
wrote:

> The script only works for XLNet models, you will need to tweak it for
> other models
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8674#issuecomment-733059560>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHYMJADG3HM6UCRSXGS56ULSRPIM3ANCNFSM4T4CBVKQ>
> .
>

Just to confirm that I am doing the right thing, what is the correct script
in the language-modeling folder for fine-tuning Transformer-XL and CTRL? I
am using run_clm.py for both of them currently but it keeps returning the
same error message for both. See error below:

For Transformer-XL:

[INFO|modeling_utils.py:1065] 2020-11-24 15:55:30,982 >> All the
weights of TransfoXLLMHeadModel were initialized from the model
checkpoint at transfo-xl-wt103.
If your task is similar to the task the model of the checkpoint was
trained on, you can already use TransfoXLLMHeadModel for predictions
without further training.
  2%|▏         | 6/311 [00:13<11:44,  2.31s/ba]Traceback (most recent
call last):
  File "run_clm.py", line 351, in <module>
    main()
  File "run_clm.py", line 261, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py",
line 303, in map
    for k, dataset in self.items()
  File "/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py",
line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py",
line 1259, in map
    update_data=update_data,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py",
line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/datasets/fingerprint.py",
line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py",
line 1520, in _map_single
    batch, indices, check_same_num_examples=len(self.list_indexes()) >
0, offset=offset
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py",
line 1438, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices
else function(*fn_args, **fn_kwargs)
  File "run_clm.py", line 254, in tokenize_function
    return tokenizer(examples[text_column_name])
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py",
line 2214, in __call__
    **kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py",
line 2399, in batch_encode_plus
    **kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py",
line 567, in _batch_encode_plus
    verbose=verbose,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py",
line 630, in _batch_prepare_for_model
    return_attention_mask=return_attention_mask,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py",
line 2531, in pad
    f"type of {first_element} unknown: {type(first_element)}. "
ValueError: type of [] unknown: <class 'list'>. Should be one of a
python, numpy, pytorch or tensorflow object.
  2%|▏         | 6/311 [00:16<13:48,  2.72s/ba]


For CTRL:

[INFO|modeling_utils.py:1065] 2020-11-24 15:52:33,705 >> All the
weights of CTRLLMHeadModel were initialized from the model checkpoint
at ctrl.
If your task is similar to the task the model of the checkpoint was
trained on, you can already use CTRLLMHeadModel for predictions
without further training.
  0%|          | 0/311 [00:00<?,
?ba/s][WARNING|tokenization_utils_base.py:2736] 2020-11-24
15:52:39,268 >> Token indices sequence length is longer than the
specified maximum sequence length for this model (293 > 256). Running
this sequence through the model will result in indexing errors
  2%|▏         | 6/311 [00:05<04:21,  1.17ba/s]Traceback (most recent
call last):
  File "run_clm.py", line 351, in <module>
    main()
  File "run_clm.py", line 261, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py",
line 303, in map
    for k, dataset in self.items()
  File "/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py",
line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py",
line 1259, in map
    update_data=update_data,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py",
line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/datasets/fingerprint.py",
line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py",
line 1520, in _map_single
    batch, indices, check_same_num_examples=len(self.list_indexes()) >
0, offset=offset
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py",
line 1438, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices
else function(*fn_args, **fn_kwargs)
  File "run_clm.py", line 254, in tokenize_function
    return tokenizer(examples[text_column_name])
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py",
line 2214, in __call__
    **kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py",
line 2399, in batch_encode_plus
    **kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py",
line 567, in _batch_encode_plus
    verbose=verbose,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py",
line 630, in _batch_prepare_for_model
    return_attention_mask=return_attention_mask,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py",
line 2531, in pad
    f"type of {first_element} unknown: {type(first_element)}. "
ValueError: type of [] unknown: <class 'list'>. Should be one of a
python, numpy, pytorch or tensorflow object.
  2%|▏         | 6/311 [00:08<07:26,  1.46s/ba]




On Tue, Nov 24, 2020 at 10:50 AM Adaku Uchendu <adaku2@umbc.edu> wrote:

> I see. Thank you, Sylvain.
>
> On Tue, Nov 24, 2020 at 10:46 AM Sylvain Gugger <notifications@github.com>
> wrote:
>
>> The script only works for XLNet models, you will need to tweak it for
>> other models
>>
>> —
>> You are receiving this because you authored the thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/transformers/issues/8674#issuecomment-733059560>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AHYMJADG3HM6UCRSXGS56ULSRPIM3ANCNFSM4T4CBVKQ>
>> .
>>
>
>
> --
> *Adaku Uchendu*
>
> *McNair Scholar*
> *Mathematics major*
> *Statistic minor *
> *Math Lab Tutor*
> *Pre-Calculus LA*
> *University of Maryland, Baltimore County *
> *Class of 2018*
>
It looks like it comes from a bug in the slow tokenizers that can't handle an empty sequence at the beginning. We're looking into it. 
Thank you, Sylvain.

On Tue, Nov 24, 2020 at 11:50 AM Sylvain Gugger <notifications@github.com>
wrote:

> It looks like it comes from a bug in the slow tokenizers that can't handle
> an empty sequence at the beginning. We're looking into it.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8674#issuecomment-733103967>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHYMJAHQJ4FQ24C6JWITIO3SRPP53ANCNFSM4T4CBVKQ>
> .
>
Hi Sylvain, 

I just wanted to know if the slow tokenizer bug is no longer a problem?
Thank you  
It seems to be solved on master, I didn't try on the v4 release but is might also be solved there too.
Hi Sylvain,

I just tried to re-run my code with the recent changes and I still got the same error. Just to make sure I am doing the correct thing, I have attached my code below. And I am using the transformer version 4.0 with the current github transformer repo.
Thank you.


cd language-modeling/
python run_clm.py \
    --model_type=transfo-xl \
    --model_name_or_path=transfo-xl-wt103 \
    --train_file='/content/drive/My Drive/finetuned_models/train.txt' \
    --validation_file='/content/drive/My Drive/finetuned_models/valid.txt' \
    --save_total_limit=5 \
    --num_train_epochs=1.0 \
    --do_train \
    --do_eval \
    --per_gpu_train_batch_size=2 \
    --per_gpu_eval_batch_size=2 \
    --output_dir='/content/drive/My Drive/finetuned_models/transformer_xl'

Resolving patch:
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -2663,10 +2663,17 @@ def pad(
         # If we have PyTorch/TF/NumPy tensors/arrays as inputs, we cast them as python objects
         # and rebuild them afterwards if no return_tensors is specified
         # Note that we lose the specific device the tensor may be on for PyTorch
+
         first_element = encoded_inputs["input_ids"][0]
-        if isinstance(first_element, (list, tuple)) and first_element:
-            first_element = first_element[0]
-        if not isinstance(first_element, int):
+        if isinstance(first_element, (list, tuple)):
+            # first_element might be an empty list/tuple in some edge cases so we grab the first non empty element.
+            index = 0
+            while len(encoded_inputs["input_ids"][index]) == 0:
+                index += 1
+            if index < len(encoded_inputs["input_ids"]):
+                first_element = encoded_inputs["input_ids"][index][0]
+        # At this state, if `first_element` is still a list/tuple, it's an empty one so there is nothing to do.
+        if not isinstance(first_element, (int, list, tuple)):
             if is_tf_available() and isinstance(first_element, tf.Tensor):
                 return_tensors = "tf" if return_tensors is None else return_tensors
             elif is_torch_available() and isinstance(first_element, torch.Tensor):


Test patch: