T5: Finetuning the language modelling objective on a  new dataset

# ❓ Questions & Help

I was wondering if there is a way I could fine tune T5 model on my own dataset. There are scripts for fine tuning help for GPT2, BERT and XLNET in language modelling examples, so was thinking if that could be extended to T5 as well?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Create Model-card for LIMIT-BERT

model card

Related to pretrained model cards

# What does this PR do?

Adds a model card Readme for the LIMIT-BERT language model


can u help me out with how to input custom data files in RAG retriever and the data format

sign me up, Sam

_Originally posted by @stas00 in https://github.com/huggingface/transformers/issues/7715#issuecomment-706769068_


Fix trainer callback

Fix a bug that happends when subclassing Trainer and
overwriting evaluate() without calling prediciton_loop()

# What does this PR do?

Fixes #7702

@sgugger


wrong decoder_input_ids[:,0] for MarianMT models ?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux
- Python version: 3.7.0
- PyTorch version (GPU?): 1.6.0
- Using GPU in script?: Yes

### Who can help: @sshleifer

## Information

Model I am using (Bert, XLNet ...): MarianMTModel

The problem arises when using:
* [x] the official example scripts: (give details below)

```from transformers import MarianTokenizer
tok = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de')
src_texts = [ "I am a small frog.", "Tom asked his teacher for advice."]
tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."]  # optional
batch_enc: BatchEncoding = tok.prepare_seq2seq_batch(src_texts, tgt_texts=tgt_texts)
# model(**batch) should work
```

model(**batch)  doesn't work as intended because [shift_tokens_right](https://github.com/huggingface/transformers/blob/03ec02a667d5ed3075ea65b9f89ef7135e97f6b4/src/transformers/modeling_bart.py#L226) adds eos token to generate the target sequence.

```
shift_tokens_right(batch["labels"], model.config.pad_token_id)
```
returns
```
[[0,   105,   495,    53,  5324, 17279,   649,     3],
 [0,  2136,  8818,   715,  5832,    91,   688,     3]]
```
instead of
```
[[58100,   105,   495,    53,  5324, 17279,   649,     3],
 [58100,  2136,  8818,   715,  5832,    91,   688,     3]]
```
Here, "58100" is the decoder_start_token_id.


fixed typo in warning line 207.

replace 'men_len' with 'mem_len' to match parameter name

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


The input training data files (multiple files in glob format).

Very often Corpus comes in [split files (book-large-p1.txt, book-large-p2.txt)](https://huggingface.co/datasets/bookcorpus); Also splitting large files to smaller files can prevent [language_modeling's tokenizer going out of memory](https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/datasets/language_modeling.py#L67) in environment like Colab that does not have swap memory and limited to Standard (12Gb) or High RAM (25Gb) instances.

To avoid making any assumption and prematurely truncate the file to avoid such error, we add support to concatenate training data on Dataset level. User can split files to multiple 512Mb, in that case language_modeling's tokenizer would be less to go out of memory.

In addition, it would even further enhance the memory limitation if we keep [pytorch tensor in memory](https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/datasets/language_modeling.py#L88) instead of python list. We leave this in future work.

@LysandreJik @sgugger


Hosted Inference API for Token Classification doesn't Highlight Tokens correctly

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

 albert, bert, GPT2, XLM: @LysandreJik
 Model Cards: @julien-c
 examples/token-classification: @stefan-it
## Information

Model I am using (Bert, XLNet ...):
Bert

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

https://huggingface.co/Rostlab/prot_bert_bfd_ss3?text=N+L+Y+I+Q+W+L+K+D+G+G+P+S+S+G+R+P+P+P+S
https://huggingface.co/Rostlab/prot_bert_bfd_ss3?text=T+G+N+L+Y+I+Q+W+L+K+D+G+G+P+S+S+G+R+P+P+P+S+A+T+G

## Expected behavior

When the Hosted inference API finds a token tag for special tokens like "[CLS] and [SEP]" also occurs with next or previous tokens, it doesn't highlight it and tag it properly.

Example:
<img width="617" alt="Screenshot 2020-10-11 at 23 31 35" src="https://user-images.githubusercontent.com/6087313/95690715-f17aba80-0c19-11eb-803d-c439cd9b137d.png">
Because token "N" had the same token group as the previous special token "[CLS]", it was not highlighted. However, it was detected correctly.


examples/rag: test coverage, tiny model

Feature request

Request for a new feature

Help wanted

Extra attention is needed, help appreciated

Tests

Related to tests

Disclaimer: I don't know this code very well, this may be much harder than it seems.
Blocking PR: #7713

[`examples/rag/finetune.py`, `examples/rag/finetune.sh`, `eval_rag.py`] do not seem to be tested at all.
It would be good to have a `test_finetune.py` like `examples/seq2seq` that tested these.

cc @stas00 if interested, rag is a cool new retrieval model https://arxiv.org/pdf/2005.11401.pdf


Fix typo in all model docs

# What does this PR do?

Like #7703 but for all other models (maked-> masked)


rag examples tests fail

```
================================================================ ERRORS =================================================================
______________________________________ ERROR collecting examples/rag/test_distributed_retriever.py ______________________________________
ImportError while importing test module '/Users/shleifer/transformers_fork/examples/rag/test_distributed_retriever.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
examples/rag/test_distributed_retriever.py:26: in <module>
    from examples.rag.distributed_retriever import RagPyTorchDistributedRetriever  # noqa: E402 # isort:skip
E   ModuleNotFoundError: No module named 'examples.rag'
```


fix examples/rag imports, tests

Before
```bash
pytest examples/rag
```
fails with
```
================================================================ ERRORS =================================================================
______________________________________ ERROR collecting examples/rag/test_distributed_retriever.py ______________________________________
ImportError while importing test module '/Users/shleifer/transformers_fork/examples/rag/test_distributed_retriever.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
examples/rag/test_distributed_retriever.py:26: in <module>
    from examples.rag.distributed_retriever import RagPyTorchDistributedRetriever  # noqa: E402 # isort:skip
E   ModuleNotFoundError: No module named 'examples.rag'
```

After, the same command passes.

The fix was to change
`from examples.rag.file_name -> from filename`
and add the same `sys.path` magic that `examples/seq2seq/` uses.

### TODO
- test the scripts in `examples/rag/README.md` after this change


2 Deberta test failures

I suspect these are related to recent tokenizer changes:

https://github.com/huggingface/transformers/runs/1236753957?check_suite_focus=true

```
FAILED tests/test_tokenization_deberta.py::DebertaTokenizationTest::test_torch_encode_plus_sent_to_model
FAILED tests/test_modeling_deberta.py::DebertaModelIntegrationTest::test_inference_classification_head
```


2 RAG test failures

The two failures here look related to recent tokenizer changes:

https://github.com/huggingface/transformers/runs/1236753957?check_suite_focus=true

```
=========================== short test summary info ============================

FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_sequence_generate_batch
FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_token_generate_batch
```


[marian] Automate Tatoeba-Challenge conversion

This allows conversion of marian models from the Tatoeba-Challenge repo through the command line,
with instructions at `scripts/tatoeba/README.md`. This was previously impossible.
Tests are in `examples/` because the conversion requires examples dependencies (wget and pandas).

In my opinion, it would be a lot of work for very little benefit to remove these dependencies.

The goal of this PR is to allow @jorgtied to upload his own models.


Fine Tuning SciBERT NLI model

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Hi,
I am a novice in the domain of fine-tuning any Transformer models using my own dataset. I want to fine-tune the SciBERT NLI model(https://huggingface.co/gsarti/scibert-nli) using my dataset. The dataset formation is
S1 [SEP] S2 [SEP] Inference

I am not sure how to fine-tune my dataset on the SciBERT NLI model(https://huggingface.co/gsarti/scibert-nli).

I am sorry for asking a very simple question. But any suggestion or link helps me. Thanks in advance. -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Hi,
I am a novice in the domain of fine-tuning any Transformer models using my own dataset. I want to fine-tune the SciBERT NLI model(https://huggingface.co/gsarti/scibert-nli) using my dataset. The dataset formation is
S1 [SEP] S2 [SEP] Inference

I am not sure how to fine-tune my dataset on the SciBERT NLI model(https://huggingface.co/gsarti/scibert-nli).

I am sorry for asking a very simple question. But any suggestion or link helps me. Thanks in advance.


[NEW MODEL] Multilingual document embeddings: cT-LASER

# 🌟 New model addition

## Model description

Multilingual document embeddings by adapting the LASER architecture (which is based on BiLSTM) to transformer architectures.

## Open source status

* [x] the model implementation is available: open source implementation is here: https://github.com/ever4244/tfm_laser_0520
* [ ] the model weights are available: i haven't found them
* [x] who are the authors: @ever4244


run_tf_text_classification.py for custom dataset

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: master
- Platform: ubuntu 18.04
- Python version: 3.6
- PyTorch version (GPU?): 2080ti
- Tensorflow version (GPU?): 2080ti
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: No

### Who can help
@stefan-it
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): Longformer

The problem arises when using: run_tf_text_classification.py

The tasks I am working on is: customize dataset for text classification

## To reproduce
create a csv file like below
```
exp.csv
id,sentence1,sentence2
0,you,he
0,you,he
0,you,he
0,you,he
0,you,he
0,you,he
0,you,he
0,you,he
0,you,he
```
run official script (though actually I split them in jupyter notebook cell, but the code is the same)
python run_tf_text_classification.py \
  --train_file exp.csv \ ### training dataset file location (mandatory if running with --do_train option)
  --dev_file exp.csv \ ### development dataset file location (mandatory if running with --do_eval option)
  --test_file exp.csv \ ### test dataset file location (mandatory if running with --do_predict option)
  --label_column_id 0 \ ### which column corresponds to the labels
  --model_name_or_path  allenai/longformer-base-4096\
  --output_dir model \
  --num_train_epochs 4 \
  --per_device_train_batch_size 16 \
  --per_device_eval_batch_size 32 \
  --do_train \
  --do_eval \
  --do_predict \
  --logging_steps 10 \
  --evaluate_during_training \
  --save_steps 10 \
  --overwrite_output_dir \
  --max_seq_length 128

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-43-1bd059d64d85> in <module>
      5     tokenizer=tokenizer,
      6     label_column_id=0,
----> 7     max_seq_length=200,
      8 )

<ipython-input-42-7c289105c656> in get_tfds(train_file, eval_file, test_file, tokenizer, label_column_id, max_seq_length)
     43                     padding="max_length",
     44                 ),
---> 45                 batched=True,
     46             )
     47     def gen_train():

~/anaconda3/envs/longtext-longformer/lib/python3.6/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1254                 fn_kwargs=fn_kwargs,
   1255                 new_fingerprint=new_fingerprint,
-> 1256                 update_data=update_data,
   1257             )
   1258         else:

~/anaconda3/envs/longtext-longformer/lib/python3.6/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    154             "output_all_columns": self._output_all_columns,
    155         }
--> 156         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    157         if new_format["columns"] is not None:
    158             new_format["columns"] = list(set(out.column_names) - unformatted_columns)

~/anaconda3/envs/longtext-longformer/lib/python3.6/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    161             # Call actual function
    162
--> 163             out = func(self, *args, **kwargs)
    164
    165             # Update fingerprint of in-place transforms + update in-place history of transforms

~/anaconda3/envs/longtext-longformer/lib/python3.6/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, update_data)
   1515                     try:
   1516                         batch = apply_function_on_filtered_inputs(
-> 1517                             batch, indices, check_same_num_examples=len(self.list_indexes()) > 0, offset=offset
   1518                         )
   1519                     except NumExamplesMismatch:

~/anaconda3/envs/longtext-longformer/lib/python3.6/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1433                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1434             processed_inputs = (
-> 1435                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1436             )
   1437             if not update_data:

<ipython-input-42-7c289105c656> in <lambda>(example)
     41                     truncation=True,
     42                     max_length=max_seq_length,
---> 43                     padding="max_length",
     44                 ),
     45                 batched=True,

~/anaconda3/envs/longtext-longformer/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2211             return_length=return_length,
   2212             verbose=verbose,
-> 2213             **kwargs,
   2214         )
   2215

~/anaconda3/envs/longtext-longformer/lib/python3.6/site-packages/transformers/tokenization_utils.py in _batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
    558                 ids, pair_ids = ids_or_pair_ids, None
    559             else:
--> 560                 ids, pair_ids = ids_or_pair_ids
    561
    562             first_ids = get_input_ids(ids)

ValueError: too many values to unpack (expected 2)
```
### suggestion for fix:
in line 60 - 69, my suggestion is as comment.
```
        for k in files.keys():
            transformed_ds[k] = ds[k].map(
                lambda example: tokenizer.batch_encode_plus(
                    (example[features_name[0]], features_name[1]), # it should be (example[features_name[0]], example[features_name[1]])
                    truncation=True,
                    max_length=max_seq_length,
                    padding="max_length",
                ),
                batched=True, # batched need to be set as True, I don't why batched=True doesn't work
            )
```
<!-- A clear and concise description of what you would expect to happen. -->


Recording training loss and perplexity during training

I'm fine-tuning GPT-2 text generation with the following command on Colab:
python run_language_modeling.py \
    --output_dir=$OUTPUT_DIR \
    --model_type=gpt2 \
    --model_name_or_path=$MODEL_NAME \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --per_gpu_train_batch_size=1 \
    --save_steps=-1 \
    --num_train_epochs=5

I was wondering if I can record the training loss, perplexity ... etc per epoch into a csv file or saved as a variable on Colab?
Thank you so much!


SQuAD example docs inaccurately suggest settings for bert-large-uncased on a single V100

Hi!

In the SQuAD example docs, it's noted that ( https://github.com/huggingface/transformers/blob/3f42eb979f7bd20448ff6b15ab316d63f5489a6f/docs/source/examples.md#fine-tuning-bert-on-squad10 ) `This example code fine-tunes BERT on the SQuAD1.0 dataset. It runs in 24 min (with BERT-base) or 68 min (with BERT-large) on a single tesla V100 16GB.`

Could just be me, but I don't think the example as provided works with `s/bert-base-uncased/bert-large-uncased/`---even with FP16, you run into a GPU OOM.

It might be useful to revisit this recommendation and/or remove it.


Corrected typo: maked → masked

# What does this PR do?

Fixed small typo in the BERT documentation.

## Before submitting
✅ This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@LysandreJik @sgugger


Trainer callback breaks old code

https://github.com/huggingface/transformers/blob/ba4bbd92bcb55febbfa06aaa1551738388ec7eb0/src/transformers/trainer_callback.py#L438-L447

Currently it depends on the fact that evaluate() will first call `self.prediction_loop` https://github.com/huggingface/transformers/blob/ba4bbd92bcb55febbfa06aaa1551738388ec7eb0/src/transformers/trainer.py#L1181

which will then call `.callback_handler.on_prediction_step `
https://github.com/huggingface/transformers/blob/ba4bbd92bcb55febbfa06aaa1551738388ec7eb0/src/transformers/trainer.py#L1270

But in my old code (3.1.0), I subclass Trainer and overwrite evaluate(), without calling self.prediction_loop.

And results in this error:
```
self.prediction_bar.close()
AttributeError: 'NoneType' object has no attribute 'close'
```

I propose we add `on_predict_begin`  and `on_predict_end`.


Strange error while using the `LongformerForMultipleChoice`

Hello,

I am trying to use `LongformerForMultipleChoice` model, and the code I am using is the following:
```python
# import the pre-trained HuggingFace Longformer tokenizer.
longformer_tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')

# get the pre-trained HuggingFace Longformer
best_model_longformer = LongformerForMultipleChoice.from_pretrained('allenai/longformer-base-4096',
                                                         output_hidden_states = True)

# my multiple choice question has 4 options.
question_list = [main_question, main_question, main_question, main_question]
options_list = [option1, option2, option3, option4]
mc_labels = torch.tensor([my_answer])

encoded_dict = longformer_tokenizer(question_list, options_list,
                             return_tensors = 'pt',
                             add_prefix_space = True,
                             padding = True)
input_hidden_state = best_model_longformer(
                       **{k: v.unsqueeze(0) for k,v in encoded_dict.items()},
                        labels = mc_labels)[2][0][:,:,:].detach()
```

and I am getting the error below:
```
/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py:71: UserWarning: This overload of nonzero is deprecated:
        nonzero()
Consider using one of the following signatures instead:
        nonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)
  sep_token_indices = (input_ids == sep_token_id).nonzero()
Traceback (most recent call last):
  File "SEED_125_V20_15_LONGFORMER.py", line 427, in <module>
    main_function('/home/ec2-user/G1G2.txt','/home/ec2-user/G1G2_answer_num.txt', num_iter)
  File "SEED_125_V20_15_LONGFORMER.py", line 389, in main_function
    best_model_longformer)
  File "SEED_125_V20_15_LONGFORMER.py", line 198, in fill_MC_loss_accuracy_tensor
    input_hidden_state = best_model_longformer(**{k: v.unsqueeze(0) for k,v in encoded_dict.items()}, labels = mc_labels)[2][0][:,:,:].detach()
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 1808, in forward
    loss = loss_fct(reshaped_logits, labels)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 948, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2422, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2218, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 1 is out of bounds.
```
How can I fix this error?

I also tried solving this issue with:
```python
# group question_list and options_list as a single list rather than specifying them seperately
encoded_dict = longformer_tokenizer([question_list, options_list],
                             return_tensors = 'pt',
                             add_prefix_space = True,
                             padding = True)
```
But this generates a different error, saying:
```
ValueError: 2 expected but found 1
```
PS: I don't think my Longformer model is correctly getting that my multiple-choice questions have 4 options...is there any way to make Longformer to take my mutiple-choice questions with the ones with 4 options (instead of 2 options)?

Thank you.

PS: I am interested in extracting hidden embedding more than the error or the logit themselves


GPT2DoubleHeadsModel documentation example question (error in documentation)?

Hello,
I was reading the documentation for the GPT2DoubleHeadsModel, and I have a question.
In the documentation, the example shown is:
```python
import torch
from transformers import GPT2Tokenizer, GPT2DoubleHeadsModel
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2DoubleHeadsModel.from_pretrained('gpt2, return_dict=True)

# Add a [CLS] to the vocabulary (we should train it also!)
num_added_tokens = tokenizer.add_special_tokens({'cls_token': '[CLS]'})
embedding_layer = model.resize_token_embeddings(len(tokenizer))  # Update the model embeddings with the new vocabulary size
choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]
encoded_choices = [tokenizer.encode(s) for s in choices]
cls_token_location = [tokens.index(tokenizer.cls_token_id) for tokens in encoded_choices]
input_ids = torch.tensor(encoded_choices).unsqueeze(0)  # Batch size: 1, number of choices: 2
mc_token_ids = torch.tensor([cls_token_location])  # Batch size: 1
outputs = model(input_ids, mc_token_ids=mc_token_ids)
lm_logits = outputs.lm_logits
mc_logits = outputs.mc_logits
```
Now, I don't see any main question statement in this example, although I see two multiple-choice options. The way I used to use `GPT2DoubleHeadsModel` is I first did `tokenizer(question_statement, option_statement)` and do `encoded_dict['input_ids']` to extract `input_ids` and similarly `encoded_dict['token_type_ids']` to extract the `token_type_ids`. Has this changed? I am getting an impression that the example is wrong (maybe the example could apply for BERT, but not GPT2DoubleHeadsModel). Is this an error in the documentation? I thought that, since GPT-2 does causal language modeling, question statement and option statement has to be encoded together, and put the `[CLS]` token at the end (usually), so that GPT-2 can apply the causal language modeling process to solve the multiple-choice problem.

Reading this again, I think the examples for BERTForMultipleChoice and GPT2DoubleHeadsModel are flipped.

 Thanks,


Fix check for xla in PreTrainedModel.save_pretrained()

# What does this PR do?

Added is_torch_tpu_available() to the condition for saving a model as xla model when calling `PreTrainedModel.save_pretrained()`
The `xla_device` property of `config` can also be `True` on a non-xla device, when loading a checkpoint that was previously trained and saved on xla.

Loading a model that was trained on xla was fixed previously with #5636 , this PR fixes the problem of saving such a model again.

Fixes #7695

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.


MLflow Trainer Callback

# 🚀 Feature request

A callback to log hyperparameters, metrics and cofings/weights to MLFlow, like the existing wandb and Tensorboard callbacks.

## Motivation

I use MLFlow as my primary experiment tracking tool. It is convenient to run on a remote server and log the results from any of your training machines, andit also facilitates collaboration.

Trainer is an amazing tool, it makes it very simple to train models, however, the only way to modify the training loop to include custom logging seems to add a callback.

## Your contribution

I can contribute a PR.


tokenizers dependency warning: `transformers 3.3.1 has requirement tokenizers==0.8.1.rc2, but you'll have tokenizers 0.9.0`

## Environment info
- `transformers` version: 3.3.1
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 tokenizers: @mfuntowicz

## Information

Hey all, it'll probably be fixed soon but when updating to `tokenziers` 0.9.0 with `pip install tokenizers --upgrade` I get:

`ERROR: transformers 3.3.1 has requirement tokenizers==0.8.1.rc2, but you'll have tokenizers 0.9.0 which is incompatible.`

## To reproduce

Steps to reproduce the behavior:

1. pip Install transformers --upgrade
2. pip install tokenizers --upgrade

## Expected behavior

Expected compatibility between transformers 3.3.1 and tokenizers 0.9.0


Minor spelling corrections in docstrings. "information" is uncountable in English and has no plural.


save_pretrained() does not check if xla is available

## Environment info

- `transformers` version: 3.3.1
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.4
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [ X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. load any model trained on TPU with `BertModel.from_pretrained(tpu_checkpoint_path)`
2. run/train the model - works fine
3. save the model with `model.save_pretrained(save_path)`

```
line 720, in save_pretrained
    import torch_xla.core.xla_model as xm
ModuleNotFoundError: No module named 'torch_xla'
```
## Expected behavior

I am pretraining a LM on TPU, and for the downstream task fine-tuning I load the saved checkpoints on a non-TPU device.
Loading works fine now (#5636), but saving again does not.

`save_pretrained` should check whether the device is still xla - the original config attribute that is used in `save_pretrained` to check for the device persists when loading the xla model on another device:
```
getattr(config, 'xla_device')
True
```
It is easy to fix by changing the config attribute `setattr(config, 'xla_device', False)` in the script, but I would still consider it a bug.


Fix docstring in AutoModel class

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes doc string for the class AutoModel

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

 @sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


How to get the word embedding  after pre-training ? for example, a embedding matrix

I am excited on this great model. And I want to get the word embedding . Where shold I find the file from output or should I change to code to do this?


Fail to run text classification example with run_tf_text_classification

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Ubuntu 18.04
- Python version: 3.6.9
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.3.1
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Distributed

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

Text classification with own dataset

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

I use my own datasets {train/dev/test}/.csv and run `run_tf_text_classification.py`, the training seems OK, while error occurs while evaluation as below:

> 2020-10-10 07:33:15.368292: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at resource_variable_ops.cc:537 : Not found: Resource localhost/_AnonymousVar110/N10tensorflow3VarE does not exist.
Traceback (most recent call last):
  File "run_tf_text_classification.py", line 292, in <module>
    main()
  File "run_tf_text_classification.py", line 267, in main
    trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 592, in train
    self.evaluate()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 438, in evaluate
    output = self.prediction_loop(eval_ds, steps, num_examples, description="Evaluation")
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 327, in prediction_loop
    logits = self.distributed_prediction_steps(batch)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 814, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.NotFoundError:  Resource localhost/_AnonymousVar110/N10tensorflow3VarE does not exist.
	 [[node AssignAddVariableOp (defined at /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:457) ]] [Op:__inference_distributed_prediction_steps_11885]

Function call stack:
distributed_prediction_steps


Seq2Seq Example with Bart not Saving Best Model

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@sshleifer

## Information

Model I am using (Bert, XLNet ...): Bart

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

I am using a slightly modified version of the examples/seq2seq/finetune_bart_tiny.sh script, where I just add the `--val_check_interval 0.1 --do_predict` flags to the finetune.py call:

```
python finetune.py \
--data_dir=cnn_tiny/ \
--model_name_or_path=sshleifer/bart-tiny-random \
--learning_rate=3e-5 \
--train_batch_size=2 \
--eval_batch_size=2 \
--output_dir=$OUTPUT_DIR \
--num_train_epochs=1  \
--gpus=0 \
--val_check_interval 0.1 \
--do_train --do_predict "$@"
```
Which is supposed to save the best performing model based on the val_check_interval and then evaluate the model, as is done in the regular `finetune.sh` script (thought the error is also in this one as well, I am using the tiny version so that it is easier to see the issue).

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: tiny-cnn
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Go through this google colab: https://colab.research.google.com/drive/1xtyvXI6gNAJpSkqYi_0ieWkMFRw3OSm2?usp=sharing

```
._cnn_tiny
cnn_tiny/
cnn_tiny/._train.target
cnn_tiny/train.target
cnn_tiny/._train.source
cnn_tiny/train.source
cnn_tiny/._val.source
cnn_tiny/val.source
cnn_tiny/._val.target
cnn_tiny/val.target
cnn_tiny/._test.source
cnn_tiny/test.source
cnn_tiny/._test.target
cnn_tiny/test.target
Epoch 0:  17%|█▋        | 1/6 [00:00<00:02,  2.20it/s, loss=10.839, v_num=1]
Validating: 0it [00:00, ?it/s]
Epoch 0:  33%|███▎      | 2/6 [00:00<00:01,  2.02it/s, loss=10.839, v_num=1]
Epoch 0:  50%|█████     | 3/6 [00:01<00:01,  2.07it/s, loss=10.839, v_num=1]
Epoch 0:  67%|██████▋   | 4/6 [00:01<00:00,  2.33it/s, loss=10.837, v_num=1]
Validating: 0it [00:00, ?it/s]
Epoch 0:  83%|████████▎ | 5/6 [00:02<00:00,  2.24it/s, loss=10.837, v_num=1]
Epoch 0: 100%|██████████| 6/6 [00:02<00:00,  2.28it/s, loss=10.837, v_num=1]
Epoch 0: 100%|██████████| 6/6 [00:02<00:00,  2.28it/s, loss=10.837, v_num=1]
--2020-10-10 02:28:52--  https://cdn-datasets.huggingface.co/summarization/cnn_tiny.tgz
Resolving cdn-datasets.huggingface.co (cdn-datasets.huggingface.co)... 13.227.209.120, 13.227.209.109, 13.227.209.124, ...
Connecting to cdn-datasets.huggingface.co (cdn-datasets.huggingface.co)|13.227.209.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23131 (23K) [application/x-tar]
Saving to: ‘cnn_tiny.tgz’

     0K .......... .......... ..                              100% 44.4M=0s

2020-10-10 02:28:52 (44.4 MB/s) - ‘cnn_tiny.tgz’ saved [23131/23131]

2020-10-10 02:28:54.290821: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
/usr/local/lib/python3.6/dist-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: The validation_epoch_end should not return anything as of 9.1.to log, use self.log(...) or self.write(...) directly in the LightningModule
  warnings.warn(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: The {log:dict keyword} was deprecated in 0.9.1 and will be removed in 1.0.0
Please use self.log(...) inside the lightningModule instead.

# log on a step or aggregate epoch metric to the logger and/or progress bar
# (inside LightningModule)
self.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True)
  warnings.warn(*args, **kwargs)
Traceback (most recent call last):
  File "finetune.py", line 440, in <module>
    main(args)
  File "finetune.py", line 429, in main
    trainer.test()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 728, in test
    results = self.__test_using_best_weights(ckpt_path, test_dataloaders)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 740, in __test_using_best_weights
    'ckpt_path is "best", but ModelCheckpoint is not configured to save the best model.'
pytorch_lightning.utilities.exceptions.MisconfigurationException: ckpt_path is "best", but ModelCheckpoint is not configured to save the best model
```

## Expected behavior
The script should save the model with the best performing validation loss and should then use this saved model for evaluation against a test set. This is the same case for the regular `finetune.sh` script. This was working as of Oct 4/5th, but stopped sometime after.

Any help with this issue would be greatly appreciated!


RAG Tokenizer erroring out

## Environment info

- `transformers` version: 3.3.1
- Platform: Linux-5.4.0-48-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

@ola13 @mfuntowicz

## Information

Hi- I am trying to get the RAG running, however I am getting the error when I follow the instructions here: <https://huggingface.co/facebook/rag-token-nq>

Particularly, the error message is as follows:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-35cd6a2213c0> in <module>
      1 from transformers import AutoTokenizer, AutoModelWithLMHead
      2
----> 3 tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-nq")

~/src/transformers/src/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    258                 return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    259             else:
--> 260                 return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    261
    262         raise ValueError(

~/src/transformers/src/transformers/tokenization_rag.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
     61         print(config.generator)
     62         print("***")
---> 63         generator = AutoTokenizer.from_pretrained(generator_path, config=config.generator)
     64         return cls(question_encoder=question_encoder, generator=generator)
     65

~/src/transformers/src/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    258                 return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    259             else:
--> 260                 return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    261
    262         raise ValueError(

~/src/transformers/src/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1557
   1558         return cls._from_pretrained(
-> 1559             resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs
   1560         )
   1561

~/src/transformers/src/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs)
   1648
   1649         # Add supplementary tokens.
-> 1650         special_tokens = tokenizer.all_special_tokens
   1651         if added_tokens_file is not None:
   1652             with open(added_tokens_file, encoding="utf-8") as added_tokens_handle:

~/src/transformers/src/transformers/tokenization_utils_base.py in all_special_tokens(self)
   1026         Convert tokens of :obj:`tokenizers.AddedToken` type to string.
   1027         """
-> 1028         all_toks = [str(s) for s in self.all_special_tokens_extended]
   1029         return all_toks
   1030

~/src/transformers/src/transformers/tokenization_utils_base.py in all_special_tokens_extended(self)
   1046         logger.info(all_toks)
   1047         print(all_toks)
-> 1048         all_toks = list(OrderedDict.fromkeys(all_toks))
   1049         return all_toks
   1050

TypeError: unhashable type: 'dict'
```

 `all_toks` variable looks as follows. Obviously, it is a dictionary and `OrderedDict.fromkeys` doesn't like it.
```
[{'content': '<s>', 'single_word': False, 'lstrip': False, 'rstrip': False, 'normalized': True}, {'content': '</s>', 'single_word': False, 'lstrip': False, 'rstrip': False, 'normalized': True}, {'content': '<unk>', 'single_word': False, 'lstrip': False, 'rstrip': False, 'normalized': True}, {'content': '</s>', 'single_word': False, 'lstrip': False, 'rstrip': False, 'normalized': True}, {'content': '<pad>', 'single_word': False, 'lstrip': False, 'rstrip': False, 'normalized': True}, {'content': '<s>', 'single_word': False, 'lstrip': False, 'rstrip': False, 'normalized': True}, {'content': '<mask>', 'single_word': False, 'lstrip': True, 'rstrip': False, 'normalized': True}]
```
I will be digging deeper, hoping that I am doing an obvious mistake.

## To reproduce

```
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-nq")
```

## Expected behavior

It should load the tokenizer!

Thank you.


Fix flaky test in test_trainer

# What does this PR do?

As investigated with @LysandreJik today, the corresponding test was flaky because `loggin_dir` has a default that depends of time (so the test fails if the two Trainer are instantiated just before a new minute and just after).


Adds license information for default and distilbert models

model card

Related to pretrained model cards

# What does this PR do?

Adds license information for default and `distilbert*` models. A follow up to #7668.

- Apache 2.0 for `distilbert*` based on https://github.com/huggingface/transformers/issues/3357#issuecomment-614856396
- MIT for `facebook/bart-large-mnli` based on https://github.com/huggingface/transformers/issues/7668#issuecomment-706064737

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case. #7668
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Model Cards: @julien-c


Fix title level in Blenderbot doc

# What does this PR do?

The navigation bar is a bit crazy because the documentation of BlenderBot puts title and sections at the same level. This PR fixes that.
@LysandreJik merging as soon as it's green because it's a small fix, tagging you so you're aware.


When downloading RAG dpr indexes,  there is a pickle file loading error

When I try to finetune RAG with following code:

```
self.config = RagConfig.from_pretrained("facebook/rag-token-base", n_docs=2, use_dummy_dataset=False)
self.tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-base")
self.tokenizer.pad_token_id =  AutoTokenizer.from_pretrained("facebook/bart-large").pad_token_id
retriever = RagRetriever.from_pretrained("facebook/rag-token-base" use_dummy_dataset=False)
self.model = RagTokenForGeneration.from_pretrained("facebook/rag-token-base", retriever=retriever, config=self.config)
```
An error occurs, which seems that a downloaded pickle file cannot be loaded. I put the error message below.
I assigned 200GB memory so it should not be an memory issue. I'm not sure if this is a trivial error due to my own implementation bugs or it is more common. Thank you very much!!

--------------
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/numpy/lib/npyio.py", line 447, in load return pickle.load(fid, **pickle_kwargs)
_pickle.UnpicklingError: pickle data was truncated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/builder.py", line 553, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/builder.py", line 841, in _prepare_split
    generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/tqdm/std.py", line 1130, in __iter__
    for obj in iterable:
  File ".cache/huggingface/modules/datasets_modules/datasets/wiki_dpr/14b973bf2a456087ff69c0fd34526684eed22e48e0dfce4338f9a22b965ce7c2/wiki_dpr.py", line 132, in _generate_examples
    vecs = np.load(open(vectors_files.pop(0), "rb"), allow_pickle=True)
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/numpy/lib/npyio.py", line 450, in load
    "Failed to interpret file %s as a pickle" % repr(file))
OSError: Failed to interpret file <_io.BufferedReader name='.cache/huggingface/datasets/downloads/cd4183aaa482e0e3724cb8b2efafc6c762914aabed38c16a41f922ff7d5e90f9'> as a pickle

Traceback (most recent call last):
  File "src/finetune.py", line 432, in <module>
    main(args)
  File "src/finetune.py", line 371, in main
    model: SummarizationModule = SummarizationModule(args)
  File "src/finetune.py", line 73, in __init__
    super().__init__(hparams, num_labels=None, mode=self.mode, **kwargs)
  File "src/lightning_base.py", line 130, in __init__
    retriever = RagRetriever.from_pretrained("facebook/rag-token-base", use_dummy_dataset=False)
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/retrieval_rag.py", line 310, in from_pretrained
    config, question_encoder_tokenizer=question_encoder_tokenizer, generator_tokenizer=generator_tokenizer
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/retrieval_rag.py", line 301, in __init__
    self.init_retrieval()
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/retrieval_rag.py", line 327, in init_retrieval
    self.index.init_index()
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/retrieval_rag.py", line 241, in init_index
    dummy=self.use_dummy_dataset,
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/builder.py", line 555, in _download_and_prepare
    raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))
OSError: Cannot find data file.
srun: error: node006: task 0: Exited with exit code 1


Using PaddingStrategy and TruncationStrategy throws an UnboundLocalError in tokenizers

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@mfuntowicz I am almost sure I am using it right. But after looking at the code I found that there are two variables that are being accessed before assignment. Thanks in advance.

## Information

This is exactly what I am doing:

I am trying to load a tokenizer using `AutoTokenizer` and encode a single string. I am using a pretrained tokenizer `distilbert-base-uncased`

## To reproduce

Steps to reproduce the behavior:

This is exactly what I am running:

```
from transformers import AutoTokenizer
from transformers.tokenization_utils_base import PaddingStrategy

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased", use_fast=True)
encoded_sentence = tokenizer.encode_plus(
        "some input text",
        return_attention_mask=True,
        padding=PaddingStrategy.MAX_LENGTH,
        add_special_tokens=True,
        max_length=20,
        return_token_type_ids=True
    )
print(encoded_sentence)
```

This throws the following error:

```
Traceback (most recent call last):
  File "......", line 22, in <module>
    return_token_type_ids=True
  File ".../.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2029, in encode_plus
    **kwargs,
  File "..../.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1837, in _get_padding_truncation_strategies
    if padding_strategy != PaddingStrategy.DO_NOT_PAD and (not self.pad_token or self.pad_token_id < 0):
UnboundLocalError: local variable 'padding_strategy' referenced before assignment
```
I have padded some of the paths to hide my local directories.

The same thing happens for truncation too. E.g.,

```
from transformers import AutoTokenizer
from transformers.tokenization_utils_base import TruncationStrategy
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased", use_fast=True)
encoded_sentence = tokenizer.encode_plus(
        "some input text",
        return_attention_mask=True,
        add_special_tokens=True,
        max_length=20,
        return_token_type_ids=True,
        truncation=TruncationStrategy.LONGEST_FIRST
    )
print(encoded_sentence)
```

This raises the following error:

```
Traceback (most recent call last):
  File ".../scratch.py", line 22, in <module>
    truncation=TruncationStrategy.LONGEST_FIRST
  File ".../.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2029, in encode_plus
    **kwargs,
  File ".../.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1846, in _get_padding_truncation_strategies
    truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE
UnboundLocalError: local variable 'truncation_strategy' referenced before assignment
```

## Expected behavior

This should not throw an exception like this. I looked at the code as well. I kind of know what is the actual issue.


Error with running run_language_modeling.py on GCP TPU

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: GCP
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0
- Tensorflow version (GPU?): -
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: Yes/NO

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@sgugger @LysandreJik
## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ run_language_modeling.py ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [fine tuning BERT on Wikitext103 ] my own task or dataset: (give details below)

## To reproduce

I am trying to fine-tune Bert on the Wikitext-103 dataset by running the example code run_language_modeling.py on Google cloud TPU v3-8 using xla_spawn.py launcher. I tried both num_cores =1 and num_cores > 1, but neither worked properly.

Steps to reproduce the behavior:

1. python xla_spawn.py --num_cores 8 \
 run_language_modeling.py \
--model_name_or_path=bert-base-uncased \
--do_train \
--train_data_file\
--do_eval \
--eval_data_file\
--mlm\
--per_device_train_batch_size=4

The full output is long and you can find it [here](https://gofile.io/d/tjovN1). Here is the begining of error message:

Iteration:   0%|          | 1/28026 [00:00<3:41:44,  2.11it/s]2020-10-09 16:05:45.623409: W
  2449 tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:160] RPC failed with status = "Unavailable: Socket closed" and grpc_error_string = "{"created":"@1602259545.623281190","description":"Error received from peer ipv4:10.48.142.250:8470","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}", maybe retrying the RPC
2020-10-09 16:06:09.642115: E    2449 tensorflow/compiler/xla/xla_client/xla_util.cc:76] >>> Dumping Computation 0
2020-10-09 16:06:09.642213: E    2449 tensorflow/compiler/xla/xla_client/xla_util.cc:76] HloModule SyncTensorsGraph.23679, input_output_alias={ {0}: (39, {}, may-alias), {1}: (37, {}, may-alias),
......

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Running without error!
<!-- A clear and concise description of what you would expect to happen. -->
Thanks for any help.


Gpt1 for sequence classification

# What does this PR do?

Adds sequence classification architecture for GPT-1,
Strongly based on modifications made in #7501

Fixes #7623 (issue) (Partially)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@LysandreJik Here is the new PR without that merge problem, let me know if there is anything that should be changed =)

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Fine-tuning

I read lot of articles on pre-training vs fine-tuning but I am still not able to get the meaning in the context of transformer models, I understand the pre-training is training the dataset from scratch , while (point of confusion) the fine-tuning is using the pre-trained model/data and add on the top of that with our own custom dataset?
Then it comes the downstream tasks, any fine line of distinctions among these three terms pre-training, finetuning and downstream tasks
please clarify


Delete extra test file in repo root


Better links for models in README and doc index

# What does this PR do?

This PR fixes the links to the docs for unreleased models and makes the automatic copy to the index.rst a little bit better (by using relative links so there is no jump in version).
It adds instructions in the setup.py to clean the master in the links for unreleased models at the time of a release, we will just need to remember to have the right links in the README in a PR that adds a new model.

<!-- Remove if not applicable -->

Fixes #7657


TFEncoderDecoder

# 🚀 Feature request

Tensorflow version for the `EncoderDecoder` class for sequence to sequence text generation.

## Motivation
I am replicating [this](https://arxiv.org/pdf/1907.12461.pdf) paper which studies several combinations of encoder and decoder with pretrained checkpoints. Most of it is implemented very nicely in the current API, but is not available for tensorflow. The closest thing I found was `TFT5ForConditionalGeneration` which works nicely. Are there any plans to extend this for other pretrained models such as Bert, Roberta, GPT(2)?

## Your contribution
Happy to work on a PR if someone can provide some pointers on where to start and best practices to include new models in the existing API.


Fix dataset cardinality

# What does this PR do?

This PR fixes an issue with the generic text classification example where the size of the datasets were not properly set.

Fixes #7637


Batch and smart batch support for pipelines.

# 🚀 Feature request

## Motivation
I want to use `TextClassificationPipeline` to classify a large number of texts.
The naive approach is:

```python
model = AutoModelForSequenceClassification.from_pretrained(model_dir)
tokenizer = AutoTokenizer.from_pretrained(model_dir)
pipeline = TextClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    framework="pt",
    device=0,
)
results = pipeline(unlabeled_text_list)
```

But this gives me CUDA OOM error when `unlabeled_text_list` is long.

What about adding batch support that lets you specify the batch size and maybe also support for multiprocessing tokenization?

When possible smart batching would be nice. See this: https://github.com/UKPLab/sentence-transformers/issues/454#issuecomment-699496454

What do you think?


TFTrainer doesn't work

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik
 Trainer: @sgugger
 tensorflow: @jplu
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)
Protein Sequence dataset

## To reproduce

Steps to reproduce the behavior:
https://colab.research.google.com/drive/1v0FMM_iuRSixvDaoHaiP77pel7qkoYL8?usp=sharing
```
WARNING:tensorflow:TPU system grpc://10.12.199.226:8470 has already been initialized. Reinitializing the TPU can cause previously created variables on TPU to be lost.
WARNING:tensorflow:TPU system grpc://10.12.199.226:8470 has already been initialized. Reinitializing the TPU can cause previously created variables on TPU to be lost.
INFO:tensorflow:Initializing the TPU system: grpc://10.12.199.226:8470
INFO:tensorflow:Initializing the TPU system: grpc://10.12.199.226:8470
INFO:tensorflow:Clearing out eager caches
INFO:tensorflow:Clearing out eager caches
INFO:tensorflow:Finished initializing TPU system.
INFO:tensorflow:Finished initializing TPU system.
WARNING:absl:`tf.distribute.experimental.TPUStrategy` is deprecated, please use  the non experimental symbol `tf.distribute.TPUStrategy` instead.
INFO:tensorflow:Found TPU system:
INFO:tensorflow:Found TPU system:
INFO:tensorflow:*** Num TPU Cores: 8
INFO:tensorflow:*** Num TPU Cores: 8
INFO:tensorflow:*** Num TPU Workers: 1
INFO:tensorflow:*** Num TPU Workers: 1
INFO:tensorflow:*** Num TPU Cores Per Worker: 8
INFO:tensorflow:*** Num TPU Cores Per Worker: 8
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)
INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec453cbf60>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec453cb710>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec453cb7b8>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b73128>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b730b8>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b73048>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec453cb9b0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec453cb5c0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec453cbef0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec453cb518>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec453cb4e0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b734e0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b73470>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b73400>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b73390>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b73320>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b732b0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b73240>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
Exception ignored in: <bound method EagerResourceDeleter.__del__ of <tensorflow.python.ops.resource_variable_ops.EagerResourceDeleter object at 0x7fec44b731d0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 293, in __del__
    self._handle, ignore_lookup_error=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 252, in destroy_resource_op
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: stream is uninitialized or in an error state [Op:DestroyResourceOp]
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
<ipython-input-18-06617a21566a> in <module>()
     11
     12 with training_args.strategy.scope():
---> 13   model = TFAutoModelForSequenceClassification.from_pretrained(model_name, from_pt=True)
     14
     15 trainer = TFTrainer(

26 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in shape(self)
   1165         # `_tensor_shape` is declared and defined in the definition of
   1166         # `EagerTensor`, in C.
-> 1167         self._tensor_shape = tensor_shape.TensorShape(self._shape_tuple())
   1168       except core._NotOkStatusException as e:
   1169         six.raise_from(core._status_to_exception(e.code, e.message), None)

InternalError: RET_CHECK failure (platforms/xla/service/jellyfish/bounds_check.cc:427) allocation_size_words <= std::numeric_limits<int32>::max()
```

## Expected behavior

I am following the example for fine-tuning custom dataset:
https://huggingface.co/transformers/custom_datasets.html

It works with Pytorch, but with tensorflow it doesn't work. Using TPU it gives the above error, and using GPU it just doesn't start.

Any idea what I did wrong ?


Add FAVOR+ / Performer attention

# 🌟 FAVOR+ / Performer attention addition

Are there any plans to add this new attention approximation block to Transformers library?

## Model description
The new attention mechanism with linear time and space complexity was introduced in
 _Rethinking Attention with Performers_ [[https://arxiv.org/abs/2009.14794](https://arxiv.org/abs/2009.14794)].
Authors of the paper claim that the new attention mechanism is backward-compatbile with already existing models
> Backwards compatibility with pretrained models is available as a benefit from softmax approximation, via small finetuning (required due to error propagation)

<!-- Important information -->

## Open source status

* [x] the model implementation is available: it's an original Trax implementation from Google: https://github.com/google-research/google-research/tree/master/performer/fast_self_attention
* [ ] the model weights are available: probably not required as it's a building block for models rather than fully new architecture
* [x] who are the authors: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller


Correctly tokenize sentence pairs

Hey,

I saw different ways to tokenize sentence pairs and the intuitive one is not shown here:

https://huggingface.co/transformers/preprocessing.html#preprocessing-pairs-of-sentences

So, I am asking here if I do right.

I encode pairs of sentence using list of lists. Instead handing over two seperate lists for each sentences, I handover a list of list, where each element is a list with  a single sentence pair. So:

pairs=[[sen1, sen2],[sen1,sen2],....]

Is this hopefully right too?


squad data preprocessor error (list index out of range) while finetuning bert on squad 1.1

run_squad.py throws this error on squad v1.1 dataset

Traceback (most recent call last):
  File "run_squad.py", line 820, in <module>
    main()
  File "run_squad.py", line 762, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
  File "run_squad.py", line 446, in load_and_cache_examples
    examples = processor.get_train_examples(args.data_dir, filename=args.train_file)
  File "/home/din/question_answering_deepQA/venv_indic_deepQA/lib/python3.6/site-packages/transformers/data/processors/squad.py", line 602, in get_train_examples
    return self._create_examples(input_data, "train")
  File "/home/din/question_answering_deepQA/venv_indic_deepQA/lib/python3.6/site-packages/transformers/data/processors/squad.py", line 656, in _create_examples
    answers=answers,
  File "/home/din/question_answering_deepQA/venv_indic_deepQA/lib/python3.6/site-packages/transformers/data/processors/squad.py", line 729, in __init__
    self.start_position = char_to_word_offset[start_position_character]
IndexError: list index out of range