Bug SqueezeBERT stops with no error

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Ubuntu
- Python version: anaconda python 3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in the script?:

The GPUs available were :
```
Geforce GTX 980 4gb
Geforce GTX Titan 12gb
```
```
transformers == 4.1.1
torch==1.7.0
torchvision == 0.8.1
```

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): SqueezeBERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
using [Sequence Classification with IMDb Reviews
](https://huggingface.co/transformers/custom_datasets.html#seq-imdb) example I have made my own script
* [x] my own modified scripts: (give details below)
The only changes I have made in this script are

1.  to use Yelp dataset,
2. use SqueezeBERT instead of DistilBERT,
3. also do a 5label sentiment...

```python
training_args = TrainingArguments(
    output_dir='./SqueezeBERT_10ep_result',          # output directory
    per_device_train_batch_size=3,  # batch size per device during training
    per_device_eval_batch_size=3,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./SqueezeBERT_10ep_log',            # directory for storing logs
    logging_steps=500,
    num_train_epochs=10,              # total number of training epochs
    evaluation_strategy="epoch",
    do_train=True,
    do_eval=True,
)

model = SqueezeBertForSequenceClassification.from_pretrained('squeezebert/squeezebert-mnli-headless', return_dict=True)
model.num_labels = 5
model.classifier = nn.Linear(768,5)

from sklearn.metrics import accuracy_score, precision_recall_fscore_support

def compute_metrics(pred):

    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='macro')
    acc = accuracy_score(labels, preds)

    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

print("Displaying model architecture... !\n")
print(model)
print("Training model starting...!\n")

trainer = Trainer(
    model=model,             # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset,             # evaluation dataset
    compute_metrics=compute_metrics,
)

trainer.train()

```
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Using the yelp full dataset

## To reproduce

Steps to reproduce the behavior:

1. Run the scripts as mentioned
2. Reaching Epoch 3, it will suddenly stop using the GPU and although no error is showing up, nothing changes...
3. the Last checkpoint that saves is `checkpoint-310000`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
it should have just go on and finished the training process.


Update the README of the text classification example

# What does this PR do?

This PR updates the main README of the examples folder and the one in the text classification example to take into account the recent changes in the scripts. In particular, I re-ran the command shown for all tasks with/without FP16 to make a clean table of results.

I moved all stuff about distributed training/TPUs in the general README of the examples as all example scripts now use Trainer, so have this working out of the box.


mBART finetuned on XSUM

## Environment info

- `transformers` version: 4.1.0.dev0
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0.dev20201216+cu110 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: distributed

### Who can help

 mBART: @patrickvonplaten
 examples/seq2seq: @patil-suraj

## Information

Model I am using: mBART

The problem arises when using:
* [ X ] the official example scripts:
 I used the official seq2seq training example here:
 https://github.com/huggingface/transformers/tree/master/examples/seq2seq
* [ X ] my own modified scripts: (give details below)
my training script is as follows (no changes to finetune_trainer.py):
```shell
python -m torch.distributed.launch --nproc_per_node=2 finetune_trainer.py \
    --data_dir "./xsum" \
    --output_dir "./my_models" \
    --overwrite_output_dir \
    --model_name_or_path "facebook/mbart-large-cc25" \
    --fp16 \
    --freeze_encoder \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --learning_rate=3e-5 \
    --do_train --do_eval --do_predict \
    --evaluation_strategy steps \
    --predict_with_generate \
    --n_val 1000 \
    --max_target_length=60 \
    --val_max_target_length=60 \
    --test_max_target_length=100 \
    "$@"
```
The tasks I am working on is:
* [ X ] XSUM

## To reproduce

Steps to reproduce the behavior:

1. follow https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md
2. launch training script with my modifications (batch_size, freeze_encoder, max_target_length ..)
3. inference on two texts (french and english) using the following code:
```python
def sum_mbart_xsum(text):
    print("---------------------------------------------------------------------------------")
    print("                                MBART large xsum                                  ")
    print("---------------------------------------------------------------------------------")

    tokenizer = MBartTokenizer.from_pretrained("/home/mohamed/Desktop/Summarization/mbart-xsum")
    model = MBartForConditionalGeneration.from_pretrained("/home/mohamed/Desktop/Summarization/mbart-xsum")

    article_input_ids = tokenizer.batch_encode_plus([text], return_tensors='pt', max_length=1024, truncation=True)[
        'input_ids']
    summary_ids = model.generate(article_input_ids,
                                 num_beams=6,
                                 length_penalty=1.0,
                                 max_length=142,
                                 no_repeat_ngram_size=3)

    summary_txt = tokenizer.decode(summary_ids.squeeze(), skip_special_tokens=True)
    return summary_txt

```
## Results
1- eval/test results:
```jsonc
{
    "epoch": 3.0,
    "test_gen_len": 28.1,
    "test_loss": 1.7692,
    "test_n_ojbs": -1,
    "test_rouge1": 32.7618,
    "test_rouge2": 12.022,
    "test_rougeL": 25.6512,
    "test_rougeLsum": 25.6499,
    "test_runtime": 2778.8939,
    "test_samples_per_second": -0.0,
    "train_n_ojbs": -1,
    "train_runtime": 94633.1507,
    "train_samples_per_second": -0.0,
    "val_gen_len": 28.0,
    "val_loss": 1.7993,
    "val_n_ojbs": 1000,
    "val_rouge1": 32.9862,
    "val_rouge2": 11.528,
    "val_rougeL": 25.6517,
    "val_rougeLsum": 25.7055,
    "val_runtime": 267.0092,
    "val_samples_per_second": 3.745
}
```
2- Inference:
* Out of context summarizations (gives sth related to training data) --> (sth wrong with my finetuning configuration or inference function?)
* For French texts, results are in English and poor summary --> (why the language changes?)

## Expected behavior

My main objective of finetuning mBART on Xsum is to evaluate the multilingual level of mBART. Basically answering the following question: Should I finetune on a dataset with multiple languages to be able to summarize in multiple languages? Or the multilingual characteristic is preserved with mBART and just finetune on english (xsum) only
Current problems:
1- (inference results and questions above)
2- Why `facebook/bart-large-xsum` understands french (even if bart is trained on english)?


run_mlm.py crashes when saving model checkpoint

## Environment info

- `transformers` version: 4.0.1
- Platform: Google Cloud
- Python version: 3.6.10
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?):
- Using GPU in script?: NO. Using TPU
- Using distributed or parallel set-up in script?: YES

### Who can help
@LysandreJik  @mfuntowicz @sgugger

## Information

I'm trying to train al Albert model from scratch, with a custom tokenizer, on Google Cloud TPUS. The problem arises when saving the model checkpoints, more specifically when trying to save the tokenizer. I'm using your example script run_mlm.py.

The problem arises when using:
* [ ] the official example scripts: run_mlm.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Masked Language Modelling.

## To reproduce

Steps to reproduce the behavior:

1. Run run_mlm.py with the following params: python transformers/examples/xla_spawn.py --num_cores 8 \
    transformers/examples/language-modeling/run_mlm.py \
    --model_type albert \
    --train_file texts_train.txt  \
    --validation_file good_texts_valid.txt \
    --output_dir modelo_prueba \
    --tokenizer_name ./tokenizadores/definitivo  \
    --overwrite_output_dir \
    --line_by_line \
    --pad_to_max_len \
    --do_train \
    --do_eval \
    --evaluation_strategy steps \
    --per_device_train_batch_size 32 \
    --per_device_eval_batch_size 32 \
    --learning_rate 1e-3 \
    --max_steps 500 \
    --save_steps 100 \
    --save_total_limit 15 \
    --overwrite_cache \
    --max_seq_length 512 \
    --eval_accumulation_steps 10 \
    --logging_steps 100 \
    --config_name ./config/albert-base-v2.json \

At step 100, the following error arises:

```
INFO|trainer.py:1141] 2020-12-21 15:46:34,157 >> Saving model checkpoint to modelo_prueba/checkpoint-100
[INFO|configuration_utils.py:281] 2020-12-21 15:46:34,158 >> Configuration saved in modelo_prueba/checkpoint-100/config.json
[INFO|modeling_utils.py:741] 2020-12-21 15:46:34,556 >> Model weights saved in modelo_prueba/checkpoint-100/pytorch_model.bin
Exception in device=TPU:0: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 330, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 324, in _start_fn
    fn(gindex, *args)
  File "/home/alejandro_vaca/transformers/examples/language-modeling/run_mlm.py", line 405, in _mp_fn
    main()
  File "/home/alejandro_vaca/transformers/examples/language-modeling/run_mlm.py", line 379, in main
    trainer.train(model_path=model_path)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 777, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 848, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 869, in _save_checkpoint
    self.save_model(output_dir)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 1135, in save_model
    self._save_tpu(output_dir)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 1157, in _save_tpu
    self.tokenizer.save_pretrained(output_dir)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1972, in save_pretrained
    filename_prefix=filename_prefix,
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py", line 524, in _save_pretrained
    vocab_files = self.save_vocabulary(save_directory, filename_prefix=filename_prefix)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/models/albert/tokenization_albert_fast.py", line 252, in save_vocabulary
    if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

## Expected behavior

The expected behavior is that the script doesn't crash. Moreover, it's completely unnecessary to save the tokenizer in trainer.py, as the tokenizer is already trained and doesn't need to be saved again.


Fix TF template

# What does this PR do?

Fix the TF template for the new einsum dense layer.


[MPNet] Add slow to fast tokenizer converter

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9194

This PR adds a converter from slow to fast MPNetTokenizers. This way fast tokenizers can be correctly serialized and loaded again. To prevent future issues like #9194, we should maybe think about not allowing to add a "FastTokenizer" without a corresponding converter...what do you think @sgugger, @LysandreJik, @thomwolf ?

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


command line_by_line missing in https://github.com/huggingface/transformers/tree/master/examples/language-modeling

## Environment info
- `transformers` version: 3.5.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@LysandreJik @patrickvonplaten @TevenLeScao

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
https://github.com/huggingface/transformers/tree/master/examples/language-modeling

## To reproduce

In the old version of the script ``run_clm.py`` called ``run_language_modeling`` there was an argument ``line_by_line`` which allowed to read the data by putting each sequence on a line. This argument seems to be missing in the newer version
``run_clm.py``.

## Expected behavior
Perhaps there is an argument that has replaced ``line_by_line``  but I don't really see that. Sorry if I missed something.


[T5] Fix warning for changed EncDec Attention Bias weight

In this PR: https://github.com/huggingface/transformers/pull/8518 a bug was fixed that removed an unnecessary weight from the T5 Cross Attention layer.
In the following, this layer was added to the wrong "ignore_weight" list. This weight will never be missing since it doesn't exist in the model anymore it can only be "not used" since it's still present in saved checkpoints. This PR fixes the incorrect warning by placing the regex in the correct list.


add base model classes to  bart subclassed models

# What does this PR do?

This PR adds base model classes for `MBart`, `Pegasus` and `Blenderbot`, and adds these in `MODEL_MAPPING` `dict`.
This will enable to load these models using the `AutoModel` class and `pipelines`.

Right now these models can't be loaded using `pipeline` since pipeline relies on the `AutoModel` class.
https://github.com/huggingface/transformers/blob/a4b21cdd20328f71448123ce7c962a78a5d75612/src/transformers/pipelines.py#L105-L110


Generate function does not work with GPU

Hello,

I want to use generate function with single GPU. Specifically, I fine tuned a GPT-2 model (on GPU) and subsequently, I want to generate text with it.

When I run this

```
input_ids.to(device)
sample_output = model.generate(
    input_ids,
    do_sample=True,
    max_length=150,
    top_k=50,
    top_p=0.92
)
```
I get this error

`RuntimeError: Input, output and indices must be on the current device`

When i move the model and the input `.to('cpu')`, it works.


Differences between original implementation and HuggingFace implementation

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: **4.0.0**
- Platform: **Windows**
- Python version: **3.6.5**
- PyTorch version (GPU?): **1.6.0+cu101**
- Tensorflow version (GPU?): **-**
- Using GPU in script?: **Yes**
- Using distributed or parallel set-up in script?: **No**

### Who can help
**@stefan-it**
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

The model I am using (Bert, XLNet ...): LayoutLMforTokenClassification

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
**My own modified script**

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
**My own dataset**

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

This is more of a question rather than an issue.
When I trained the layoutlm model using my data and I used the tokenclassification model from huggingface, I got a small drop in performance. I wanted to ask if there are any differences between the two models? I have kept the hyper-parameters to be exactly the same in both cases.
Two key points where I found the differences were:
(1). When taking in the dataset - in the Microsoft version, there is a concept called "segment_ids" which is not a parameter in the huggingface layoutlm documentation.
(2). I loaded both the models and printed the number of layers in both, I saw that there is 1 extra layer called layoutlm.embeddings.position_ids in the huggingface implementation.

I am trying to find out the reason for the drop in performance. Hence, wanted to find out if there is any difference between the model implementations itself. It would be great help if you could help explain the two differences I found!

Thanks!


Can't lazy initialize BART model on GPU

## Environment info
- `transformers` version: 3.5.1
- Platform: Linux-4.14.81.bm.15-amd64-x86_64-with-debian-9.11
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
 Bart: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Bart

We have finetuned a BART model for Seq2SeqLM, and want to serve it using a Python web microframework. This microframework uses `multiprocessing` underneath, so we wish to lazily initialize the model on the GPU on the first call. However, in `modeling_bart.py`, some of the layers already call CUDA in the main process:

https://github.com/huggingface/transformers/blob/f38c4ad302dd34faef1137b13e9636f4408b0462/src/transformers/models/bart/modeling_bart.py#L125

So trying to do so results in the following error:

```
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```

## To reproduce

```
class Model:
    def __init__(self, path):
        self.path = path
        self.initialized = False

    def init(self):
        if not self.initialized:
            self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
            self.model = AutoModelForSeq2SeqLM.from_pretrained(self.path)
            self.model.to(self.device)
            self.model.eval()
            self.initialized = True
```

```
model = Model("bart-base")
# in multiprocessing subprocess...
model.init()

# RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```

Sorry about the pseudocode, the microframework used is proprietary.

Commenting out the linked lines removes the error.

## Expected behavior

Able to call `model.cuda()` from subprocess.


n_gpus is set to 1 in case of distributed training on multiple gpus, how to access to the correct n_gpus

Hi
I am using transformer 4.1.1 on multiple gpus with distributed launch script below, the n_gpu set in this case is 1 instead of actual n_gpus, could you tell me how I can access to the total number of ranks calling the launcher with? Is there any variable in huggingface library which could show the number of gpus the script is called with? Here is the command I run:

`export BS=4; CUDA_VISIBLE_DEVICES=0,1  USE_TF=0   python -m torch.distributed.launch --nproc_per_node=2 --master_port=9915  finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler  --task translation --val_max_target_length 128 --warmup_steps 500 --n_train 500
`

thanks


n_gpu=1 in case of using distributed pytorch launcher


allowing the user to set booleans with HfArgumentParser

Hi
this is more general issue, if the user defines some booleans as arguments like --flag false/true once calling the finetune_trainer.py one would get following error as below with HfArgumentParser, it would be great to allow users set explicit values for booleans too, currently this is only possible with config files. thanks

```

raceback (most recent call last):
Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 329, in <module>
  File "finetune_t5_trainer.py", line 329, in <module>
        main()main()

  File "finetune_t5_trainer.py", line 48, in main
  File "finetune_t5_trainer.py", line 48, in main
        model_args, data_args, training_args, adapter_args = parser.parse_args_into_dataclasses()model_args, data_args, training_args, adapter_args = parser.parse_args_into_dataclasses()

  File "/opt/conda/envs/internship/lib/python3.7/site-packages/transformers-3.5.1-py3.7.egg/transformers/hf_argparser.py", line 144, in parse_args_into_dataclasses
  File "/opt/conda/envs/internship/lib/python3.7/site-packages/transformers-3.5.1-py3.7.egg/transformers/hf_argparser.py", line 144, in parse_args_into_dataclasses
        raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")

ValueErrorValueError: : Some specified arguments are not used by the HfArgumentParser: ['True', 'true', 'False', 'False', 'True', 'True', 'True', 'True', 'True', 'True', 'True', '--add_adapters_in_decoder', 'False']Some specified arguments are not used by the HfArgumentParser: ['True', 'true', 'False', 'False', 'True', 'True', 'True', 'True', 'True', 'True', 'True', '--add_adapters_in_decoder', 'False']

Traceback (most recent call last):
  File "/opt/conda/envs/internship/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/internship/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/internship/lib/python3.7/site-packages/torch-1.7.0-py3.7-linux-x86_64.egg/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/internship/lib/python3.7/site-packages/torch-1.7.0-py3.7-linux-x86_64.egg/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/internship/bin/python', '-u', 'finetune_t5_trainer.py', '--local_rank=1', '--model_name_or_path', 't5-small', '--tokenizer_name', 't5-small', '--learning_rate', '1e-2', '--output_dir', 'outputs/test', '--max_source_length', '128', '--max_target_length', '128', '--val_max_target_length', '128', '--test_max_target_length', '128', '--num_train_epochs', '10', '--warmup_steps', '500', '--eval_steps', '200', '--overwrite_output_dir', 'True', '--tasks', '[scitail,', 'boolq]', '--eval_tasks', '[rte,', 'boolq]', '--sampling', 'true', '--label_smoothing', '0.1', '--freeze_encoder', 'False', '--freeze_embeds', 'False', '--per_device_train_batch_size', '64', '--per_device_eval_batch_size', '64', '--save_steps', '20', '--logging_first_step', 'True', '--logging_steps', '200', '--save_total_limit', '1', '--train_adapters', 'True', '--adapter_config_name', 'parametric-meta-adapter', '--temperature', '10', '--do_eval', 'True', '--predict_with_generate', 'True', '--n_train', '10', '--task_embedding_dir', 'test_data/task_embeddings/n-train-all', '--task_embedding_dim', '512', '--n_val', '10', '--n_train', '10', '--do_finetune', 'True', '--do_train', 'True', '--n_finetune', '100', '--eval_output_dir', 'outputs/eval_test', '--reduction_factor', '16', '--non_linearity', 'relu', '--train_task_embeddings', 'True', '--projected_task_embedding_dim', '512', '--add_adapters_in_decoder', 'False', '--unfreeze_lm_head', '--unfreeze_layer_norms']' returned non-zero exit status 1.
```


passing config file to train with on multiple gpus

## Environment info
- transformers 3.5.1
- Python version: 3.7
- PyTorch version (GPU?): 1.6
- Tensorflow version (GPU?): -
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help
 Trainer: @sgugger
 T5: @patrickvonplaten
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 documentation: @sgugger

## Information
Hi, I want to run finetune_trainer with multiple gpus with a config file, I am using the following command:

`export BS=4; CUDA_VISIBLE_DEVICES=0,1  USE_TF=0   python -m torch.distributed.launch --nproc_per_node=2 --master_port=9910  finetune_t5_trainer.py configs/experiments/test.json
`
 getting this error, looks like config file is not passed properly in this case, the command on one gpu works, could you tell me please how I can call the codes with config, it would be great to have it in the read me. thanks

```
inetune_t5_trainer.py: error: the following arguments are required: --model_name_or_path, --output_dir
usage: finetune_t5_trainer.py [-h] --model_name_or_path MODEL_NAME_OR_PATH
                              [--not_load_t5_checkpoint]
                              [--config_name CONFIG_NAME]
                              [--tokenizer_name TOKENIZER_NAME]
                              [--cache_dir CACHE_DIR] [--freeze_encoder]
                              [--freeze_embeds] [--freeze_model_but_lm_head]
                              [--unfreeze_lm_head]
                              [--freeze_model_but_task_embeddings]
                              [--unfreeze_layer_norms] [--sampling]
                              [--tasks TASKS [TASKS ...]]
                              [--eval_tasks EVAL_TASKS [EVAL_TASKS ...]]
                              [--adapters ADAPTERS [ADAPTERS ...]]
                              [--max_source_length MAX_SOURCE_LENGTH]
                              [--max_target_length MAX_TARGET_LENGTH]
                              [--val_max_target_length VAL_MAX_TARGET_LENGTH]
                              [--test_max_target_length TEST_MAX_TARGET_LENGTH]
                              [--n_train N_TRAIN] [--n_val N_VAL]
                              [--n_test N_TEST] [--eval_beams EVAL_BEAMS]
                              [--no_ignore_pad_token_for_loss]
                              [--n_finetune N_FINETUNE] --output_dir
                              OUTPUT_DIR [--overwrite_output_dir] [--do_train]
                              [--do_eval] [--do_predict]
                              [--evaluate_during_training]
                              [--evaluation_strategy {EvaluationStrategy.NO,EvaluationStrategy.STEPS,EvaluationStrategy.EPOCH}]
                              [--prediction_loss_only]
                              [--per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE]
                              [--per_device_eval_batch_size PER_DEVICE_EVAL_BATCH_SIZE]
                              [--per_gpu_train_batch_size PER_GPU_TRAIN_BATCH_SIZE]
                              [--per_gpu_eval_batch_size PER_GPU_EVAL_BATCH_SIZE]
                              [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS]
                              [--eval_accumulation_steps EVAL_ACCUMULATION_STEPS]
                              [--learning_rate LEARNING_RATE]
                              [--weight_decay WEIGHT_DECAY]
                              [--adam_beta1 ADAM_BETA1]
                              [--adam_beta2 ADAM_BETA2]
                              [--adam_epsilon ADAM_EPSILON]
                              [--max_grad_norm MAX_GRAD_NORM]
                              [--num_train_epochs NUM_TRAIN_EPOCHS]
                              [--max_steps MAX_STEPS]
                              [--warmup_steps WARMUP_STEPS]
                              [--logging_dir LOGGING_DIR]
                              [--logging_first_step]
                              [--logging_steps LOGGING_STEPS]
                              [--save_steps SAVE_STEPS]
                              [--save_total_limit SAVE_TOTAL_LIMIT]
                              [--no_cuda] [--seed SEED] [--fp16]
                              [--fp16_opt_level FP16_OPT_LEVEL]
                              [--local_rank LOCAL_RANK]
                              [--tpu_num_cores TPU_NUM_CORES]
                              [--tpu_metrics_debug] [--debug]
                              [--dataloader_drop_last]
                              [--eval_steps EVAL_STEPS]
                              [--dataloader_num_workers DATALOADER_NUM_WORKERS]
                              [--past_index PAST_INDEX] [--run_name RUN_NAME]
                              [--disable_tqdm DISABLE_TQDM]
                              [--no_remove_unused_columns]
                              [--label_names LABEL_NAMES [LABEL_NAMES ...]]
                              [--load_best_model_at_end]
                              [--metric_for_best_model METRIC_FOR_BEST_MODEL]
                              [--greater_is_better GREATER_IS_BETTER]
                              [--label_smoothing LABEL_SMOOTHING]
                              [--sortish_sampler] [--predict_with_generate]
                              [--adafactor]
                              [--encoder_layerdrop ENCODER_LAYERDROP]
                              [--decoder_layerdrop DECODER_LAYERDROP]
                              [--dropout DROPOUT]
                              [--attention_dropout ATTENTION_DROPOUT]
                              [--lr_scheduler LR_SCHEDULER]
                              [--fixed_length_emb]
                              [--encoder_projection ENCODER_PROJECTION]
                              [--encoder_pooling ENCODER_POOLING]
                              [--projection_length PROJECTION_LENGTH]
                              [--only_projection_bottleneck]
                              [--concat_projection_token]
                              [--gcs_bucket GCS_BUCKET]
                              [--temperature TEMPERATURE] [--train_adapters]
                              [--do_finetune] [--parametric_task_embedding]
                              [--eval_output_dir EVAL_OUTPUT_DIR]
                              [--generate_classifier_weights]
                              [--adapter_config_name ADAPTER_CONFIG_NAME]
                              [--task_embedding_dir TASK_EMBEDDING_DIR]
                              [--task_embedding_dim TASK_EMBEDDING_DIM]
                              [--add_layer_norm_before_adapter]
                              [--no_add_layer_norm_after_adapter]
                              [--hidden_dim HIDDEN_DIM]
                              [--reduction_factor REDUCTION_FACTOR]
                              [--train_task_embeddings]
                              [--non_linearity NON_LINEARITY]
                              [--projected_task_embedding_dim PROJECTED_TASK_EMBEDDING_DIM]
                              [--no_add_adapters_in_decoder]
finetune_t5_trainer.py: error: the following arguments are required: --model_name_or_path, --output_dir
Traceback (most recent call last):
  File "/opt/conda/envs/internship/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/internship/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/internship/lib/python3.7/site-packages/torch-1.7.0-py3.7-linux-x86_64.egg/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/internship/lib/python3.7/site-packages/torch-1.7.0-py3.7-linux-x86_64.egg/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/internship/bin/python', '-u', 'finetune_t5_trainer.py', '--local_rank=1', 'configs/experiments/test.json']' returned non-zero exit status 2.
```


Does provide any scripts about how to convert mpnet pretrain model to transformers pretrain one?

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

I want to convert my pretrain chinese mpnet with the fairseq format to transformers format

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->