Layout lm tf 2

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds TF version of LayoutLM for issue [(10312)](https://github.com/huggingface/transformers/issues/10312)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Document Trainer limitation on custom models

# What does this PR do?

As discussed in #10629, documenting the limitations of the `Trainer` when working with custom models.


Issues with Multi-GPU

- `transformers` version: 4.3.3
- Platform: Linux-4.15.0-132-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes, multi GeForce RTX 2080 Ti GPUs
- Using distributed or parallel set-up in script?: DataParallel
- NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2

I have tried to run the IMDb review sequence classification from https://huggingface.co/transformers/custom_datasets.html on two GPUs, using `DataParallel`:

```
import os
os.environ["CUDA_VISIBLE_DEVICES"]="6,7"
import time
import torch
import torch.nn as nn
from pathlib import Path
from sklearn.model_selection import train_test_split
from transformers import DistilBertTokenizerFast
from transformers import DistilBertForSequenceClassification, Trainer, TrainingArguments

def read_imdb_split(split_dir):
    split_dir = Path(split_dir)
    texts = []
    labels = []
    for label_dir in ["pos", "neg"]:
        for text_file in (split_dir/label_dir).iterdir():
            texts.append(text_file.read_text())
            labels.append(0 if label_dir is "neg" else 1)

    return texts, labels

class IMDbDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx])
        return item

    def __len__(self):
        return len(self.labels)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
n_gpus = torch.cuda.device_count()

train_texts, train_labels = read_imdb_split('aclImdb/train')
test_texts, test_labels = read_imdb_split('aclImdb/test')
train_texts, val_texts, train_labels, val_labels = train_test_split(train_texts, train_labels, test_size=.2)

tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
train_encodings = tokenizer(train_texts, truncation=True, padding=True)
val_encodings = tokenizer(val_texts, truncation=True, padding=True)
test_encodings = tokenizer(test_texts, truncation=True, padding=True)

train_dataset = IMDbDataset(train_encodings, train_labels)
val_dataset = IMDbDataset(val_encodings, val_labels)
test_dataset = IMDbDataset(test_encodings, test_labels)

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
)

model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased")
if n_gpus > 1:
    model = nn.DataParallel(model)
model.to(device)

trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset             # evaluation dataset
)

trainer.train()
```

For **torch==1.8.0**, no matter I use single GPU or multi GPU, I encounter the same CUDA error (shown below). For **torch==1.7.1**, I am able to run the code on single GPU with no issue. However, with multi-GPU, the `Input, output and indices must be on the current device` error occurs (also shown below).

With **torch==1.8.0**:
```
2021-03-10 19:16:07.624155: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']
- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  0%|                                                                                | 0/1875 [00:00<?, ?it/s]Traceback (most recent call last):
  File "test.py", line 80, in <module>
    trainer.train()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 940, in train
    tr_loss += self.training_step(model, inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 1304, in training_step
    loss = self.compute_loss(model, inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 1334, in compute_loss
    outputs = model(**inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 167, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 177, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 167, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 177, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 623, in forward
    return_dict=return_dict,
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 487, in forward
    return_dict=return_dict,
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 309, in forward
    x=hidden_state, attn_mask=attn_mask, head_mask=head_mask[i], output_attentions=output_attentions
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 256, in forward
    output_attentions=output_attentions,
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 177, in forward
    q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`
```

With **torch==1.7.1**:
```
2021-03-10 19:22:14.938302: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']
- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  0%|                                                                                | 0/1875 [00:00<?, ?it/s]Traceback (most recent call last):
  File "test.py", line 80, in <module>
    trainer.train()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 940, in train
    tr_loss += self.training_step(model, inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 1304, in training_step
    loss = self.compute_loss(model, inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 1334, in compute_loss
    outputs = model(**inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 623, in forward
    return_dict=return_dict,
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 480, in forward
    inputs_embeds = self.embeddings(input_ids)  # (bs, seq_length, dim)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 107, in forward
    word_embeddings = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Input, output and indices must be on the current device
```

With **torch==1.5.0**:
```
2021-03-10 19:23:51.586005: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']
- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  0%|                                                                                | 0/1875 [00:00<?, ?it/s]Traceback (most recent call last):
  File "test.py", line 80, in <module>
    trainer.train()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 940, in train
    tr_loss += self.training_step(model, inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 1304, in training_step
    loss = self.compute_loss(model, inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/trainer.py", line 1334, in compute_loss
    outputs = model(**inputs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 623, in forward
    return_dict=return_dict,
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 480, in forward
    inputs_embeds = self.embeddings(input_ids)  # (bs, seq_length, dim)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 107, in forward
    word_embeddings = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/mnt/sdb/env1/lib/python3.6/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:403
```


Extend trainer logging for sm

# What does this PR do?

adds a helper function to `logging.py` to add a native logging handler if needed (`add_handler`). Adds in the `Trainer` a logging `StreamHandler(sys.stdout)` with `sys.stdout` when training is run on sagemaker to forward logs.


Ensure metric results are JSON-serializable

# What does this PR do?

Metrics returned from numpy (with an `np.mean()` for instance) are not real Python floats but `np.float32` (or other type) objects that are not serializable. This causes problems when the metrics are saved in JSON format in the `Trainer`, for instance when using `load_best_model_at_end`. This PR fixes that by recursively applying a `.item()` on the metrics dictionary.

Fixes #10299


Help using Speech2Text

Hey @patil-suraj (and anyone who can help),

Sorry, I'm still a beginner compared to the rest of the folks here so sorry if my question is a little basic.

But I'm trying to build a pipeline to manually transcribe Youtube videos (that aren't transcribed correctly by Google) and I was considering using your [model ](https://huggingface.co/facebook/s2t-small-librispeech-asr)for it.

Here's my unfinished code on  [Google Colab](https://colab.research.google.com/drive/15SgSw1KmD-sxdf6Zd953fJIPSmisHd6M?usp=sharing); the last line throws an error:

```
!pip install git+https://github.com/huggingface/transformers
!pip install youtube-dl path.py soundfile librosa sentencepiece torchaudio

import youtube_dl
from path import Path as Path
import tempfile
import textwrap
import librosa
import soundfile as sf
import torch
from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration

model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr")
processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr")
wrapper = textwrap.TextWrapper(width=70)

mydir = tempfile.TemporaryDirectory()
dirname = mydir.name + "/tmp.wav"

!youtube-dl -o $dirname -ci -f 'bestvideo[ext=mp4]+bestaudio' -x --audio-format wav https://www.youtube.com/watch?v=d5yfUuHYWho

filename = dirname + ".wav"

speech, rate = sf.read(filename)
speech = librosa.resample(speech.T, rate, 16000)

features = processor(speech, sampling_rate=16000, padding=True, return_tensors="pt")
```

And here's the error produced:

```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-8-8fc3e2d943e0> in <module>()
----> 1 features = processor(speech, sampling_rate=16000, padding=True, return_tensors="pt")

5 frames
/usr/local/lib/python3.7/dist-packages/torchaudio/compliance/kaldi.py in _get_waveform_and_window_properties(waveform, channel, sample_frequency, frame_shift, frame_length, round_to_power_of_two, preemphasis_coefficient)
    147     assert 2 <= window_size <= len(
    148         waveform), ('choose a window size {} that is [2, {}]'
--> 149                     .format(window_size, len(waveform)))
    150     assert 0 < window_shift, '`window_shift` must be greater than 0'
    151     assert padded_window_size % 2 == 0, 'the padded `window_size` must be divisible by two.' \

AssertionError: choose a window size 400 that is [2, 2]
```

Can anyone point me in the right direction?  Thanks.


I get different results everytime I run run_squad.py

Is it possible to have deterministic results using the run_squad.py script?
It has the set_seed() method but still, it gives different results every time I run it.

How can I get same results across all runs?


Using `label` in Trainer leads to TypeError

## Environment info

- `transformers` version: 4.3.3
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Not explicitly.
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik

## Information
My dataset is defined as follows:
```python
"""Implements MNIST Dataset"""
from torch.utils.data import Dataset
from torchvision import datasets, transforms
from torchvision.transforms import Grayscale, ToTensor, Normalize

class Mnist(Dataset):
    def __init__(self, config):
        self.config = config

        transformations = [Grayscale(num_output_channels=1),ToTensor(),Normalize(mean=[0.0],std=[1.0])]
        self.transform = (
            transforms.Compose(transformations)
        )

        self.dataset = datasets.MNIST(
            config.load_dataset_args.path,
            download=True,
            train=self.config.split == "train",
            transform=self.transform,
        )

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, example_idx):
        # essential to return as dict, hence the roundabout way of loading the dataset
        img, label = self.dataset[example_idx]
        return {"image": img, "label": label}
```
Model I am using - a custom CNN, defined as follows:

```python
"""Implementation of a custom CNN with random weights."""

from torch.nn import (
    BatchNorm2d,
    Conv2d,
    Linear,
    MaxPool2d,
    Module,
    ReLU,
    Sequential,
    CrossEntropyLoss,
)

class SimpleCnn(Module):
    def __init__(self):
        super(SimpleCnn, self).__init__()
        self.cnn_layers = Sequential(
            Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(32),
            ReLU(),
            MaxPool2d(kernel_size=2, stride=2),
            Conv2d(32, 32, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(32),
            ReLU(),
            MaxPool2d(kernel_size=2, stride=2),
            Conv2d(32, 32, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(32),
            ReLU(),
            MaxPool2d(kernel_size=2, stride=2),
        )
        self.linear_layers = Linear(32 * 3 * 3, 10)

        self.loss_fn = CrossEntropyLoss()

    def forward(self, image, label=None):
        out = self.cnn_layers(image)
        out = out.view(out.size(0), -1)
        out = self.linear_layers(out)
        if label is not None:
            loss = self.loss_fn(out, label)
            return loss, out
        return out

```
The problem arises when using:
Trainer with a custom `label_names` as `['label']`. I provide `label_names` as `['label']` in `TrainingArguments`. The following error occurs on `trainer.train()`:

```python

Traceback (most recent call last):
  File "hf_train.py", line 97, in <module>
    trainer.train()
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 943, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1307, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1337, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'labels'
```
I tried printing batch keys, using `torch.utils.data.DataLoader` inside, and after the function call to `get_train_dataloader` in `trainer.py`:

```
dict_keys(['image', 'label']) #Inside
dict_keys(['labels', 'image']) #Immediately after call
```
I don't understand how it gets converted to `labels` on its own.

## To reproduce

Steps to reproduce the behavior:

1. Load any dataset with one output key as `['label']`.
2. Provide `['label']` as the label_names to `TrainingArguments`
3. Run `trainer.train()`.

One can also try using `load_dataset('mnist')` directly from the `datasets` library. This error will get thrown.

This is not expected. Strangely enough, changing every `'label'` to `'class_label'` or `'labels'` works perfectly. I don't know why this would happen.


expanduser path in Trainer

the `output_dir` passed to TrainingArguments is not expanded (the behaviour is probably the same for logging_dir)
### Who can help

Library:

- trainer: @sgugger

## To reproduce
Directly using os.makedirs but this is what happens in Trainer
```py
In [7]: !mkdir ~/foo

In [8]: !cd ~/foo
/mnt/beegfs/home/lerner/foo

In [10]: os.makedirs("~/bar")

In [14]: !realpath "~/bar"
/mnt/beegfs/home/lerner/foo/~/bar

```

## To fix
Call os.path.expanduser before making dir
```py
In [10]: os.makedirs(os.path.expanduser("~/bar"))

In [18]: cd
/mnt/beegfs/home/lerner

In [21]: !realpath bar
/mnt/beegfs/home/lerner/bar
```


considering `pad_to_multiple_of` for run_mlm.py

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.3
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.8
- Tensorflow version (GPU?): -
- Using GPU in script?: -
- Using distributed or parallel set-up in script?: -

### Who can help
@sgugger, @patil-suraj

## Information
I have seen in huggingface codes such as run_seq2seq.py that for the case of fp16, they pad to multiple of 8 like below, perhpas for efficiency purpose:

```
 data_collator = DataCollatorForSeq2Seq(
        tokenizer,
        label_pad_token_id=label_pad_token_id,
        pad_to_multiple_of=8 if training_args.fp16 else None,
       )
```

For run_mlm.py codes, the padding condition in case of fp16 is not considered, could it be a bug and possibly performance could get better in run_mlm.py if this was set? kindly appreciate  having a look.

thanks


Average checkpoints

Is it possible to average weights of several checkpoints? Some thing like https://github.com/pytorch/fairseq/blob/master/scripts/average_checkpoints.py


Model "deberta-v2--xxlarge-mnli" doesn't work!!!

Whenever i try to load the tokenizer by
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-v2-xxlarge-mnli')
```
it returns this issue:
```
config_class = CONFIG_MAPPING[config_dict["model_type"]]
KeyError: 'deberta-v2'
```

Is this model not available in Transformers 4.3.3 (the latest release)?


Copy tokenizer files in each of their repo

# What does this PR do?

This PR cleans the maps in the tokenizer files to make sure each checkpoint has the proper tokenization files. This will allow us to remove custom code that mapped some checkpoints to special files (like BART using RoBERTa vocab files) and take full advantage of the versioning systems for those checkpoints. All checkpoints changed have been properly copied in the corresponding model repos in parallel.

For instance, to accomodate the move on the fast BART tokenizers, the following commits have been on the model hub:
- in [facebook/bart-base](https://huggingface.co/facebook/bart-base/commit/c2469fb7e666a5c5629a161f17c9ef23c85217f7)
- in [facebook/bart-large](https://huggingface.co/facebook/bart-large/commit/22fa33834dccc11df99c4fc5fcc96c67f806dfdb)
- in [facebook/bart-large-mnli](https://huggingface.co/facebook/bart-large-mnli/commit/6a35c499ad1087bad8d9c348a05b1fa10c5ad47d)
- in [facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn/commit/18614750f248f300641757e8e44e6afce801d664)
- in [facebook/bart-large-xsum](https://huggingface.co/facebook/bart-large-xsum/commit/96ea79a741cd376cdc8a740b225330773da151f0)
- in [yjernite/bart_eli5](https://huggingface.co/yjernite/bart_eli5/commit/38797dd2ef06f5542c6f7db853518703f6b3da21)

In the PR I've also uniformized the way the maps are structured across models, to make it easier to alter (and ultimately remove) them in the future via automatic scripts.


Invalid pytorch_model.bin for TAPAS-large

Hi,
I was downloading `google/tapas-large` binaries from [HF models](https://huggingface.co/google/tapas-large/tree/main) and for pytorch_model.bin, a zip file was getting downloaded which is not a `bin` file like other models.

folder structure is like - `archive/data/.*File`, `archive/data.pkl` ,` archive/version.File`. Same for tapas-base as well.
These cannot be used for loading a model (`TapasForQuestionAnswering.from_pretrained('path/to/binaries directory')`)

plz suggest how to proceed further..


wav2vec2: adding single-char tokens to tokenizer causes tokenization mistakes

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.8.0-44-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

### Who can help

@patrickvonplaten and @LysandreJik

Issue is probably related to interactions of the following:

https://github.com/huggingface/transformers/blob/9a8c168f56fe3c0e21d554a577ac03beb004ef89/src/transformers/tokenization_utils.py#L213

https://github.com/huggingface/transformers/blob/11fdde02719dbd20651c9f43cc6f54959fc6ede6/src/transformers/tokenization_utils.py#L352

https://github.com/huggingface/transformers/blob/cb38ffcc5e0ae2fac653342ac36dc75c15ea178f/src/transformers/models/wav2vec2/tokenization_wav2vec2.py#L184

This is a corner case: `add_tokens` adds new tokens to `self.unique_no_split_tokens` -- causing `tokenize()` to skip calling `Wav2Vec2CTCTokenizer._tokenize()`

This is probably not the case with most tokenizers since their vocab includes most, if not all, commonly used single-characters tokens without including them in `self.unique_no_split_tokens`. I faced this while debugging my code for https://github.com/huggingface/transformers/pull/10581 to add support for Buckwalter Arabic transliteration. The issue is not limited to adding single-char tokens but rather when words (space-separated) start or end with a newly added token.

## Information

Model I am using (Bert, XLNet ...): wav2vec2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: adding tokens to ASR vocab

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: training an ASR with extended vocab

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import Wav2Vec2Processor

tokenizer = Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base').tokenizer
tokenizer.add_tokens('x')
token_ids = tokenizer('C x A').input_ids
decoded = tokenizer.decode(token_ids)
print(decoded, token_ids)
# CxA [19, 32, 7]
```

## Expected behavior

Should have printed `C x A [19, 4, 32, 4, 7]`


Fixes an issue in `text-classification` where MNLI eval/test datasets are not being preprocessed.

# What does this PR do?
In https://github.com/huggingface/transformers/commit/dfd16af8322788e6dd58e8396e0d6f2f5312bf99 for `run_glue.py`, `{train|eval|test}_dataset` was split out and preprocessed individually. However, this misses `datasets["{validation|test}_mismatched"]` which is appended to the `{eval|test}_dataset` only when MNLI is used.

When running evaluation on MNLI, that means we eventually hit an un-preprocessed dataset which leads to a stack trace like this:
```
Traceback (most recent call last):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "/transformers/examples/text-classification/run_glue.py", line 532, in _mp_fn
    main()
  File "/transformers/examples/text-classification/run_glue.py", line 493, in main
    metrics = trainer.evaluate(eval_dataset=eval_dataset)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 1657, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 1788, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 1899, in prediction_step
    loss, outputs = self.compute_loss(model, inputs, return_outputs=True)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 1458, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 625, in forward
    return_dict=return_dict,
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 471, in forward
    raise ValueError("You have to specify either input_ids or inputs_embeds")
ValueError: You have to specify either input_ids or inputs_embeds
```

This commit resolves this by moving the `dataset.map(preprocess...)` to the beginning.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # 10620

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


MNLI eval/test dataset is not being preprocessed in `run_glue.py`

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.9.0-14-amd64-x86_64-with-debian-9.13
- Python version: 3.6.10
- PyTorch version (GPU?): 1.8.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no, using TPU
- Using distributed or parallel set-up in script?: distributed

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

N/A, I have a fix upcoming 👍

## Information

Model I am using (Bert, XLNet ...): Any model within `examples/text-classification/run_glue.py` that uses MNLI

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (MNLI)
* [ ] my own task or dataset: (give details below)

Essentially, the issue is that in https://github.com/huggingface/transformers/commit/dfd16af8322788e6dd58e8396e0d6f2f5312bf99 for `run_glue.py`, `{train|eval|test}_dataset` was split out and preprocessed individually. However, this misses `datasets["{validation|test}_mismatched"]` which is appended to the `{eval|test}_dataset` only when MNLI is used.

## To reproduce

Steps to reproduce the behavior:

1. Run the `run_glue.py` example on an MNLI dataset and include eval. The full command I'm using on a v2-8 TPU is:

```
python examples/xla_spawn.py --num_cores 8 examples/text-classification/run_glue.py --logging_dir=./tensorboard-metrics --task_name MNLI --cache_dir ./cache_dir --do_eval --max_seq_length 128 --learning_rate 3e-5 --output_dir MNLI --logging_steps 30 --save_steps 3000 --tpu_metrics_debug --model_name_or_path bert-base-cased --per_device_eval_batch_size 64 --overwrite_output_dir
```

This results in:
```
Traceback (most recent call last):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "/transformers/examples/text-classification/run_glue.py", line 532, in _mp_fn
    main()
  File "/transformers/examples/text-classification/run_glue.py", line 493, in main
    metrics = trainer.evaluate(eval_dataset=eval_dataset)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 1657, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 1788, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 1899, in prediction_step
    loss, outputs = self.compute_loss(model, inputs, return_outputs=True)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 1458, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 625, in forward
    return_dict=return_dict,
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 471, in forward
    raise ValueError("You have to specify either input_ids or inputs_embeds")
ValueError: You have to specify either input_ids or inputs_embeds
```

## Expected behavior

Dataset should be preprocessed for the entirety of the dataset.

Fix: https://github.com/huggingface/transformers/pull/10621


wav2vec2: `convert_tokens_to_string` contracts legitimately repeated characters

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.8.0-44-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

### Who can help

@patrickvonplaten - issue is most probably due to https://github.com/huggingface/transformers/blob/cb38ffcc5e0ae2fac653342ac36dc75c15ea178f/src/transformers/models/wav2vec2/tokenization_wav2vec2.py#L203

## Information

Model I am using (Bert, XLNet ...): wav2vec2

The problem arises when using:
* [x] the official example scripts: run_asr.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: wav2vec2
* [ ] my own task or dataset: (give details below)

## To reproduce
```python
from transformers import Wav2Vec2Processor

tokenizer = Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base').tokenizer
tokenizer.decode(tokenizer('CARRY').input_ids)
# CARY
```

Decoder should have returned `'CARRY'` instead.


Run_qa  crashes because of parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))

## Environment info
- `transformers` version: 4.3.3
- Platform: linux
- Python version:3.7, 3.8, 3.9 reproed across all three
- PyTorch version (GPU?): 1.7, tried 1.8 with same behavior
- Tensorflow version (GPU?):N/A
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: Yes 2 gpu

### Who can help
@sgugger, @patil-suraj
## Information

Model I am using (Bert, XLNet ...): bert-base-uncased

The problem arises when using:
* [ X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
SQUAD 1.0
## To reproduce

Steps to reproduce the behavior:

1. Install clean transformers environment
2. run the run_qa.py script with instructions as specified
3. crash
If you go ahead and create a new environment and install the most recent version of the transformer and try to run the run_qa.py script(SQUAD) it crashes because of a parser issue.

python run_qa.py   --model_name_or_path bert-base-uncased  --dataset_name squad  --do_train  --per_device_train_batch_size 8  --learning_rate 3e-5  --max_seq_length 384  --doc_stride 128  --output_dir output  --overwrite_output_dir  --cache_dir cache  --preprocessing_num_workers 4  --seed 42  --num_train_epochs 1
Traceback (most recent call last):
  File "run_qa.py", line 1095, in <module>
    main()
  File "run_qa.py", line 902, in main
    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
  File "/home/spacemanidol/miniconda3/envs/sparseml/lib/python3.7/site-packages/transformers/hf_argparser.py", line 52, in __init__
    self._add_dataclass_arguments(dtype)
  File "/home/spacemanidol/miniconda3/envs/sparseml/lib/python3.7/site-packages/transformers/hf_argparser.py", line 93, in _add_dataclass_arguments
    elif hasattr(field.type, "__origin__") and issubclass(field.type.__origin__, List):
  File "/home/spacemanidol/miniconda3/envs/sparseml/lib/python3.7/typing.py", line 721, in __subclasscheck__
    return issubclass(cls, self.__origin__)
TypeError: issubclass() arg 1 must be a clas
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Run and produce a BERT-QA model


Request: Ignore Dataset transforms when iterating to the most recent checkpoint when resuming training

# 🚀 Feature request

It'd be great if, when resuming training from a checkpoint and using a Dataset with a format/transform function applied, the dataset's format/transform function could be ignored while iterating up to the last checkpoint step.

@lhoestq @sgugger

## Motivation

I doubt it's much of an issue most of the time, but I've started playing with `dataset.set_transform()` for doing some heavy preprocessing, and just iterating through samples to the current checkpoint step can take a ridiculously long time compared to a dataset without a transform applied. And I don't think there's any case where the transformed sample would be used, right?

See [this conversation in the forum](https://discuss.huggingface.co/t/understanding-set-transform/3740/6?u=jncasey) for more backstory and my rudimentary thoughts on how I'd accomplish it.

## Your contribution

I'm hesitant to try updating any of the trainer code myself since it's so complicated, and needs to cover so many edge cases I'm not familiar with.


changing ".view()" to ".reshape()" for pytorch

New Version of PyTorch uses ".reshape()" instead of ".view()".
There might be some issue if still using ".view()".


Fix tests of TrainerCallback

# What does this PR do?

When introducing the `report_to` argument, I must have messed something up. Bottomline is that the tests of `TrainerCallback` can fail depending on what is installed in the env (TensorBoard for instance), this PR fixes that.


Not able to convert T5 tf checkpoints

Hi,
I was trying to convert some tf checkpoints for T5 into PyTorch using ```transformers-cli convert```, and am getting the following error:

> Traceback (most recent call last):
  File "/home/william18026/miniconda3/bin/transformers-cli", line 32, in <module>
    service.run()
  File "/home/william18026/miniconda3/lib/python3.7/site-packages/transformers/commands/convert.py", line 158, in run
    raise ValueError("--model_type should be selected in the list [bert, gpt, gpt2, transfo_xl, xlnet, xlm]")
ValueError: --model_type should be selected in the list [bert, gpt, gpt2, transfo_xl, xlnet, xlm]

My initial attempt was with transformers==4.3.3, but then I also tried using the source version (4.4.0dev) and the editable clone, but got the same error with all. T5 seems to have been added to the pipeline in #9654, but for some reason, it's not working at the user (/my) end.

What could I be doing wrong?


OOM issues with save_pretrained models

Posted this issue to the HuggingFace forums without a response.

Having a weird issue with DialoGPT Large model deployment. From PyTorch 1.8.0 and Transformers 4.3.3 using model.save_pretrained and tokenizer.save_pretrained, the exported pytorch_model.bin is almost twice the size of the model card repo and results in OOM on a reasonably equipped machine that when using the standard transformers download process it works fine (I am building a CI pipeline to containerize the model hence the pre-populated model requirement):

```
Model card:

pytorch_model.bin 1.6GB

model.save_pretrained and tokenizer.save_pretrained:

-rw-r--r-- 1 jrandel jrandel  800 Mar  6 16:51 config.json
-rw-r--r-- 1 jrandel jrandel 446K Mar  6 16:51 merges.txt
-rw-r--r-- 1 jrandel jrandel 3.0G Mar  6 16:51 pytorch_model.bin
-rw-r--r-- 1 jrandel jrandel  357 Mar  6 16:51 special_tokens_map.json
-rw-r--r-- 1 jrandel jrandel  580 Mar  6 16:51 tokenizer_config.json
-rw-r--r-- 1 jrandel jrandel 780K Mar  6 16:51 vocab.json
```

When I download the model card files directly however, I’m getting the following errors:

```
curl -L https://huggingface.co/microsoft/DialoGPT-large/resolve/main/config.json -o ./model/config.json
curl -L https://huggingface.co/microsoft/DialoGPT-large/resolve/main/pytorch_model.bin -o ./model/pytorch_model.bin
curl https://huggingface.co/microsoft/DialoGPT-large/resolve/main/tokenizer_config.json -o ./model/tokenizer_config.json
curl https://huggingface.co/microsoft/DialoGPT-large/resolve/main/config.json -o ./model/config.json
curl https://huggingface.co/microsoft/DialoGPT-large/resolve/main/merges.txt -o ./model/merges.txt
curl https://huggingface.co/microsoft/DialoGPT-large/resolve/main/special_tokens_map.json -o ./model/special_tokens_map.json
curl https://huggingface.co/microsoft/DialoGPT-large/resolve/main/vocab.json -o ./model/vocab.json
<snip>
    tokenizer = AutoTokenizer.from_pretrained("model/")
  File "/var/lang/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 395, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/var/lang/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1788, in from_pretrained
    return cls._from_pretrained(
  File "/var/lang/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1801, in _from_pretrained
    slow_tokenizer = (cls.slow_tokenizer_class)._from_pretrained(
  File "/var/lang/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1876, in _from_pretrained
    special_tokens_map = json.load(special_tokens_map_handle)
  File "/var/lang/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/var/lang/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/var/lang/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/var/lang/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/runtime/bootstrap.py", line 481, in <module>
    main()
  File "/var/runtime/bootstrap.py", line 458, in main
    lambda_runtime_client.post_init_error(to_json(error_result))
  File "/var/runtime/lambda_runtime_client.py", line 42, in post_init_error
    response = runtime_connection.getresponse()
  File "/var/lang/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/var/lang/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/var/lang/lib/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
time="2021-03-08T09:01:39.33" level=warning msg="First fatal error stored in appctx: Runtime.ExitError"
time="2021-03-08T09:01:39.33" level=warning msg="Process 14(bootstrap) exited: Runtime exited with error: exit status 1"
time="2021-03-08T09:01:39.33" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2021-03-08T09:01:39.33" level=warning msg="Failed to send default error response: ErrInvalidInvokeID"
time="2021-03-08T09:01:39.33" level=error msg="INIT DONE failed: Runtime.ExitError"
time="2021-03-08T09:01:39.33" level=warning msg="Reset initiated: ReserveFail"
```

So what would be causing the large file variance between save_pretrained models and the model card repo? And any ideas why the directly downloaded model card files aren’t working in this example?

Thanks in advance


Implementing efficient self attention in T5

# 🌟 New model addition

My teammates and I (including @ice-americano) would like to use efficient self attention methods such as Linformer, Performer and Nystromformer

## Model description

These new methods serve as approximations of regular attention, but reduce complexity from quadratic in the inputs to linear.  We would like to add a parameter to T5 where users can specify an efficient attention method to use instead of regular attention.  Ideally, this would be implemented across all models, but the models tend to have varying implementations of attention, rendering this generalization fairly tedious.

## Open source status

* [x] the model implementation is available: repos are https://github.com/mlpen and https://github.com/lucidrains/performer-pytorch
* [ ] the model weights are available: N/A
* [x] who are the authors: @mlpen and @lucidrains


split seq2seq script into summarization & translation

keeping the original script for tests cc @stas00
fix #10164


Trigger add sm information

The PR adds functionality to identify more telemetry information when training is run.


SortedDL for contiguous LM

Hi there,
I am currently implementing LM re-training of a RoBERTa model using the `Trainer` API. Since I have a huge training corpus, I was wondering if there is a functionality in the `Trainer` or the corresponding `DataCollatorForLanguageModeling` that allows for sorted batching as in `fastai`?

More precisely, I would like to feed in all my training data as a contiguous text stream and let the respective functions handle sorted batching irrespective of the sequence length of the individual sequences.

Best,
Simon


Image feature extractor design

# What does this PR do?

Here we can discuss how to design the `ImageFeatureExtractor` class, and the `ViTFeatureExtractor` subclass.

The hierarchy looks as follows:
`FeatureExtractorMixin` -> `ImageFeatureExtractor` -> `ViTFeatureExtractor`. The `FeatureExtractorMixin` defines common properties among `SequenceFeatureExtractors` (for speech recognition) and `ImageFeatureExtractors` (for vision related tasks), namely saving utilities and the general `BatchFeature` class.

Notes:
- `ImageFeatureExtractor` is based on `SequenceFeatureExtractor`, but with some changes: renamed `max_length` to `max_resolution`, renamed `PaddingStrategy.LONGEST` to `PaddingStrategy.LARGEST` (to pad to the resolution of the largest image in a batch), renamed `PaddingStrategy.MAX_LENGTH` to `PaddingStrategy.MAX_RESOLUTION`.
- Currently, this `ImageFeatureExtractor` class defines common properties among feature extractors for vision models, which are now `image_mean`, `image_std` and `padding_value`. Each concrete FeatureExtractor then provides values for these 3 attributes, and defines any additional attributes.
- Currently, the `ImageFeatureExtractor` class only defines `pad` and `_pad` methods (which should be updated to work for 2D images), but I guess we can add general image transformation methods (such as resize, normalize), and maybe also a `__call__` method. These are now all defined in `ViTFeatureExtractor`.

## Who can review?

@patrickvonplaten @patil-suraj @LysandreJik


Can't load config for hosted model, works when downloaded

I have recently (19 hours ago) uploaded a new model to huggingface: https://huggingface.co/EMBEDDIA/sloberta

When attempting to load it with `model = AutoModelForMaskedLM.from_pretrained("EMBEDDIA/sloberta")`  I get the following error:

```
Traceback (most recent call last):
  File "/home/mulcar/.conda/envs/transformerslatest/lib/python3.8/site-packages/transformers/configuration_utils.py", line 353, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mulcar/.conda/envs/transformerslatest/lib/python3.8/site-packages/transformers/modeling_auto.py", line 1105, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/home/mulcar/.conda/envs/transformerslatest/lib/python3.8/site-packages/transformers/configuration_auto.py", line 272, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/mulcar/.conda/envs/transformerslatest/lib/python3.8/site-packages/transformers/configuration_utils.py", line 362, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'EMBEDDIA/sloberta'. Make sure that:

- 'EMBEDDIA/sloberta' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'EMBEDDIA/sloberta' is the correct path to a directory containing a config.json file
```

If I download the model from the huggingface.co, eg. cloning the model's repo, it loads perfectly fine. Is there a waiting time before a new model is completely added to the system? Or is there an other issue going on?


[M2M100] remove final_logits_bias

# What does this PR do?

M2M100 does not need `final_logits_bias`, this PR removes it from the `M2M100ForConditionalGeneration`


Fix cross-attention head mask for Torch encoder-decoder models

1. This PR fixes head masking for the cross-attention module in the following models:

- BART,
- Blenderbot,
- Blenderbot_small,
- FSMT,
- LED,
- M2M_100,
- Marian,
- MBart,
- Pegasus.
- T5

2. This PR also contains slight changes in docstrings so that it will be clear that `head_mask` is related to the config of an encoder and the shape of `decoder_head_mask` and `cross_head_mask` depends on the config of a decoder.

3. This PR enables `test_headmasking` for M2M_100 model.

**Reviewers:** @patrickvonplaten @patil-suraj

Fixes: #10540


fix flaky m2m100 test

# What does this PR do?

The `test_retain_grad_hidden_states_attentions` test is sometimes failing for `M2M100`, with the error
` AttributeError: 'NoneType' object has no attribute 'retain_grad'`

 This is because of `layerdrop` sometimes a layer is skipped and the `encoder_attenion/decoder_attentions/cross_attentions` can be `None`.

This PR sets the `config.encoder_layerdrop` and `config.decoder_layerdrop` to 0  in tests to make the tests deterministic.


AlbertForSequenceClassification random output

I use AlbertForSequenceClassification interface as follows:
`import torch
from transformers import  BertTokenizer,AlbertConfig,AlbertForSequenceClassification
import numpy
pretrained = "./albert_chinese_base"
tokenizer = BertTokenizer.from_pretrained(pretrained)
config = AlbertConfig.from_json_file('./albert_chinese_base/config.json')
config.output_hidden_states = True
model = AlbertForSequenceClassification.from_pretrained(pretrained, config = config)
inputtext = "今天心情情很好啊，买了很多东西，我特别喜欢，终于有了自己喜欢的电子产品，这次总算可以好好学习了"
max_length = 128
tokenized_text=tokenizer.encode_plus(inputtext, add_special_tokens = True, # add [CLS], [SEP]
                                 max_length = max_length, # max length of the text that can go to BERT
                                 pad_to_max_length = True, # add [PAD] tokens
                                 return_attention_mask = True, # add attention mask to not focus on pad tokens
                                 return_tensors="pt")
outputs=model(input_ids=tokenized_text["input_ids"], token_type_ids=tokenized_text["token_type_ids"], attention_mask=tokenized_text["attention_mask"])
print(outputs.logits)`

but, when I run this code, an error occured as follows:

`Some weights of the model checkpoint at ./albert_chinese_base were not used when initializing AlbertForSequenceClassification: ['predictions.bias', 'predictions.LayerNorm.weight', 'predictions.LayerNorm.bias', 'predictions.dense.weight', 'predictions.dense.bias', 'predictions.decoder.weight', 'predictions.decoder.bias']

-This IS expected if you are initializing AlbertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).

-This IS NOT expected if you are initializing AlbertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of AlbertForSequenceClassification were not initialized from the model checkpoint at ./albert_chinese_base and are newly initialized: ['classifier.weight', 'classifier.bias']`

And I print outputs.logits value, found that the value is different every time you run,the logits value like this:
tensor([[0.3077, 0.1200]], grad_fn=)
tensor([[-0.3245, -0.3117]], grad_fn=)
so, I wonder AlbertForSequenceClassification model is not initialized correctly the value of ['classifier.weight', 'classifier.bias'] and
random value every time I run, and BertForSequenceClassification result output is right,so how can I slove the problem?


[examples template] added max_sample args and metrics changes

# What does this PR do?

This PR adds the same as https://github.com/huggingface/transformers/pull/10551 and https://github.com/huggingface/transformers/pull/10436 to the cookie-cutter template.

Fixes #10423

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## review:
@stas00


Speedup tf tests

Fyi @sgugger @patrickvonplaten @stas00, I'm temporarily marking these tests as slow as they take more than 1h30 minutes and prevent the CI from completing, therefore preventing any relevant information from the TF tests.

I'm working on improving the CI times so this is temporary (will revert by Friday).


[docs] How to solve "Title level inconsistent" sphinx error

This PR documents an easy solution to the "Title level inconsistent" puzzle when adding a new sub-section to an `.rst` doc.

@sgugger


Pass encoder outputs into GenerationMixin

# What does this PR do?

For encoder-decoder models such as T5, `GenerationMixin.generate()` currently runs both the encoder and the decoder. This PR allows one to pass already-computed `encoder_outputs` into this method, thus only the decoder will be run.

The flexibility to skip the encoder in the generation utilities is useful for several different scenarios, such as:
- The T5 encoder can encode `inputs_embeds` instead of `input_ids`. However, this is not possible within the generation utilities because only `input_ids` is accepted. With the changes in this PR, one can encode the `inputs_embeds` separately, and pass the encoder outputs to `generate()`. This is a partial solution to the issue https://github.com/huggingface/transformers/issues/6535.
- In some applications, the same encoder outputs are reused in different decoding processes. It would be computationally efficient not having to recompute the encoder outputs.
- This would also allow altering the encoder outputs for the purpose of incorporating additional information, etc. (In general, I think it is good practice to offer this option, where the encoding process is "decoupled" from the generation utilities.)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Check layer types for Optimizer construction

# What does this PR do?

As pointed out on the [forum](https://discuss.huggingface.co/t/parameter-groups-and-gpt2-layernorm/4239), `Trainer` currently excludes form weight decay layernorm layers by using a name pattern, which is not consistently followed by all models.

This PR actual checks the layer types and adds some tests.


No model card for  roberta-large-finetuned-wsc

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

I can not finetune this model `roberta-large-finetuned-wsc`, it doesn't have model card.

## Your contribution
Please fix this!


Fairscale FSDP fix model save

# What does this PR do?

This PR fixes the fact training with fairscale fully-sharded wrapper was hanging: it looks like recent changes in fairscale make a synchronization during the model state dict call, which results in the training hanging if we don't call that state_dict method on all processes. This PR addresses that.


Fix version control with anchors

# What does this PR do?

In urls containing an anchor (such as https://huggingface.co/transformers/master/installation.html#caching-models), the version controller was not finding the right version (basically because the page wasn't ending with .html). This PR fixes that.

Fixes #10559


[FeatureExtractorSavingUtils] Refactor PretrainedFeatureExtractor

# What does this PR do?

This PR refactors the class `PreTrainedFeatureExtractor`. The following changes are done to move functionality that is shared between sequence and image feature extractors into a separate file. This should unblock the PRs of [DETR](https://github.com/huggingface/transformers/pull/9998), [VIT](https://github.com/huggingface/transformers/pull/10513), and [CLIP](https://github.com/huggingface/transformers/pull/10426)

- `PreTrainedFeatureExtractor` is renamed to `PreTrainedSequenceFeatureExtractor` because it implicitly assumed that the it will treat only sequential inputs (a.k.a sequence of float values or a sequence of float vectors). `PreTrainedFeatureExtractor` was too general
- All functionality that is shared between Image and Speech feature extractors (which IMO all relates to "saving" utilities) is moved to a `FeatureExtractorSavingUtilsMixin`
- `BatchFeature` is moved from the `feature_extraction_sequence_utils.py` to `feature_extraction_common_utils.py` to be used by the `PreTrainedImageFeatureExtractor` class as well
- The tests are refactored accordingly

The following things were assumed before applying the changes.

- In the mid-term future there will only be three modalities in HF: text, sequential features (value sequence, vector sequence), image features (2D non-sequential array)
- Models, such as ViT, DETR & CLIP will call their "preprocessor" `VITFeatureExtractor`, .... IMO, feature extractor is also a fitting name for image recognition (see: https://en.wikipedia.org/wiki/Feature_extraction) so that it is assumed that for image-text or image-only models there will be a `PreTrainedImageFeatureExtractor`, a `VITFeatureExtractor`, (and maybe a VITTokenizer & VITProcessor as well, but not necessary). For vision-text models that do require both a tokenizer and a feature extractor such as CLIP it is assumed that the classes `CLIPFeatureExtractor` and `CLIPTokenizer` are wrapped into a `CLIPProcessor` class similar to `Wav2Vec2Processor`. I think this is the most important assumption that is taken here, so we should make sure we are on the same page here @LysandreJik @sgugger @patil-suraj @NielsRogge
- Image - Text or Image - only models won't require a `BatchImageFeature` or `BatchImage`, but can just use `BatchFeature`. From looking at the code in @NielsRogge's PR here: https://github.com/huggingface/transformers/pull/10513 this seems to be the case.

# Backwards compatibility:

The class `PreTrainedFeatureExtractor` was accessible via:

```python
from transformers import PreTrainedFeatureExtractor
```

but is now replaced by `PreTrainedSequenceFeatureExtractor`. However, since `PreTrainedFeatureExtractor` so far was only available on master, this change is OK IMO.


Enable torch 1.8.0 on GPU CI

This enables torch 1.8.0 on the GPU CI, and disables the torch-scatter tests today as they're creating issues and blocking the CI pipeline.


CUBLAS_STATUS_INTERNAL_ERROR at examples/question-answering/run_qa.py

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.10.20-1-lts-x86_64-with-glibc2.2.5
- Python version: 3.8.3
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:  No

### Who can help
- @LysandreJik
- @sgugger

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
```
#!/bin/bash

python3 -m venv env
source env/bin/activate
pip install torch
pip install datasets
git clone https://github.com/huggingface/transformers.git
pip install -e transformers/
python transformers/examples/question-answering/run_qa.py \
	--model_name_or_path bert-base-uncased \
	--dataset_name squad \
	--do_train \
	--do_eval \
	--per_device_train_batch_size 1 \
	--learning_rate 3e-5 \
	--num_train_epochs 4 \
	--max_seq_length 384 \
	--doc_stride 128 \
	--output_dir ./models/
```
```
03/08/2021 09:54:08 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 2distributed training: False, 16-bits training: False
03/08/2021 09:54:08 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir=./models/, overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=False, evaluation_strategy=IntervalStrategy.NO, prediction_loss_only=False, per_device_train_batch_size=1, per_device_eval_batch_size=8, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=3e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=4.0, max_steps=-1, lr_scheduler_type=SchedulerType.LINEAR, warmup_ratio=0.0, warmup_steps=0, logging_dir=runs/Mar08_09-54-06_inf-105-gpu-1, logging_strategy=IntervalStrategy.STEPS, logging_first_step=False, logging_steps=500, save_strategy=IntervalStrategy.STEPS, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level=O1, fp16_backend=auto, fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name=./models/, disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, report_to=[], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, _n_gpu=2)
03/08/2021 09:54:08 - WARNING - datasets.builder -   Reusing dataset squad (/home/blozano/.cache/huggingface/datasets/squad/plain_text/1.0.0/0fd9e01360d229a22adfe0ab7e2dd2adc6e2b3d6d3db03636a51235947d4c6e9)
[INFO|configuration_utils.py:463] 2021-03-08 09:54:09,206 >> loading configuration file https://huggingface.co/bert-base-uncased/resolve/main/config.json from cache at /home/blozano/.cache/huggingface/transformers/3c61d016573b14f7f008c02c4e51a366c67ab274726fe2910691e2a761acf43e.637c6035640bacb831febcc2b7f7bee0a96f9b30c2d7e9ef84082d9f252f3170
[INFO|configuration_utils.py:499] 2021-03-08 09:54:09,207 >> Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.4.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

[INFO|configuration_utils.py:463] 2021-03-08 09:54:09,509 >> loading configuration file https://huggingface.co/bert-base-uncased/resolve/main/config.json from cache at /home/blozano/.cache/huggingface/transformers/3c61d016573b14f7f008c02c4e51a366c67ab274726fe2910691e2a761acf43e.637c6035640bacb831febcc2b7f7bee0a96f9b30c2d7e9ef84082d9f252f3170
[INFO|configuration_utils.py:499] 2021-03-08 09:54:09,510 >> Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.4.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

[INFO|tokenization_utils_base.py:1721] 2021-03-08 09:54:10,138 >> loading file https://huggingface.co/bert-base-uncased/resolve/main/vocab.txt from cache at /home/blozano/.cache/huggingface/transformers/45c3f7a79a80e1cf0a489e5c62b43f173c15db47864303a55d623bb3c96f72a5.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99
[INFO|tokenization_utils_base.py:1721] 2021-03-08 09:54:10,138 >> loading file https://huggingface.co/bert-base-uncased/resolve/main/tokenizer.json from cache at /home/blozano/.cache/huggingface/transformers/534479488c54aeaf9c3406f647aa2ec13648c06771ffe269edabebd4c412da1d.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4
[INFO|modeling_utils.py:1051] 2021-03-08 09:54:10,501 >> loading weights file https://huggingface.co/bert-base-uncased/resolve/main/pytorch_model.bin from cache at /home/blozano/.cache/huggingface/transformers/a8041bf617d7f94ea26d15e218abd04afc2004805632abc0ed2066aa16d50d04.faf6ea826ae9c5867d12b22257f9877e6b8367890837bd60f7c54a29633f7f2f
[WARNING|modeling_utils.py:1158] 2021-03-08 09:54:12,594 >> Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[WARNING|modeling_utils.py:1169] 2021-03-08 09:54:12,594 >> Some weights of BertForQuestionAnswering were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
03/08/2021 09:54:12 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/blozano/.cache/huggingface/datasets/squad/plain_text/1.0.0/0fd9e01360d229a22adfe0ab7e2dd2adc6e2b3d6d3db03636a51235947d4c6e9/cache-a560de6b2f76743b.arrow
03/08/2021 09:54:12 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/blozano/.cache/huggingface/datasets/squad/plain_text/1.0.0/0fd9e01360d229a22adfe0ab7e2dd2adc6e2b3d6d3db03636a51235947d4c6e9/cache-15b011eed342eca6.arrow
[INFO|trainer.py:471] 2021-03-08 09:54:15,885 >> The following columns in the evaluation set  don't have a corresponding argument in `BertForQuestionAnswering.forward` and have been ignored: example_id, offset_mapping.
[INFO|trainer.py:929] 2021-03-08 09:54:15,937 >> ***** Running training *****
[INFO|trainer.py:930] 2021-03-08 09:54:15,937 >>   Num examples = 88524
[INFO|trainer.py:931] 2021-03-08 09:54:15,937 >>   Num Epochs = 4
[INFO|trainer.py:932] 2021-03-08 09:54:15,937 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:933] 2021-03-08 09:54:15,937 >>   Total train batch size (w. parallel, distributed & accumulation) = 2
[INFO|trainer.py:934] 2021-03-08 09:54:15,937 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:935] 2021-03-08 09:54:15,937 >>   Total optimization steps = 177048
  0%|                                                                                | 0/177048 [00:00<?, ?it/s]Traceback (most recent call last):
  File "transformers/examples/question-answering/run_qa.py", line 507, in <module>
    main()
  File "transformers/examples/question-answering/run_qa.py", line 481, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/blozano/finetune_qa/transformers/src/transformers/trainer.py", line 1036, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/blozano/finetune_qa/transformers/src/transformers/trainer.py", line 1420, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/blozano/finetune_qa/transformers/src/transformers/trainer.py", line 1452, in compute_loss
    outputs = model(**inputs)
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 167, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 177, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/blozano/finetune_qa/transformers/src/transformers/models/bert/modeling_bert.py", line 1775, in forward
    outputs = self.bert(
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/blozano/finetune_qa/transformers/src/transformers/models/bert/modeling_bert.py", line 971, in forward
    encoder_outputs = self.encoder(
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/blozano/finetune_qa/transformers/src/transformers/models/bert/modeling_bert.py", line 568, in forward
    layer_outputs = layer_module(
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/blozano/finetune_qa/transformers/src/transformers/models/bert/modeling_bert.py", line 456, in forward
    self_attention_outputs = self.attention(
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/blozano/finetune_qa/transformers/src/transformers/models/bert/modeling_bert.py", line 387, in forward
    self_outputs = self.self(
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/blozano/finetune_qa/transformers/src/transformers/models/bert/modeling_bert.py", line 253, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/blozano/finetune_qa/env/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

```
```
NVIDIA-SMI 460.56       Driver Version: 460.56       CUDA Version: 11.2
```

## Expected behavior

The expected default behavior as stated in transformers/examples/question-answering/README.md


Fix typo in docstring for pipeline

# What does this PR do?
Fixed typo in docstring for pipeline ("conversation" -> "conversational")
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[M2M100] fix positional embeddings

# What does this PR do?

The torchscript tests for `M2M100` are failing on master. This is because the `weights` in `M2M100SinusoidalPositionalEmbedding` are initially not on the same device as the rest of the parameters.

The PR makes the `weights` as `nn.Parameter` so they'll be on the same device.


Small question about BertForMaskedLM usage on TF model

Hi everyone,
I was using BertForMaskedLM to predict possible candidate words in the content.

For example:
**cat like to drink [MASK], so am I.**

on [tf1 bert-pretrained model](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip)
output would be
`[[{'sequence': '[CLS] cat like to drink it, so am i. [SEP]', 'score': 0.22914603352546692, 'token': 2009, 'token_str': 'i t'}, {'sequence': '[CLS] cat like to drink water, so am i. [SEP]', 'score': 0.1088637188076973, 'token': 2300, 'token_str': 'w a t e r'}, {'sequence': '[CLS] cat like to drink blood, so am i. [SEP]', 'score': 0.1075243279337883, 'token': 2668, 'token_str': 'b l o o d'}]]`

However, in [tf2 bert-pretrained model](https://storage.googleapis.com/cloud-tpu-checkpoints/bert/keras_bert/uncased_L-12_H-768_A-12.tar.gz)
output gives,
`[[{'sequence': '[CLS] cat like to drink∘, so am i. [SEP]', 'score': 0.0002078865800285712, 'token': 30126, 'token_str': '# # ∘'}, {'sequence': '[CLS] cat like to drink zinc, so am i. [SEP]', 'score': 0.00020266805950086564, 'token': 15813, 'token_str': 'z i n c'}, {'sequence': '[CLS] cat like to drink organic, so am i. [SEP]', 'score': 0.00019718357361853123, 'token': 7554, 'token_str': 'o r g a n i c'}]]
`
It seems tf2 will randomly give unreasonable words, but I have no idea why causing this.

I notice the implementation of BERT on TF1 and TF2 are different (I think so? since tf2 use keras as main core implementation), however, does the architecture vary from TF1 pretrained model to TF2 that cause BertForMaskedLM give different embedding?  Does BertForMaskedLM support reading weight on TF2 model?

The code I am using are something like this
```
config = BertConfig.from_pretrained("./pretrained/tf2")
config.is_decoder=False

tokenizer = BertTokenizer.from_pretrained("./pretrained/tf2")
model = BertForMaskedLM.from_pretrained("./pretrained/tf2",config=config,from_tf=True)
```
tf1 checkpoint in FileZilla
![tf1](https://user-images.githubusercontent.com/56808566/110295862-96b08800-802c-11eb-985e-90b12449a21a.PNG)
tf2
![tf2](https://user-images.githubusercontent.com/56808566/110295918-a760fe00-802c-11eb-84a5-b35dd101da7f.PNG)

Am I using wrong API(like should I use other API instead other than BertForMaskedLM) or any place I have to change to make it work?
Any suggestion or reflection are sincerely grateful, thanks in advance!


Can't reproduce xlm-roberta-large finetuned result on XNLI

# ❓ Questions & Help

I'm trying to finetune `xlm-roberta-large` on MNLI English training data and make zero-shot classification on XNLI dataset.
However, I found that `xlm-roberta-large` is super sensitive to hyper parameters. The reported average accuracy is 80.9, while my model can only achieve 79.74, which is 1% less than the reported accuracy.

I used Adam optimizer with 5e-6 learning rate and the batch size is 16.  Any one can suggest better hyperparameters to reproduce the XNLI result of `xlm-roberta-large`?