## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

 `transformers` version: 3.0.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?:  True
- Using distributed or parallel set-up in script?: No

### Who can help
albert, bert, xlm: @LysandreJik
trainer: @sgugger

## Information

Model I am using (XLMRobertaForSequenceClassification):

The problem arises when using:
* [ *] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [* ] my own task or dataset: (give details below)

## To reproduce

# This training code is based on the `run_glue.py` script here:
# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128

for epoch_i in range(0, epochs):

    # ========================================
    #               Training
    # ========================================

    # Perform one full pass over the training set.

    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')

    # Measure how long the training epoch takes.
    t0 = time.time()

    # Reset the total loss for this epoch.
    total_train_loss = 0

    # Put the model into training mode. Don't be mislead--the call to
    # `train` just changes the *mode*, it doesn't *perform* the training.
    # `dropout` and `batchnorm` layers behave differently during training
    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)
    model.train()

    # For each batch of training data...
    for step, batch in enumerate(train_dataloader):

        # Progress update every 40 batches.
        if step % 40 == 0 and not step == 0:
            # Calculate elapsed time in minutes.
            elapsed = format_time(time.time() - t0)

            # Report progress.
            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))

        # Unpack this training batch from our dataloader.
        #
        # As we unpack the batch, we'll also copy each tensor to the GPU using the
        # `to` method.
        #
        # `batch` contains three pytorch tensors:
        #   [0]: input ids
        #   [1]: attention masks
        #   [2]: labels
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)

        # Always clear any previously calculated gradients before performing a
        # backward pass. PyTorch doesn't do this automatically because
        # accumulating the gradients is "convenient while training RNNs".
        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)
        model.zero_grad()

        # Perform a forward pass (evaluate the model on this training batch).
        # The documentation for this `model` function is here:
        # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
        # It returns different numbers of parameters depending on what arguments
        # arge given and what flags are set. For our useage here, it returns
        # the loss (because we provided labels) and the "logits"--the model
        # outputs prior to activation.
        loss, logits = model(b_input_ids,
                             token_type_ids=None,
                             attention_mask=b_input_mask,
                             labels=b_labels)

        # Accumulate the training loss over all of the batches so that we can
        # calculate the average loss at the end. `loss` is a Tensor containing a
        # single value; the `.item()` function just returns the Python value
        # from the tensor.
        total_train_loss += loss.item()

        # Perform a backward pass to calculate the gradients.
        loss.backward()

        # Clip the norm of the gradients to 1.0.
        # This is to help prevent the "exploding gradients" problem.
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        # Update parameters and take a step using the computed gradient.
        # The optimizer dictates the "update rule"--how the parameters are
        # modified based on their gradients, the learning rate, etc.
        optimizer.step()

        # Update the learning rate.
        scheduler.step()

    # Calculate the average loss over all of the batches.
    avg_train_loss = total_train_loss / len(train_dataloader)

    # Measure how long this epoch took.
    training_time = format_time(time.time() - t0)

    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(training_time))

    # ========================================
    #               Validation
    # ========================================
    # After the completion of each training epoch, measure our performance on
    # our validation set.

    print("")
    print("Running Validation...")

    t0 = time.time()

    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    model.eval()

    # Tracking variables
    total_eval_accuracy = 0
    total_eval_loss = 0
    nb_eval_steps = 0

    # Evaluate data for one epoch
    for batch in validation_dataloader:

        # Unpack this training batch from our dataloader.
        #
        # As we unpack the batch, we'll also copy each tensor to the GPU using
        # the `to` method.
        #
        # `batch` contains three pytorch tensors:
        #   [0]: input ids
        #   [1]: attention masks
        #   [2]: labels
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)

        # Tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training).
        with torch.no_grad():

            # Forward pass, calculate logit predictions.
            # token_type_ids is the same as the "segment ids", which
            # differentiates sentence 1 and 2 in 2-sentence tasks.
            # The documentation for this `model` function is here:
            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
            # Get the "logits" output by the model. The "logits" are the output
            # values prior to applying an activation function like the softmax.
            (loss, logits) = model(b_input_ids,
                                   token_type_ids=None,
                                   attention_mask=b_input_mask,
                                   labels=b_labels)

        # Accumulate the validation loss.
        total_eval_loss += loss.item()

        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.to('cpu').numpy()

        # Calculate the accuracy for this batch of test sentences, and
        # accumulate it over all batches.
        total_eval_accuracy += flat_accuracy(logits, label_ids)

    # Report the final accuracy for this validation run.
    avg_val_accuracy = total_eval_accuracy / len(validation_dataloader)
    print("  Accuracy: {0:.2f}".format(avg_val_accuracy))

    # Calculate the average loss over all of the batches.
    avg_val_loss = total_eval_loss / len(validation_dataloader)

    # Measure how long the validation run took.
    validation_time = format_time(time.time() - t0)

    print("  Validation Loss: {0:.2f}".format(avg_val_loss))
    print("  Validation took: {:}".format(validation_time))

    # Record all statistics from this epoch.
    training_stats.append(
        {
            'epoch': epoch_i + 1,
            'Training Loss': avg_train_loss,
            'Valid. Loss': avg_val_loss,
            'Valid. Accur.': avg_val_accuracy,
            'Training Time': training_time,
            'Validation Time': validation_time
        }
    )

print("")
print("Training complete!")

print("Total training took {:} (h:mm:ss)".format(format_time(time.time()-total_t0)))

## Expected behavior
THE ERROR:
======== Epoch 1 / 4 ========
Training...
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-92-840aefe69c26> in <module>()
     85                              token_type_ids=None,
     86                              attention_mask=b_input_mask,
---> 87                              labels=b_labels)
     88
     89         # Accumulate the training loss over all of the batches so that we can

ValueError: too many values to unpack (expected 2)

 I am working an a four-label classification task (on an Arabic dataset). I have run this code before but it is not working now. I get this error: ValueError: too many values to unpack (expected 2). I have not changed any of the steps or the preprocessing but it raises up this error this time. The tensor for each label instance is [0,1,2,3]. The error points to the labels parameter. Could you please suggest solutions?


LayoutLM Tensorflow model

# 🚀 Feature request

It would be great if there was a TF version of the layoutlm. I see there are scripts in the repo to convert PyTorch checkpoints to TF models but I think the requirement is to have a TF model architecture to be able to load PyTorch model's wights in it.

## Motivation
We are using TF in production and we'd love to be able to use the layoutlm.

## Your contribution
I am happy to tackle the conversion.

I was wondering if there are instructions on how to do the conversion properly so that it can be added to the repo.


Matrix multiplication error for ReformerModelWithLMHead when tie_word_embeddings is True

## Environment info

- `transformers` version: 4.3.2
- Platform: Windows 10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.0 cpu-only
- Tensorflow version (GPU?): not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

- longformer, reformer, transfoxl, xlnet: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Reformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the following script:
```python
import torch
from transformers import ReformerConfig, ReformerModelWithLMHead

config = ReformerConfig(is_decoder=True, tie_word_embeddings=True)
model = ReformerModelWithLMHead(config)

inp = torch.randint(0, 100, (1, 4096))
out = model(inp)
```

2. The error:
```
Traceback (most recent call last):
  File "./test.py", line 8, in <module>
    out = model(inp)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\transformers\models\reformer\modeling_reformer.py", line 2248, in forward
    logits = self.lm_head(sequence_output)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\transformers\models\reformer\modeling_reformer.py", line 1761, in forward
    return apply_chunking_to_forward(self.forward_chunk, self.chunk_size_lm_head, self.seq_len_dim, hidden_states)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\transformers\modeling_utils.py", line 1787, in apply_chunking_to_forward
    return forward_fn(*input_tensors)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\transformers\models\reformer\modeling_reformer.py", line 1764, in forward_chunk
    hidden_states = self.decoder(hidden_states)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\xe442\anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: mat1 and mat2 shapes cannot be multiplied (4096x512 and 256x320)
```

## Expected behavior

There should be no errors.


[Trainer] implement gradient_accumulation_steps support in DeepSpeed integration

This PR:

Fixes in a bug:
- `lr_scheduler.step()` shouldn't be called under DeepSpeed - it's already called in its `optimizer.step()` internally - so it was moving through the scheduler rate change at twice the speed :(

Adds support for `gradient_accumulation_steps`:
* makes `gradient_accumulation_steps` work with deepspeed - for nuances see: https://github.com/microsoft/DeepSpeed/issues/776 - it required a lot of `if` / `if nots` - not helping the readability of the trainer  - and took a lot of trial and error to figure out - but what to do
* adds a corresponding doc
* adds a first serious quality test for DeepSpeed that measures that `gradient_accumulation_steps` works - modelled after `test_trainer.py`'s own `test_gradient_accumulation` and extends it to compare loss as well, and also tests that the optimizer actually kicked in - with fp16 deepspeed it normally takes a few dozen steps before it kicks in with dynamic scaling enabled.
* extends `testing_utils` with a `mockenv_context` which is similar to `@mockenv`, but which can be used inside the test as context manager if multiple env vars need to be tested - `@mockenv` is only useful as a decorator. At the end I think I don't really need it as using the same env worked for all tests, but it might come handy if ports don't get released fast enough and then the test will use different ports - I'm concerned about CIs. And it's easier to re-use the class-wide env, rather hardcoding or creating a global variable - so it's just cleaner too.

Suggestion/Question:
* `get_regression_trainer` is very awesome! But importing it from a test file is not great - probably should move it and its components into a utilities file - `testing_utils.py` or create a new one `testing_training_utils.py` and in the future add other trainer-testing specific utils in there? Though this should be dealt with in a separate PR.

@sgugger


[Example] Using label_smoothing_factor raise error when evaluating model

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: Ubuntu 20.04
- Python version: 3.8
- PyTorch version (GPU): 1.6.0

### Who can help

Library:
- pipelines: @LysandreJik

Examples:
- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## Information

Model I am using BERT:

The problem arises when using:
* [x] the official example scripts: https://github.com/huggingface/transformers/blob/master/examples/legacy/token-classification/run_ner.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. I run the old script run_ner.py with default label_smoothing_factor = 0.0.  It works well.

2. I add label_smoothing_factor = 0.1 to JSON config file.
`{
    "data_dir": "/home/dzungle/NER/data/",
    "train_file": "/home/dzungle/NER/data/train.csv",
    "validation_file": "/home/dzungle/data/dev.csv",
    "model_name_or_path": "emilyalsentzer/Bio_ClinicalBERT",
    "output_dir": "/home/dzungle/NER/models/",
    "label_smoothing_factor": 0.1,
    "max_seq_length": 256,
    "num_train_epochs": 1,
    "per_device_train_batch_size": 8,
    "gradient_accumulation_steps": 4,
    "per_device_eval_batch_size": 1,
    "save_steps": 1000,
    "eval_steps" : 50,
    "save_total_limit":1,
    "seed": 1,
    "do_train": true,
    "do_eval": true,
    "do_predict": true,
    "overwrite_output_dir" : true,
    "evaluate_during_training" : true
}`

3. I run the script and it works well for training but got an error when evaluating.

**Error:**
```
Traceback (most recent call last):
  File "run_ner.py", line 333, in <module>
    main()
  File "run_ner.py", line 282, in main
    result = trainer.evaluate()
  File "/home/dzungle/miniconda3/envs/hppi/lib/python3.8/site-packages/transformers/trainer.py", line 1604, in evaluate
    output = self.prediction_loop(
  File "/home/dzungle/miniconda3/envs/hppi/lib/python3.8/site-packages/transformers/trainer.py", line 1742, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/home/dzungle/miniconda3/envs/hppi/lib/python3.8/site-packages/transformers/trainer.py", line 1874, in prediction_step
    labels = nested_detach(tuple(inputs.get(name) for name in self.label_names))
  File "/home/dzungle/miniconda3/envs/hppi/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 111, in nested_detach
    return type(tensors)(nested_detach(t) for t in tensors)
  File "/home/dzungle/miniconda3/envs/hppi/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 111, in <genexpr>
    return type(tensors)(nested_detach(t) for t in tensors)
  File "/home/dzungle/miniconda3/envs/hppi/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 112, in nested_detach
    return tensors.detach()
AttributeError: 'NoneType' object has no attribute 'detach'
```
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
As I known, label_smoothing_factor is a new feature of recent transformers version. I would expect that the script with label_smoothing_factor=0.1 works well as using default value 0.0.


[ci] don't fail when there are no zombies

fixes:
```
Run pkill -f tests; pkill -f examples
4
Error: Process completed with exit code 1.
```

Didn't think that it'd `exit(1)` when there is nothing to kill

@sgugger


pretraining objective of T5 model

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Hi, it would be great to have pretraining of T5 model implemented. Currently, run_mlm.py script does not support it.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

T5 is the SOTA model and having pertaining would be very helpful to the community.


Issue Loading bert-based-german-cased

Message on the website is:
 Can't load tokenizer using from_pretrained, please update its configuration: 400 Client Error: Bad Request for url: https://int-deepset-models-bert.s3.eu-central-1.amazonaws.com/pytorch/bert-base-german-cased-vocab.txt


Documentation of the decode method is missing

The tokenizer documentation [page](https://huggingface.co/transformers/main_classes/tokenizer.html) is generated from the following files:

- tokenization_utils_base.py
- tokenization_utils_fast.py
- tokenization_utils.py

At least the documentation of the decode method is missing even if it is properly documented in the [source file](https://github.com/huggingface/transformers/blob/9a7e63729f3ff6ddf065fd0d443421e46b1a2ffb/src/transformers/tokenization_utils_base.py#L3099).

@sgugger
Could you please have a look?


fixes #10303

# What does this PR do?

Fixes #10303

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Documentation: @sgugger


convert_tokens_to_string documentation bug

The [documentation](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.convert_tokens_to_string) states that convert_tokens_to_string would convert _a sequence of token ids in a single string._

That is actually not correct as it converts a sequence of tokens. The method that converts a sequence of token ids is the decode method.

Documentation: @sgugger


Tensorflow not found but i can import it

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: macOS-11.2.1-arm64-arm-64bit
- Python version: 3.8.6
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

@jplu, @patrickvonplaten, @LysandreJik

## Description
When i import transformers i get message bellow.

## To reproduce

Steps to reproduce the behavior:

1. install tf for mac m1(https://github.com/apple/tensorflow_macos)
2. install transformers
3. import transformers

Message when i import transformers:
```
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
```

## Expected behavior
I'd like to make transformers find tensorflow.


[WIP] Add Megatron-11B

# What does this PR do?

Fixes #9560

This PR introduces the Megatron model as described in https://github.com/pytorch/fairseq/blob/master/examples/megatron_11b/README.md
This one will probably be fun to test with DeepSpeed, as @stas00 mentioned it's referenced a lot in its docs :smile:

It's important to mention that there are actually two independent implementations of [Megatron-LM](https://arxiv.org/pdf/1909.08053.pdf):
* The one described in the original paper belongs to NVIDIA (https://github.com/NVIDIA/Megatron-LM), but they released only a 345M checkpoint. It's also based on a rewrite of GPT2 and is not compatible with the current huggingface implementation due to minor changes, like LayerNorm reordering (see https://github.com/NVIDIA/Megatron-LM/issues/37).
* [Fairseq](https://github.com/pytorch/fairseq/blob/master/examples/megatron_11b/README.md), on the other hand, uses its own GPT2 version based on their encoder-decoder framework (with the encoder removed) and it does release the colossal 11B pretrained model.

After some tinkering I realized that fairseq's checkpoint is already pretty compatible with the existing BART port. So, based on that and the fact that NVIDIA doesn't plan on releasing the 3B and 8B checkpoints, **I chose to port only the fairseq version**.

**NOTE:** The original fairseq implementation requires an 8-GPU server to even load the model weights, so I just load the checkpoints manually one by one and merge the model-parallelized tensors into single-model ones.

### How to reproduce the conversion

1. First, find a server with _at least 85GB of RAM_, this model is huge!
2. Next, download and untar the checkpoint:
```
# WARNING: this file is 19GB
wget https://dl.fbaipublicfiles.com/fairseq/models/model_parallel/megatron_11b.tar.gz
tar -xzvf megatron_11b.tar.gz
wget -P ./megatron_11b/ 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/encoder.json'
wget -P ./megatron_11b/ 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/vocab.bpe'
```
3. Run the conversion script
```
python convert_megatron_original_pytorch_checkpoint_to_pytorch.py --fairseq_path /path/to/megatron_11b --pytorch_dump_path /path/to/megatron_hf_dump
```
4. The conversion script will load the model-parallel shards of the checkpoint, group the sharded parameters and concatenate the weights, so that the [fairseq.ModelParallelTransformerLanguageModel](https://github.com/pytorch/fairseq/blob/3b27ed7996b0315f471c795cf9b7dfcc18467cbe/fairseq/model_parallel/models/transformer_lm.py) `state_dict` can be easily loaded into a CPU-compatible [faiseq.TransformerLanguageModel](https://github.com/pytorch/fairseq/blob/3b27ed7996b0315f471c795cf9b7dfcc18467cbe/fairseq/models/transformer_lm.py). The de-parallelisation is based on ParlAI's [conversion script](https://github.com/facebookresearch/ParlAI/blob/abfb771ac4ed2966d6f3ea22c7a38e4ebc9cc0f0/parlai/agents/bart/convert_fairseq_to_parlai.py#L258-L307).
5. Then the script will initialize the huggingface Megatron model and load the converted `state_dict` into it.

### Here's how Megatron differs from the existing BART/MBART implemenations:

1. The most controversial difference, IMO, is the missing encoder, since it's a decoder-only model. For now, I decided to remove the encoder parts inherited from MBART, bit left the encoder-dependent parts in the decoder (e.g. `encoder_hidden_states`, `encoder_attention_mask`) and the cross-attention to simplify the review process on your end.
2. Megatron uses `SinusoidalPositionalEmbedding` instead of learned ones, so I just yanked those from FSMT :smile:
3. Megatron does not have a `layernorm_embedding`
4. Minor detail: the `self_attn_layer_norm` is applied before self-attention (like in MBART) instead of after (like in BART).

### Important questions regarding the API:

1. What should be done about the missing encoder? I think the `decoder` variable can be left as is, since it's compatible with the fairseq checkpoint keys, but the `encoder_*` references in the code bother me a lot. We need to somehow strike a balance between `Copied from` and removing the unused parts.
2. I think the position of `self_attn_layer_norm` should be a parameter in the config, similar to `decoder_normalize_before=True` in faiseq. This will close the not-so-obvious difference between BART and MBART.
3. The existence of `layernorm_embedding` can also be parametrized, similar to `layernorm_embedding=False` in fairseq.

### Quick LM test

You can test out the model's capabilities like so (again, you'll probably need _at least 85GB RAM_, there's some weird memory duplication happening somewhere, this should not need more than 50):
```
from transformers import MegatronForCausalLM, MegatronTokenizer, TextGenerationPipeline

tokenizer = MegatronTokenizer.from_pretrained("megatron-11b")
model = MegatronForCausalLM.from_pretrained("anton-l/megatron-11b")

def generate(prompt, max_length=40, num_beams=5, num_return=3):
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids
    outputs = model.generate(
        input_ids=input_ids, num_beams=num_beams, num_return_sequences=num_return, max_length=max_length
    )
    decoded = tokenizer.batch_decode(outputs, skip_special_tokens=True)
    return decoded

print(generate("Before boarding your rocket to Mars, remember to pack these items: "))
```
```
['Before boarding your rocket to Mars, remember to pack these items: 1. A parachute.',
 'Before boarding your rocket to Mars, remember to pack these items: 1. A parachute $100 bill2. A copy of your passport3. A copy of your passport444',
 'Before boarding your rocket to Mars, remember to pack these items: 1. A parachute $1 million dollars dollars dollars dollars dollars dollars dollars dollars dollars dollars dollars dollars dollars dollars dollars dollars dollars']
```

To be honest, I'm not too impressed with its text-generation power. :smile: I guess it's either that the model was too large to train it for enough steps, or I missed something during the conversion. The original implementation does not have a text-generation script (or any non-wikitext results, for that matter), so I'm kinda in the dark here.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten, @patil-suraj


unexpected keyword argument 'forced_bos_token_id' when using mbart-large-50-many-to-many-mmt

When I try to run the example on the model card, I get this error;

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-88d049aaf9c0> in <module>
      5 tokenizer.src_lang = "hi_IN"
      6 encoded_hi = tokenizer(article_hi, return_tensors="pt")
----> 7 generated_tokens = model.generate(
      8     **encoded_hi,
      9     forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"]

~/opt/Python-3.8.2/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     24         def decorate_context(*args, **kwargs):
     25             with self.__class__():
---> 26                 return func(*args, **kwargs)
     27         return cast(F, decorate_context)
     28

~/opt/Python-3.8.2/lib/python3.8/site-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, **model_kwargs)
    831         if self.config.is_encoder_decoder:
    832             # add encoder_outputs to model_kwargs
--> 833             model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
    834
    835             # set input_ids as decoder_input_ids

~/opt/Python-3.8.2/lib/python3.8/site-packages/transformers/generation_utils.py in _prepare_encoder_decoder_kwargs_for_generation(self, input_ids, model_kwargs)
    376             argument: value for argument, value in model_kwargs.items() if not argument.startswith("decoder_")
    377         }
--> 378         model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
    379         return model_kwargs
    380

~/opt/Python-3.8.2/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

TypeError: forward() got an unexpected keyword argument 'forced_bos_token_id'
```

Looking at the code in the master repository, I can't see the generate function taking that argument anywhere at all so I'm unsure how to proceed with this.

_Originally posted by @IamAdiSri in https://github.com/huggingface/tokenizers/issues/633#issuecomment-781689632_


Object of type 'int64' is not JSON serializable in Trainer.save_checkpoint

I am using the recent run_ner.py example script to train an NER model. I want to evaluate the performance of the model during training and use the following command for training:

```
python3 run_ner.py
--model_name_or_path bert-base-uncased
--dataset_name conll2003
--return_entity_level_metrics
--output_dir conll-tmp
--overwrite_output_dir
--do_train
--do_eval
--do_predict
--evaluation_strategy steps
--logging_steps 10
--eval_steps 10
--load_best_model_at_end
```
I run the command in the current docker image huggingface/transformers-pytorch-gpu
However, I get the following error:
```
Traceback (most recent call last):
File "run_ner.py", line 470, in main()
File "run_ner.py", line 404, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)                                                                                File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 983, in train                                                                                self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)                                                                                File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1062, in _maybe_log_save_evaluate                                                                  self._save_checkpoint(model, trial, metrics=metrics)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1126, in _save_checkpoint                                                                          self.state.save_to_json(os.path.join(output_dir, "trainer_state.json"))                                                                                File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_callback.py", line 95, in save_to_json                                                                       json_string = json.dumps(dataclasses.asdict(self), indent=2, sort_keys=True) + "\n"                                                                                File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps                                                                                **kw).encode(obj)
File "/usr/lib/python3.6/json/encoder.py", line 201, in encode                                                                                chunks = list(chunks)
File "/usr/lib/python3.6/json/encoder.py", line 430, in _iterencode                                                                                yield from _iterencode_dict(o, _current_indent_level)
File "/usr/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict                                                                                yield from chunks
File "/usr/lib/python3.6/json/encoder.py", line 325, in _iterencode_list                                                                                yield from chunks
File "/usr/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict                                                                                yield from chunks
File "/usr/lib/python3.6/json/encoder.py", line 437, in _iterencode                                                                                o = _default(o)
File "/usr/lib/python3.6/json/encoder.py", line 180, in default                                                                                o.__class__.__name__)
TypeError: Object of type 'int64' is not JSON serializable
--
```


Converting fairseq NMT to transformers misses model weight

Hi there, question about fairseq NMT model ([FSMT](https://huggingface.co/transformers/model_doc/fsmt.html)) conversion.

I tried to convert my own fairseq-nmt model ([`transformer_wmt_en_de`](https://github.com/pytorch/fairseq/blob/master/fairseq/models/transformer.py#L1046)) based on [this conversion script](https://github.com/huggingface/transformers/blob/master/src/transformers/models/fsmt/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py).
However, `decoder.embed_out` weight is missing after converting fairseq model to transformers FSMT model. This parameter exists when not specifing  `--share-all-embeddings` or `--share-decoder-input-output-embed`, while official fairseq wmt models do not have `decoder.embed_out` weight because specifying `--share-all-embedding`.
https://github.com/pytorch/fairseq/issues/2537

Are there any solution or tips to converting own fairseq model?


AutoTokenizer from pretrained BERT throws TypeError when encoding certain input

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:  4.3.2
- Platform: Arch Linux
- Python version: 3.9.1
- PyTorch version (GPU?): 1.7.1, no
- Tensorflow version (GPU?): Not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
Guess from git blame: @LysandreJik , @thomwolf @n1t0

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
When I use a pretrained BERT tokenizer, it throws a TypeError on singleton input or input containing ø/æ/å.

It was discovered when I used the pretrained `Maltehb/danish-bert-botxo` which would fail in the below way on any input containing Danish characters (ø/æ/å), but I also realized that it happens with the standard `bert-base-uncased` as shown below.

Steps to reproduce the behavior:
1. Run these line
```py
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokenizer.encode(["hello", "world"])                          # <--- This works
tokenizer.encode(["hello"])                                   # <--- This throws the below shown stack trace
tokenizer.encode(["dette", "er", "en", "sø"])                 # <--- This throws the same error
```
Stack trace
```py
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-ef056deb5f59> in <module>
----> 1 tokenizer.encode(["hello"])

~/.venv/lib/python3.9/site-packages/transformers/tokenization_utils_base.py in encode(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, return_tensors, **kwargs)
   2102                 ``convert_tokens_to_ids`` method).
   2103         """
-> 2104         encoded_inputs = self.encode_plus(
   2105             text,
   2106             text_pair=text_pair,

~/.venv/lib/python3.9/site-packages/transformers/tokenization_utils_base.py in encode_plus(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2418         )
   2419
-> 2420         return self._encode_plus(
   2421             text=text,
   2422             text_pair=text_pair,

~/.venv/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py in _encode_plus(self, text, text_pair, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
    453
    454         batched_input = [(text, text_pair)] if text_pair else [text]
--> 455         batched_output = self._batch_encode_plus(
    456             batched_input,
    457             is_split_into_words=is_split_into_words,

~/.venv/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py in _batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose)
    380         )
    381
--> 382         encodings = self._tokenizer.encode_batch(
    383             batch_text_or_text_pairs,
    384             add_special_tokens=add_special_tokens,

TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I expect the model not to throw a type error when the types are the same.
I also expected that the tokenization would produce id's.

[This issue](https://github.com/alexandrainst/danlp/issues/113) is caused by the above
I am grateful for the software and thank you in advance for the help!


[predict] AttributeError: 'Seq2SeqTrainer' object has no attribute 'metrics_format'

Hi everybody

When using mbart for machine translation prediction, i got:

Traceback (most recent call last):
  File "/Users/lishuqi/Desktop/WAT2021/transformers-master/examples/seq2seq/run_seq2seq.py", line 667, in <module>
    main()
  File "/Users/lishuqi/Desktop/WAT2021/transformers-master/examples/seq2seq/run_seq2seq.py", line 637, in main
    metrics_formatted = trainer.metrics_format(metrics)
AttributeError: 'Seq2SeqTrainer' object has no attribute 'metrics_format'

Am I doing something wrong with the translation?
@patil-suraj


[examples/seq2seq] defensive programming + expand/correct README

This PR deals with the new s2s script and its usage - mostly documentation.

This PR:

`run_seq2seq.py`:
* checks for invalid column names

`README.md`:
* largely expands the document explaining and exemplifying the supported formats
* documents the nuances of t5 and mbart translation - I hope we fix this on the programmatical level in the future
* fixes examples where scores were bad - all examples were verified to work and provide good scores, including the custom files, which were far from easy to figure out. Hopefully now it'll be easier.
* makes the examples quick to complete by running only a short sample - this is important to notice breakages, e.g. in eval stage - nobody is going to wait for train to complete in hours.
* adds cnn/daily mail dataset
* recovers one preprocessed dataset from the last s2s incarnation recommendation: it is offered for high bleu scores (the other 3 are either identical or are just slightly worse than the preprocessed ones - full porting status: https://github.com/huggingface/transformers/issues/10044)

@patil-suraj, @sgugger


Marian input decoding bug

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Marian

Language I am using the model on (English, Chinese ...): English, German

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I'm examining Marian models to translate text. I noticed `convert_tokens_to_string` method uses `spm_target` which can be problematic if we want to decode source text.

Here is my script:

```
from transformers import MarianTokenizer, MarianModel

tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de')
model = MarianModel.from_pretrained('Helsinki-NLP/opus-mt-en-de')

input_text = "I was eating lunch when he saw me"
target_text = "Ich aß gerade zu Mittag, als er mich sah"

input_tokenized = tokenizer(input_text, return_tensors='pt')
with tokenizer.as_target_tokenizer():
    target_tokenized = tokenizer(target_text, return_tensors='pt')

print(tokenizer.decode(input_tokenized.data['input_ids'][0]))
with tokenizer.as_target_tokenizer():
    print(tokenizer.decode(target_tokenized.data['input_ids'][0]))

```

stdout:

```
I was▁eating▁lunch▁when he▁saw me
Ich aß gerade zu Mittag, als er mich sah

```

As you can see the input text is not decoded correctly since `spm_target` is used. A potential fix is to use `current_spm` and let `as_target_tokenizer` context manager decide which spm should be used (similar to text encoding):

```
def convert_tokens_to_string(self, tokens: List[str]) -> str:
    return self.current_spm.DecodePieces(tokens)
```

I can PR the fix if needed.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: master branch (f6e53e3c2bafb37c861db71a4b28c304403af92b)
- Python version: 3.7.4
- PyTorch version (GPU?): 1.7.1 (False)


[pretrained] model classes aren't checking the arch of the pretrained model it loads

While comparing different models trained on xsum (most of which are Bart) I made a mistake and passed "google/pegasus-xsum" to `BartForConditionalGeneration`

```
BartForConditionalGeneration.from_pretrained("google/pegasus-xsum")
```

I got:

```
Some weights of the model checkpoint at google/pegasus-xsum were not used when initializing BartForConditionalGeneration: ['model.encoder.layer_norm.weight', 'model.encoder.layer_norm.bias', 'model.decoder.layer_norm.weight', 'model.decoder.layer_norm.bias']
- This IS expected if you are initializing BartForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BartForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BartForConditionalGeneration were not initialized from the model checkpoint at google/pegasus-xsum and are newly initialized: ['model.encoder.embed_positions.weight', 'model.encoder.layernorm_embedding.weight', 'model.encoder.layernorm_embedding.bias', 'model.decoder.embed_positions.weight', 'model.decoder.layernorm_embedding.weight', 'model.decoder.layernorm_embedding.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "./bart-summarize2.py", line 8, in <module>
    tokenizer = BartTokenizer.from_pretrained(mname)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1788, in from_pretrained
    return cls._from_pretrained(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1860, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/roberta/tokenization_roberta.py", line 159, in __init__
    super().__init__(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/gpt2/tokenization_gpt2.py", line 179, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

Any reason why the model class doesn't check that it's being fed a wrong architecture? It could detect that and give a corresponding error message, rather than spitting random errors like above? I was pretty sure it was a bug in pegasus model until I noticed that pegasus != Bart.

Thanks.

@LysandreJik


[examples s2s] AttributeError: 'MBartTokenizerFast' object has no attribute 'tgt_lang'

After this PR https://github.com/huggingface/transformers/pull/10205 This is still broken for other models:

```
python examples/seq2seq/run_seq2seq.py     --model_name_or_path facebook/mbart-large-en-ro      --do_train     --do_eval     --task translation_en_to_ro     --dataset_name wmt16     --dataset_config_name ro-en     --source_prefix "translate English to Romanian: "     --output_dir /tmp/tst-translation     --per_device_train_batch_size=16     --per_device_eval_batch_size=16     --overwrite_output_dir     --predict_with_generate     --max_train_samples 500     --max_val_samples 500
```

```
Traceback (most recent call last):
  File "examples/seq2seq/run_seq2seq.py", line 668, in <module>
    main()
  File "examples/seq2seq/run_seq2seq.py", line 469, in main
    train_dataset = train_dataset.map(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/arrow_dataset.py", line 1120, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/arrow_dataset.py", line 1091, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "examples/seq2seq/run_seq2seq.py", line 450, in preprocess_function
    with tokenizer.as_target_tokenizer():
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/mbart/tokenization_mbart_fast.py", line 193, in as_target_tokenizer
    self.set_tgt_lang_special_tokens(self.tgt_lang)
AttributeError: 'MBartTokenizerFast' object has no attribute 'tgt_lang'
```

@patil-suraj, @sgugger


Fix example links in the task summary

# What does this PR do?

This PR fixes (and adds or removes) the links shown in the task summary.

Fixes #10288


Trainer train continues after resume_from_checkpoint on a checkpoint with early stop

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
When continuing training from checkpoint, Trainer does not check if the checkpoint terminated with an `self.control.should_training_stop == True`.

`self.control.should_training_stop == True` holds when:
1. `state.global_step >= state.max_steps`
    * training does not resume on `resume_from_checkpoint` due to recovering steps information (`state.global_step`) from checkpoint state 👍
2. Due to early stopping condition True
    * training resumes as no mechanism to find previous early stopping state 👎
    * even `early_stopping_patience_counter` is restarted from 0 on `EarlyStoppingCallback` init, irrespective of `resume_from_checkpoint` 👎

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@sgugger as issue in Trainer.

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Initialize `Trainer.train` with `resume_from_checkpoint` pointing to a checkpoint that stopped due to early stopping

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Training should not happen as the checkpoint loaded had stopped due to early stopping.
<!-- A clear and concise description of what you would expect to happen. -->


Masking issues with GPT2LMHeadModel.generate()

Is this intended behavior, that padding a sentence and attention_mask it will not give the exact same generation result comparing to the same sentence unpadded?

Edit: [This notebook](https://colab.research.google.com/drive/1oyFRFigtSNUYwKO1EQPRHfEqke0-F6_N?usp=sharing) demonstrates this, with the newest version available on colab. I realized that I didn't turn sampling off with the example below but the colab one has sampling off.

```
>>> gpt2_model = transformers.GPT2LMHeadModel.from_pretrained("gpt2")
>>> gpt2_model.generate(torch.tensor([[100,200,300]]))
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[100, 200, 300,  84,  12,  75,  84,  12,  75,  84,  12,  75,  84,  12,
          75,  84,  12,  75,  84,  12]])
>>> gpt2_model.generate(torch.tensor([[100,200,300,50256]]),attention_mask=torch.tensor([[1,1,1,0]]))
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[  100,   200,   300, 50256,   198,   198,     7,    16,     8,   383,
          3381,   366,    75,     1,  1724,   262,  4129,   286,   262,  4731]])
```


Minor documentation issue

## Minor issue in the Fine tuning docs
In the "Named Entity Recognition" section of the "Summary of tasks" documentation page there are some bad links.
Here is a link to the section: https://huggingface.co/transformers/task_summary.html#named-entity-recognition

The text in question is:
##Named Entity Recognition
Named Entity Recognition (NER) is the task of classifying tokens according to a class, for example, identifying a token as a person, an organisation or a location. An example of a named entity recognition dataset is the CoNLL-2003 dataset, which is entirely based on that task. If you would like to fine-tune a model on an NER task, you may leverage the run_ner.py (PyTorch), run_pl_ner.py (leveraging pytorch-lightning) or the run_tf_ner.py (TensorFlow) scripts.

## ISSUE:
The links for the following give a 404 error:
       run_pl_ner.py, run_tf_ner.py


Deprecate prepare_seq2seq_batch

# What does this PR do?

This PR officially deprecates `prepare_seq2seq_batch` to prepare its removal in Transformers v5. As discussed before, the proper way to prepare data for sequence-to-sequence tasks is to:
- call the tokenizer on the inputs
- call the tokenizers on the targets inside the context manager `as_target_tokenizer`

When only dealing with input texts without targets, just using the tokenizer call works perfectly well.

For `mBART` and `mBART50` tokenizers the source and target language can be specified at init or changed at any time by setting the attributes `.src_lang` and `.tgt_lang`.

Here is a full example showing how to port old code using `prepare_seq2seq_batch` to the new way in the case of an mBART tokenizer (remove the mentiones of `src_lang` and `tgt_lang` for other tokenizers:
```
tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-en-ro')
batch = tokenizer.prepare_seq2seq_batch(src_texts, tgt_texts, padding=True, truncation=True, src_lang="en_XX", tgt_lang="ro_RO", return_tensors="pt")
```
becomes
```
tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-en-ro', src_lang="en_XX", tgt_lang="ro_RO")
batch = tokenizer(src_texts, padding=True, truncation=True, return_tensors="pt")
with tokenizer.as_target_tokenizer():
    targets = tokenizer(tgt_texts, padding=True, truncation=True, return_tensors="pt")
batch["labels"] = targets["input_ids"]
```
The languages can be changed at any time with
```
tokenizer.src_lang = new_src_code
tokenizer.tgt_lang = new_tgt_code
```

This PR fixes a few things in `MBartTokenizer` and `MBartTokenizerFast` for the new API to work completely and removes all mentions of `prepare_seq2seq_batch` from the documentation and tests (except the test of that method in the common tests). It was already not used anymore in the seq2seq example `run_seq2seq`.


* Introduce save_strategy training argument

* collapse EvaluationStrategy and LoggingStrategy into a single TimeStrategy enum

* modify tests to use modified enum
# What does this PR do?
1. Introduce new `save_strategy` argument to decide on interval between 2 model saves during training.
2. Introduce a unified enum `TimeStrategy` which is used across `evaluation_strategy`, `logging_strategy` and `save_strategy`.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [Discussed during PR for logging_strategy.](https://github.com/huggingface/transformers/pull/10267)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@sgugger


Random Word Replacement Probability

Hi,

It appears that the token masking function replaces tokens with random words 50% of the time instead of the commented 10%.

https://github.com/huggingface/transformers/blob/709c86b5a925f1efe650e24ee8b1f52bdc5a3acb/src/transformers/data/data_collator.py#L381


Patch zero shot distillation script cuda issue

Quick patch to #10244 replacing an accidental deletion of `.cuda()` when using cuda. Causes error with multi-GPU.


Clean TF BART and TF Seq2Ses template

# What does this PR do?

This PR aims to clean TF BART and the TF Seq2Seq template by adding explicit keyword arguments, typing and update the documentation in the model implementation to make it easier to understand and read.


[tests] tests/test_trainer_distributed.py intermittent failure

Tests

Related to tests

`tests/test_trainer_distributed.py` fails occasionally on multi-gpu github runner CI and as a result doesn't free up the 29500 default distributed port.

This could be caused by an occasional deadlock discusses in testing_utils.py's `_stream_subprocess``. When debugging one such zombie it was stuck in `exec(eval(sys.stdin.readline()))`

Note that other similar tests under `examples` don't exhibit the same behavior - perhaps it somehow has to do with this being a different script that it runs (this test runs its own file as the distributed script).

The bt of the subsequent failures is long and confusing, as there are several mixed failures, but it's all really one failure: `Address already in use` since the previous distributed run of the same test didn't free up this port.

A quick check should show which process is bound to it:
```
netstat -tulpn | grep :29500
```

The full bt:
```
NCCL_DEBUG=INFO pytest -sv tests/test_trainer_distributed.py
================================================================ test session starts ================================================================
platform linux -- Python 3.7.4, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python
cachedir: .pytest_cache
rootdir: /home/github_actions/actions-runner/_work/transformers/transformers
plugins: xdist-2.2.1, forked-1.3.0
collected 1 item

tests/test_trainer_distributed.py::TestTrainerDistributed::test_trainer
Running:  /home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python -m torch.distributed.launch --nproc_per_node=2 /home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py --output_dir /tmp/tmp2k265qn5
stderr: Traceback (most recent call last):
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py", line 82, in <module>
stderr:     training_args = parser.parse_args_into_dataclasses()[0]
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
stderr:     obj = dtype(**inputs)
stderr:   File "<string>", line 61, in __init__
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 491, in __post_init__
stderr:     if is_torch_available() and self.device.type != "cuda" and self.fp16:
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 620, in device
stderr:     return self._setup_devices
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1359, in __get__
stderr:     cached = self.fget(obj)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 605, in _setup_devices
stderr:     torch.distributed.init_process_group(backend="nccl")
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 436, in init_process_group
stderr:     store, rank, world_size = next(rendezvous_iterator)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 179, in _env_rendezvous_handler
stderr:     store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
stderr: RuntimeError: Address already in use
stderr: Traceback (most recent call last):
stderr:   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
stderr:     "__main__", mod_spec)
stderr:   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
stderr:     exec(code, run_globals)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
stderr:     main()
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
stderr:     cmd=cmd)
stderr: subprocess.CalledProcessError: Command '['/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python', '-u', '/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py', '--local_rank=1', '--output_dir', '/tmp/tmp2k265qn5']' returned non-zero exit status 1.
stdout: *****************************************
stdout: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
stdout: *****************************************
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO Bootstrap : Using [0]ens6:10.128.0.66<0>
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
stdout:
stdout: multi-gpu-ci-runner:18062:18062 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO NET/Socket : Using [0]ens6:10.128.0.66<0>
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO Using network Socket
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout:
stdout: multi-gpu-ci-runner:18062:18089 [1] include/socket.h:403 NCCL WARN Connect to 10.128.0.66<52523> failed : Connection refused
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO bootstrap.cc:95 -> 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO bootstrap.cc:309 -> 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO init.cc:555 -> 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO init.cc:840 -> 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO group.cc:73 -> 2 [Async thread]
stderr: Traceback (most recent call last):
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py", line 82, in <module>
stderr:     training_args = parser.parse_args_into_dataclasses()[0]
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
stderr:     obj = dtype(**inputs)
stderr:   File "<string>", line 61, in __init__
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 491, in __post_init__
stderr:     if is_torch_available() and self.device.type != "cuda" and self.fp16:
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 620, in device
stderr:     return self._setup_devices
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1359, in __get__
stderr:     cached = self.fget(obj)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 605, in _setup_devices
stderr:     torch.distributed.init_process_group(backend="nccl")
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
stderr:     barrier()
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
stderr:     work = _default_pg.barrier()
stderr: RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
FAILED

===================================================================== FAILURES ======================================================================
________________________________________________________ TestTrainerDistributed.test_trainer ________________________________________________________

self = <tests.test_trainer_distributed.TestTrainerDistributed testMethod=test_trainer>

    @require_torch_multi_gpu
    def test_trainer(self):

        distributed_args = f"""
            -m torch.distributed.launch
            --nproc_per_node={torch.cuda.device_count()}
            {self.test_file_dir}/test_trainer_distributed.py
        """.split()
        output_dir = self.get_auto_remove_tmp_dir()
        args = f"--output_dir {output_dir}".split()
        cmd = [sys.executable] + distributed_args + args
>       execute_subprocess_async(cmd, env=self.get_env())

tests/test_trainer_distributed.py:72:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = ['/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python', '-m', 'torch.distributed.launc.../github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py', '--output_dir', ...]
env = {'HOME': '/home/github_actions', 'KMP_DUPLICATE_LIB_OK': 'True', 'KMP_INIT_AT_FORK': 'FALSE', 'LANG': 'C.UTF-8', ...}, stdin = None
timeout = 180, quiet = False, echo = True

    def execute_subprocess_async(cmd, env=None, stdin=None, timeout=180, quiet=False, echo=True) -> _RunOutput:

        loop = asyncio.get_event_loop()
        result = loop.run_until_complete(
            _stream_subprocess(cmd, env=env, stdin=stdin, timeout=timeout, quiet=quiet, echo=echo)
        )

        cmd_str = " ".join(cmd)
        if result.returncode > 0:
            stderr = "\n".join(result.stderr)
            raise RuntimeError(
>               f"'{cmd_str}' failed with returncode {result.returncode}\n\n"
                f"The combined stderr from workers follows:\n{stderr}"
            )
```

A short term workaround could be to randomize the port, so this test won't trumple upon its previous zombie.
```
+        from random import randint
+        master_port = 2950 + randint(1, 99)
         distributed_args = f"""
             -m torch.distributed.launch
             --nproc_per_node={torch.cuda.device_count()}
+            --master_port {master_port}
             {self.test_file_dir}/test_trainer_distributed.py
         """.split()
```

but this is a band-aid and a real solution is needed. It also will be an issue with any other distributed tests that rely on the same default port number.

I will keep on monitoring the issue.

Meanwhile this PR https://github.com/huggingface/transformers/pull/10281 should help with preventing the incremental number of zombies from scheduled runs.

It's difficult to debug w/o being able to reproduce this problem at will.


[CI] Kill any run-away pytest processes

As discussed on slack  this PR proposes to change github runner to kill any run-away pytest processes before starting a new job.

@LysandreJik


Trainer.train argument resume_from_last_checkpoint

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
`Trainer.train` accepts `resume_from_checkpoint` argument, which requires the user to explicitly provide the checkpoint location to continue training from.
`resume_from_last_checkpoint` can be useful to resume training by picking the latest checkpoint from `output_dir` of the `TrainingArguments` passed.

## Motivation

1. The checkpoint directory is created by the library, so user needs to navigate to the directory to find the value to provide for `resume_from_checkpoint`
2. User may just want to resume from the last valid checkpoint since their training got disrupted previously (a common scenario for someone to want to resume training). All they know is the `output_dir` they provided initially

This motivates to provide a `resume_from_last_checkpoint=True` to the `Trainer.train(...)` call, which will pick the latest checkpoint from `args.output_dir`. FYI `get_last_checkpoint` function from `trainer_utils` can be used to do exactly the same.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
I can raise a PR if it is a useful feature to have!


Performance of mbart-large-50-many-to-many-mmt on de/fr/it

Hi everybody

I am using ` mbart-large-50-many-to-many-mmt` and I am running into the following problem.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0 (installed from source)
- Platform: Linux
- Python version: 3.8.5
- PyTorch version (CPU): 1.7.1

### Who can help
@patrickvonplaten, @patil-suraj

## Information

I am using the `mbart-large-50-many-to-many-mmt` model for translation and it works as expected when translating German to English but when translating to other languages such as French or Italian it seems broken. I am using the same code as highlighted in the model card.

```Python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

input_text = "Der Mars hat einen neuen Besucher, wenn auch einen robotischen: \
Nach einer mehr als 472 Millionen Kilometer langen Reise setzte am Donnerstagabend \
das amerikanische Roboterfahrzeug Perseverance sanft im Marsstaub auf. "

tokenizer.src_lang = "de_DE"
encoded = tokenizer(input_text, return_tensors="pt")
generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)[0]

#--> '</s>en_XX Mars has a new visitor, but also a robotic one: After a journey of more than 472 million kilometers, the American robotic vehicle Perseverance gently set off in Mars dust on Thursday evening.</s>'

tokenizer.src_lang = "de_DE"
encoded = tokenizer(input_text, return_tensors="pt")
generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)[0]

#--> '</s>fr_XX On Mars, on Mars, on Mars, on Mars, on Mars, on Mars, on Mars, on Mars, on Mars.</s>'

tokenizer.src_lang = "de_DE"
encoded = tokenizer(input_text, return_tensors="pt")
generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.lang_code_to_id["it_IT"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)[0]

#--> '</s>it_IT Mars has a new visitor, anche robotico: After a journey di più di 472 milioni di chilometri, Thursday evening, the American robot vehicle Perseverance si è calmato in the dust of Mars.</s>'
```

Am I doing something wrong with the translation or is the performance on these languages expected to be worse?


Improving training time for Marian MT model with the Trainer

## Environment info
- Platform: Linux-4.19.0-14-cloud-amd64-x86_64-with-debian-10.7
- Python version: 3.7.9
- PyTorch version (GPU): 1.7.1
- Using GPU in script?: Yes (2 Tesla V100 GPUs with 16160MiB memory)
- CUDA Version: 11.0
- transformers: 4.3.2
- datasets: 1.3.0

### Who can help
@lhoestq @sgugger @sshleifer

## Information
The model I am using is a [MarianMTModel](https://huggingface.co/transformers/model_doc/marian.html#marianmtmodel) to train a machine translation model for Italian to Dutch. In order to so I perform the following steps:

- Split the data
1. The dataset originally consists of 2 parts, 1 text file containing Italian sentences and 1 text file containing the corresponding Dutch sentences. (42,940,499 sentences). These text files are combined in a pandas dataframe with a source and target column and is written to a csv file. Based on this [issue](https://github.com/huggingface/datasets/issues/610#issuecomment-691672919), which solved out-of-memory issues, I split the dataframe in 1000 csv chunks.
2. The csv files are loaded with the load_dataset method (which results in a 7.5G csv-train.arrow file) from the datasets library as follows:
```python
train_files = glob.glob(data_folder + 'shards/data_chunk_train_*') # list of the individual csv files
train_dataset = load_dataset('csv', data_files=train_files, split='train', cache_dir=cache_folder, keep_in_memory=True)
```
I use the keep_in_memory=True here to hopefully make things faster during training.

- Tokenization
1. At first, I to batch tokenized all the sentences with the map function. However, this resulted in 16 * 32G cache-files and gave a training time of 2000 hours. So I changed this to use the set_transform method in the latest release from datasets as follows:
``` python
def encode(example):
	return tokenizer.prepare_seq2seq_batch(src_texts=example['source'], tgt_texts=example['target'], padding='max_length', max_length=512)
train_dataset.set_transform(encode)
```

I use the max_length here to tokenize every sentence to the same size.
The tokenizer is a MarianTokenizer (where the spm files and vocab are trained with sentencepiece) and is defined as follows:
```python
tokenizer = MarianTokenizer(vocab='tokenizer/vocab.json', source_spm='tokenizer/source.model',
target_spm='tokenizer/target.model', source_lang='it', target_lang='nl', model_max_length=512)
```

- Model
1. A MarianMTModel is configured with the following MarianConfig (same config as the pretrained MarianMT models):
``` python
model = MarianConfig(decoder_layers=6, encoder_layers=6, d_model= 512, decoder_attention_heads=8,
decoder_ffn_dim=2048, decoder_layerdrop=0.0, encoder_attention_heads=8, encoder_ffn_dim=2048,
encoder_layerdrop=0.0, max_position_embeddings=512)

model = MarianMTModel(configuration)
```

- Training
1. After I loaded the data, created the MarianTokenizer and configured the MarianMTmodel I started training with the Trainer. I used the following trainingArguments:
``` python
training_args = TrainingArguments(num_train_epochs=3,  per_device_train_batch_size=12,
per_device_eval_batch_size=12, warmup_steps=100,  weight_decay=0.01, logging_dir='./logs', logging_steps=5000,
save_steps=10000, disable_tqdm=False, logging_first_step=True, fp16=True, remove_unused_columns = False)
```

I was not able to increase the batch size, as this gave out of memory errors on the GPU. And finally started training:
```python
trainer = Trainer(
	model=model,                         # the instantiated 🤗 Transformers model to be trained
	args=training_args,                  # training arguments, defined above
	train_dataset=train_dataset,         # training dataset
	eval_dataset=dev_dataset # contains 50,000 samples
)
trainer.train()
```

## Expected behavior
I expect the model to train within a reasonable amount of time (i.e. a couple of days). However, the training process is going to take about 500 hours:
0 %|                          | 219/5367564 [01:18<531:01:44,  2.81it/s]

I was wondering if this is expected behaviour that it takes that long to train. Could you please give me any suggestions on how to modify this to make it faster?


ImportError: cannot import name 'pipeline' from 'transformers' (unknown location)

Hi, I created an env with conda, installed TF, then installed PyTorch, then "pip install git+https://github.com/huggingface/transformers", but when I ran 'python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I hate you'))"', it gave me the ImportError. How can I resolve this?


Move the TF NER example

# What does this PR do?

This PR moves the `run_tf_ner.py` example into the legacy folder because it uses the "legacy" way to train a model with the `utils_ner.py` file.


Fix squad processor for TF

# What does this PR do?

This PR fixes the Squad processor that prepares and creates a `tf.data.dataset` to be able to be used in the `TFTrainer` through the `run_tf_squad.py` example script.

There were two issues:

1. The `token_type_ids` was forced to be `True` in the tokenizer output even if this argument was not part of the tokenizer's property `model_input_names`. Now it always belongs to the created dataset.
2. The `input_processing` method that parses the inputs of a TF model doesn't allow arguments that are not part of the signature, then the extra features have been removed from the created dataset.

# Fixes issue

#10246


Rework the AMP for TF XLNet

# What does this PR do?

This PR reworks the AMP of XLNet to remove some useless casts for better and less confusing AMP compliancy.


ElectraForQuestionAnswering with SQuADHead

# 🚀 Feature request

<!-- -->
Implement ElectraForQuestionAnswering as described in the paper. https://arxiv.org/abs/2003.10555

## Motivation
<!-- -->
In the original implementation, the authors use question answering module from XLNet rather than simple linear layer. There is a huge gap of performance between these 2 question answering module, especially on Squad2.0 like tasks. I suggest to follow the original implementation and rename the one with simple linear layer to ElectraForQuestionAnsweringSimple.

## Your contribution

<!--  -->
My team and I have implemented it using SQuADHead from modeling utils. I can submit a PR and make other necessary changes.

Code example:
```
from transformers import ElectraModel, ElectraPreTrainedModel
from transformers.modeling_utils import SQuADHead

class ElectraForQuestionAnswering(ElectraPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.start_n_top = config.start_n_top
        self.end_n_top = config.end_n_top

        self.electra = ElectraModel(config)
        self.squad_head = SQuADHead(config)
        self.init_weights()

    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        head_mask=None,
        start_positions=None,
        end_positions=None,
        is_impossible=None,
        cls_index=None,
        p_mask=None,
        return_dict=None):

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        transformer_outputs = self.electra(
            input_ids=input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            head_mask=head_mask,
            return_dict=return_dict,
        )
        hidden_states = transformer_outputs[0]

        return self.squad_head(hidden_states=hidden_states,
                                             start_positions=start_positions,
                                             end_positions=end_positions,
                                             cls_index=cls_index,
                                             is_impossible=is_impossible,
                                             p_mask=p_mask,
                                             return_dict=return_dict,
                                              )
```


Summarization of long text with T5 seems to output random memory content

Hello everyone,
I'm trying to summarizing a long text (~1800 words) using the T5 model. I set the max-length and min_length parameters as well, and when I do so, the output seems to contain random memory content...

Here my code:

#len_text=1755
    inputs = tokenizer.encode("summarize: " + proc_text, return_tensors="pt", max_length= (len_text + 1), truncation=True)
    outputs = model.generate(
    inputs,
    max_length=round(len_text/3), #~590 words
    min_length=round(len_text/5),  #~350 words
    no_repeat_ngram_size=2,
    length_penalty=2.0,
    num_beams=4,
    early_stopping=False)

And here the output (~150 words only):

the key is to deploy predictive maintenance on assets where it makes sense. a combination of machine learning and data driven analytics can be used to plan, analyze, plan and expand across an enterprise to gain real savings and improvements. to achieve high operational efficiency and availability, ensuring that all assets are performing at peak performance with high availability and the lowest possible maintenance costs, companies are provided with some compelling options to manage their assets. the cost of adopting the wrong strategy can actually introduce failures in themselves. many companies in the process industry and energy sectors are still- - . s-»  - nrhn hh gra,[&..._/*—s...  '– [“?;e and(**” (---]" & – / _ »: " *',--..,.-...[[__[*-_-"-&-s-–&&_-(-//-n'-,n-d-'rs/d&m––re»â«_ââ-ââ–â? â€[“?.”123467891012—:’...;‘’’–‘‘'&’-«‘’&–——‘–-—–==– ‘‘-’—-e, ‘’l’ — e–’_—_–_&—&/–... ‘o ’ ‘–/— ‘—’.– “ ‘& ‘ rere i–“– (–,–e’,’ (‘—e&,—...—/& (&.&...’/ ‘...– =–./’=- ‘- «– „– «-»–»- (— (-)-‘&'–«–*–: ‘e-aiia–n–o–s–enyen–i—== ‘=&=?&*&# ;&e : r– and engtd&‘ ‘ ‘ee,,&n&y-r&o-in–y–d’; ‘n’e ‘s&

I attached the full text I'm attempting to summarize as well.

Thanks!

[brief.txt](https://github.com/huggingface/transformers/files/6007898/brief.txt)


[test] fix func signature

This PR makes a small fix where the func argument cannot be `None` as it's used w/o checking it's `None`.

@sgugger


[ISSUES.md] propose using google colab to reproduce problems

It makes the reproduction process much faster if a user supplies a google colab notebook where we can see the problem.

This PR adds this suggestion to the existing how-to list.

@sgugger, @LysandreJik


Language Modeling Task (GPT2 / CLM) Does Not Generate Line Breaks?

The legacy run_language_modeling.py script produced output that respected line breaks in the train_data_file. The updated run_clm.py script does not. I imagine this is likely due to how the dataset is processed in the new script, but if it is, how do I intervene and fix it?

## Environment info

- general environment: Google Colab
- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
Models:

- gpt2: @patrickvonplaten, @LysandreJik

Library:

- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [x] the official example scripts: run_clm.py | run_language_modeling.py
* [x] my own modified scripts: colab notebooks that use these scripts

The tasks I am working on is:
* [x] my own task or dataset: Tiny Shakespeare (from text file)

## To reproduce

Steps to reproduce the behavior:

1. Download https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
2. python run_clm.py with --train_file set to input.txt
3. Instantiate finetuned GPT2 model and use model.generate to create new sequence

Colab notebooks may be found below:
Original (with legacy run_language_modeling.py):
https://colab.research.google.com/drive/1ieS4TuaFNJhuunaAM9wVmyp-n8Yx9_la?usp=sharing

Updated (with updated run_clm.py):
https://colab.research.google.com/drive/1dqIzv7WLk7sDOmFhLdMDhyKCIEcvw3lB?usp=sharing

## Expected behavior

When using the legacy run_language_modeling.py script, the output is as expected, with the correct line breaks:

When running the updated run_clm.py script, line breaks are conspicuously missing:

Is there a straightforward way to remedy this?

My thanks as always for this wonderful repo, all your hard work, and any assistance you might be able to provide.


[trainer] implement support for full fp16 in evaluation/predict

This PR allows users to use `model.half()` in evaluation/predict, which may or may not deliver results identical to fp32 eval. The outcome depends on how the model was trained and the application. e.g. if I use `--label_smoothing` with t5-small I get `eval loss = nan`, but bleu scores are exactly the same as with fp32.

### Need

Besides users asking for it in the past, the real need that prompted me to implement this is based on this Issue:  https://github.com/huggingface/transformers/issues/10161. To explain - DeepSpeed trains in fp16, while keeping master copy of fp32 weights on cpu, which allows fitting a model like t5-11b (45GB in params) onto a 40GB gpu (only 22.5GB in fp16). But then the user wants to eval and deepspeed is of no use here at the moment. So we need to give a way to users to run full fp16 in eval, which is what this PR proposes.

This PR:
* [x] adds `is_in_train` public Trainer attribute which helps to tell whether `evaluation` is running on its own, or called from `train`
* [x] adds `--fp16_full_eval` to enable the full fp16 mode under eval/predict (while `full-fp16` would read better, I picked the name starting with `--fp16_` to align/group well with other3 `--fp16_*` args.
* [x] adds the first test that measures gpu mem deltas - let's hope it proves to work across different gpus

The logic is a bit tricky since we must not `model.to(device)` before `model.half()` or otherwise the model loading will OOM, but I hope I was able to keep it simple and not error-prone. Perhaps instead of replaying `place_on_device` logic at the end of `train` in the deepspeed clean up section - it'd be better to re-play the full logic in the `predict_loop`? So that each stage can decide at its beginning how and when to put the model on device.

A few small related fixes:
* [x] fixes `_wrap_model` to do nothing under deepspeed
* [x] fixes `--fp16` help to remove apex-only comment, as it's outdated.

Questions:
* [ ] Should I add a log, saying that half is used at `model.half()` activation
* [ ] I put it inside `prediction_step` which seems to be the right place, it won't run if it's a re-entrant eval-inside-train
* [ ] as the `inputs` are `ints` I don't think we need to switch them to `half()` as well.

@sgugger


Introduce logging_strategy training argument
in TrainingArguments and TFTrainingArguments. (#9838)

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->
1. Introduce a `logging_strategy` argument in TrainingArguments.
2. Define a LoggingStrategy enumeration. This is similar to `EvalStrategy`.

<!-- Remove if not applicable -->

Fixes #9838

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [Link to issue](https://github.com/huggingface/transformers/issues/9838).
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
As changes in trainer: @sgugger