`pip install -e .[dev]` in Python 3.9.1+ fails because `jaxlib==0.1.55` cannot be found

## Environment info

- `transformers` version: 4.2.0dev0 (the error is during the installation)
- Platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.10
- Python version: 3.9.1 (the error occurs) -> 3.8.0 (the error does not occur)
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 documentation: @sgugger

## Information

This is the report of a bug that I encountered during the [dev] version of `transformers`.
I try to create a conda environment to install `transformer [dev]` by `pip install -e .[dev]`,
but failed due to the `jaxlib` version.

## To reproduce

Git clone the forked `transformers` and update it to be `This branch is even with huggingface:master.`

``` sh
  $  git clone git@github.com:forest1988/transformers.git forest1988_transformers
  $  cd forest1988_transformers/
  $  git remote add upstream https://github.com/huggingface/transformers.git
  $  git pull upstream main
  $  git pull upstream master
  $  git push origin master
```

Create a new conda env.

``` sh
  $  conda create -n transformers-for-contribute
  $  conda activate transformers-for-contribute
```

Then, try to install `transformers [dev]` by `pip install -e .[dev]`.

``` sh
(transformers-for-contribute) ****@**** $ conda install pip
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: ****/.pyenv/versions/anaconda3-2020.07/envs/transformers-for-contribute

  added / updated specs:
    - pip

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2020.12.8  |       h06a4308_0         121 KB
    certifi-2020.12.5          |   py39h06a4308_0         140 KB
    openssl-1.1.1i             |       h27cfd23_0         2.5 MB
    pip-20.3.3                 |   py39h06a4308_0         1.8 MB
    python-3.9.1               |       hdb3f193_2        18.1 MB
    setuptools-51.0.0          |   py39h06a4308_2         726 KB
    wheel-0.36.2               |     pyhd3eb1b0_0          33 KB
    ------------------------------------------------------------
                                           Total:        23.4 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.12.8-h06a4308_0
  certifi            pkgs/main/linux-64::certifi-2020.12.5-py39h06a4308_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1i-h27cfd23_0
  pip                pkgs/main/linux-64::pip-20.3.3-py39h06a4308_0
  python             pkgs/main/linux-64::python-3.9.1-hdb3f193_2
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  setuptools         pkgs/main/linux-64::setuptools-51.0.0-py39h06a4308_2
  sqlite             pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  tzdata             pkgs/main/noarch::tzdata-2020d-h14c3975_0
  wheel              pkgs/main/noarch::wheel-0.36.2-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3

Proceed ([y]/n)? y

Downloading and Extracting Packages
pip-20.3.3           | 1.8 MB    | ################################################################################################################################################################# | 100%
ca-certificates-2020 | 121 KB    | ################################################################################################################################################################# | 100%
python-3.9.1         | 18.1 MB   | ################################################################################################################################################################# | 100%
certifi-2020.12.5    | 140 KB    | ################################################################################################################################################################# | 100%
setuptools-51.0.0    | 726 KB    | ################################################################################################################################################################# | 100%
wheel-0.36.2         | 33 KB     | ################################################################################################################################################################# | 100%
openssl-1.1.1i       | 2.5 MB    | ################################################################################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(transformers-for-contribute) ****@**** $ pwd
****/workspace/Clone/forest1988_transformers
(transformers-for-contribute) ****@**** $ pip install -e ".[dev]"
Obtaining file:///****/workspace/Clone/forest1988_transformers
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Could not find a version that satisfies the requirement jaxlib==0.1.55; extra == "dev" (from transformers[dev])
ERROR: No matching distribution found for jaxlib==0.1.55; extra == "dev"
(transformers-for-contribute) ****@**** $ pip install jaxlib==0.1.55
ERROR: Could not find a version that satisfies the requirement jaxlib==0.1.55
ERROR: No matching distribution found for jaxlib==0.1.55
```

When I downgraded the python to 3.8 by `conda install python==3.8`, then `pip install -e ".[dev]"` works.

I tried other versions of python installed via conda:
- `conda install python==3.7` : OK
- `conda install python==3.9` : the same error occurs

## Expected behavior

Depending on the version of python we are using, we may find that the version of `jaxlib` specified in [`setup.py`](https://github.com/huggingface/transformers/blob/master/setup.py) is missing, and it causes `pip install -e .[dev]` failure.

For the `transformers [dev]`, is it better not to use python 3.9+? (I apologize if I missed the explanation)

If I change `jaxlib==0.1.55` to `jaxlib>=0.1.55` in `setup.py`, will it cause problems elsewhere?


[trainer] group fp16 args together

this PR proposes a purely cosmetic change that puts all the fp16 args together - so they are easier to manage/read

@sgugger


[autoformatters] wrapping destroying items/lists

# 🚀 Feature request

Would it be possible to make the auto-wrappers respect items/lists?

e.g. ended up with:
```
        """
        number of training steps is either 1. args.max_steps if --max_steps > 1 2. else derive from dataset if we can
        get its size
        """
```
not only it's broken, it's unreadable.

The original was:
```
        """
        number of training steps is either
        1. args.max_steps if --max_steps > 1
        2. else derive from dataset if we can get its size
        """
```

Ideally it should not remove new lines before bullets */- and numbers 1.

I also am not sure why there is a need to merge lines when the writer meant them to be shorter. I get the shortening, but why can't short lines be left alone. Which would be the case in this example. It looks like the only way I can enforce readable content is to inject paragraphs.

Thank you!

@sgugger


Allow example to use a revision and work with private models

# What does this PR do?

This PR adds the ability to:
- pick a particular revision for a model checkpoint
- use private models when the user is logged in
with the example scripts.

Just did `run_glue` as a proof of concept, will duplicate to all examples and the new example template if this suits everyone.


Unable to train xlnet with tensorflow

## Environment info
- `transformers` version: '2.0.0'
- Platform: jupyter notebook
- Python version: 3.7.6
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.1.0 GPU
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger, @TevenLeScao, @jplu

## Information

Model I am using (Bert, XLNet ...): XLNet

The problem arises when using:
my own modified scripts: (give details below)

The tasks I am working on is:
my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
# I get my input, output from a dataframe. It's just a series of text and a series of
# integers representing classes.
x = df['description']
y_label =  pd.Categorical(df['target'])
y_cat = y_label.categories
y = y_label.codes
n_label = len(y_cat)

# I use the tokenizer. Then convert it to a numpy array
xlnet_tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
train_tokenized_inputs = [xlnet_tokenizer.encode(text)
         for text in x.values.tolist()]
# It needs to be at least 1 and no more than 2000
train_max_length = max(1,min(np.array([len(inp) for inp in train_tokenized_inputs]).max(), 2000))
train_padded_inputs = (tf.keras.preprocessing.sequence.pad_sequences(train_tokenized_inputs, maxlen=train_max_length,
                                                           value=0,
                                                      padding='post', truncating='post',dtype='int32'))

# I use the xlnet model
clf = TFAutoModelForSequenceClassification.from_pretrained("xlnet-base-cased", num_labels=n_label)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
clf.compile(optimizer='adam',loss=loss)
clf.fit(x=train_padded_inputs, y=y,
    batch_size=32,
    epochs=1,
    verbose=1,
    callbacks=None,
    validation_split=0.2,
    validation_data=None,
    shuffle=True,
    class_weight=None,
    sample_weight=None,
    initial_epoch=0,
    steps_per_epoch=None,
    validation_steps=None,
    validation_freq=1,
    max_queue_size=10,
    workers=1,
    use_multiprocessing=False,)
```

The error message is:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-68-c147be84f56e> in <module>
     15     max_queue_size=10,
     16     workers=1,
---> 17     use_multiprocessing=False,)

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    817         max_queue_size=max_queue_size,
    818         workers=workers,
--> 819         use_multiprocessing=use_multiprocessing)
    820
    821   def evaluate(self,

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    233           max_queue_size=max_queue_size,
    234           workers=workers,
--> 235           use_multiprocessing=use_multiprocessing)
    236
    237       total_samples = _get_total_number_of_samples(training_data_adapter)

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in _process_training_inputs(model, x, y, batch_size, epochs, sample_weights, class_weights, steps_per_epoch, validation_split, validation_data, validation_steps, shuffle, distribution_strategy, max_queue_size, workers, use_multiprocessing)
    550         batch_size=batch_size,
    551         check_steps=False,
--> 552         steps=steps_per_epoch)
    553     (x, y, sample_weights,
    554      val_x, val_y,

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, batch_size, check_steps, steps_name, steps, validation_split, shuffle, extract_tensors_from_dataset)
   2344     # First, we build the model on the fly if necessary.
   2345     if not self.inputs:
-> 2346       all_inputs, y_input, dict_inputs = self._build_model_with_inputs(x, y)
   2347       is_build_called = True
   2348     else:

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in _build_model_with_inputs(self, inputs, targets)
   2570     else:
   2571       cast_inputs = inputs
-> 2572     self._set_inputs(cast_inputs)
   2573     return processed_inputs, targets, is_dict_inputs
   2574

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in _set_inputs(self, inputs, outputs, training)
   2657           kwargs['training'] = training
   2658       try:
-> 2659         outputs = self(inputs, **kwargs)
   2660       except NotImplementedError:
   2661         # This Model or a submodel is dynamic and hasn't overridden

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    771                     not base_layer_utils.is_in_eager_or_tf_function()):
    772                   with auto_control_deps.AutomaticControlDependencies() as acd:
--> 773                     outputs = call_fn(cast_inputs, *args, **kwargs)
    774                     # Wrap Tensors in `outputs` in `tf.identity` to avoid
    775                     # circular dependencies.

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--> 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

TypeError: in converted code:

    /opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_xlnet.py:916 call  *
        output = self.sequence_summary(output)
    /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py:773 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    /opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py:459 call  *
        output = self.first_dropout(output)
    /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py:416 converted_call
        return py_builtins.overload_of(f)(*args)

    TypeError: 'NoneType' object is not callable
```

In addition, I tried to use TFTrainer in case I could solve my problem with it.

`from transformers import TFTrainer`

Gets this error message

```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-51-aece35bcf827> in <module>
----> 1 from transformers import TFTrainer

ImportError: cannot import name 'TFTrainer' from 'transformers' (/opt/conda/lib/python3.7/site-packages/transformers/__init__.py)
```

## Expected behavior

I expect the code to run and the model to be fine-tuned on my dataset.

I expect that I shouldn't need the TFTrainer as the explanation on huggingface.co says the model is a standard tensorflow 2 layer. But I expect that I should be able to import it.


Retrieval Collapse when fine-tuning RAG

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: latest production version
- Platform:
- Python version: 3.8
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 RAG: @patrickvonplaten, @lhoestq
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Fine-tune RAG on FEVER dataset
2. Notice that the same documents are retrieved every time

We are trying to fine-tune RAG on the FEVER dataset.  We follow the steps in the paper (we think) to a 'T' (RAG plus BART classifier head).  However, when we fine-tune, retrieval "collapses" (a term used in the paper) so that all queries retrieve the same irrelevant documents.  As a sanity check, we fine-tuned with a frozen retriever, and achieved similar results (72%) to what the paper achieves with frozen retriever.  Thus, it appears that perhaps there is a bug in HF's implementation of the retriever (and its gradients) that is causing this.  Alternatively, perhaps there is an obvious mistake in our config of the retriever.  Do you have any insights into this?  Thanks!


Add head_mask/decoder_head_mask for BART

Description:
This PR adds `head_mask` and `decoder_head_mask` for BART PyTorch implementation according to BERT implementation.

Motivation:
According to HuggingFace's websites "There is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT (that some call “BERTology”)." This PR enables to mask attention heads in encoder and decoder models exactly like for BERT. This PR thus creates an opportunity to study the importance of attention heads in encoder-decoder BERT-like model.

Reviewer: @patrickvonplaten


added head_mask/decoder_head_mask for BART


Bump notebook from 6.1.4 to 6.1.5 in /examples/research_projects/lxmert

dependencies

Pull requests that update a dependency file

Bumps [notebook](https://github.com/jupyter/jupyterhub) from 6.1.4 to 6.1.5.

Commits

See full diff in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=notebook&package-manager=pip&previous-version=6.1.4&new-version=6.1.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/huggingface/transformers/network/alerts).


Put back LXMert example

# What does this PR do?

During the example reorganization, LXMert seems to have slipped into the cracks and got accidentally deleted. This PR puts it back.

Fixes #9309


Generate Function - Manual decoder_input_ids Error (Bart, Pegasus)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Google Colab
- Python version: 3.6.9

### Who can help

@patrickvonplaten

## To reproduce

Link to the forum discussion: [https://discuss.huggingface.co/t/rewriting-generate-function-for-manual-decoder-input/3034/3](https://discuss.huggingface.co/t/rewriting-generate-function-for-manual-decoder-input/3034/3)

Steps to reproduce the behavior:

```python
!pip install transformers==4.1.1
!pip install sentencepiece

from transformers import BartTokenizer, BartForConditionalGeneration
tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')

# OR
'''
from transformers import PegasusTokenizer, PegasusForConditionalGeneration
tokenizer = PegasusTokenizer.from_pretrained('google/pegasus-large')
model = PegasusForConditionalGeneration.from_pretrained('google/pegasus-large')
'''

text = "this is a sample text"

input_ids = tokenizer(text, return_tensors="pt").input_ids

decoder_input_ids = tokenizer("<s> Anatomy is", return_tensors="pt", add_special_tokens=False).input_ids

output = model.generate(input_ids, decoder_input_ids=decoder_input_ids, num_beams=4, num_return_sequences=4)

print("With decoder_input_ids num_beams=4", tokenizer.batch_decode(output, skip_special_tokens=True))

output = model.generate(input_ids, num_beams=4, num_return_sequences=4)

print("Without decoder_input_ids num_beams=4", tokenizer.batch_decode(output, skip_special_tokens=True))
```

Error:
```
TypeError                                 Traceback (most recent call last)

<ipython-input-38-271e60997201> in <module>()
      2 decoder_input_ids = tokenizer("<s> Anatomy is", return_tensors="pt", add_special_tokens=False).input_ids
      3
----> 4 output = model.generate(input_ids, decoder_input_ids=decoder_input_ids, num_beams=4, num_return_sequences=4)
      5
      6 print("With decoder_input_ids num_beams=4", tokenizer.batch_decode(output, skip_special_tokens=True))

2 frames

/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     24         def decorate_context(*args, **kwargs):
     25             with self.__class__():
---> 26                 return func(*args, **kwargs)
     27         return cast(F, decorate_context)
     28

/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, **model_kwargs)
    610                 pad_token_id=pad_token_id,
    611                 eos_token_id=eos_token_id,
--> 612                 **model_kwargs,
    613             )
    614

/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py in beam_search(self, input_ids, beam_scorer, logits_processor, max_length, pad_token_id, eos_token_id, **model_kwargs)
   1041
   1042         while cur_len < max_length:
-> 1043             model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
   1044
   1045             outputs = self(**model_inputs, return_dict=True)

TypeError: prepare_inputs_for_generation() got multiple values for argument 'decoder_input_ids'
```


How to use Longformer for summarization

Hi - Do you have a sample code on how to use Longformer for summarization tasks.


trainer.predict() returns different values from model.logits

Hi dear authors!

When I was using my **fine-tuned bert model** to do the sequence classification task, I found the values returned by `trainer.predict(test_dataset)` were very different from what I got from `model(**test_encodings)`. I did not find messages describing what the `predictions` actually are in the documents, so I'm not seeing what `trainer.predict()` returns. Could you please help me explain a bit more?

Here are some of my codes
- predicts with `model(**test_encodings)`
```python
def _predict_with_np(text_a, text_b, tokenizer, model):
    scores = [0, 1]
    encoded_input = tokenizer((text_a, text_b),
                              truncation=True, padding=True,
                              return_tensors="pt")
    output = model(**encoded_input)
    logit = output.logits[0]
    softmax_score = F.softmax(logit,dim=-1)
    score = scores[torch.argmax(softmax_score)]

    return score, logit, softmax_score

def predict_with_np(params, printing=False, NUM=0):

    texts = load_data()

    print("===> Loading fine-tuned model and tokenizer...")
    model = BertForSequenceClassification.from_pretrained(TUNED_MODEL)
    tokenizer = BertTokenizer.from_pretrained(TUNED_TOKENIZER)

    print("===> Classifying...")
    for i, text in enumerate(texts):
            if NUM > 0 and i > NUM:
                break
            text_a, text_b = text
            text_a = text_a.strip()
            text_b = text_b.strip()
            score, logit, softmax_score = _predict_with_np(text_a, text_b, tokenizer, model)
```

- predicts with `trainer.predict()`
```python
def predict_with_hf(params):
    test_dataset = load_pt_data()

    print("===> Loading Model and Training Arguments...")
    model = BertForSequenceClassification.from_pretrained(TUNED_MODEL)

    training_args = TrainingArguments(
        run_name=params.run_name,
        disable_tqdm=True,
        fp16=params.fp16,
        gradient_accumulation_steps=params.gradient_accumulation_steps,
        do_train=False,
        do_eval=False,
        do_predict=True,
        output_dir=params.output_dir,
    )

    print("===> Predicting...")
    trainer = Trainer(
        model=model,
        args=training_args,
        eval_dataset=test_dataset
    )

    results = {}
    logger.info("*** Predict ***")
    result = trainer.predict(test_dataset)
    output_pred_file = os.path.join(training_args.output_dir, "pred_results.txt")
    with open(output_pred_file, "w") as writer:
        logger.info("***** Pred results *****")
        for pred in result.predictions:
            logger.info("  predictions = %s", pred)
            writer.write("predictions = %s\n" % pred)
```

With these two versions, I got outputs like this:
```json
text_a: "Don't worry. I'll take care of it."
text_b: "Why so long?"
score: 0
logits: [0.3749077320098877, -0.15262120962142944]
softmax: [0.6289066076278687, 0.37109342217445374]
predictions: [-0.04395686  0.29134133]
```

I've read several lines of code inside `src/trainer.py` so I guess predictions are supposed to be logits. But actually, they are away different from the logits I have here.

Am I calculating things in the wrong way, or are the predictions designed to be something else?

Thanks for reading my long questions!


CUDA runtime error during benchmarking

Running `transformers/examples/benchmarking/run_benchmark.py` with any type of model, with multi-processing gives the following error:

```
1 / 1
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/THCGeneral.cpp line=54 error=3 : initialization error
cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/THCGeneral.cpp:54
cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/THCGeneral.cpp:54
Traceback (most recent call last):
  File "run_benchmark.py", line 47, in <module>
    main()
  File "run_benchmark.py", line 43, in main
    benchmark.run()
  File "/home/dock/.conda/envs/torch/lib/python3.7/site-packages/transformers/benchmark/benchmark_utils.py", line 709, in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
ValueError: too many values to unpack (expected 2)
```
It looks like `self.inference_memory` function is returning the string `N/A`. Everything works fine when `no_multi_processing` option is selected.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Ubuntu 18.04.1 LTS
- Python version: 3.7.9
- PyTorch version (GPU?): 1.2.0 with GPU support
- Tensorflow version (GPU?): None
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: I guess so

### Who can help
@patrickvonplaten
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): GPT2, DistilGPT2

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)


run_glue.py with XLNet model on CoLA dataset reaches 0 accuracy

## Environment info
- `transformers` version: 4.1.1
- Platform: Linux
- Python version: 3.6.10
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@TevenLeScao

## Information
Model I am using: XLNet

The problem arises when using:
*  The official example scripts of `run_glue.py`

The tasks I am working on is:
* an official GLUE: CoLA

## To reproduce

Steps to reproduce the behavior:
I am using the "run_glue" cmd as described here: https://github.com/huggingface/transformers/tree/master/examples/text-classification
`python run_glue.py --task_name cola  --model_name_or_path xlnet-base-cased --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --do_predict --overwrite_output_dir`

That's the results I get:
```
[p]$ cat res_cola_xlnet.txt
eval_loss = 0.612945020198822
eval_matthews_correlation = 0.0
epoch = 3.0
```

## Expected behavior

Results > 0


wrong output for Bert-larged-uncased

I am running Bert for the pytorch version:

from transformers import BertConfig, BertTokenizer, BertModel

config_class, model_class, tokenizer_class = (BertConfig, BertModel, BertTokenizer)

transformer_config = config_class.from_pretrained(pretrained_model + "/bert_config.json")
tokenizer = tokenizer_class.from_pretrained(pretrained_model, do_lower_case = True)
transformer_model = model_class.from_pretrained(pretrained_model, config=transformer_config)

last_hidden_states, pooling_output =transformer_model(input_ids_tensor, attention_mask_tensor, segment_ids_tensor)

The output of 'transformer_model' -- last_hidden_states, should be a Tensor. But the result is 'last_hidden_state', that means it is a string object. What's wrong with it ?


Simplify  marian distillation script

Simplify marian distillation script, by adding a suggested MAX_LEN and using finetune.py directly.


`run_glue.py` fails when using my own dataset of regression task

## Environment info

- `transformers` version: 4.1.1
- Platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik
 examples/token-classification: @stefan-it

(Excuse me if I'm asking someone who is not in charge. I couldn't find `examples/text-classification` in the list.)

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

It seems that an error occurs when I use `run_glue.py` with my own dataset of regression task.

``` sh
CUDA_VISIBLE_DEVICES=0 python run_glue.py  \
  --model_name_or_path bert-base-cased \
  --train_file ****.csv \
  --validation_file ****.csv \
  --do_train \
  --do_eval \
  --max_seq_length 64 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs **** \
  --logging_steps **** \
  --save_steps **** \
  --save_total_limit **** \
  --output_dir ****/v4.1.1/****
```

An example of the train/valid CSV file is as below:

``` csv
id,label,sentence1
__id_as_string__,3.0,__string__
```

Sorry for the lack of details. I use this heavily masked notation to take into account the licensing of the dataset.
You can see that the columns contain `label` and `sentence1`, and the value of `label` is `float`.
I confirmed that `is_regression` is `True` in this case.

The error message says:

``` sh
Traceback (most recent call last):
  File "run_glue.py", line 419, in <module>
    main()
  File "run_glue.py", line 293, in main
    label_to_id = {v: i for i, v in enumerate(label_list)}
UnboundLocalError: local variable 'label_list' referenced before assignment
```

It seems that the case `data_args.task_name is None` and `is_regression is True` has not been considered in the example.
Excuse me if I misunderstand something.

https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue.py#L277

```
    if (
        model.config.label2id != PretrainedConfig(num_labels=num_labels).label2id
        and data_args.task_name is not None
        and is_regression
    ):
        # Some have all caps in their config, some don't.
        label_name_to_id = {k.lower(): v for k, v in model.config.label2id.items()}
        if list(sorted(label_name_to_id.keys())) == list(sorted(label_list)):
            label_to_id = {i: label_name_to_id[label_list[i]] for i in range(num_labels)}
        else:
            logger.warn(
                "Your model seems to have been trained with labels, but they don't match the dataset: ",
                f"model labels: {list(sorted(label_name_to_id.keys()))}, dataset labels: {list(sorted(label_list))}."
                "\nIgnoring the model labels as a result.",
            )
    elif data_args.task_name is None:
        label_to_id = {v: i for i, v in enumerate(label_list)}
```

When I modified the last two lines as below, I could go to the next step.
May I ask you that is it the correct way to avoid the error?

```
    elif data_args.task_name is None:
        # No definition for 'data_args.task_name is None' and 'is_regression is True'?
        if not is_regression:
            label_to_id = {v: i for i, v in enumerate(label_list)}
```

## Expected behavior

`run_glue.py` can be used for our own dataset of regression task.


Model inputs and outputs are ``None`` when converting fine-tuned gpt2 to Tensorflow?

Hi,

I've fine tuned a distilgpt2 model using my own text using ``run_language_modeling.py`` and its working fine after training and ``run_generation.py`` script produces the expected results.

Now I want to convert this to a Tensorflow Lite model and did so by using the following

```python
from transformers import *

CHECKPOINT_PATH = '/content/drive/My Drive/gpt2_finetuned_models/checkpoint-2500'

model = GPT2LMHeadModel.from_pretrained("distilgpt2")
model.save_pretrained(CHECKPOINT_PATH)
model = TFGPT2LMHeadModel.from_pretrained(CHECKPOINT_PATH, from_pt=True)
```

But I dont think I'm doing this right as after conversion, when I write

```python
print(model.inputs)
print(model.outputs)
```

I get

```
None
None
```

But I still went ahead with the TFLite conversion using :

```python
import tensorflow as tf

input_spec = tf.TensorSpec([1, 64], tf.int32)
model._set_inputs(input_spec, training=False)

converter = tf.lite.TFLiteConverter.from_keras_model(model)

# FP16 quantization:
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]

tflite_model = converter.convert()

open("/content/gpt2-fp16.tflite", "wb").write(tflite_model)
```

But does not work and when using the generated ``tflite`` model I get the error:

> tensorflow/lite/kernels/kernel_util.cc:249 d1 == d2 || d1 == 1 || d2 == 1 was not true.

Which I'm sure has something to to with my model not converting properly and getting ``None`` for input/output.

Does anyone have any idea how to fix this?

Thanks


Similar usage of `past_key_values` in CausalLM and Seq2SeqLM

# 🚀 Feature request

It seems GPT-2 and BartDecoder has a different style of `past_key_values`.

In GPT-2, `past_key_values` is explained as below:
(the explanation is from https://huggingface.co/transformers/model_doc/gpt2.html#gpt2model)

```
(parameters)
past_key_values (List[torch.FloatTensor] of length config.n_layers) – Contains precomputed hidden-states (key and values in the attention blocks) as computed by the model (see past_key_values output below). Can be used to speed up sequential decoding. The input_ids which have their past given to this model should not be passed as input_ids as they have already been computed.

(returns)
past_key_values (List[torch.FloatTensor], optional, returned when use_cache=True is passed or when config.use_cache=True) – List of torch.FloatTensor of length config.n_layers, with each tensor of shape (2, batch_size, num_heads, sequence_length, embed_size_per_head)).

Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.
```

In BartDecoder and its inner BartDecoderLayer, `past_key_values` is explained and treated as below:
(the explanation is from https://huggingface.co/transformers/model_doc/bart.html#bartmodel)

```
past_key_values (Tuple[Tuple[torch.Tensor]] of length config.n_layers with each tuple having 2 tuples each of which has 2 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) –

Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up decoding.
```

`v4.1.1 modeling_bart`
https://github.com/huggingface/transformers/blob/v4.1.1/src/transformers/models/bart/modeling_bart.py

``` python
# in BartDecoder

        for idx, decoder_layer in enumerate(self.layers):
            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
            if output_hidden_states:
                all_hidden_states += (hidden_states,)
            dropout_probability = random.uniform(0, 1)
            if self.training and (dropout_probability < self.layerdrop):
                continue

            past_key_value = past_key_values[idx] if past_key_values is not None else None

            hidden_states, layer_self_attn, present_key_value, layer_cross_attn = decoder_layer(
                hidden_states,
                attention_mask=combined_attention_mask,
                encoder_hidden_states=encoder_hidden_states,
                encoder_attention_mask=encoder_attention_mask,
                past_key_value=past_key_value,
                output_attentions=output_attentions,
            )
```

``` python
# in BartDecoderLayer

        # Self Attention
        # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
        self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
        # add present self-attn cache to positions 1,2 of present_key_value tuple
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            past_key_value=self_attn_past_key_value,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
        )
        hidden_states = F.dropout(hidden_states, p=self.dropout, training=self.training)
        hidden_states = residual + hidden_states
        if not self.normalize_before:
            hidden_states = self.self_attn_layer_norm(hidden_states)

        # Cross-Attention Block
        cross_attn_present_key_value = None
        cross_attn_weights = None
        if encoder_hidden_states is not None:
            residual = hidden_states
            if self.normalize_before:
                hidden_states = self.encoder_attn_layer_norm(hidden_states)

            # cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
            cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
            hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
                hidden_states=hidden_states,
                key_value_states=encoder_hidden_states,
                attention_mask=encoder_attention_mask,
                past_key_value=cross_attn_past_key_value,
                output_attentions=output_attentions,
            )
            hidden_states = F.dropout(hidden_states, p=self.dropout, training=self.training)
            hidden_states = residual + hidden_states
            if not self.normalize_before:
                hidden_states = self.encoder_attn_layer_norm(hidden_states)

            # add cross-attn to positions 3,4 of present_key_value tuple
            present_key_value = present_key_value + cross_attn_present_key_value
```

## Motivation

It seems that one of the aims of the refactoring of Bart by @patrickvonplaten https://github.com/huggingface/transformers/pull/8900 is "Allow to use BartEncoder and BartDecoder separately from the BartModel".

I appreciate this very much and would love to treat `BartDecoder` as well as `gpt2`, but I feel that the difference in the handling of `past_key_values` is a barrier.

In `gpt2`, `past_key_value` in `past_key_values` is `torch.tensor` with each tensor of shape (2, batch_size, num_heads, sequence_length, embed_size_per_head)).
However, in `Bart`, `past_key_value` in `past_key_values` is `Tuple[torch.Tensor]` and `self_atten` part is not a tensor but a "2 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head))".

If we want to handle `self_attn_past_key_value` in `Bart` like that in `gpt2`, is it the right way to concatename the 2 tensors in `past_key_value`?
Or, is there the other correct way to treat it?

Thank you in advance.


[trainer] self.model_wrapped + _model_unwrap

This PR adds:

* [x] adds `self.model_wrapped` - to have access to the outmost module regardless of how many times it was wrapped (e.g. under DeepSpeed there is a double wrapping ` DDP(Deepspeed(Transformers Model))`)
* [x]  makes sure that `self.model` is always set to the normal model
* [x] fixes a bug where under `model_parallel` `self.model` was not set (twice)!
* [x] simplifies the `model_init` checking logic
* [x] replaces `_actual_model` which couldn't handle multiple wrapping levels with `_model_unwrap` which can and integrate it

Please ignore the small mentions of DeepSpeed, this PR is split of from https://github.com/huggingface/transformers/pull/9211 to get all the non-DeepSpeed related changes into a separate review to make things a bit easier on the reviewers as it was suggested by @sgugger. This PR was made by copying from the other PR and manually removing all the added deepspeed code.

If possible, let's get it in asap so that I could rebase and we could move on with the DeepSpeed PR. Thank you very much!

@sgugger


This PR adds:

* [x] self.wrapped

https://github.com/huggingface/transformers/pull/9211

in progress


Conditional Generation using input_embeds instead of input_ids

Hi @patrickvonplaten!

When using input_embeds instead of input_ids as inputs to the BartForConditionalGeneration model, I am not able to generate the result. Could you please take a look? The same code works with GPT2 in place of Bart. Thanks!

Here is the script
```
import torch
from transformers import BartForConditionalGeneration, BartTokenizer

model_path = "facebook/bart-large"
model = BartForConditionalGeneration.from_pretrained(model_path, output_hidden_states=True)
tokenizer = BartTokenizer.from_pretrained(model_path)

text = "I disapprove of what you <mask> , but"
input_ids = tokenizer.encode_plus(text, return_tensors='pt')['input_ids']

with torch.no_grad():
    x = model.get_input_embeddings()(input_ids).squeeze()

model(inputs_embeds = x)
```

Here is the Traceback I got
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-21-e17be5532b10> in <module>()
     18     x = model.get_input_embeddings()(input_ids).squeeze()
     19
---> 20 model(inputs_embeds = x)
     21

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/bart/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
   1244             output_attentions=output_attentions,
   1245             output_hidden_states=output_hidden_states,
-> 1246             return_dict=return_dict,
   1247         )
   1248         lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/bart/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
   1079         # -> is this used for backward compatibility
   1080         if decoder_input_ids is None and decoder_inputs_embeds is None:
-> 1081             decoder_input_ids = shift_tokens_right(input_ids, self.config.pad_token_id)
   1082
   1083         output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions

/usr/local/lib/python3.6/dist-packages/transformers/models/bart/modeling_bart.py in shift_tokens_right(input_ids, pad_token_id)
     67     Shift input ids one token to the right, and wrap the last non pad token (usually <eos>).
     68     """
---> 69     prev_output_tokens = input_ids.clone()
     70
     71     assert pad_token_id is not None, "self.model.config.pad_token_id has to be defined."

AttributeError: 'NoneType' object has no attribute 'clone'
```

- `transformers` version: 4.1.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.0 (True)


Where is the impact when output_attentions=True?

Is there any impact regarding performance (training/fine-tuning time, GPU memory, batch size, etc.) when  `output_attentions=True`?

```python
self.bert_encoder = BertModel.from_pretrained(
            hparams.architecture, # "bert-base-uncased"
            output_attentions=True)
```


replace apex.normalization.FusedLayerNorm with torch.nn.LayerNorm

This PR proposes to drop `apex.normalization.FusedLayerNorm` in favor of faster `torch.nn.LayerNorm`.

1. For performance and background details please see the discussions in https://github.com/huggingface/transformers/issues/9377
2. It's also needed for https://github.com/huggingface/transformers/pull/9384 since `apex.normalization.FusedLayerNorm` corrupts data under model parallel https://github.com/NVIDIA/apex/issues/1022

Fixes: #9377

@LysandreJik, @sgugger, @patrickvonplaten


[logging] autoflush

This PR proposes to:

* auto-flush `transformers` logging

When using logging for tracing signals from different parts of the code and which could be mixed with print debug this aids to get all the logging events synchronized.

I don't think this change will introduce any performance impacts.

If it helps someone here is the code I used to sync `transformers` logging with various other debug prints.

I was porting bart to MP and I needed to trace that the device switching happens correctly and I added a bunch of `logger.info` calls inside `modeling_bart.py` and also had some other helpers `print` debug messages which weren't logger based:

```

# auto flush std streams
from sys import stdout, stderr
def stdout_write_flush(args, w=stderr.write): w(args); stderr.flush()
def stderr_write_flush(args, w=stderr.write): w(args); stderr.flush()
stdout.write = stdout_write_flush
stderr.write = stderr_write_flush

from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig

import logging
import transformers.utils.logging
import transformers.models.bart.modeling_bart

# I wanted a shorter simpler format
handlers = transformers.utils.logging._get_library_root_logger().handlers
for handler in handlers:
    formatter = logging.Formatter("[%(funcName)s] %(message)s")
    handler.setFormatter(formatter)

transformers.models.bart.modeling_bart.logger.setLevel(transformers.logging.INFO)

# then all the model creation and generate() goes next
```

@LysandreJik, @sgugger, @patrickvonplaten


[model parallelism] Bart goes parallel

Model Parallel

Model Parallelilsm Implementations

WIP

Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

This PR implements model parallelism (MP) in Bart.

This is the latest incarnation of generalization of the MP in `transformers`, based on @alexorona's original work. I have done some of it already in https://github.com/huggingface/transformers/pull/9323 and this PR builds upon the other one. It's slightly complex what to merge when, but this PR is independent and can be merged on its own.

For reviewers I propose to read things in this order:
1. https://github.com/huggingface/transformers/pull/9316
2. https://github.com/huggingface/transformers/pull/9323
3. this PR
4. Additional important design discussions https://github.com/huggingface/transformers/issues/8771

If all is in agreement, I propose:

1. ☐ merging this PR first,
2. ☐ then I'll backport the new code from this PR to https://github.com/huggingface/transformers/pull/9323 and we merge that.
3. ☐ then we handle gpt2, which I haven't touched yet. Perhaps @alexorona could help there if his time permits or one of us.
4. ☐ complete Bart's other heads (can be item 3) and `deparallelize` - the latter is not really needed in practice so will handle those when dust around design settles.
5. ☐ add Bart to trainer's supported for `--model_parallel` flags
6. ☐ write tests for `model_parallel_utils.py`
7. ☐ meanwhile we can polish the concept of device maps which will require a review of all architectures `transformers` has implemented.

Actually first we need to merge smaller bits:
1. https://github.com/huggingface/transformers/pull/9347
2. https://github.com/huggingface/transformers/pull/9386

---------

So this PR:

* [x] Implements MP in Bart based on discussions in all of the threads/PRs listed above. Only `BartForConditionalGeneration` at the moment while we are sorting out the API. But the bulk of the work is done, since `BartModel` has all in place.

* [x] switches to the concept of `main_device` rather than `(first|last)_device` so the first device of encoder becomes the main_device and almost everything happens there (`embeddings`, `lm_head`, etc), and other devices are used exclusively for encoder and decoder purposes.

* [x] switches to a more explicit `device_map` that can support non-symmetrical models (different number of layers in encoder and decoder).  It can also handle different types of maps. See the demo at the end this post for details.

* [x] further improves the magical `to()` functions that can operate on any type of variable except opaque objects. Can be used to put the inputs on the correct devices either automatically via a `forward` decorator or explicitly inside `forward`. We could use either or both.

* [x] adds a bunch of debug functions that make it easy to trace device IDs of variables, params and whole layers.

* [x] further improves the device map validation function

* [x] improves tests

* [x] needs to remove  apex.normalization.FusedLayerNorm as it's buggy under MP (corrupts data) per https://github.com/huggingface/transformers/issues/9377 a dedicated to removal PR is https://github.com/huggingface/transformers/pull/9386

Here is a quick demo (you will need 2 gpus to run it):

```
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig

#mname = "sshleifer/tinier_bart"
mname = "sshleifer/distilbart-xsum-6-6"

model = BartForConditionalGeneration.from_pretrained(mname)
tokenizer = BartTokenizer.from_pretrained(mname)

sentences = ["I'm sitting here in a boring room. It's just another rainy Sunday afternoon. I'm wasting my time I got nothing to do. I'm hanging around I'm waiting for you. But nothing ever happens. And I wonder."]
inputs = tokenizer(sentences, max_length=1024, return_tensors='pt', truncation="longest_first")

device_maps_flat = {
    "sshleifer/tinier_bart": {
        "encoder": {0: [0, 1] },
        "decoder": {1: [0] },
    },
    "sshleifer/distilbart-xsum-6-6": {
        "encoder": {0: [0, 1, 2, 3, 4, 5] },
        "decoder": {1: [0, 1, 2, 3, 4, 5] },
    },
}

device_maps_split = {
    "sshleifer/tinier_bart": {
        "encoder": {0: [0],
                    1: [1],
                    },
        "decoder": {1: [0] },
    },
    "sshleifer/distilbart-xsum-6-6": {
        "encoder": {0: [0, 1, 2],
                    1: [3, 4, 5],
                    },
        "decoder": {0: [0, 1, 2],
                    1: [3, 4, 5],
                    },
    },
}

# 3 different ways (2 different device maps and 1 autogenerated device map)
model.parallelize() # autogenerated
#model.parallelize(device_maps_flat[mname])
#model.parallelize(device_maps_split[mname])

inputs = inputs.to("cuda:0")
# Generate Summary
summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=25, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])
# prints: [" I'm sitting in a room where I'm waiting for something to happen."]
```

You can see from the demo, that when calling `model.parallelize` you can skip the `device_map` arg altogether and the model will generate the right one. Or you can provide one that:
1. gives some gpus exclusively to encoder and others to decoder
2. splits the model horizontally so that the encoder uses all gpus so so the decoder
the model transparently handles all the remappings

Note, the user still needs to put the data on the `main_device`, so perhaps that will eventually become not hardcoded via:
```
# inputs = inputs.to("cuda:0")
inputs = inputs.to(model.main_device)
```

As we have been discussing elsewhere the device map format is unstable yet. So I propose we document it as unstable yet, but the users can rely on the autogenerated device map which requires no input from the user (i.e. calling `model.parallelize() ) - if it changes it'll happen transparently for the user.

Also note that in situations of Trainer-based scripts, like `finetune_trainer.py`, the user has no way to supply such device map at the moment so in effect the model generates the map on the fly as in the above para.

Fixes: #8344

@LysandreJik, @patrickvonplaten, @sgugger, @alexorona


[Marian] Doc says `config.add_bias_logits=True`, but config is has `config.add_bias_logits=False`

**Question**:

In the docs, it is written that Marian has (contrary to Bart), `config.add_bias_logits=True`: https://huggingface.co/transformers/model_doc/marian.html#implementation-notes. But when looking into the code:
https://github.com/huggingface/transformers/blob/b01f451ca38695c60175b34d245997ef4d18231d/src/transformers/models/marian/configuration_marian.py#L25 Marian has the exact same default config as Bart and also Marian's config files online have `config.add_bias_logits=False` - see:
https://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/config.json

@sshleifer @patil-suraj

Is the documentation not up-to-date anymore? Because all the slow tests are passing....


[docs] Fix TF base model examples: outputs.last_hidden_states -> state

# What does this PR do?

Fixes a typo in the examples of Tensorflow-based base models, in which the returned last_hidden_state attribute of model output is incorrectly listed as "last_hidden_states".

Fixes #9376

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@julien-c @patrickvonplaten


[Docs] `past_key_values` return a tuple of tuple as a default

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR corrects the docs regarding `past_key_values`. `past_key_values` should always be of type `Tuple[...]`.

Fixes #9380

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


BartModel's `past_key_values` seems to have different explanations in input_doc and output_doc

## Environment info

- `transformers` version: 4.1.1
- Platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
 Bart: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Bart

The problem arises in [the document](https://huggingface.co/transformers/model_doc/bart.html) of BartModel and BartForConditionalGeneration

## To reproduce

Thank you for kindly answering my question https://github.com/huggingface/transformers/issues/9298.
I'm now trying to use Bart in transformers v4.1.1.

I'd like to make use of `past_key_values`, which seems to have been the major change of the refactoring https://github.com/huggingface/transformers/pull/8900,
but I am a bit confused about the type and shape of it.

About the input of the `forward` function, it is explained as:

```
past_key_values (Tuple[Tuple[torch.Tensor]] of length config.n_layers with each tuple having 2 tuples each of which has 2 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) –

Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up decoding.
```

About the output, it is explained as:

```
past_key_values (List[torch.FloatTensor], optional, returned when use_cache=True is passed or when config.use_cache=True) – List of torch.FloatTensor of length config.n_layers, with each tensor of shape (2, batch_size, num_heads, sequence_length, embed_size_per_head)).

Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see past_key_values input) to speed up sequential decoding.
```

I think it will be natural if the input `past_key_values` and the output `past_key_values` have the same format and the output can be used as the input in the next step.

If my understanding is correct, the document of the input is generated with `BART_INPUTS_DOCSTRING`, and the output is from `Seq2SeqModelOutput`.

```
    @add_start_docstrings_to_model_forward(BART_INPUTS_DOCSTRING)
    @add_code_sample_docstrings(
        tokenizer_class=_TOKENIZER_FOR_DOC,
        checkpoint="facebook/bart-large",
        output_type=Seq2SeqModelOutput,
        config_class=_CONFIG_FOR_DOC,
```
I'm sorry if I'm wrong, but maybe the `Seq2SeqModelOutput` documentation hasn't been updated for refactoring?
(When I look at the [git log](https://github.com/huggingface/transformers/commits/88ef8893cd649cc2b4adb9885aba88c750118cff/src/transformers/modeling_outputs.py), I cannot find the related commit.)

I apologize if the difference in input/output format is due to some intention.

If you don't mind, I'd like to ask one more question.
In the refactoring of Bart, the `BartDecoderLayer` (renamed from `DecoderLayer`) seems to be updated as below:

``` python
        # make sure decoder uni-directional self-attn at 1st position and cross-attn at 2nd position.
        present_key_value = (self_attn_present_key_value, cross_attn_present_key_value)

        return (
            hidden_states,
            self_attn_weights,
            present_key_value,
            cross_attn_weights,
        )
```

And in the `BartDecoder`, cache is updated as below:

``` python
            if use_cache:
                next_decoder_cache += (present_key_value,)
...

        next_cache = next_decoder_cache if use_cache else None
```

Does it mean the Bart (and other Seq2Seq Language Models) have both `selt_atten_present_key_value` and `cross_attn_present_key_value` in `past_key_values`?

## Expected behavior

Maybe the document of `Seq2SeqModelOutput` needs to be updated.
I apologize if the difference in the input/output explanations is due to some intention.


Improve documentation coverage for Bertweet

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9035
@sgugger  added docs for Bertweet

## Before submitting

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


[Docs] Tokenizer Squad 2.0 example

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9326

This PR fixes the docs. I ran following code from (https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0) to see whether the Squad tokenization works as expected.

Concatenated code from examples:

```python
#!/usr/bin/env python3
import json
from pathlib import Path
from transformers import DistilBertTokenizerFast

def read_squad(path):
    path = Path(path)
    with open(path, 'rb') as f:
        squad_dict = json.load(f)

    contexts = []
    questions = []
    answers = []
    for group in squad_dict['data']:
        for passage in group['paragraphs']:
            context = passage['context']
            for qa in passage['qas']:
                question = qa['question']
                for answer in qa['answers']:
                    contexts.append(context)
                    questions.append(question)
                    answers.append(answer)

    return contexts, questions, answers

train_contexts, train_questions, train_answers = read_squad('train-v2.0.json')
val_contexts, val_questions, val_answers = read_squad('dev-v2.0.json')

def add_end_idx(answers, contexts):
    for answer, context in zip(answers, contexts):
        gold_text = answer['text']
        start_idx = answer['answer_start']
        end_idx = start_idx + len(gold_text)

        # sometimes squad answers are off by a character or two – fix this
        if context[start_idx:end_idx] == gold_text:
            answer['answer_end'] = end_idx
        elif context[start_idx-1:end_idx-1] == gold_text:
            answer['answer_start'] = start_idx - 1
            answer['answer_end'] = end_idx - 1     # When the gold label is off by one character
        elif context[start_idx-2:end_idx-2] == gold_text:
            answer['answer_start'] = start_idx - 2
            answer['answer_end'] = end_idx - 2     # When the gold label is off by two characters

add_end_idx(train_answers, train_contexts)
add_end_idx(val_answers, val_contexts)

tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')

train_encodings = tokenizer(train_contexts, train_questions, truncation=True, padding=True)
val_encodings = tokenizer(val_contexts, val_questions, truncation=True, padding=True)

def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
            start_positions.append(encodings.char_to_token(i, answers[i]['answer_start']))
            end_positions.append(encodings.char_to_token(i, answers[i]['answer_end'] - 1))

            # if None, the answer passage has been truncated
            if start_positions[-1] is None:
                start_positions[-1] = tokenizer.model_max_length
            if end_positions[-1] is None:
                end_positions[-1] = tokenizer.model_max_length
        encodings.update({'start_positions': start_positions, 'end_positions': end_positions})

add_token_positions(train_encodings, train_answers)
add_token_positions(val_encodings, val_answers)
```

Then I checked that the tokenization is correct with this helper function for a couple of ids:

```python
def show_answer(idx):
    print("Tokenized", tokenizer.decode(train_encodings['input_ids'][idx][train_encodings['start_positions'][idx]: train_encodings['end_positions'][idx]]))
    print("Real", train_answers[idx]['text'])
```

It turns out that the tokenization was almost always incorrect:

1) The standard case should not be:

```python
encodings.char_to_token(i, answers[i]['answer_end'] - 1)
```

, but

```python
encodings.char_to_token(i, answers[i]['answer_end'])
```

2)

It might happen that `char_to_token` points to a space character which has no corresponding token and is therefore `None`. In this case the character after the space should be used.

The fix proposed in the PR corrects this behavior.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


replacing apex.normalization.FusedLayerNorm with torch.nn.LayerNorm

It seems that time has arrived to drop `apex.normalization.FusedLayerNorm` in favor of `torch.nn.LayerNorm`

1. the latter was ported more than a year ago from apex  https://github.com/pytorch/pytorch/pull/27634 (around pt-1.4)
2. it's faster than the apex according to my benchmarks https://github.com/pytorch/pytorch/issues/37713#issuecomment-753434842 (**33% faster on rtx-3090!**, 10% faster on gtx-1070)

**but note:** this same benchmark run here https://github.com/pytorch/fairseq/issues/2012#issuecomment-622607286 on V100 reports the opposite  - that the native is slower (pt-1.5). So it might help to run this very quick benchmark on other cards and compare. In particular if you have access to V100 please report back the findings at this thread: https://github.com/pytorch/pytorch/issues/37713

The main reason for this need is that `apex.normalization.FusedLayerNorm` is buggy (corrupts memory) when it comes to switching devices, which is done a lot under Model Parallel. https://github.com/NVIDIA/apex/issues/1022
With  `apex.normalization.FusedLayerNorm` things fail a lot under MP and requires sticking `torch.cuda.set_device(id)` in many many places as a workaround :( Since this overload is used at model's init time it's not possible to not use it under MP as the latter gets activate after model's init.

I will use that workaround if you find out that apex is faster still on some important-to-consider hardware. And, of course, in that case please report back to the pytorch team so that they could fix it. Otherwise apex support is pretty much no more and it's just a matter of time before apex will be unusable.

The models that need that change are bart/fsmt/prophetnet.

@patrickvonplaten, @LysandreJik


[docs] TFRobertaModel example: last_hidden_states -> last_hidden_state

This is a documentation error on the currently deployed version of https://huggingface.co/transformers/

### Who can help

 examples/distillation: @VictorSanh
 documentation: @sgugger

## Information

Model I am using: TFRoberta

The problem arises when using:
* [x] the official example scripts

## To reproduce

Steps to reproduce the behavior:

1. View the code example at https://huggingface.co/transformers/model_doc/roberta.html#tfrobertamodel
2. `last_hidden_states = outputs.last_hidden_states` should be `last_hidden_states = outputs.last_hidden_state`

The current incorrect spelling will yield an error. I apologize that I was not able to find that line in the repo, otherwise I would submit a PR.


Fix Typo

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
I fixed typo in the comment.

## Before submitting
- [ V] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form:

```
Category 1 10000
Category 2  2000
Category 3   400
Category 4   300
Category 5   100

```
The dataset consists of text with data labeled into one of the five categories. I am trying to use the pretrained models like BERT for the classification task but the model fails to identify the categories 3-5 .I have tried to apply class weights in the loss criterion however it doesn't help much although it gives better performance as compared to simple fine tuning of the pretrained models. I have came to know about SMOTE and other methods in order to handle the class imbalance issues . But since most of the transformer models expect the inputs as text which are later tokenized by their respective tokenizers I am not able to do any kind of oversampling . If there is a workaround for this thing I would be interested to know about it.


how to evaluate models on SUPER_GLUE benchmark?

Hi, I am trying to evaluate models on SUPER_GLUE benchmark.
However, I can load SUPER_GLUE dataset from Transformer but I cant find any metrics of this benchmark.
Is there any script like _**superglue_metrics.py**_ that can evaluate models on superglue?

thanks a lot! :)


Why does datasets get imported when running "from transformers.models.roberta.tokenization_roberta_fast import RobertaTokenizerFast"

When running

    from transformers.models.roberta.tokenization_roberta_fast import RobertaTokenizerFast

I get this warning:

    ImportWarning: To use `datasets`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition.  If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.

Why is datasets getting imported, when we import the tokenizer?

Thanks! :)


Excessive GPU-GPU communication with GPT2 making multi-GPU training slow?

Benchmarks

Issues related to Memory regressions in tests and scripts

Summary: on a multi-GPU system, training GPT2 seems to scale poorly unless a very fast GPU-GPU interconnect like NVLink is available. In particular, without NVLink using two GPUs is *slower* than using just one GPU.

## Environment info

- `transformers` version: 4.1.1
- Platform: Linux-5.8.0-rc7-custom-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0.dev20201214+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No?
- Hardware: 2 x NVIDIA RTX 3090 w/NVLink

### Who can help

Maybe @LysandreJik or @patrickvonplaten ?

## Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The script is a pretty basic example of training a medium-size GPT2 from scratch. The script is here: https://panda.moyix.net/~moyix/train_csrc.py

The dataset and tokenized vocab:
* Dataset: https://panda.moyix.net/~moyix/plainsrc_all.txt.gz (718M, gzipped)
* Vocab: https://panda.moyix.net/~moyix/csrc_vocab.tar.gz

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

Training a GPT2 language model on C source code.

## To reproduce

Run with only one GPU: `CUDA_VISIBLE_DEVICES=0 python train_csrc.py`

Run with two GPUs, NVLink disabled: `NCCL_P2P_DISABLE=1 python train_csrc.py`

Run with two GPUs and NVLink enabled: `python train_csrc.py`

Here is some benchmarking I did with my dataset on transformers 3.3.1 and 4.1.1 (note the difference in ETA is just because 3.3.1 only seems to report the ETA for the current epoch):

Version|NVLINK|GPUs|ETA|Perf
--------|--------|-----|-----|-----
4.1.1 |  Yes  |  2GPU |  419:52:28 |  1.94it/s
4.1.1 |  No   |  2GPU | 1025:06:27 |  1.26s/it
4.1.1 |  N/A  |  1GPU |  599:14:57 |  2.72it/s
3.3.1 |  Yes  |  2GPU |   83:46:51 |  1.94it/s
3.3.1 |  No   |  2GPU |  204:54:22 |  1.26s/it
3.3.1 |  N/A  |  1GPU |  119:02:34 |  2.73it/s

You can see that using two GPUs is actually slower than using a single GPU, unless NVLink is available (599 hours for 1 GPU vs 1025 hours for two GPUs). So presumably there is a large amount of GPU-GPU communication going on?

## Expected behavior

Scaling should be roughly linear with the number of GPUs. Unfortunately I am not very familiar with the implementation details of GPT2 in Huggingface, but others report roughly linear scaling with Transformer models like BERT so it should work here as well: https://towardsdatascience.com/training-bert-at-a-university-eedcf940c754

Although I have  a system with NVLink at home, this issue is still affecting me because I would like to be able to run this on the university HPC cluster, where most nodes do not have NVLink.


Custom train/validation file not supported in  run_qa.py

**Environment info**
transformers version: 4.0.1
Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.10
Python version: 3.8.5
PyTorch version (GPU?): 1.7.1+cu110 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: yes

I am trying to pass custom dataset or modified squad dataset (in valid squad format only) using parameters
  --train_file = train-v1.1.json \
  --validation_file = dev-v1.1.json \
but it does not work for me   g

from the official documentation, **https://github.com/huggingface/transformers/tree/master/examples/question-answering**
this script runs fine:
```
python run_qa.py \
  --model_name_or_path bert-base-uncased \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```

but if I use the below script:
```
python run_qa.py \
  --model_name_or_path bert-base-uncased \
  --train_file = train-v1.1.json \
  --validation_file = dev-v1.1.json \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 16 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /data1/debug_squad1/
```

**for data**  train-v1.1.json.  dev-v1.1.json /  train.csv, dev.csv
error:
```
2020-12-31 12:00:59.821145: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2020-12-31 12:00:59.821182: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "run_qa.py", line 469, in <module>
    main()
  File "run_qa.py", line 159, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/media/data2/anaconda/envs/bertQA-env/lib/python3.8/site-packages/transformers/hf_argparser.py", line 135, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 16, in __init__
  File "run_qa.py", line 142, in __post_init__
    assert extension in ["csv", "json"], "`train_file` should be a csv or a json file."
AssertionError: `train_file` should be a csv or a json file.
```

the train_file, validation_file is a valid parameter in run_qa.py file.
Can someone please help with how can we train on specific dataset?


TF >= 2.3 cleaning

# What does this PR do?

The minimal TF version has recently been fixed to >=2.3, this PR remove all the <2.3 calls, mostly replacing experimental features by their stable ones.


Fix utils on Windows

# What does this PR do?

This PR fixes `check_repo` for Windows execution.