[Good first issue] DistilBERT TensorFlow Integration tests

TensorFlow

Anything TensorFlow

Tests

Related to tests

The TensorFlow implementation of the DistilBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_distilbert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_distilbert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Good first issue] MPNet PyTorch Integration tests

PyTorch

Anything PyTorch

Tests

Related to tests

The PyTorch implementation of the MPNet model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_mpnet.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_mpnet.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Good first issue] LXMERT PyTorch Integration tests

PyTorch

Anything PyTorch

Tests

Related to tests

The PyTorch implementation of the LXMERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_lxmert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_lxmert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Good first issue] FlauBERT PyTorch Integration tests

PyTorch

Anything PyTorch

Tests

Related to tests

The PyTorch implementation of the FlauBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_flaubert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_flaubert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.
- The TensorFlow implementation already has an integration test, which is visible here:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_flaubert.py#L342-L370

This test can be translated to PyTorch.


[Good first issue] ELECTRA PyTorch Integration tests

PyTorch

Anything PyTorch

Tests

Related to tests

The PyTorch implementation of the ELECTRA model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_electra.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_electra.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.
- The TensorFlow implementation already has an integration test, which is visible here:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_electra.py#L253-L267

This test can be translated to PyTorch.


[Good first issue] DistilBERT PyTorch Integration tests

PyTorch

Anything PyTorch

Tests

Related to tests

The PyTorch implementation of the DistilBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_distilbert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_distilbert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Good first issue] BERT Generation PyTorch Integration tests

PyTorch

Anything PyTorch

Tests

Related to tests

The PyTorch implementation of the BERT for generation model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_bert_generation.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert_generation.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Good first issue] ALBERT TensorFlow Integration tests

TensorFlow

Anything TensorFlow

Tests

Related to tests

The TensorFlow implementation of the ALBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_albert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_albert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Good first issue] ALBERT PyTorch Integration tests

PyTorch

Anything PyTorch

Tests

Related to tests

The PyTorch implementation of the ALBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_albert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_albert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Bart models] fix typo in naming

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Creds go to @ratthachat for spotting it!

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


ALBERT Tokenizer integration test

Implements an integration test for the ALBERT tokenizer.


Fix Longformer and LED

# What does this PR do?

This PR fix TF Longformer and LED when `inputs_embeds`/`decoder_inputs_embeds` are used as main input instead of `input_ids`/`decoder_input_ids`.

Here a quick test that shows the bug for Longformer:
```python
from transformers.models.longformer.modeling_tf_longformer import TFLongformerMainLayer
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from transformers import LongformerConfig
import tensorflow as tf
import numpy as np

class CustomLongFormer(tf.keras.layers.Layer):
    def __init__(self, name='longformer', **kwargs):
        super().__init__(name=name, **kwargs)
        config = LongformerConfig(attention_window=4, num_hidden_layers=1, vocab_size=10)
        self.longformer = TFLongformerMainLayer(config)

    def call(self, inputs):
        x = self.longformer(inputs)[0]
        return x

longformer = CustomLongFormer()
inputs_embeds = Input(shape=(None, None), dtype='float32', name="inputs_embeds")
output = longformer({"inputs_embeds": inputs_embeds})
output = Dense(9, activation='softmax')(output)
model = Model({"inputs_embeds": inputs_embeds}, output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

x = np.array([np.random.uniform(0,1, (3, 768))] * 100)
y = np.array([[1]*3] * 100)
model.fit(x=x, y=y, epochs=10, batch_size=4, validation_split=0.1)
```

And the one for LED:
```python
from transformers.models.led.modeling_tf_led import TFLEDMainLayer
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from transformers import LEDConfig
import tensorflow as tf
import numpy as np

class CustomLED(tf.keras.layers.Layer):
    def __init__(self, name='longformer', **kwargs):
        super().__init__(name=name, **kwargs)
        config = LEDConfig(attention_window=4, num_hidden_layers=1, vocab_size=10)
        self.led = TFLEDMainLayer(config)

    def call(self, inputs):
        x = self.led(inputs)[0]
        return x

led = CustomLED()
inputs_embeds = Input(shape=(None, None), dtype='float32', name="inputs_embeds")
decoder_inputs_embeds = Input(shape=(None, None), dtype='float32', name="decoder_inputs_embeds")
output = led({"inputs_embeds": inputs_embeds, "decoder_inputs_embeds": decoder_inputs_embeds})
output = Dense(9, activation='softmax')(output)
model = Model({"inputs_embeds": inputs_embeds, "decoder_inputs_embeds": decoder_inputs_embeds}, output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

x = np.array([np.random.uniform(0,1, (3, 1024))] * 100)
y = np.array([[1]*3] * 100)
model.fit(x={"inputs_embeds": x, "decoder_inputs_embeds": x}, y=y, epochs=10, batch_size=4, validation_split=0.1)
```

The reason is because the graph compiled is different than the one when the usual `input_ids`/`decoder_input_ids` are used. Knowing that we are not testing this case (in graph execution) other models might be involved in a similar bug. Hence, I put in my TODO list to create a test that will test if all the models can be used with different combinaison of inputs in graph mode.

# Fix issue
#9864


Converting pretrained tf2 bert model to pytorch model for using FillMaskPipeline

Sincerely greeting to every NLP researchers and HuggingFace members,
I am currently working on masked word prediction in a sentence like what FillMaskPipeline did.
(i.e. This is a  [MASK] test.  Where the result of [MASK] might gives
{'sequence': '[CLS]This is a [MASK] test.[SEP]', 'score': floating point, 'token': integer, 'token_str': string }
as one of the candidate output)

In my past few works I downloaded bert-base-chinese as my base model, which can be found in [original bert](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip), then keep running the pretraining steps on my custom dataset using run_pretraining [script](https://github.com/google-research/bert/blob/master/run_pretraining.py), finally using [transformers-cli](https://huggingface.co/transformers/converting_tensorflow_models.html) to convert my bert_model.ckpt to pytorch_model.bin and apply FillMaskPipeline to do masked word prediction.
Everything work fine when I use the script under tensorflow-gpu==1.13.1 and CUDA 10.0 with GTX 1080 Ti. However, in current task I want to switch from 1080ti to RTX3090 to have larger batch size but in [Access to checkpoint](https://github.com/tensorflow/models/tree/master/official/nlp/bert#access-to-pretrained-checkpoints) the bert-base-chinese model haven't release yet, so I first convert the tf1 bert-base-chinese model to tf2 by using this [script](https://github.com/tensorflow/models/blob/master/official/nlp/bert/tf2_encoder_checkpoint_converter.py) with following args,

!python tf2_encoder_checkpoint_converter.py --checkpoint_to_convert=$BASE_DIR/bert_model.ckpt --converted_checkpoint_path=tmp/ --bert_config_file=$BASE_DIR/bert_config.json

after model being converted I use tf2 [create_pretraining_data.py](https://github.com/tensorflow/models/blob/master/official/nlp/data/create_pretraining_data.py) on my dataset and run [run_pretraining.py](https://github.com/tensorflow/models/blob/master/official/nlp/bert/run_pretraining.py). The problem I encounter now is after finished training I have no idea how to do masked prediction task on my data now.
I tried [converting ckpt script](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/convert_bert_original_tf2_checkpoint_to_pytorch.py) but notice the BertSelfAttention layer can not be convert due to MLM heads, also I didn't found any related script doing similar work as FillMaskPipeline in tensorflow, my questions are,

- Is it possible to convert pretrained tf2 model to pytorch without any addition modification on convert_bert_original_tf2_checkpoint_to_pytorch.py (I have no idea how to move forward and I also try to ignore the layer that can not convert but it raise the error as following png)
![image](https://user-images.githubusercontent.com/56808566/106563015-206cc180-6566-11eb-9356-000d9281df64.png)

- Is there any related script in tensorflow 1 and tensorflow 2 I can refer to, I spent some time finding related posts but with no luck. I merely understand some tensorflow so if anyone can give me direction on doing masked prediction would be much more appreciated.

### System Info
My environments are: (using pipreqs)
Python version: 3.7.5
CUDA used to build PyTorch: cuda_11.0_bu.TC445_37.28845127_0
OS: Ubuntu 18.04.5 LTS
GCC version: 7.5.0

### Versions of relevant libraries: (using pipreqs)
numpy==1.19.5
transformers==4.2.2
six==1.15.0
torch==1.7.1+cu110
tensorflow_gpu==2.2.0
gin_config==0.1.1
absl_py==0.11.0
tensorflow_hub==0.11.0
sentencepiece==0.1.94
absl==0.0
torchvision==0.8.2+cu110
bert4keras==0.9.9
gin==0.1.006
tensorflow==2.4.1

Any suggestions, helps, and relative post that I can dig into would be appreciated. Thanks in advance!


[wip] [pipeline parallel] t5 - experiment #2

WIP

Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

The first attempt at t5/pp using pytorch-nightly Pipe https://github.com/huggingface/transformers/pull/9765 was successful to a degree, but at the moment can't be combined with any other Parallel solutions.

All the examples of Pipeline conversion use trivial examples or models that lend easily to being converted to `Sequential`. `transformers` models or at least `t5` doesn't easily lend to this transformation due to complex intertwined logic and a huge number of variables passed around.

The main challenge: In order to build a Pipeline one needs to convert the Module stack into a `Sequential` list.

So in the case of t5, we need to convert this logic:
```
T5ForConditionalGeneration->
  logic
  T5Stack->
     logic
     loop(T5Block, T5Block, T5Block, ...) ->
     logic
  logic
  T5Stack->
     logic
     loop(T5Block, T5Block, T5Block, ...) ->
     logic
  logic
```
into
```
Pipe(
  Sequential(
    T5ForConditionalGeneration,
    T5ForConditionalGeneration_p1,
    T5Stack,
    T5Stack_p1,
    T5Block,
    T5Block,
    T5Block,
    ...
    T5Stack_p2,
    T5ForConditionalGeneration_p2,
    T5Stack,
    T5Stack_p1,
    T5Block,
    T5Block,
    T5Block,
    ...
    T5Stack_p2,
    T5ForConditionalGeneration_p3,
  )
)
```
I think we don't need to Sequentialize any further beyond T5Block, but we will have to see down the road.

Problems:
1. Can't change the structure of the model because of the pre-trained weights.
2. The inputs/outputs are very complicated because the entry into the Pipeline (first and last stages) can only be a tuple of pure Tensors.
3. The inputs/outputs besides required to be Tensors have to expose first dimension to be batch-dimension since it slices all inputs and restores all outputs on that dimension on the way to/from forward (but only on the very first and last stages of the sequence)

I did successfully implement a t5-pipeline version https://github.com/huggingface/transformers/pull/9765  that uses 2 shorter pipes, as it was natural to convert a loop over `T5Block`s to `Sequential` and it now looks like this
```
T5ForConditionalGeneration->
  logic
  T5Stack-> Pipe(Sequential(T5Block, T5Block, T5Block))
  logic
  T5Stack-> Pipe(Sequential(T5Block, T5Block, T5Block))
  logic
```
using pytorch pipe in a very painful way overcoming problem n2. But it's doubtful this approach will work with any other 1D Parallel side (e.g. combining with Sharded DDP) - definitely doesn't work with DeepSpeed Zero-DP.

But that implementation won't work with DeepSpeed pipeline - it has to be Sequential from the top-level. Not sure about fairscale yet.

So I'm trying again, this time starting by just trying to Sequentialize the layers while overcoming problem n1.

If you do look at the code, please ignore everything in the diff but `modeling_t5.py` (and I removed a lot of the model parallel code as it is getting in the way and it won't be needed if we figure out the pipe - since `pipe(chunks=1) == naive vertical MP`, so we get all the complex things that MP currently does for free. But we have to do even more complicated things instead. Naive vertical MP appears to be trivial compared to the changes required to make pipe work.

You can see the process of conversion in this PR, I Sequentialized:
1. the `T5Block`-loop
2. the 2nd half of `T5Stack`,

now I need to continue breaking up the structure upstream. At this stage there is no Pipe in the code, the 1st main difficulty is to Sequentilize the layers.

If you want to see just how I converted the `T5Block`-loop into Sequential, it is this commit - might be easier to see: https://github.com/huggingface/transformers/pull/9940/commits/4c0ea522157f693bccce80c4cbecc24019186676 The input/output have to be the same because Sequential sends the output of one stage to the input of another.

If you have some brilliant ideas that I'm perhaps missing at how to easily Sequentialize t5 layers I'm all ears.

@patrickvonplaten, @sgugger, @LysandreJik


Can't import pipeline

- `transformers` 4.2
- Platform: MacOS
- Python version: 3.7.9
- PyTorch version (GPU?): CPU
- Tensorflow version (GPU?): CPU
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
- Pip Version: Latest

I can't import the pipeline function:
```
from transformers import pipeline
```

Gives the following error:
```Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'pipeline' from 'transformers' (unknown location)
```


trainer_seq2seq.py Question

Hi, does trainer_seq2seq.py in transformers/src support multi GPU training? Thank you


ConvBERT: minor fixes for conversion script

Hi,

the conversion script for ConvBERT throws the following error message when using it:

```bash
Traceback (most recent call last):
  File "convert_convbert_original_tf1_checkpoint_to_pytorch.py", line 19, in <module>
    from ...utils import logging
ImportError: attempted relative import with no known parent package
```

I fixed that error, as well as using the correct name for the configuration file argument.

Additionally, I just found that the configuration files from the [YituTech](https://huggingface.co/YituTech) organization for ConvBERT from aren't correct, because they use:

```json
"model_type": "conv_bert",
```

instead of:

```json
"model_type": "convbert",
```

(This currently results in a `KeyError: 'conv_bert'` error).


Use compute_loss in prediction_step

# What does this PR do?

As requested in #9915, this PR uses `compute_loss` in the `prediction_step` method of `Trainer`, so it properly computes losses when the user have customized the way to do that. It does require a new argument to `compute_loss` to return the outputs on top of the loss for the prediction loop, so users that want to use this feature will have to tweak their subclass a little bit, but there is no breaking change.


Bump numpy

# What does this PR do?

As pointed out on the [forums](https://discuss.huggingface.co/t/typeerror-full-like-got-an-unexpected-keyword-argument-shape/2981), the method `np.full_like` used in the evaluation of the `Trainer` with the argument `shape=` does not work for all versions of numpy. According to the [numpy documentation]() it was introduced in version 1.17 only, so this PR bumps the setup to that version.

If for some reason we don't want to have a minimum version of numpy, I can try to find another way to do the same thing in `Trainer`.


Possible bug in `prepare_for_model` when using fast tokenizers

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.5.1
- `tokenizers` version: 0.9.3
- Platform: Linux
- Python version: 3.7.2
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

Also confirmed with `transformers==4.2.2` & `tokenizers==0.9.4`

### Who can help
tokenizers: @n1t0, @LysandreJik
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

I am building my own data preprocessing script which requires me to first know the number of tokens in each sentence before, then match sentences pairs and prepare them as input for a model, in this case BERT. I would like to use the fast tokenizer to speed things up in large datasets, however, I encounter the next assertion problem which is not true since I do provide `return_special_tokens_mask=True` in the function call.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

The following code snippet reproduces the problem for me:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)
s1 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("Halt! Who goes there?"))
s2 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("It is I, Arthur, son of Uther Pendragon, from the castle of Camelot. King of the Britons, defeator of the Saxons, sovereign of all England!"))
tokenizer.prepare_for_model(s1, s2, return_special_tokens_mask=True)
```

I get the following assertion error when running the code:
```
AssertionError                            Traceback (most recent call last)
<ipython-input-5-889164fb3ae8> in <module>
      2 s1 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("Halt! Who goes there?"))
      3 s2 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("It is I, Arthur, son of Uther Pendragon, from the castle of Camelot. King of the Britons, defeator of the Saxons, sovereign of all England!"))
----> 4 tokenizer.prepare_for_model(s1, s2, return_special_tokens_mask=True)

~/p/.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in prepare_for_model(self, ids, pair_ids, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, prepend_batch_axis, **kwargs)
   2724         if return_special_tokens_mask:
   2725             if add_special_tokens:
-> 2726                 encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids)
   2727             else:
   2728                 encoded_inputs["special_tokens_mask"] = [0] * len(sequence)

~/p/.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in get_special_tokens_mask(self, token_ids_0, token_ids_1, already_has_special_tokens)
   3031         """
   3032         assert already_has_special_tokens and token_ids_1 is None, (
-> 3033             "You cannot use ``already_has_special_tokens=False`` with this tokenizer. "
   3034             "Please use a slow (full python) tokenizer to activate this argument."
   3035             "Or set `return_special_token_mask=True` when calling the encoding method "

AssertionError: You cannot use ``already_has_special_tokens=False`` with this tokenizer. Please use a slow (full python) tokenizer to activate this argument.Or set `return_special_token_mask=True` when calling the encoding method to get the special tokens mask in any tokenizer.
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
For the fast tokenizer to provide the same output as the slow (python) tokenizer:
```python
{'input_ids': [101, 9190, 999, 2040, 3632, 2045, 1029, 102, 2009, 2003, 1045, 1010, 4300, 1010, 2365, 1997, 21183, 5886, 7279, 7265, 7446, 1010, 2013, 1996, 3317, 1997, 19130, 4140, 1012, 2332, 1997, 1996, 28101, 5644, 1010, 4154, 2953, 1997, 1996, 28267, 1010, 11074, 1997, 2035, 2563, 999, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'special_tokens_mask': [1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

<!-- A clear and concise description of what you would expect to happen. -->

## Possible Fix
Make the following change in:
https://github.com/huggingface/transformers/blob/d1b14c9b548de34b6606946482946008622967db/src/transformers/tokenization_utils_base.py#L2855
Since we know beforehand if the special tokens were added and the text pair was already concatenated with or without special tokens, I think the following change will be valid, however, I didn't test it beyond my own use case
```python
        if return_special_tokens_mask:
            if add_special_tokens:
                encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(sequence, already_has_special_tokens=True)
            else:
                encoded_inputs["special_tokens_mask"] = [0] * len(sequence)
```


Fix 9918

# What does this PR do?

This PR addresses the problem shown in #9918 by:
- adding the documentation of the `encode` method to the `PreTrainedTokenizer` and `PreTrainedTokenizerFast` (it is in all their subclasses already)
- adding the "What are input IDs" link where missing in some models docstrings.

In passing, I uncovered a failure of the doc styling script on DPR, so this PR also fixes that.

Fixes #9918


[2D Parallelism] Tracking feasibility

Model Parallel

Model Parallelilsm Implementations

### Background

ZeRO-DP (ZeRO Data Parallel) and PP (Pipeline Parallelism) provide each a great memory saving over multiple GPUs. Each 1D allows for a much more efficient utilization of the gpu memory, but it's still not enough for very big models - sometimes not even feasible with any of the existing hardware. e.g. a model that's 45GB-big with just model params (t5-11b) can't fit even on a 40GB GPU.

The next stage in Model Parallelism that can enable loading bigger models onto smaller hardware is 2D Parallelism. That's combining Pipeline Parallelism (PP) with ZeRO-DP.

3D Parallelism is possible too and it requires adding a horizontal MP (ala [Megatron-LM](https://github.com/NVIDIA/Megatron-LM), but we don't quite have any way to implement that yet. Need to study Megatron-LM first. So starting with a relatively low hanging fruit of 2D.

------------------

### Tracking

We have 3 implementations that provide the required components to build 2D Parallelism:

1. DeepSpeed (**DS**)
2. FairScale (**FS**)
3. PyTorch (native) (**PT**)

and the purpose of this issue is to track the feasibility/status/inter-operability in each one of them. And also which parts have been back-ported to PyTorch core.

Plus it tracks the status of where transformers models are at with regards to the above 3 implementations.

The 2 main questions are:

1. native 2D: how do we integrate a native PP with native ZeRO-DP (sharded) (e.g. can fairscale PP work with fairscale ZeRO-DP)
2. inter-operability 2D: is there a chance one implementation of PP/ZeRO-DP could work with one or both others ZeRO-DP/PP (e.g. can fairscale PP work with DeepSpeed ZeRO-DP).

------------------

### Notes

* 3D Parallelism is possible too and it requires adding a horizontal MP (ala Megatron-LM), but we don't quite have any way to implement that yet. Need to study Megatron-LM first. So starting with low hanging fruit of 2D.

* MPU = Model Parallel Unit - a little helper module that helps each 1D to know which gpu groups it can use for PP, which for MP, which for DP. So that one 1D doesn't interfere with another 1D. e.g. in the case of 4 gpus and PP+DP, one may want:
   ```
         pp
   dp0 [0, 1]
   dp1 [2, 3]
   ```

   So here there are 2 pipelines: 0-1, and 2-3, and DP sees gpus 0 and 2 as the entry points.

--------------------------

### TLDR

ZeRO-DP / PP inter-operability status

|    | DS | FS | PT |
|----|----|----|----|
| DS |  :heavy_check_mark:  | :question:  |  :x: |
| FS |  :question: | :question:  |  :question: |
| PT | :x:| :question:  |  :question: |

--------------------------

### 1. DeepSpeed

1D status:
* [x] [PP](https://www.deepspeed.ai/tutorials/pipeline/)
* [x] [ZeRO-DP](https://www.deepspeed.ai/tutorials/zero/)

2D native status:
* [ ] :question: native PP + ZeRO-DP - untested yet, as it requires porting transformers to native PP first

2D inter-operability status:
- [ ] :x: pytorch PP + DeepSpeed ZeRO-DP. I tried using pytorch PP with DeepSpeed ZeRO-DP and couldn't figure out how to make it work: https://github.com/microsoft/DeepSpeed/issues/710
- [ ] :question: fairscale PP + DeepSpeed ZeRO-DP  (unknown)

Important components:
 * [original megatron-lm MPU](https://github.com/microsoft/DeepSpeedExamples/blob/master/Megatron-LM/mpu/initialize.py)
 * [WIP DeepSpeed MPU](https://github.com/jeffra/DSE/blob/megatron-deepspeed-pipeline/megatron/mpu/initialize.py)

--------------------------

### 2. FairScale

Just started gather information on this one - will update once I have it.

1D status:
* [x] [PP](https://fairscale.readthedocs.io/en/latest/tutorials/pipe.html)
* [x] [ZeRO-DP](https://fairscale.readthedocs.io/en/latest/tutorials/oss.html)

2D native status:
* [ ] :question: native PP + ZeRO-DP - gathering info https://github.com/facebookresearch/fairscale/issues/351

2D inter-operability status:
- [ ] :question:  pytorch PP + fairscale ZeRO-DP gathering info
- [ ] :question:  DeepSpeed PP + fairscale ZeRO-DP gathering info

Important components:
* [MPU](https://github.com/facebookresearch/fairscale/blob/master/fairscale/nn/model_parallel/initialize.py#L41)

--------------------------

### 3. PyTorch

pytorch has been integrating from what I understand primarily fairscale version into its core.

1D status:
* [x] [PP](https://pytorch.org/docs/master/pipeline.html) - experimental support. have PoC t5 working: https://github.com/huggingface/transformers/pull/9765 [example](https://github.com/pytorch/pytorch/blob/master/benchmarks/distributed/pipeline/pipe.py)
* [ ] ZeRO-DP - plans to implement that (primarily integrating fairscale implementation)

2D native status:
- [ ] :grey_exclamation: native PP + ZeRO-DP (Pytorch ZeRO-DP doesn't exists yet)

2D inter-operability status:
- [ ] :grey_exclamation: DeepSpeed PP + Pytorch ZeRO-DP (Pytorch ZeRO-DP doesn't exists yet)
- [ ] :grey_exclamation: fairscale PP + Pytorch ZeRO-DP  (Pytorch ZeRO-DP doesn't exists yet)

Important components:
* MPU:  ?

Ported components:
* ZeRO-DP stage 1: ZeroRedundancyOptimizer: an implementation of a standalone sharded optimizer wrapper https://github.com/pytorch/pytorch/pull/46750

Issues to track:
* The main discussion around integrating Deepspeed ZeRO into pytorch core: https://github.com/pytorch/pytorch/issues/42849

--------------------

### Transformers

To make 2D Parallelism working we need to of course support all these stages in `transformers`, so here is a status on what we have working or what is a work in progress. Some components (like bart-mp) work but are unmerged since we are still unsure how to move forward project-wide.

* ZeRO-DP
   - [x] works across all models with fairscale and DeepSpeed integrated.

* Naive vertical MP (aka PP w/ a single stage)
   - [x] t5
   - [x] gpt2
   - [ ] bart - unmerged https://github.com/huggingface/transformers/pull/9384

* Pytorch PP
   - [ ] t5 - unmerged https://github.com/huggingface/transformers/pull/9765

* Horizontal MP - unresearched!


Hyperparameter search w/ RayTune BrokenPipeError: [Errno 32] Broken pipe

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7
- Using GPU in script?: Yes

### Who can help
- ray/raytune: @richardliaw, @amogkam
- trainer: @sgugger

### Information

Model I am using (Bert, XLNet ...): sshleifer/distilbart-cnn-12-6
Dataset: dummy XSUM (50 samples in train, 5 samples in val)

## To reproduce

I have tried `trainer.train` with the exact same parameters and it works just fine.

I am trying to do a hyperparameter search with the Seq2SeqTrainer and RayTune. For now I am just trying a dummy search with 2 different learning rates and 2 different gradient accumulation steps. Here is my code:

```
def hp_objective(metrics):
    loss = metrics.pop('eval_loss', None)
    _ = metrics.pop('epoch', None)
    _ = metrics.pop('eval_gen_len', None)

    return np.sum(list(metrics.values()))

def hp_space(trial):
    from ray import tune

    return {
    'learning_rate': tune.choice([1e-5, 1e-4]),
    'gradient_accumulation_steps': tune.choice([4, 8])
    }

def model_init():
            model = AutoModelForSeq2SeqLM.from_pretrained(
                model_args.model_name_or_path,
                config=config,
                cache_dir=model_args.cache_dir,
            )
            # use task specific params
            use_task_specific_params(model, data_args.task)

            # set num_beams for evaluation
            if data_args.eval_beams is None:
                data_args.eval_beams = model.config.num_beams

            # set decoder_start_token_id for MBart
            if model.config.decoder_start_token_id is None and isinstance(tokenizer, MBartTokenizer):
                assert (
                    data_args.tgt_lang is not None and data_args.src_lang is not None
                ), "mBart requires --tgt_lang and --src_lang"
                model.config.decoder_start_token_id = tokenizer.lang_code_to_id[data_args.tgt_lang]

            if model_args.freeze_embeds:
                freeze_embeds(model)
            if model_args.freeze_encoder:
                freeze_params(model.get_encoder())
                assert_all_frozen(model.get_encoder())

            return model

trainer = Seq2SeqTrainer(
            model_init=model_init,
            config=config,
            args=training_args,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            data_collator=Seq2SeqDataCollator(tokenizer, data_args, training_args.tpu_num_cores),
            compute_metrics=compute_metrics_fn,
            data_args=data_args)

logger.info("*** Hyperparameters Search ***")
start_time = time.time()

trainer.hyperparameter_search(
            direction = "maximize",
            compute_objective = hp_objective,
            hp_space = hp_space,
            backend = "ray",
            resources_per_trial = {'gpu': 1})
```

And I get the following error:

```
02/01/2021 16:26:31 - INFO - __main__ -   *** Hyperparameters Search ***
02/01/2021 16:26:31 - INFO - ray.tune.ray_trial_executor -   Initializing Ray automatically.For cluster usage or custom Ray initialization, call `ray.init(...)` before `tune.run`.
2021-02-01 16:26:33,796	INFO services.py:1173 -- View the Ray dashboard at http://127.0.0.1:8265
tcmalloc: large alloc 1236656128 bytes == 0x7f275ac1a000 @  0x7f2ab6d16615 0x591e47 0x4cc179 0x4cc2db 0x566a71 0x5a4cd1 0x5a4fb8 0x7f2a1e822c7c 0x7f2a1e829bfa 0x7f2a1e829fe2 0x7f2a1e82b34f 0x7f2a1e828a39 0x7f2a1e829afc 0x7f2a1e82b34f 0x7f2a1e828a39 0x7f2a1e829fe2 0x7f2a1e82b34f 0x7f2a1e828a39 0x7f2a1e82ad13 0x7f2a1e82b43c 0x7f2a1e828a39 0x7f2a1e829f45 0x7f2a1e82b7fa 0x7f2a1e828a39 0x7f2a1e82ad5e 0x7f2a1e82b43c 0x7f2a1e828a39 0x7f2a1e829f45 0x7f2a1e82b7fa 0x7f2a1e828a39 0x7f2a1e82ad5e
tcmalloc: large alloc 1545822208 bytes == 0x7f26fe9e4000 @  0x7f2ab6d16615 0x591e47 0x4cc179 0x4cc2db 0x566a71 0x5a4cd1 0x5a4fb8 0x7f2a1e822c7c 0x7f2a1e829bfa 0x7f2a1e829fe2 0x7f2a1e82b34f 0x7f2a1e828a39 0x7f2a1e829afc 0x7f2a1e82b34f 0x7f2a1e828a39 0x7f2a1e829fe2 0x7f2a1e82b34f 0x7f2a1e828a39 0x7f2a1e82ad13 0x7f2a1e82b43c 0x7f2a1e828a39 0x7f2a1e829f45 0x7f2a1e82b7fa 0x7f2a1e828a39 0x7f2a1e82ad5e 0x7f2a1e82b43c 0x7f2a1e828a39 0x7f2a1e829f45 0x7f2a1e82b7fa 0x7f2a1e828a39 0x7f2a1e82ad13
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 706, in send_packed_command
    sendall(self._sock, item)
  File "/usr/local/lib/python3.6/dist-packages/redis/_compat.py", line 9, in sendall
    return sock.sendall(*args, **kwargs)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/content/drive/My Drive/MAGMA: Summarization/transformers/examples/seq2seq/finetune_trainer.py", line 436, in <module>
    main()
  File "/content/drive/My Drive/MAGMA: Summarization/transformers/examples/seq2seq/finetune_trainer.py", line 351, in main
    resources_per_trial = {'gpu': 1})
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1077, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/integrations.py", line 252, in run_hp_search_ray
    analysis = ray.tune.run(_objective, config=trainer.hp_space(None), num_samples=n_trials, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/tune.py", line 325, in run
    restore=restore)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/experiment.py", line 149, in __init__
    self._run_identifier = Experiment.register_if_needed(run)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/experiment.py", line 287, in register_if_needed
    register_trainable(name, run_object)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/registry.py", line 71, in register_trainable
    _global_registry.register(TRAINABLE_CLASS, name, trainable)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/registry.py", line 124, in register
    self.flush_values()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/registry.py", line 146, in flush_values
    _internal_kv_put(_make_key(category, key), value, overwrite=True)
  File "/usr/local/lib/python3.6/dist-packages/ray/experimental/internal_kv.py", line 27, in _internal_kv_put
    updated = worker.redis_client.hset(key, "value", value)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3050, in hset
    return self.execute_command('HSET', name, *items)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 900, in execute_command
    conn.send_command(*args)
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 726, in send_command
    check_health=kwargs.get('check_health', True))
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 718, in send_packed_command
    (errno, errmsg))
redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe.
02/01/2021 16:26:48 - INFO - wandb.sdk.internal.internal -   Internal process exited
```

## Expected behavior

The Trainer should run a hyperparameter search with the 8 different combinations of `learning_rate` and `gradient_accumulation_steps`.


Hyperparameter search w/ Optuna CUDA out of memory

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Google Colab and Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7
- Using GPU in script?: Yes

### Who can help

- trainer: @sgugger
- Optuna: ???

## Information

Model I am using (Bert, XLNet ...): sshleifer/distilbart-cnn-12-6

## To reproduce

I am running an hyperparameter search with Optuna. I get a CUDA OOM error even if `per_device_train_batch_size` is set to 1 and the only parameters that I change are `learning_rate` and `gradient_accumulation_steps`. I have the same problem both with Google Colab and Ubuntu. Both of this environments have a 15 GB GPU.

The code I am running:

```
def hp_objective(metrics):
    loss = metrics.pop('eval_loss', None)
    _ = metrics.pop('epoch', None)
    _ = metrics.pop('eval_gen_len', None)

    return np.sum(list(metrics.values()))

def hp_space(trial):

    return {
    'learning_rate': trial.suggest_float('learning_rate', 1e-5, 1e-2, log=True),
    'gradient_accumulation_steps':\
        trial.suggest_categorical('gradient_accumulation_steps', [4, 8]),
    }

def model_init():
            model = AutoModelForSeq2SeqLM.from_pretrained(
                model_args.model_name_or_path,
                config=config,
                cache_dir=model_args.cache_dir,
            )
            # use task specific params
            use_task_specific_params(model, data_args.task)

            # set num_beams for evaluation
            if data_args.eval_beams is None:
                data_args.eval_beams = model.config.num_beams

            # set decoder_start_token_id for MBart
            if model.config.decoder_start_token_id is None and isinstance(tokenizer, MBartTokenizer):
                assert (
                    data_args.tgt_lang is not None and data_args.src_lang is not None
                ), "mBart requires --tgt_lang and --src_lang"
                model.config.decoder_start_token_id = tokenizer.lang_code_to_id[data_args.tgt_lang]

            if model_args.freeze_embeds:
                freeze_embeds(model)
            if model_args.freeze_encoder:
                freeze_params(model.get_encoder())
                assert_all_frozen(model.get_encoder())

            return model

trainer = Seq2SeqTrainer(
            model_init=model_init,
            config=config,
            args=training_args,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            data_collator=Seq2SeqDataCollator(tokenizer, data_args, training_args.tpu_num_cores),
            compute_metrics=compute_metrics_fn,
            data_args=data_args,
        )

logger.info("*** Hyperparameters Search ***")
start_time = time.time()

trainer.hyperparameter_search(
            direction = "maximize",
            compute_objective = hp_objective,
            hp_space = hp_space,
            backend = "optuna")
```
The error:

```
[INFO|modeling_utils.py:1149] 2021-02-01 15:24:41,640 >> All the weights of BartForConditionalGeneration were initialized from the model checkpoint at sshleifer/distilbart-cnn-12-6.
If your task is similar to the task the model of the checkpoint was trained on, you can already use BartForConditionalGeneration for predictions without further training.
02/01/2021 15:24:41 - INFO - utils -   using task specific params for summarization: {'early_stopping': True, 'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'no_repeat_ngram_size': 3, 'num_beams': 4}

[W 2021-02-01 15:24:42,103] Trial 8 failed because of the following error: RuntimeError('CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 14.76 GiB total capacity; 13.66 GiB already allocated; 13.75 MiB free; 13.83 GiB reserved in total by PyTorch)',)
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/optuna/_optimize.py", line 211, in _run_trial
    value_or_values = func(trial)
  File "/home/ubuntu/transformers/src/transformers/integrations.py", line 168, in _objective
    trainer.train(model_path=model_path, trial=trial)
  File "/home/ubuntu/transformers/src/transformers/trainer.py", line 622, in train
    self.model = model.to(self.args.device)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 612, in to
    return self._apply(convert)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 381, in _apply
    param_applied = fn(param)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 610, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 14.76 GiB total capacity; 13.66 GiB already allocated; 13.75 MiB free; 13.83 GiB reserved in total by PyTorch)
Traceback (most recent call last):
  File "/home/ubuntu/transformers/examples/seq2seq/finetune_trainer.py", line 435, in <module>
    main()
  File "/home/ubuntu/transformers/examples/seq2seq/finetune_trainer.py", line 350, in main
    backend = "optuna")
  File "/home/ubuntu/transformers/src/transformers/trainer.py", line 1077, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/home/ubuntu/transformers/src/transformers/integrations.py", line 178, in run_hp_search_optuna
    study.optimize(_objective, n_trials=n_trials, timeout=timeout, n_jobs=n_jobs)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/optuna/study.py", line 385, in optimize
    show_progress_bar=show_progress_bar,
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/optuna/_optimize.py", line 73, in _optimize
    progress_bar=progress_bar,
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/optuna/_optimize.py", line 164, in _optimize_sequential
    trial = _run_trial(study, func, catch)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/optuna/_optimize.py", line 262, in _run_trial
    raise func_err
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/optuna/_optimize.py", line 211, in _run_trial
    value_or_values = func(trial)
  File "/home/ubuntu/transformers/src/transformers/integrations.py", line 168, in _objective
    trainer.train(model_path=model_path, trial=trial)
  File "/home/ubuntu/transformers/src/transformers/trainer.py", line 622, in train
    self.model = model.to(self.args.device)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 612, in to
    return self._apply(convert)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 381, in _apply
    param_applied = fn(param)
  File "/home/ubuntu/miniconda3/envs/magma/lib/python3.6/site-packages/torch/nn/modules/module.py", line 610, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 14.76 GiB total capacity; 13.66 GiB already allocated; 13.75 MiB free; 13.83 GiB reserved in total by PyTorch)
```

## Expected behavior

The GPU should never go OOM, since the batch size is 1 in all trials.


[Tokenizer Utils Base] Make pad function more flexible

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Currently, tokenizers forces the dict to be padded to have an `input_ids` key. This restricts transformers tokenizers too much for models outside of NLP, such as Wav2Vec2: https://github.com/huggingface/transformers/pull/9659/files?file-filters%5B%5D=.py
As discussed offline, the cleanest approach is to add `input_ids` to the class attribute `model_input_names` and enforce a certain order. This is ensured by a test and a couple of comments that make the reader aware of it.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Missing None verification in the CLM language modeling example

Here: https://github.com/huggingface/transformers/blob/1682804ebd504d3381523116773583a52f35afd1/examples/language-modeling/run_clm.py#L230, data_args.train_file can be None (as it is checked some lines above). Therefore, there should be a check to see if it is the case or not.


Deploying a transformers pipeline into Google Cloud AI-Platform prediction

I am trying to deploy the model "distilbert-base-uncased-finetuned-sst-2-english" into Google Cloud AI-platform with a [customer prediction routine](https://cloud.google.com/ai-platform/prediction/docs/custom-prediction-routines).

The code stays pretty simple but I encounter an issue when deploying the model.

```
Create Version failed. Bad model detected with error: "Failed to load model: Unexpected error when loading the model: problem in predictor - DistributionNotFound: The 'tqdm>=4.27' distribution was not found and is required by this application, \nTry: pip install transformers -U or pip install -e '.[dev]' if you're working with git master (Error code: 0)"
```

My setup.py files is:
```python
from setuptools import setup

setup(
    name='customerPredictionCustomerReview',
    version='0.1',
    scripts=['predictor.py'],
    install_requires=["transformers==4.2.2"],
)
```

My application is just using the sentiment-analysis pipeline and one model.
```python
        model_path = os.path.join(model_dir, 'distilbert-base-uncased-finetuned-sst-2-english')
        classify = pipeline("sentiment-analysis", model=model_path, tokenizer=model_path)
```

I am wondering if you know why the others dependencies of transformers are not being installed properly. I have also tried to add `tqdm` to the setup `install_requires` but it didn't work, I got the same error. Would you have an idea here?

In addition, do you maybe suggest another way to deploy the model than the one I used?

Thank you in advance


Implementing ELECTRIC training for ELECTRA

Feature request

Request for a new feature

# 🚀 Feature request

Google released Electric this summer at EMNLP (see: [here](https://www.aclweb.org/anthology/2020.emnlp-main.20.pdf)). Electric is like ELECTRA, but trained using a Noise Contrastive Estimation loss instead of a negative sampling loss.

## Motivation

Electric is well-suited for modeling perplexity scores, and can model these very efficiently. Modeling these perplexity scores using BERT requires N passes over the input sentence, where N is the number of tokes in the sentence (see [here](https://arxiv.org/abs/1910.14659)).

## Your contribution

Electric has been implemented in the Google Electra repository. From I can see, moving from an Electra to Electric-style training is not a huge code change, but I'm not that familiar with the inner workings of transformers to be able to make a judgment call on this.


[docs] fix auto model docs

# What does this PR do?

Small doc fixes for auto model classes.


Fix bart conversion script

# What does this PR do?
Fix import and add the `make_linear_from_emb` function in the script.


Tensorflow doc changes on loss output size

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9771, by changing the documentation to correctly state the size of the output loss function.

I did not change the documentation for TFSeq2SeqQuestionAnsweringModelOutput and TFSeq2SeqSequenceClassifierOutput, as I could not find any code using this, so I was unsure what the correct output size would be.

I also fixed a few instances where I found the documentation referring to torch.LongTensor when tf.tensor should be used.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@jplu

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[Templates] Add template "call-for-model" markdown and "call-for-big-bird" markdown

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds a template to generate a "call-for-model" sheet and also adds one for [BigBird](https://github.com/google-research/bigbird)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Would you like to add convert the generator script by ConvBert

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

I excite to find that transformer add the support fort ConvBert, but I found that it just provide the script about convert the discriminator, would you like to support for convert the generator of Convbert like the Electra, I train both electra convbert and masklm convbert.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


AttributeError: module 'torch.utils' has no attribute 'checkpoint' for fine tune LED

hello, I fine tuned my own LED model by following this [notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing#scrollTo=tLM3niQqhEzP) and I saved it using

```python
 led.save_pretrained("longformer2Bart")
 tokenizer.save_pretrained("longformer2Bart")
```

however, whenever I try testing that model using something like this

```python
from transformers import LEDTokenizer, LEDForConditionalGeneration
model = LEDForConditionalGeneration.from_pretrained("longformer2Bart")
tokenizer = LEDTokenizer.from_pretrained("longformer2Bart")

article = """(CNN)James Holmes made his introduction to the world in a Colorado cinema filled with spectators watching a midnight showing of the new Batman movie, "The Dark Knight Rises," in June 2012. The moment became one of the deadliest shootings in U.S. history. Holmes is accused of opening fire on the crowd, killing 12 people and injuring or maiming 70 others in Aurora, a suburb of Denver. Holmes appeared like a comic book character: He resembled the Joker, with red-orange hair, similar to the late actor Heath Ledger\'s portrayal of the villain in an earlier Batman movie, authorities said. But Holmes was hardly a cartoon. Authorities said he wore body armor and carried several guns, including an AR-15 rifle, with lots of ammo. He also wore a gas mask. Holmes says he was insane at the time of the shootings, and that is his legal defense and court plea: not guilty by reason of insanity. Prosecutors aren\'t swayed and will seek the death penalty. Opening statements in his trial are scheduled to begin Monday. Holmes admits to the shootings but says he was suffering "a psychotic episode" at the time,  according to court papers filed in July 2013 by the state public defenders, Daniel King and Tamara A. Brady. Evidence "revealed thus far in the case supports the defense\'s position that Mr. Holmes suffers from a severe mental illness and was in the throes of a psychotic episode when he committed the acts that resulted in the tragic loss of life and injuries sustained by moviegoers on July 20, 2012," the public defenders wrote. Holmes no longer looks like a dazed Joker, as he did in his first appearance before a judge in 2012. He appeared dramatically different in January when jury selection began for his trial: 9,000 potential jurors were summoned for duty, described as one of the nation\'s largest jury calls. Holmes now has a cleaner look, with a mustache, button-down shirt and khaki pants. In January, he had a beard and eyeglasses. If this new image sounds like one of an academician, it may be because Holmes, now 27, once was one. Just before the shooting, Holmes was a doctoral student in neuroscience, and he was studying how the brain works, with his schooling funded by a U.S. government grant. Yet for all his learning, Holmes apparently lacked the capacity to command his own mind, according to the case against him. A jury will ultimately decide Holmes\' fate. That panel is made up of 12 jurors and 12 alternates. They are 19 women and five men, and almost all are white and middle-aged. The trial could last until autumn. When jury summonses were issued in January, each potential juror stood a 0.2% chance of being selected, District Attorney George Brauchler told the final jury this month. He described the approaching trial as "four to five months of a horrible roller coaster through the worst haunted house you can imagine." The jury will have to render verdicts on each of the 165 counts against Holmes, including murder and attempted murder charges. Meanwhile, victims and their relatives are challenging all media outlets "to stop the gratuitous use of the name and likeness of mass killers, thereby depriving violent individuals the media celebrity and media spotlight they so crave," the No Notoriety group says. They are joined by victims from eight other mass shootings in recent U.S. history. Raised in central coastal California and in San Diego, James Eagan Holmes is the son of a mathematician father noted for his work at the FICO firm that provides credit scores and a registered nurse mother, according to the U-T San Diego newspaper. Holmes also has a sister, Chris, a musician, who\'s five years younger, the newspaper said. His childhood classmates remember him as a clean-cut, bespectacled boy with an "exemplary" character who "never gave any trouble, and never got in trouble himself," The Salinas Californian reported. His family then moved down the California coast, where Holmes grew up in the San Diego-area neighborhood of Rancho Peñasquitos, which a neighbor described as "kind of like Mayberry," the San Diego newspaper said. Holmes attended Westview High School, which says its school district sits in "a primarily middle- to upper-middle-income residential community." There, Holmes ran cross-country, played soccer and later worked at a biotechnology internship at the Salk Institute and Miramar College, which attracts academically talented students. By then, his peers described him as standoffish and a bit of a wiseacre, the San Diego newspaper said. Holmes attended college fairly close to home, in a neighboring area known as Southern California\'s "inland empire" because it\'s more than an hour\'s drive from the coast, in a warm, low-desert climate. He entered the University of California, Riverside, in 2006 as a scholarship student. In 2008 he was a summer camp counselor for disadvantaged children, age 7 to 14, at Camp Max Straus, run by Jewish Big Brothers Big Sisters of Los Angeles. He graduated from UC Riverside in 2010 with the highest honors and a bachelor\'s degree in neuroscience. "Academically, he was at the top of the top," Chancellor Timothy P. White said. He seemed destined for even higher achievement. By 2011, he had enrolled as a doctoral student in the neuroscience program at the University of Colorado Anschutz Medical Campus in Aurora, the largest academic health center in the Rocky Mountain region. The doctoral in neuroscience program attended by Holmes focuses on how the brain works, with an emphasis on processing of information, behavior, learning and memory. Holmes was one of six pre-thesis Ph.D. students in the program who were awarded a neuroscience training grant from the National Institutes of Health. The grant rewards outstanding neuroscientists who will make major contributions to neurobiology. A syllabus that listed Holmes as a student at the medical school shows he was to have delivered a presentation about microRNA biomarkers. But Holmes struggled, and his own mental health took an ominous turn. In March 2012, he told a classmate he wanted to kill people, and that he would do so "when his life was over," court documents said. Holmes was "denied access to the school after June 12, 2012, after he made threats to a professor," according to court documents. About that time, Holmes was a patient of University of Colorado psychiatrist Lynne Fenton. Fenton was so concerned about Holmes\' behavior that she mentioned it to her colleagues, saying he could be a danger to others, CNN affiliate KMGH-TV reported, citing sources with knowledge of the investigation. Fenton\'s concerns surfaced in early June, sources told the Denver station. Holmes began to fantasize about killing "a lot of people" in early June, nearly six weeks before the shootings, the station reported, citing unidentified sources familiar with the investigation. Holmes\' psychiatrist contacted several members of a "behavioral evaluation and threat assessment" team to say Holmes could be a danger to others, the station reported. At issue was whether to order Holmes held for 72 hours to be evaluated by mental health professionals, the station reported. "Fenton made initial phone calls about engaging the BETA team" in "the first 10 days" of June, but it "never came together" because in the period Fenton was having conversations with team members, Holmes began the process of dropping out of school, a source told KMGH. Defense attorneys have rejected the prosecution\'s assertions that Holmes was barred from campus. Citing statements from the university, Holmes\' attorneys have argued that his access was revoked because that\'s normal procedure when a student drops enrollment. What caused this turn for the worse for Holmes has yet to be clearly detailed. In the months before the shooting, he bought four weapons and more than 6,000 rounds of ammunition, authorities said. Police said he also booby-trapped his third-floor apartment with explosives, but police weren\'t fooled. After Holmes was caught in the cinema parking lot immediately after the shooting, bomb technicians went to the apartment and neutralized the explosives. No one was injured at the apartment building. Nine minutes before Holmes went into the movie theater, he called a University of Colorado switchboard, public defender Brady has said in court. The number he called can be used to get in contact with faculty members during off hours, Brady said. Court documents have also revealed that investigators have obtained text messages that Holmes exchanged with someone before the shooting. That person was not named, and the content of the texts has not been made public. According to The New York Times, Holmes sent a text message to a fellow graduate student, a woman, about two weeks before the shooting. She asked if he had left Aurora yet, reported the newspaper, which didn\'t identify her. No, he had two months left on his lease, Holmes wrote back, according to the Times. He asked if she had heard of "dysphoric mania," a form of bipolar disorder marked by the highs of mania and the dark and sometimes paranoid delusions of major depression. The woman asked if the disorder could be managed with treatment. "It was," Holmes wrote her, according to the Times. But he warned she should stay away from him "because I am bad news," the newspaper reported. It was her last contact with Holmes. After the shooting, Holmes\' family issued a brief statement: "Our hearts go out to those who were involved in this tragedy and to the families and friends of those involved," they said, without giving any information about their son. Since then, prosecutors have refused to offer a plea deal to Holmes. For Holmes, "justice is death," said Brauchler, the district attorney. In December, Holmes\' parents, who will be attending the trial, issued another statement: They asked that their son\'s life be spared and that he be sent to an institution for mentally ill people for the rest of his life, if he\'s found not guilty by reason of insanity. "He is not a monster," Robert and Arlene Holmes wrote, saying the death penalty is "morally wrong, especially when the condemned is mentally ill." "He is a human being gripped by a severe mental illness," the parents said. The matter will be settled by the jury. CNN\'s Ana Cabrera and Sara Weisfeldt contributed to this report from Denver."""

input_ids = tokenizer(article, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```

I get the following error
```
AttributeError                            Traceback (most recent call last)
<ipython-input-16-6227477597c7> in <module>
      8
      9 input_ids = tokenizer(article, return_tensors="pt").input_ids
---> 10 output_ids = model.generate(input_ids)
     11
     12 # print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

~/.virtualenvs/insights2/lib/python3.6/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     24                 try:
     25                     with self:
---> 26                         x = next(gen)
     27                     yield x
     28                 except StopIteration:

~/.virtualenvs/insights2/lib/python3.6/site-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, **model_kwargs)
    831         if self.config.is_encoder_decoder:
    832             # add encoder_outputs to model_kwargs
--> 833             model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
    834
    835             # set input_ids as decoder_input_ids

~/.virtualenvs/insights2/lib/python3.6/site-packages/transformers/generation_utils.py in _prepare_encoder_decoder_kwargs_for_generation(self, input_ids, model_kwargs)
    376             argument: value for argument, value in model_kwargs.items() if not argument.startswith("decoder_")
    377         }
--> 378         model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
    379         return model_kwargs
    380

~/.virtualenvs/insights2/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725                 self._forward_hooks.values()):
    726             hook_result = hook(self, input, result)
--> 727             if hook_result is not None:
    728                 result = hook_result
    729         if (len(self._backward_hooks) > 0) or (len(_global_backward_hooks) > 0):

~/.virtualenvs/insights2/lib/python3.6/site-packages/transformers/models/led/modeling_led.py in forward(self, input_ids, attention_mask, global_attention_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
   1703                         return custom_forward
   1704
-> 1705                     layer_outputs = torch.utils.checkpoint.checkpoint(
   1706                         create_custom_forward(encoder_layer),
   1707                         hidden_states,

AttributeError: module 'torch.utils' has no attribute 'checkpoint'
```
I don't run into this error if I try loading the `patrickvonplaten/led-large-16384-pubmed` model. Not sure if I saved the model incorrectly, @patrickvonplaten or the rest of the community, I'd greatly appreciate any help with this


[doc] transformers.PreTrainedTokenizer.encode() doesn't get resolved to its doc

There are a few

> See: transformers.PreTrainedTokenizer.encode()

in the docstrings, but they don't resolve to anything in the online docs, since the `encode` method is in `PreTrainedTokenizerBase`
as it can be seen: https://huggingface.co/transformers/internal/tokenization_utils.html#transformers.tokenization_utils_base.PreTrainedTokenizerBase.encode

Should sphinx be able to resolve inheritance and still point to the right doc, or must the docs be modified to say:

> See: transformers.PreTrainedTokenizerBase.encode()

instead?

There are 117 of these.

Example:

https://huggingface.co/transformers/model_doc/t5.html#transformers.T5Model.forward

> Indices can be obtained using T5Tokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.__call__() for detail.

the `encode` method doesn't get a link.

-----------------------

Also it seems that all modules have this

> `What are input IDs? <../glossary.html#input-ids>`__

but not where `input_ids` are documented, instead after the mask - shouldn't that line be 2 records up?

![snapshot_18](https://user-images.githubusercontent.com/10676103/106406034-e5934e00-63ec-11eb-8ddd-7a56d4e70732.png)

@sgugger


distilbert: fix creation of sinusoidal embeddings

Hi,

similar issue as reported by @stas00 with BART, see #8226.

The creation of sinusoidal embeddings is currently not working on PyTorch 1.8+.

It fails with:

```bash
  File "/mnt/europeana-bert/flair/flair/embeddings/token.py", line 820, in __init__
    self.model = AutoModel.from_pretrained(model, config=config, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/modeling_auto.py", line 728, in from_pretrained
    return MODEL_MAPPING[type(config)].from_pretrained(
  File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1034, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 419, in __init__
    self.embeddings = Embeddings(config)  # Embeddings
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 88, in __init__
    create_sinusoidal_embeddings(
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 76, in create_sinusoidal_embeddings
    out[:, 0::2] = torch.FloatTensor(np.sin(position_enc[:, 0::2]))
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
```

I've seen this problem when trying to train a model in Flair with DistilBERT as feature-based embeddings, as well as when training a DistilBERT model from scratch using the official example.

It can be reproduced in a `nvcr.io/nvidia/pytorch:20.12-py3` container, that comes with PyTorch 1.8.


RAG + DPR model performance issues

Hi.
I am trying to reproduce the results obtained in the Retriever Augmented Generation paper for Question Answering on the Natural Questions (NQ) Dataset (Exact Match accuracy 44%).
However, I am not able to reproduce them.

Can someone kindly let me know which DPR model, DPR dataset and RAG dataset was used to obtain the 44% EM Accuracy on NQ dataset?

Thanks.


prediction_step() is not using compute_loss()

Hi @sgugger , I think there is an issue with `prediction_step` in `trainer.py`. The problem arises when implementing a custom loss function that requires reshaping input labels. In `training_step`, for loss calculation, it calls `compute_loss()` which is totally fine but in the `prediction_step` it calculates loss without `compute_loss()` function. This inconsistency causes some issues. Would you think it would be better to call `compute_loss()` in both cases in order to avoid this problem?

Update: As I see the code, the loss is tightly coupled with `LabelSmoother` which makes it hard to do them in a single function. If you have any suggestion, it makes me happy to contribute to Huggingface ;)

[prediction_step()](https://huggingface.co/transformers/_modules/transformers/trainer.html#Trainer.prediction_step)

[training_step()](https://huggingface.co/transformers/_modules/transformers/trainer.html#Trainer.training_step)


Hello,

I ran the follownig official example script from [longformerforquestionanswering](https://huggingface.co/transformers/model_doc/longformer.html#longformerforquestionanswering)

```
# Tokenizer
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-large-4096-finetuned-triviaqa')

# Model
model = TFLongformerForQuestionAnswering.from_pretrained('allenai/longformer-large-4096-finetuned-triviaqa')

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
encoding = tokenizer(question, text, return_tensors="pt")
input_ids = encoding["input_ids"]

# default is local attention everywhere
# the forward method will automatically set global attention on question tokens
attention_mask = encoding["attention_mask"]

outputs = model(input_ids, attention_mask=attention_mask)
start_logits = outputs.start_logits
end_logits = outputs.end_logits
all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())

answer_tokens = all_tokens[torch.argmax(start_logits) :torch.argmax(end_logits)+1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens)) # remove space prepending space token
```

But got following error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-18-4bf253125151> in <module>
      7 attention_mask = encoding["attention_mask"]
      8
----> 9 outputs = model(input_ids, attention_mask=attention_mask)
     10 start_logits = outputs.start_logits
     11 end_logits = outputs.end_logits

~\Documents\env\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in __call__(self, *args, **kwargs)
    983
    984         with ops.enable_auto_cast_variables(self._compute_dtype_object):
--> 985           outputs = call_fn(inputs, *args, **kwargs)
    986
    987         if self._activity_regularizer:

~\Documents\env\lib\site-packages\transformers\modeling_tf_longformer.py in call(self, inputs, attention_mask, global_attention_mask, token_type_ids, position_ids, inputs_embeds, output_attentions, output_hidden_states, return_dict, start_positions, end_positions, training)
   1492                 # put global attention on all tokens until `config.sep_token_id` is reached
   1493                 sep_token_indices = tf.where(input_ids == self.config.sep_token_id)
-> 1494                 global_attention_mask = _compute_global_attention_mask(shape_list(input_ids), sep_token_indices)
   1495
   1496         outputs = self.longformer(

~\Documents\env\lib\site-packages\transformers\modeling_tf_utils.py in shape_list(x)
    924         :obj:`List[int]`: The shape of the tensor as a list.
    925     """
--> 926     static = x.shape.as_list()
    927     dynamic = tf.shape(x)
    928     return [dynamic[i] if s is None else s for i, s in enumerate(static)]

AttributeError: 'torch.Size' object has no attribute 'as_list'
```


Gradient accumulation and distributed parallelism will reduce the effect?

Reference the code: "transformers/examples/legacy/question-answering/run_squad.py"
I found：
1. normal results without using distributed code, using only gradient accumulation.
2. using the distributed code with the parameter gradient_accumulation_steps=1, the effect is normal.
3. using the distributed code with the parameter gradient_accumulation_steps set to other, the experimental effect is abnormal.

What is going on, please? Thanks!


How to add more fields in TrainingArguments

I am using ```from transformers import TrainingArguments```. However, there are more training arguments in my own project. How can I add more fields (parameters) in to the ```args```? Besides, if I have some other ```Arguments Class``` that is similar to ```TrainingArguments```, how to merge them into one ```args```?


[seq2seq] fix logger format for non-main process

Currently, in `finetune_trainer.py` non-main process doesn't have any formatting at all, so we end up with:

```
[WARNING|modeling_t5.py:1645] 2021-01-30 20:01:37,246 >> [p0] got MPU
[WARNING|modeling_t5.py:1646] 2021-01-30 20:01:37,246 >> [p0] DP group [0]
[p1] got MPU
[p1] DP group [1]
```

as you can see the 2nd process in DDP misses formatting in logger.

this PR fixes it.

I looked in the take-over version `run_seq2seq.py` if it needed to be fixed too and it doesn't have these function calls, not sure why.  They appear to be needed, unless they get called elsewhere.

@sgugger, @patil-suraj


Doc title in the template

# What does this PR do?

After reviewing a few PRs post-template, I'm noticing the doc pages are always misnamed -> they should use the cased name of the model, not the uppercase version.


run_seq2seq.py : Why we pad labels with -100?

As mentioned in [this line](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py#L437), why we add -100. Can't we just keep pad_token_id ?


[seq2seq] some logging for all processes in distributed mode

In 2D Parallelism, e.g. Pipeline + DeepSpeed I need to log unique device maps per process for the user to see, but currently `logger.info()` is only activated for the main process via `if is_main_process`. Currently only in `examples/seq2seq/run_seq2seq.py`, `examples/seq2seq/finetune_trainer.py`, but it'll be needed for other scripts as well down the road.

Any idea how I could accomplish that while keeping things as they are? I guess I could use `logger.warn` as a workaround, since it's not disabled for other processes. But it's not a good approach, since it's a WARNING after all. And I don't quite want to use `print()` as it might not be what the user wants if they want things quiet.

Perhaps you have some other ideas on how I could go about doing that.

I think perhaps adding another logger that's INFO-activated for all distributed processes, and is used only occasionally when the normal logger won't do.

I think as we are getting more and more into distributed training we will need to be able to log specific things for specific processes.

Thank you.

@LysandreJik, @patrickvonplaten, @sgugger


Remove subclass for sortish sampler

# What does this PR do?

When putting the sortish sampler in the main `Trainer`, I forgot to remove the overide in `Seq2SeqTrainer` which led to an issue (see #9900). This in turn makes the old `finetune_trainer` script fail because its datasets don't have the right entries (the texts are processed during the data collation) so it requires reverting the changes in that script to use back the old `Seq2SeqTrainer`  (which is fine since that script will soon go to legacy).

Fixes #9900