fine tune patrickvonplaten/longformer2roberta-cnn_dailymail-fp16 using LED updates

I was wondering if there was anyway to fine tune the
`patrickvonplaten/longformer2roberta-cnn_dailymail-fp16` model instead of `patrickvonplaten/led-large-16384-pubmed`? When I tried fine tuning it in the past I ran into the
`TypeError: forward() got an unexpected keyword argument 'head_mask'` issue given that `EncoderDecoderModel` wasn't intended for longformer. So I'm now trying to see if I can use `LEDForConditionalGeneration` for it but I noticed when I try doing:

```python
from transformers import LEDTokenizer, LEDForConditionalGeneration

article = """(CNN)James Holmes made his introduction to the world in a Colorado cinema filled with spectators watching a midnight showing of the new Batman movie, "The Dark Knight Rises," in June 2012. The moment became one of the deadliest shootings in U.S. history. Holmes is accused of opening fire on the crowd, killing 12 people and injuring or maiming 70 others in Aurora, a suburb of Denver. Holmes appeared like a comic book character: He resembled the Joker, with red-orange hair, similar to the late actor Heath Ledger\'s portrayal of the villain in an earlier Batman movie, authorities said. But Holmes was hardly a cartoon. Authorities said he wore body armor and carried several guns, including an AR-15 rifle, with lots of ammo. He also wore a gas mask. Holmes says he was insane at the time of the shootings, and that is his legal defense and court plea: not guilty by reason of insanity. Prosecutors aren\'t swayed and will seek the death penalty. Opening statements in his trial are scheduled to begin Monday. Holmes admits to the shootings but says he was suffering "a psychotic episode" at the time,  according to court papers filed in July 2013 by the state public defenders, Daniel King and Tamara A. Brady. Evidence "revealed thus far in the case supports the defense\'s position that Mr. Holmes suffers from a severe mental illness and was in the throes of a psychotic episode when he committed the acts that resulted in the tragic loss of life and injuries sustained by moviegoers on July 20, 2012," the public defenders wrote. Holmes no longer looks like a dazed Joker, as he did in his first appearance before a judge in 2012. He appeared dramatically different in January when jury selection began for his trial: 9,000 potential jurors were summoned for duty, described as one of the nation\'s largest jury calls. Holmes now has a cleaner look, with a mustache, button-down shirt and khaki pants. In January, he had a beard and eyeglasses. If this new image sounds like one of an academician, it may be because Holmes, now 27, once was one. Just before the shooting, Holmes was a doctoral student in neuroscience, and he was studying how the brain works, with his schooling funded by a U.S. government grant. Yet for all his learning, Holmes apparently lacked the capacity to command his own mind, according to the case against him. A jury will ultimately decide Holmes\' fate. That panel is made up of 12 jurors and 12 alternates. They are 19 women and five men, and almost all are white and middle-aged. The trial could last until autumn. When jury summonses were issued in January, each potential juror stood a 0.2% chance of being selected, District Attorney George Brauchler told the final jury this month. He described the approaching trial as "four to five months of a horrible roller coaster through the worst haunted house you can imagine." The jury will have to render verdicts on each of the 165 counts against Holmes, including murder and attempted murder charges. Meanwhile, victims and their relatives are challenging all media outlets "to stop the gratuitous use of the name and likeness of mass killers, thereby depriving violent individuals the media celebrity and media spotlight they so crave," the No Notoriety group says. They are joined by victims from eight other mass shootings in recent U.S. history. Raised in central coastal California and in San Diego, James Eagan Holmes is the son of a mathematician father noted for his work at the FICO firm that provides credit scores and a registered nurse mother, according to the U-T San Diego newspaper. Holmes also has a sister, Chris, a musician, who\'s five years younger, the newspaper said. His childhood classmates remember him as a clean-cut, bespectacled boy with an "exemplary" character who "never gave any trouble, and never got in trouble himself," The Salinas Californian reported. His family then moved down the California coast, where Holmes grew up in the San Diego-area neighborhood of Rancho Peñasquitos, which a neighbor described as "kind of like Mayberry," the San Diego newspaper said. Holmes attended Westview High School, which says its school district sits in "a primarily middle- to upper-middle-income residential community." There, Holmes ran cross-country, played soccer and later worked at a biotechnology internship at the Salk Institute and Miramar College, which attracts academically talented students. By then, his peers described him as standoffish and a bit of a wiseacre, the San Diego newspaper said. Holmes attended college fairly close to home, in a neighboring area known as Southern California\'s "inland empire" because it\'s more than an hour\'s drive from the coast, in a warm, low-desert climate. He entered the University of California, Riverside, in 2006 as a scholarship student. In 2008 he was a summer camp counselor for disadvantaged children, age 7 to 14, at Camp Max Straus, run by Jewish Big Brothers Big Sisters of Los Angeles. He graduated from UC Riverside in 2010 with the highest honors and a bachelor\'s degree in neuroscience. "Academically, he was at the top of the top," Chancellor Timothy P. White said. He seemed destined for even higher achievement. By 2011, he had enrolled as a doctoral student in the neuroscience program at the University of Colorado Anschutz Medical Campus in Aurora, the largest academic health center in the Rocky Mountain region. The doctoral in neuroscience program attended by Holmes focuses on how the brain works, with an emphasis on processing of information, behavior, learning and memory. Holmes was one of six pre-thesis Ph.D. students in the program who were awarded a neuroscience training grant from the National Institutes of Health. The grant rewards outstanding neuroscientists who will make major contributions to neurobiology. A syllabus that listed Holmes as a student at the medical school shows he was to have delivered a presentation about microRNA biomarkers. But Holmes struggled, and his own mental health took an ominous turn. In March 2012, he told a classmate he wanted to kill people, and that he would do so "when his life was over," court documents said. Holmes was "denied access to the school after June 12, 2012, after he made threats to a professor," according to court documents. About that time, Holmes was a patient of University of Colorado psychiatrist Lynne Fenton. Fenton was so concerned about Holmes\' behavior that she mentioned it to her colleagues, saying he could be a danger to others, CNN affiliate KMGH-TV reported, citing sources with knowledge of the investigation. Fenton\'s concerns surfaced in early June, sources told the Denver station. Holmes began to fantasize about killing "a lot of people" in early June, nearly six weeks before the shootings, the station reported, citing unidentified sources familiar with the investigation. Holmes\' psychiatrist contacted several members of a "behavioral evaluation and threat assessment" team to say Holmes could be a danger to others, the station reported. At issue was whether to order Holmes held for 72 hours to be evaluated by mental health professionals, the station reported. "Fenton made initial phone calls about engaging the BETA team" in "the first 10 days" of June, but it "never came together" because in the period Fenton was having conversations with team members, Holmes began the process of dropping out of school, a source told KMGH. Defense attorneys have rejected the prosecution\'s assertions that Holmes was barred from campus. Citing statements from the university, Holmes\' attorneys have argued that his access was revoked because that\'s normal procedure when a student drops enrollment. What caused this turn for the worse for Holmes has yet to be clearly detailed. In the months before the shooting, he bought four weapons and more than 6,000 rounds of ammunition, authorities said. Police said he also booby-trapped his third-floor apartment with explosives, but police weren\'t fooled. After Holmes was caught in the cinema parking lot immediately after the shooting, bomb technicians went to the apartment and neutralized the explosives. No one was injured at the apartment building. Nine minutes before Holmes went into the movie theater, he called a University of Colorado switchboard, public defender Brady has said in court. The number he called can be used to get in contact with faculty members during off hours, Brady said. Court documents have also revealed that investigators have obtained text messages that Holmes exchanged with someone before the shooting. That person was not named, and the content of the texts has not been made public. According to The New York Times, Holmes sent a text message to a fellow graduate student, a woman, about two weeks before the shooting. She asked if he had left Aurora yet, reported the newspaper, which didn\'t identify her. No, he had two months left on his lease, Holmes wrote back, according to the Times. He asked if she had heard of "dysphoric mania," a form of bipolar disorder marked by the highs of mania and the dark and sometimes paranoid delusions of major depression. The woman asked if the disorder could be managed with treatment. "It was," Holmes wrote her, according to the Times. But he warned she should stay away from him "because I am bad news," the newspaper reported. It was her last contact with Holmes. After the shooting, Holmes\' family issued a brief statement: "Our hearts go out to those who were involved in this tragedy and to the families and friends of those involved," they said, without giving any information about their son. Since then, prosecutors have refused to offer a plea deal to Holmes. For Holmes, "justice is death," said Brauchler, the district attorney. In December, Holmes\' parents, who will be attending the trial, issued another statement: They asked that their son\'s life be spared and that he be sent to an institution for mentally ill people for the rest of his life, if he\'s found not guilty by reason of insanity. "He is not a monster," Robert and Arlene Holmes wrote, saying the death penalty is "morally wrong, especially when the condemned is mentally ill." "He is a human being gripped by a severe mental illness," the parents said. The matter will be settled by the jury. CNN\'s Ana Cabrera and Sara Weisfeldt contributed to this report from Denver."""

tokenizer = LEDTokenizer.from_pretrained("allenai/longformer-base-4096")
model = LEDForConditionalGeneration.from_pretrained("patrickvonplaten/longformer2roberta-cnn_dailymail-fp16")#.to("cuda").half()

input_ids = tokenizer(article, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```

I get strange results for that pretrained model
```
considerations considerations considerations lag lag lag Sith Sith Sith miracle miracle miracle Sith Sith Metropolitan Metropolitan Metropolitan Sith SithHERHERHER miracle miracle Hurt Hurt Hurt miracle miracle Joey Joey Joey Sith Sith ticking ticking ticking memorial memorial memorial tee tee tee miracle miracle Holder Holder Holder miracle miracle raspberry raspberry raspberry Sith Sithamoamoamo Sith Sith dominate dominate dominate miracle miracleDashDashDash miracle miracle scored scored scored dominate dominate Sith Sith (* (* (* dominate dominate Joey Joey miracle miracle hide hide hide miracle miracle characteristics characteristics characteristics miracle miracletighttighttight raspberry raspberry hal hal halomeveromeveromever miracle miracle ticking ticking dominate dominate Metropolitan Metropolitan dominate dominate Dek dominate dominate AWS AWS AWS sentencing sentencing sentencingCasCasCas customer customer customer Joey Joey dominate dominatetighttight miracle miracle AWS
```

 if I try using `LEDForConditionalGeneration` instead of `EncoderDecoderModel` for model `patrickvonplaten/longformer2roberta-cnn_dailymail-fp16`. Is there something I'm missing? I'd greatly appreciate any feedback/help with this


Wrong offsets mapping in XLMRobertaTokenizerFast

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.2
- Platform: Linux-4.15.0-124-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help

@mfuntowicz @thomwolf

## Information

Model I am using (Bert, XLNet ...): XLMRobertaTokenizerFast

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import XLMRobertaTokenizerFast
tokenizer = XLMRobertaTokenizerFast.from_pretrained('xlm-roberta-large')
text = "？……"
tokenized = tokenizer(text, return_offsets_mapping=True)
print('Text:', text)
print('Tokens:', tokenizer.convert_ids_to_tokens(tokenized.input_ids))
print('Mapped to:', [text[start:end] for start, end in tokenized.offset_mapping])
```

Observed behavior:

```
Text: ？……
Tokens: ['<s>', '▁?', '......', '</s>']
Mapped to: ['', '？', '？', '']
```

Expected behavior:

```
Text: ？……
Tokens: ['<s>', '▁?', '......', '</s>']
Mapped to: ['', '？', '……', '']
```

## Expected behavior

I'm using XLM-R for Chinese text, and I would expect offset mappings to work correctly even in the presence of various Unicode punctuation symbols. It looks like XLM-R tokenizes "？……" as two tokens  ('▁?' and '......'), which I would expect to map back to the appropriate locations in the input. Instead, the offset mapping from these tokens is identical.

This example is an ending of an actual sentence in the Chinese Treebank -- I removed the sentence itself because it doesn't matter for reproducing the bug.


DistilGPT2 extremely strange model behaviour

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-4.15.0-91-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0a0+1606899 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, 2
- Using distributed or parallel set-up in script?: Yes

### Who can help

 albert, bert, GPT2, XLM: @LysandreJik
 examples/distillation: @VictorSanh

## Information

Model I am using (Bert, XLNet ...): DistilGPT2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ X ] my own modified scripts: (give details below)

Had to slightly modify the official scripts to even get it to run - other than that, effectively no difference

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X ] my own task or dataset: (give details below)

Chat logs. In the format

```
[22-Jul-20 04:15 AM] User
Message

[22-Jul-20 04:15 AM] Another user
Message

[22-Jul-20 04:16 AM] Etc.
You get the point
```

## To reproduce

Steps to reproduce the behavior:

1. (Fix and) run example scripts for distillation with a custom GPT2 model.

When trying to generate, the model will generate the same text with no change, regardless of using ``pipeline`` or ``model.generate``. If attempting to use input texts, the network will output the texts immediately appended by the phrase. In the two times I've run this experiment, it was the space character twice and ``[Nov-Nov-20] User`` once (after I tried removing lines that were under a certain token value)

## Expected behavior

The model to work as a normal transformer.

<!-- A clear and concise description of what you would expect to happen. -->


--fp16 fine-tuning appears to be taking more memory (4.3.0).

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-5.4.0-62-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: 4x A100-SXM4-40GB
- Using distributed or parallel set-up in script?: Yes

### Who can help
@alexorona @stas00 @sgugger

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [ ] my own modified scripts: (give details below)

Official trainer, optionally modified by adding "model.parallelize() " after loading.  (Results shown with and without).

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Regular seq2seq on data.

Run script:
```
export BS=1; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1,2,3 ./finetune_trainer.py --model_name_or_path t5-large --output_dir output_dir --adam_eps 1e-06 --data_dir xsum --fp16 \
--do_train --learning_rate 3e-5 \
--logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 \
--overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS \
--warmup_steps 5 \

```

## Brief summary

1. When fine tuning T5, I'm observing memory usage to increase when using --fp16, though not as much as previously reported in #8403 .
2. (Optional) Possibly related: I'm trying to squeeze T5-11B in 4x40GB A100s using model parallelism.  I seemed to be able to do it yesterday on 4.1.1 with a sequence length of 128, and I remember observing a fairly moderate seqlength vs memory usage dependence (as expected from the comment here ( https://github.com/huggingface/transformers/issues/8771#issuecomment-764058315 ), though I'm not an expert and I'm not sure if this increase only applies >512 tokens, and if what I saw yesterday was a fluke/error on my part somewhere).  Today on a fresh env/pull I'm not observing this dependence (though I'm not sure why -- it might be my issue -- data is reported at the bottom side).

## To reproduce

Steps to reproduce the behavior:

1. 4.3.0, runscript as above, run with and without --fp16 option.  Different model sizes (and with/without model.parallelize() added, since I wasn't sure if that was the issue)

## Data
Below are three cases of memory usage with/without --fp16:
1. with model.parallelize()
2. without model.parallelize()   (but GPUs still visible -- extra info, I thought it was interesting it still takes up extra memory on the other GPUs)
3. without model.parallelize()   (only 1 GPU visible)
-
```
*** WITH MODEL.PARALLELIZE() ***

t5-3b, --max_source_length 128 --max_target_length 128
WITHOUT --fp16
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   28C    P0    77W / 400W |  13598MiB / 40537MiB |     33%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   29C    P0    80W / 400W |  12874MiB / 40537MiB |     25%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   24C    P0    81W / 400W |  12874MiB / 40537MiB |      4%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   25C    P0    80W / 400W |  12874MiB / 40537MiB |     23%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

WITH --fp16: (takes more memory)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   27C    P0   108W / 400W |  15138MiB / 40537MiB |      6%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   28C    P0    99W / 400W |  14214MiB / 40537MiB |      9%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   23C    P0    85W / 400W |  14214MiB / 40537MiB |     12%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   25C    P0    92W / 400W |  14216MiB / 40537MiB |     11%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

*** WITHOUT MODEL.PARALLELIZE, but all GPUs still visible ( CUDA_VISIBLE_DEVICES=0,1,2,3 ) ***
t5-large, --max_source_length 128 --max_target_length 128
WITHOUT --fp16
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   28C    P0    93W / 400W |  20362MiB / 40537MiB |      1%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   27C    P0    78W / 400W |   6046MiB / 40537MiB |      3%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   22C    P0    78W / 400W |   6046MiB / 40537MiB |      3%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   24C    P0    79W / 400W |   6022MiB / 40537MiB |      7%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

t5-large, --max_source_length 128 --max_target_length 128
WITH --fp16
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   28C    P0    91W / 400W |  20318MiB / 40537MiB |      2%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   27C    P0    80W / 400W |   7304MiB / 40537MiB |      4%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   23C    P0    78W / 400W |   7304MiB / 40537MiB |      5%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   24C    P0    79W / 400W |   7280MiB / 40537MiB |      5%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

*** WITHOUT MODEL.PARALLELIZE, ONLY 1 GPU VISIBLE ( CUDA_VISIBLE_DEVICES=0 ) ***

t5-large, --max_source_length 128 --max_target_length 128
WITHOUT --fp16
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   29C    P0   101W / 400W |  13790MiB / 40537MiB |     32%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   26C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   21C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   23C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

t5-large, --max_source_length 128 --max_target_length 128
WITH --fp16 (more memory)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:05:00.0 Off |                    0 |
| N/A   28C    P0   101W / 400W |  15012MiB / 40537MiB |     42%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:06:00.0 Off |                    0 |
| N/A   26C    P0    70W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   21C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:08:00.0 Off |                    0 |
| N/A   23C    P0    71W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

```

WRT sequence length vs data dependence, in a model parallel setting, I am observing today:
(varying --max_source_length and --max_target_length )

model, seq length, gpu0, gpu1, gpu2, gpu3

t5-large, 32 seq length, 5.7GB / 4.7GB / 4.7GB / 4.7GB
t5-large, 64 seq length, 5.7GB / 4.7GB / 4.7GB / 4.7GB
t5-large, 128 seq length, 5.8GB / 4.8GB / 4.8GB / 4.8GB
t5-large, 512 seq length, 6.0GB / 5.2GB / 5.2GB / 5.2GB

t5-3b, 64 seq length, 15.2GB / 14.3GB / 14.3GB / 14.3GB
t5-3b, 128 seq length, 15.2GB / 14.3GB / 14.3GB / 14.3GB
t5-3b, 256 seq length, 15.5GB / 14.7GB / 14.7GB / 14.7GB
t5-3b, 512 seq length, 16.2GB / 15.2GB / 15.2GB / 15.2GB

Essentially very minimal change in RAM requirements vs sequence length.  Though perhaps I have misconfigured something here.

## Expected behavior

1. Less memory usage with --fp16  (should it be about half? suggested from https://github.com/huggingface/transformers/issues/8403#issuecomment-725562117 )
2. (Optional) Nominally, smaller sequence length models taking up significantly less memory?


examples: fix XNLI url

Hi,

this PR fixes the URL for the XNLI dataset in the text classification example.

(/cc @sleepinyourhat :hugs: )


RAG Model without DPR

Hello everyone,
I am interesting in studying how RAG generator answers questions without the DPR retriever but with other passages / contexts identified by other methods.

For example in the code below

```from transformers import RagRetriever
from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

retriever = RagRetriever.from_pretrained(’./rag-token-nq’, indexed_dataset=dataset)
tokenizer = RagTokenizer.from_pretrained("./rag-token-nq")
model = RagTokenForGeneration.from_pretrained("./rag-token-nq", retriever=retriever)

input_dict = tokenizer.prepare_seq2seq_batch(“How many people live in Paris?”, “In Paris, there are 10 million people.”, return_tensors=“pt”)
input_ids = input_dict[“input_ids”]

model = RagTokenForGeneration.from_pretrained(“facebook/rag-token-nq”, retriever=retriever)

generated_ids = model.generate(input_ids=input_ids, labels=input_dict[“labels”])

generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_string)
```

In the line
```input_dict = tokenizer.prepare_seq2seq_batch(“How many people live in Paris?”, “In Paris, there are 10 million people.”, return_tensors=“pt”) ```
, I want to use “How many people live in Paris ?” as the question and “In Paris, there are 10 million people.” the passage/context which should be used to generate the answer.

Kindly let me know how to do this?

Is my understanding of the code correct and if not, how to go about it?

Thanks,
Krishanu


Error using TFAutoModelForSequenceClassification with Tensorflow 2.2.0

Hello.

I am trying to implement `TFAutoModelForSequenceClassification` in my code following the example for sequence classification as shown [here](https://huggingface.co/transformers/task_summary.html#sequence-classification)

The code is as follows:

```
from transformers import AutoTokenizer, TFAutoModelForSequenceClassification
import tensorflow as tf
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased-finetuned-mrpc")
model = TFAutoModelForSequenceClassification.from_pretrained("bert-base-cased-finetuned-mrpc")
classes = ["not paraphrase", "is paraphrase"]
sequence_0 = "The company HuggingFace is based in New York City"
sequence_1 = "Apples are especially bad for your health"
sequence_2 = "HuggingFace's headquarters are situated in Manhattan"
paraphrase = tokenizer(sequence_0, sequence_2, return_tensors="tf")
not_paraphrase = tokenizer(sequence_0, sequence_1, return_tensors="tf")
paraphrase_classification_logits = model(paraphrase)[0]
not_paraphrase_classification_logits = model(not_paraphrase)[0]
paraphrase_results = tf.nn.softmax(paraphrase_classification_logits, axis=1).numpy()[0]
not_paraphrase_results = tf.nn.softmax(not_paraphrase_classification_logits, axis=1).numpy()[0]
# Should be paraphrase
for i in range(len(classes)):
    print(f"{classes[i]}: {int(round(paraphrase_results[i] * 100))}%")
# Should not be paraphrase
for i in range(len(classes)):
    print(f"{classes[i]}: {int(round(not_paraphrase_results[i] * 100))}%")

```

Here is the error readout:
```

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-20-c48d70c01597> in <module>
      2 import tensorflow as tf
      3 tokenizer = AutoTokenizer.from_pretrained("bert-base-cased-finetuned-mrpc")
----> 4 model = TFAutoModelForSequenceClassification.from_pretrained("bert-base-cased-finetuned-mrpc")
      5 classes = ["not paraphrase", "is paraphrase"]
      6 sequence_0 = "The company HuggingFace is based in New York City"

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/models/auto/modeling_tf_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1189
   1190         if type(config) in TF_MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING.keys():
-> 1191             return TF_MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING[type(config)].from_pretrained(
   1192                 pretrained_model_name_or_path, *model_args, config=config, **kwargs
   1193             )

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1216
   1217         # Instantiate model.
-> 1218         model = cls(config, *model_args, **model_kwargs)
   1219
   1220         if from_pt:

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/models/bert/modeling_tf_bert.py in __init__(self, config, *inputs, **kwargs)
   1369
   1370         self.num_labels = config.num_labels
-> 1371         self.bert = TFBertMainLayer(config, name="bert")
   1372         self.dropout = tf.keras.layers.Dropout(config.hidden_dropout_prob)
   1373         self.classifier = tf.keras.layers.Dense(

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/modeling_tf_utils.py in wrapped_init(self, *args, **kwargs)
    105         elif isinstance(config, PretrainedConfig):
    106             if len(args) > 0:
--> 107                 initializer(self, *args, **kwargs)
    108             else:
    109                 initializer(self, config, *args, **kwargs)

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/models/bert/modeling_tf_bert.py in __init__(self, config, add_pooling_layer, **kwargs)
    590         self.return_dict = config.use_return_dict
    591         self.embeddings = TFBertEmbeddings(config, name="embeddings")
--> 592         self.encoder = TFBertEncoder(config, name="encoder")
    593         self.pooler = TFBertPooler(config, name="pooler") if add_pooling_layer else None
    594

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/models/bert/modeling_tf_bert.py in __init__(self, config, **kwargs)
    430         super().__init__(**kwargs)
    431
--> 432         self.layer = [TFBertLayer(config, name="layer_._{}".format(i)) for i in range(config.num_hidden_layers)]
    433
    434     def call(

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/models/bert/modeling_tf_bert.py in <listcomp>(.0)
    430         super().__init__(**kwargs)
    431
--> 432         self.layer = [TFBertLayer(config, name="layer_._{}".format(i)) for i in range(config.num_hidden_layers)]
    433
    434     def call(

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/models/bert/modeling_tf_bert.py in __init__(self, config, **kwargs)
    410         super().__init__(**kwargs)
    411
--> 412         self.attention = TFBertAttention(config, name="attention")
    413         self.intermediate = TFBertIntermediate(config, name="intermediate")
    414         self.bert_output = TFBertOutput(config, name="output")

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/models/bert/modeling_tf_bert.py in __init__(self, config, **kwargs)
    344         super().__init__(**kwargs)
    345
--> 346         self.self_attention = TFBertSelfAttention(config, name="self")
    347         self.dense_output = TFBertSelfOutput(config, name="output")
    348

~/.conda/envs/myenv/lib/python3.8/site-packages/transformers-4.2.2-py3.8.egg/transformers/models/bert/modeling_tf_bert.py in __init__(self, config, **kwargs)
    254         self.num_attention_heads = config.num_attention_heads
    255         self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
--> 256         self.query = tf.keras.layers.experimental.EinsumDense(
    257             equation="abc,cde->abde",
    258             output_shape=(None, config.num_attention_heads, self.attention_head_size),

AttributeError: module 'tensorflow.keras.layers.experimental' has no attribute 'EinsumDense'

```

I cannot seem to find a lot of information on solving this on Google. Any ideas? I am using a dockerized version of tensorflow 2.2.0 with Jupyter. This is a fresh install of transformers.


[fsmt] onnx triu workaround

This PR

* solves
   ```
   RuntimeError: Exporting the operator triu to ONNX opset version 12 is not supported. Please open a bug to request ONNX    export support for the missing operator.
   ```
   as reported in https://github.com/huggingface/transformers/issues/9737.
   It adds a workaround for `triu` not being supported by pytorch's onnx set, as proposed here: https://github.com/pytorch/pytorch/issues/32968#issuecomment-733240232 with some modifications to make it work with transformers. The original workaround couldn't handle a matrix with `-inf`.
* adds an onnx export test

The workaround fix is localized to fsmt, but transformers has a handful of those:

```
src/transformers/pipelines/question_answering.py:        candidates = np.tril(np.triu(outer), max_answer_len - 1)
src/transformers/models/fsmt/modeling_fsmt.py:    causal_mask = torch.triu(fill_with_neg_inf(torch.zeros(tgt_len, tgt_len)), 1).to(
src/transformers/models/transfo_xl/modeling_transfo_xl.py:            dec_attn_mask = (torch.triu(all_ones, 1 + mlen) + torch.tril(all_ones, -mask_shift_len))[:, :, None]  # -1
src/transformers/models/transfo_xl/modeling_transfo_xl.py:            dec_attn_mask = torch.triu(word_emb.new_ones((qlen, klen), dtype=torch.uint8), diagonal=1 + mlen)[
src/transformers/models/xlnet/modeling_xlnet.py:        mask_up = torch.triu(attn_mask, diagonal=1)
src/transformers/models/ctrl/modeling_ctrl.py:        mask = torch.triu(torch.ones(seq_len + past_length, seq_len + past_length), 1).to(inputs_embeds.device)
src/transformers/models/prophetnet/modeling_prophetnet.py:        left_block[stream_idx].triu_(-stream_idx + 1)
src/transformers/models/prophetnet/modeling_prophetnet.py:        causal_mask = torch.triu(causal_mask, 1)
```

Perhaps the workaround wrapper should be somewhere in the common tools and used in other places?

Or merge this to let the user who needed this in first place move forward and then refactor to other places which haven't been spoken for. But actually if you look at https://github.com/pytorch/pytorch/issues/32968 many of the me-too comments talk about `transformers`.

https://github.com/pytorch/pytorch/issues/32968 proposes other solutions too. I tried them and either they didn't work, or were inefficient as spotted by @patrickvonplaten

Fixes: https://github.com/huggingface/transformers/issues/9737

@LysandreJik, @mfuntowicz


[fsmt] Exporting the operator triu to ONNX opset version 12 is not supported

As a follow up to https://github.com/huggingface/transformers/issues/9722

```
import torch
import transformers
from transformers import convert_graph_to_onnx
from pathlib import Path

convert_graph_to_onnx.convert(
    framework="pt",
    model="facebook/wmt19-en-de",
    output=Path("encoder/en_de_trans.onnx"),
    opset=12,
    tokenizer="facebook/wmt19-en-de",
    use_external_format= False,
    pipeline_name= "translation_en_to_de",
)
```
after applying the fix from https://github.com/huggingface/transformers/pull/9736
it then crashes with:

```
Traceback (most recent call last):
  File "/mnt/nvme1/code/huggingface/transformers-master/porting/onnx2.py", line 26, in <module>
    convert_graph_to_onnx.convert(
  File "/hf/transformers-master/src/transformers/convert_graph_to_onnx.py", line 367, in convert
    convert_pytorch(nlp, opset, output, use_external_format)
  File "/hf/transformers-master/src/transformers/convert_graph_to_onnx.py", line 279, in convert_pytorch
    export(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/__init__.py", line 271, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/utils.py", line 86, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/utils.py", line 671, in _export
    _model_to_graph(model, args, verbose, input_names,
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/utils.py", line 450, in _model_to_graph
    graph = _optimize_graph(graph, operator_export_type,
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/utils.py", line 204, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/__init__.py", line 309, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/utils.py", line 970, in _run_symbolic_function
    symbolic_fn = _find_symbolic_in_registry(domain, op_name, opset_version, operator_export_type)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/utils.py", line 927, in _find_symbolic_in_registry
    return sym_registry.get_registered_op(op_name, domain, opset_version)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/onnx/symbolic_registry.py", line 112, in get_registered_op
    raise RuntimeError(msg)
RuntimeError: Exporting the operator triu to ONNX opset version 12 is not supported. Please open a bug to request ONNX export support for the missing operator.

Process finished with exit code 1
```

Need to look at workarounds proposed at https://github.com/pytorch/pytorch/issues/32968


[fsmt] token_type_ids isn't used

This PR fixes a bug discovered in https://github.com/huggingface/transformers/issues/9722

`token_type_ids` was returned by default by the tokenizer, but it isn't used by the model.

The original fsmt port was a frankenstein of bart for the model and xlm for the tokenizer, hence the discrepancy.

I thought the tests had common onnx tests, but it doesn't seem to be the case. I added a local test in the related PR:  https://github.com/huggingface/transformers/pull/9738

With this fix `convert_graph_to_onnx.convert` still fails with:
```
RuntimeError: Exporting the operator triu to ONNX opset version 12 is not supported. Please open a bug to request ONNX export support for the missing operator
```
but that's a totally different issue. Fixed in https://github.com/huggingface/transformers/pull/9738

Fixes: https://github.com/huggingface/transformers/issues/9722

@LysandreJik, @patrickvonplaten


Add `report_to` training arguments to control the integrations used

# What does this PR do?

This PR introduces a new `report_to` training argument that controls which of the multiple reporting tools to use in a training round. Currently, `Trainer` automatically uses everything installed, which can cause trouble when:
- one platform is installed but not properly set up.
- one platform is installed but the user doesn't want to use it today.

In my opinion the current behavior is too magical and does not fit our philosophy. To avoid any breaking change, the current default for this `report_to` argument is to use everything installed, but I would like to switch this to an empty list at the next major release, so the user has to opt-in the platforms they want to use.


Fixes to run_seq2seq and instructions

# What does this PR do?

This PR fixes two issues in the new `run_seq2seq` script and adds instructions on how to run it. The fixes are:
- default of `val_max_target_length` to `max_target_length` (had forgotten to do this in the initial PR)
- add an optional prefix to the source text (for T5 models)


[WIP] Small improvement in shape manipulation in t5, makes exporting

# What does this PR do?

(torchscript + ONNX) easier because shape inference is not static.

- Still requires a test that would make sure there's not regression
there.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


OSError: [Errno 116] Stale file handle

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.6
- Tensorflow version (GPU?): -
- Using GPU in script?: -
- Using distributed or parallel set-up in script?: - only 1 GPU

### Who can help
 Trainer: @sgugger
 examples/seq2seq: @patil-suraj

## Information
Hi, I am training finetune_trainer.py on on wmt dataset. I am getting the following error sometimes, do you have an idea what might cause it? thanks for any suggestion

```
  File "finetune_trainer.py", line 342, in <module>
    main()
  File "finetune_trainer.py", line 256, in main
    if (os.path.isdir(training_args.output_dir) and not training_args.optimize_from_scratch) else None,
  File "/home/jim/trainer.py", line 814, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/home/jim/trainer.py", line 885, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/home/jim/trainer.py", line 916, in _save_checkpoint
    torch.save(self.optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
  File "/home/jim/libs/anaconda3/envs/success/lib/python3.7/site-packages/torch/serialization.py", line 374, in save
    _legacy_save(obj, opened_file, pickle_module, pickle_protocol)
  File "/home/jim/libs/anaconda3/envs/success/lib/python3.7/site-packages/torch/serialization.py", line 214, in __exit__
    self.file_like.close()
OSError: [Errno 116] Stale file handle
```
Model I am using T5.

## To reproduce
this is not happening all the time, but it does happen

## Expected behavior

to run the codes


Mismatch of the mask token id of BART between fairseq and huggingface

## 🐛 Bug

The mask token id of BART is different between fairseq (torch.hub) and huggingface, and this discrepancy leads to different results in mask_filling. So I wonder which token id is actually correct.

(After checking the norm of the embedding at each mask token id, I feel that torch.hub might be correct. I have posted the same issue at fairseq github and been waiting for the reply.)

### To Reproduce
#### Code sample
```
from transformers import BartForConditionalGeneration, BartTokenizer
model = BartForConditionalGeneration.from_pretrained("facebook/bart-base", force_bos_token_to_be_generated=True)
tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
assert tokenizer.mask_token_id == 50264
example_english_phrase = "<mask> cat is <mask>."
batch = tokenizer(example_english_phrase, return_tensors='pt')
generated_ids = model.generate(batch['input_ids'],return_dict_in_generate = True, num_beams=10, num_return_sequences=1, output_scores = True)
print(" ".join(tokenizer.convert_ids_to_tokens(generated_ids[0][0])))

# </s> <s> This Ġcat Ġis Ġadorable . </s>**

import torch
bart = torch.hub.load('pytorch/fairseq', 'bart.base')
bart.eval()
assert bart.task.source_dictionary.indices["<mask>"]  == 51200
assert bart.task.source_dictionary.indices["madeupword0003"] == tokenizer.mask_token_id
# Somehow the huggingface model has a smaller vocab size, and 51200 is out of index
assert len(model.model.encoder.embed_tokens.weight) == 50265
assert len(bart.model.encoder.embed_tokens.weight) == 51201
# But the embedding at tokenizer.mask_token_id is the same between the two models
assert all(bart.model.encoder.embed_tokens.weight[tokenizer.mask_token_id] == model.model.encoder.embed_tokens.weight[tokenizer.mask_token_id])

def fill_mask(
        model,
        masked_inputs,
        topk = 1,
        match_source_len = False,
        masked_token = '<mask>',
        **generate_kwargs
):
    batch_tokens = []
    for masked_input in masked_inputs:
        assert masked_token in masked_input, \
            "please add one {} token for the input".format(masked_token)
        text_spans = masked_input.split(masked_token)
        text_spans_bpe = (' {0} '.format(masked_token)).join(
            [model.bpe.encode(text_span.rstrip()) for text_span in text_spans]
        ).strip()
        tokens = model.task.source_dictionary.encode_line(
            '<s> ' + text_spans_bpe + ' </s>',
            append_eos=False,
            add_if_not_exist=False,
        ).long()
        batch_tokens.append(tokens)
    generate_kwargs['beam'] = max(
        topk,
        generate_kwargs.get('beam', -1),
    )
    generate_kwargs['match_source_len'] = match_source_len
    batch_hypos = model.generate(batch_tokens, **generate_kwargs)
    return batch_hypos

masked_inputs=[example_english_phrase]
generate_kwargs = {}
generate_kwargs['beam'] = 10
generate_kwargs['match_source_len'] = False
batch_hypos = fill_mask(bart,masked_inputs, **generate_kwargs)
print(" ".join(tokenizer.convert_ids_to_tokens(batch_hypos[0][0]["tokens"])))

# <s> The Ġcat Ġis Ġdead . </s>**

#### replace <mask> with madeupword0003 ####
example_english_phrase = "madeupword0003 cat is madeupword0003."
masked_inputs=[example_english_phrase]
batch_hypos = fill_mask(bart,masked_inputs, masked_token = "madeupword0003", **generate_kwargs)
print(" ".join(tokenizer.convert_ids_to_tokens(batch_hypos[0][0]["tokens"])))

# <s> This Ġcat Ġis Ġadorable . </s>
```
### Environment

 - PyTorch Version: 1.5.1+cu101
 - OS (e.g., Linux): Linux
 - Python version: 3.6.10
 - transformers version: 4.2.1
 - CUDA version: 10.1

### Additional context

<!-- Add any other context about the problem here. -->


Docs suggest to use discriminator weights for ElectraForMaskedLM instead of generator

## Environment info

- `transformers` version: 4.2.2
- Platform: Linux-5.3.0-64-generic-x86_64-with-Ubuntu-19.10-eoan
- Python version: 3.7.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
 @LysandreJik

## Information

Model I am using (Bert, XLNet ...): ELECTRA

The problem arises when using:
* [x] the official example scripts: https://huggingface.co/transformers/model_doc/electra.html#codecell3
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: I want to check if a some word fits into the context of a sentence, for which I use the prediction probability of that word at the MASK position

## To reproduce

Steps to reproduce the behavior:

1. `from transformers import ElectraForMaskedLM`
2. `model = ElectraForMaskedLM.from_pretrained('google/electra-small-discriminator')`

## My issue

I get the following warning:

`Some weights of ElectraForMaskedLM were not initialized from the model checkpoint at google/electra-small-discriminator and are newly initialized: ['generator_predictions.LayerNorm.weight', 'generator_predictions.LayerNorm.bias', 'generator_predictions.dense.weight', 'generator_predictions.dense.bias', 'generator_lm_head.weight', 'generator_lm_head.bias']`

I understand that I'm loading the discriminator weights, whereas ElectraForMaskedLM needs the generator weights for the MLM output. Why do the docs tell me to use the discriminator? Am I missing something?


Changing model default for TableQuestionAnsweringPipeline.

# What does this PR do?

Niels removed his tapas model from the Hub, so we need to update the default to `google` organization

- Discussion: https://discuss.huggingface.co/t/table-question-answering-is-not-an-available-task-under-pipeline/3284/6
- Had to update the slow test that was out-of-sync I think, @LysandreJik can you confirm ?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@LysandreJik
@thomwolf

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Fix some TF slow tests

# What does this PR do?

This PR fixes several slow tests related to saved model creation.


ERROR about using layer_past and use_cache in Attention Layer of GPT2

Hi,
I am trying to use "use_cache" and "past_key_values" to speed up the decode steps.
But I have some questions about the Attention Layer, here are some codes in forward function:
[layer_past code](https://github.com/huggingface/transformers/blob/master/src/transformers/models/gpt2/modeling_gpt2.py#L230)
``` python
query = self.split_heads(query)
key = self.split_heads(key, k=True)
value = self.split_heads(value)
if layer_past is not None:
    past_key, past_value = layer_past[0].transpose(-2, -1), layer_past[1]  # transpose back cf below
    key = torch.cat((past_key, key), dim=-1)
    value = torch.cat((past_value, value), dim=-2)
```

If I send the layer_past value, it raise a size unmatched ERROR.
It shows that the shapes of "key" and "value" are not match with the attention mask.
Maybe the "key" before torch.cat has the same shape as the attention mask, but after torch.cat with past_key, the shape of "key" change.

Here is an example:
``` python
import torch
from transformers import GPT2Model, GPT2Config

config = GPT2Config()
config.use_cache = True
model = GPT2Model(config=config)

input_ids = torch.randint(0, 100, (2, 6))
attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]], dtype=torch.bool)
past_key_values = None
outputs = model(input_ids=input_ids, attention_mask=attention_mask, past_key_values=None)
logits = outputs[0]
past_key_values = outputs[1]
print(logits.size())
print(len(past_key_values))
print([_kv.size() for _kv in past_key_values])

# we get the past_key_values and add the next step decoder input.
added_input_ids = torch.randint(0, 100, (2, 1))
added_attention_mask = torch.tensor([[1], [1]], dtype=torch.bool)
input_ids = torch.cat([input_ids, added_input_ids], dim=1)
attention_mask = torch.cat([attention_mask, added_attention_mask], dim=1)
print(input_ids.size(), attention_mask.size())
outputs = model(input_ids=input_ids, attention_mask=attention_mask, past_key_values=past_key_values)
# here occur the ERROR

logits = outputs[0]
past_key_values = outputs[1]
print(logits.size())
print(len(past_key_values))
print([_kv.size() for _kv in past_key_values])
```

```
/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py in _attn(self, q, k, v, attention_mask, head_mask, output_attentions)
    175         if attention_mask is not None:
    176             # Apply the attention mask
--> 177             w = w + attention_mask
    178
    179         w = nn.Softmax(dim=-1)(w)

RuntimeError: The size of tensor a (13) must match the size of tensor b (7) at non-singleton dimension 3
```


fix T5 head mask in model_parallel

# What does this PR do?

`head_mask` in T5 is not parallelized correctly in model parallel, each layer's head mask should be put on that layer's device if it's not `None`.

Fixes #9718


AutoModel doesn't work with DPRContextEncoder

## Environment info
- `transformers` version: 4.2.2
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@lhoestq
@patrickvonplaten

## Information

If I run:
```python
model = AutoModel.from_pretrained('facebook/dpr-ctx_encoder-single-nq-base')
```

AutoModel infers `DPRQuestionEncoder` but not the correct class (i.e. `DPRContextEncoder`). Thus we can't use the correct model weights.

The output is:
```
Some weights of the model checkpoint at facebook/dpr-ctx_encoder-single-nq-base were not used when initializing DPRQuestionEncoder: ['ctx_encoder.bert_model.embeddings.word_embeddings.weight', 'ctx_encoder.bert_model.embeddings.position_embeddings.weight', 'ctx_encoder.bert_model.embeddings.token_type_embeddings.weight', 'ctx_encoder.bert_model.embeddings.LayerNorm.weight', 'ctx_encoder.bert_model.embeddings.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.0.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.0.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.0.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.0.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.0.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.0.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.0.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.0.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.0.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.0.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.0.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.0.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.0.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.0.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.0.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.0.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.1.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.1.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.1.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.1.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.1.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.1.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.1.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.1.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.1.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.1.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.1.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.1.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.1.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.1.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.1.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.1.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.2.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.2.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.2.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.2.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.2.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.2.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.2.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.2.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.2.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.2.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.2.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.2.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.2.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.2.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.2.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.2.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.3.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.3.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.3.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.3.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.3.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.3.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.3.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.3.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.3.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.3.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.3.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.3.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.3.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.3.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.3.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.3.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.4.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.4.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.4.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.4.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.4.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.4.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.4.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.4.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.4.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.4.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.4.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.4.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.4.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.4.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.4.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.4.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.5.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.5.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.5.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.5.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.5.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.5.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.5.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.5.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.5.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.5.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.5.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.5.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.5.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.5.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.5.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.5.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.6.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.6.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.6.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.6.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.6.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.6.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.6.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.6.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.6.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.6.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.6.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.6.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.6.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.6.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.6.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.6.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.7.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.7.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.7.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.7.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.7.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.7.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.7.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.7.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.7.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.7.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.7.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.7.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.7.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.7.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.7.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.7.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.8.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.8.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.8.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.8.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.8.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.8.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.8.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.8.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.8.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.8.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.8.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.8.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.8.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.8.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.8.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.8.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.9.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.9.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.9.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.9.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.9.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.9.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.9.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.9.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.9.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.9.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.9.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.9.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.9.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.9.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.9.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.9.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.10.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.10.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.10.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.10.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.10.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.10.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.10.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.10.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.10.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.10.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.10.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.10.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.10.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.10.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.10.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.10.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.11.attention.self.query.weight', 'ctx_encoder.bert_model.encoder.layer.11.attention.self.query.bias', 'ctx_encoder.bert_model.encoder.layer.11.attention.self.key.weight', 'ctx_encoder.bert_model.encoder.layer.11.attention.self.key.bias', 'ctx_encoder.bert_model.encoder.layer.11.attention.self.value.weight', 'ctx_encoder.bert_model.encoder.layer.11.attention.self.value.bias', 'ctx_encoder.bert_model.encoder.layer.11.attention.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.11.attention.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.11.attention.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.11.attention.output.LayerNorm.bias', 'ctx_encoder.bert_model.encoder.layer.11.intermediate.dense.weight', 'ctx_encoder.bert_model.encoder.layer.11.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.11.output.dense.weight', 'ctx_encoder.bert_model.encoder.layer.11.output.dense.bias', 'ctx_encoder.bert_model.encoder.layer.11.output.LayerNorm.weight', 'ctx_encoder.bert_model.encoder.layer.11.output.LayerNorm.bias', 'ctx_encoder.bert_model.pooler.dense.weight', 'ctx_encoder.bert_model.pooler.dense.bias']
- This IS expected if you are initializing DPRQuestionEncoder from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DPRQuestionEncoder from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DPRQuestionEncoder were not initialized from the model checkpoint at facebook/dpr-ctx_encoder-single-nq-base and are newly initialized: ['bert_model.embeddings.word_embeddings.weight', 'bert_model.embeddings.position_embeddings.weight', 'bert_model.embeddings.token_type_embeddings.weight', 'bert_model.embeddings.LayerNorm.weight', 'bert_model.embeddings.LayerNorm.bias', 'bert_model.encoder.layer.0.attention.self.query.weight', 'bert_model.encoder.layer.0.attention.self.query.bias', 'bert_model.encoder.layer.0.attention.self.key.weight', 'bert_model.encoder.layer.0.attention.self.key.bias', 'bert_model.encoder.layer.0.attention.self.value.weight', 'bert_model.encoder.layer.0.attention.self.value.bias', 'bert_model.encoder.layer.0.attention.output.dense.weight', 'bert_model.encoder.layer.0.attention.output.dense.bias', 'bert_model.encoder.layer.0.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.0.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.0.intermediate.dense.weight', 'bert_model.encoder.layer.0.intermediate.dense.bias', 'bert_model.encoder.layer.0.output.dense.weight', 'bert_model.encoder.layer.0.output.dense.bias', 'bert_model.encoder.layer.0.output.LayerNorm.weight', 'bert_model.encoder.layer.0.output.LayerNorm.bias', 'bert_model.encoder.layer.1.attention.self.query.weight', 'bert_model.encoder.layer.1.attention.self.query.bias', 'bert_model.encoder.layer.1.attention.self.key.weight', 'bert_model.encoder.layer.1.attention.self.key.bias', 'bert_model.encoder.layer.1.attention.self.value.weight', 'bert_model.encoder.layer.1.attention.self.value.bias', 'bert_model.encoder.layer.1.attention.output.dense.weight', 'bert_model.encoder.layer.1.attention.output.dense.bias', 'bert_model.encoder.layer.1.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.1.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.1.intermediate.dense.weight', 'bert_model.encoder.layer.1.intermediate.dense.bias', 'bert_model.encoder.layer.1.output.dense.weight', 'bert_model.encoder.layer.1.output.dense.bias', 'bert_model.encoder.layer.1.output.LayerNorm.weight', 'bert_model.encoder.layer.1.output.LayerNorm.bias', 'bert_model.encoder.layer.2.attention.self.query.weight', 'bert_model.encoder.layer.2.attention.self.query.bias', 'bert_model.encoder.layer.2.attention.self.key.weight', 'bert_model.encoder.layer.2.attention.self.key.bias', 'bert_model.encoder.layer.2.attention.self.value.weight', 'bert_model.encoder.layer.2.attention.self.value.bias', 'bert_model.encoder.layer.2.attention.output.dense.weight', 'bert_model.encoder.layer.2.attention.output.dense.bias', 'bert_model.encoder.layer.2.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.2.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.2.intermediate.dense.weight', 'bert_model.encoder.layer.2.intermediate.dense.bias', 'bert_model.encoder.layer.2.output.dense.weight', 'bert_model.encoder.layer.2.output.dense.bias', 'bert_model.encoder.layer.2.output.LayerNorm.weight', 'bert_model.encoder.layer.2.output.LayerNorm.bias', 'bert_model.encoder.layer.3.attention.self.query.weight', 'bert_model.encoder.layer.3.attention.self.query.bias', 'bert_model.encoder.layer.3.attention.self.key.weight', 'bert_model.encoder.layer.3.attention.self.key.bias', 'bert_model.encoder.layer.3.attention.self.value.weight', 'bert_model.encoder.layer.3.attention.self.value.bias', 'bert_model.encoder.layer.3.attention.output.dense.weight', 'bert_model.encoder.layer.3.attention.output.dense.bias', 'bert_model.encoder.layer.3.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.3.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.3.intermediate.dense.weight', 'bert_model.encoder.layer.3.intermediate.dense.bias', 'bert_model.encoder.layer.3.output.dense.weight', 'bert_model.encoder.layer.3.output.dense.bias', 'bert_model.encoder.layer.3.output.LayerNorm.weight', 'bert_model.encoder.layer.3.output.LayerNorm.bias', 'bert_model.encoder.layer.4.attention.self.query.weight', 'bert_model.encoder.layer.4.attention.self.query.bias', 'bert_model.encoder.layer.4.attention.self.key.weight', 'bert_model.encoder.layer.4.attention.self.key.bias', 'bert_model.encoder.layer.4.attention.self.value.weight', 'bert_model.encoder.layer.4.attention.self.value.bias', 'bert_model.encoder.layer.4.attention.output.dense.weight', 'bert_model.encoder.layer.4.attention.output.dense.bias', 'bert_model.encoder.layer.4.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.4.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.4.intermediate.dense.weight', 'bert_model.encoder.layer.4.intermediate.dense.bias', 'bert_model.encoder.layer.4.output.dense.weight', 'bert_model.encoder.layer.4.output.dense.bias', 'bert_model.encoder.layer.4.output.LayerNorm.weight', 'bert_model.encoder.layer.4.output.LayerNorm.bias', 'bert_model.encoder.layer.5.attention.self.query.weight', 'bert_model.encoder.layer.5.attention.self.query.bias', 'bert_model.encoder.layer.5.attention.self.key.weight', 'bert_model.encoder.layer.5.attention.self.key.bias', 'bert_model.encoder.layer.5.attention.self.value.weight', 'bert_model.encoder.layer.5.attention.self.value.bias', 'bert_model.encoder.layer.5.attention.output.dense.weight', 'bert_model.encoder.layer.5.attention.output.dense.bias', 'bert_model.encoder.layer.5.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.5.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.5.intermediate.dense.weight', 'bert_model.encoder.layer.5.intermediate.dense.bias', 'bert_model.encoder.layer.5.output.dense.weight', 'bert_model.encoder.layer.5.output.dense.bias', 'bert_model.encoder.layer.5.output.LayerNorm.weight', 'bert_model.encoder.layer.5.output.LayerNorm.bias', 'bert_model.encoder.layer.6.attention.self.query.weight', 'bert_model.encoder.layer.6.attention.self.query.bias', 'bert_model.encoder.layer.6.attention.self.key.weight', 'bert_model.encoder.layer.6.attention.self.key.bias', 'bert_model.encoder.layer.6.attention.self.value.weight', 'bert_model.encoder.layer.6.attention.self.value.bias', 'bert_model.encoder.layer.6.attention.output.dense.weight', 'bert_model.encoder.layer.6.attention.output.dense.bias', 'bert_model.encoder.layer.6.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.6.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.6.intermediate.dense.weight', 'bert_model.encoder.layer.6.intermediate.dense.bias', 'bert_model.encoder.layer.6.output.dense.weight', 'bert_model.encoder.layer.6.output.dense.bias', 'bert_model.encoder.layer.6.output.LayerNorm.weight', 'bert_model.encoder.layer.6.output.LayerNorm.bias', 'bert_model.encoder.layer.7.attention.self.query.weight', 'bert_model.encoder.layer.7.attention.self.query.bias', 'bert_model.encoder.layer.7.attention.self.key.weight', 'bert_model.encoder.layer.7.attention.self.key.bias', 'bert_model.encoder.layer.7.attention.self.value.weight', 'bert_model.encoder.layer.7.attention.self.value.bias', 'bert_model.encoder.layer.7.attention.output.dense.weight', 'bert_model.encoder.layer.7.attention.output.dense.bias', 'bert_model.encoder.layer.7.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.7.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.7.intermediate.dense.weight', 'bert_model.encoder.layer.7.intermediate.dense.bias', 'bert_model.encoder.layer.7.output.dense.weight', 'bert_model.encoder.layer.7.output.dense.bias', 'bert_model.encoder.layer.7.output.LayerNorm.weight', 'bert_model.encoder.layer.7.output.LayerNorm.bias', 'bert_model.encoder.layer.8.attention.self.query.weight', 'bert_model.encoder.layer.8.attention.self.query.bias', 'bert_model.encoder.layer.8.attention.self.key.weight', 'bert_model.encoder.layer.8.attention.self.key.bias', 'bert_model.encoder.layer.8.attention.self.value.weight', 'bert_model.encoder.layer.8.attention.self.value.bias', 'bert_model.encoder.layer.8.attention.output.dense.weight', 'bert_model.encoder.layer.8.attention.output.dense.bias', 'bert_model.encoder.layer.8.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.8.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.8.intermediate.dense.weight', 'bert_model.encoder.layer.8.intermediate.dense.bias', 'bert_model.encoder.layer.8.output.dense.weight', 'bert_model.encoder.layer.8.output.dense.bias', 'bert_model.encoder.layer.8.output.LayerNorm.weight', 'bert_model.encoder.layer.8.output.LayerNorm.bias', 'bert_model.encoder.layer.9.attention.self.query.weight', 'bert_model.encoder.layer.9.attention.self.query.bias', 'bert_model.encoder.layer.9.attention.self.key.weight', 'bert_model.encoder.layer.9.attention.self.key.bias', 'bert_model.encoder.layer.9.attention.self.value.weight', 'bert_model.encoder.layer.9.attention.self.value.bias', 'bert_model.encoder.layer.9.attention.output.dense.weight', 'bert_model.encoder.layer.9.attention.output.dense.bias', 'bert_model.encoder.layer.9.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.9.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.9.intermediate.dense.weight', 'bert_model.encoder.layer.9.intermediate.dense.bias', 'bert_model.encoder.layer.9.output.dense.weight', 'bert_model.encoder.layer.9.output.dense.bias', 'bert_model.encoder.layer.9.output.LayerNorm.weight', 'bert_model.encoder.layer.9.output.LayerNorm.bias', 'bert_model.encoder.layer.10.attention.self.query.weight', 'bert_model.encoder.layer.10.attention.self.query.bias', 'bert_model.encoder.layer.10.attention.self.key.weight', 'bert_model.encoder.layer.10.attention.self.key.bias', 'bert_model.encoder.layer.10.attention.self.value.weight', 'bert_model.encoder.layer.10.attention.self.value.bias', 'bert_model.encoder.layer.10.attention.output.dense.weight', 'bert_model.encoder.layer.10.attention.output.dense.bias', 'bert_model.encoder.layer.10.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.10.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.10.intermediate.dense.weight', 'bert_model.encoder.layer.10.intermediate.dense.bias', 'bert_model.encoder.layer.10.output.dense.weight', 'bert_model.encoder.layer.10.output.dense.bias', 'bert_model.encoder.layer.10.output.LayerNorm.weight', 'bert_model.encoder.layer.10.output.LayerNorm.bias', 'bert_model.encoder.layer.11.attention.self.query.weight', 'bert_model.encoder.layer.11.attention.self.query.bias', 'bert_model.encoder.layer.11.attention.self.key.weight', 'bert_model.encoder.layer.11.attention.self.key.bias', 'bert_model.encoder.layer.11.attention.self.value.weight', 'bert_model.encoder.layer.11.attention.self.value.bias', 'bert_model.encoder.layer.11.attention.output.dense.weight', 'bert_model.encoder.layer.11.attention.output.dense.bias', 'bert_model.encoder.layer.11.attention.output.LayerNorm.weight', 'bert_model.encoder.layer.11.attention.output.LayerNorm.bias', 'bert_model.encoder.layer.11.intermediate.dense.weight', 'bert_model.encoder.layer.11.intermediate.dense.bias', 'bert_model.encoder.layer.11.output.dense.weight', 'bert_model.encoder.layer.11.output.dense.bias', 'bert_model.encoder.layer.11.output.LayerNorm.weight', 'bert_model.encoder.layer.11.output.LayerNorm.bias', 'bert_model.pooler.dense.weight', 'bert_model.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

I think could be useful to generalise this behaviour and automatically detect whether the model is a context/document encoder or question encoder.


Run_ner.py falsely aligns prediction list

## Environment info
- `transformers` version: 3.5.1
- Platform: Linux-5.9.1-kd-cluster-x86_64-with-glibc2.10
- Python version: 3.8.0
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik
@stefan-it

## Information

I am using the bert-base-german-cased model, which I trained on an NER task to predict entity labels.
When I run prediction on the test dataset, BERT tells me that the maximum sequence length (300) is exhausted and so no predictions will be made on a number of items.
However, this seems to be a wrong error report. The problem appears to be that in the align_predictions function, the preds_list variable gets a wrong dimension - the predictions are made correctly, but the index is shifted, so that they point to the wrong words.
An example of what I mean:
```
Amtsgericht Ort
Leipzig Ort
Abteilung O
für O
Strafsachen O
```
becomes
```
Amtsgericht O
Leipzig O
Abteilung O
für Ort
Strafsachen Ort
```
in the preds_list.
The write_predictions_to_file function then gets tangled up by this shifted index and (I think falsely) declares a sequence length error.

Strangely, another test document works just fine, without any difference between them that I could find. No sequence length error and no false indices there.

The problem arises when using:
* [] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I'm afraid it isn't so easy to reproduce because it requires the trained model and the data.

The align_predictions function:
```
    def align_predictions(predictions: np.ndarray, label_ids: np.ndarray) -> Tuple[List[int], List[int]]:
        preds = np.argmax(predictions, axis=2)

        batch_size, seq_len = preds.shape

        out_label_list = [[] for _ in range(batch_size)]
        preds_list = [[] for _ in range(batch_size)]

        for i in range(batch_size):
            for j in range(seq_len):
                if label_ids[i, j] != nn.CrossEntropyLoss().ignore_index:
                    out_label_list[i].append(label_map[label_ids[i][j]])
                    preds_list[i].append(label_map[preds[i][j]])

        return preds_list, out_label_list
```

Console output:
```
01/21/2021 11:07:01 - INFO - filelock -   Lock 140422091422400 released on /home/IAIS/tschmude/bert_remote/examples/token-classification/Data_processing_scripts/CrossVal_Files/Rotation/Test_file_swap/cached_test_BertTokenizer_340.lock
/home/IAIS/tschmude/anaconda3/envs/bert_env_remote/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:64: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
  0%|                                                     | 0/1 [00:00<?, ?it/s]/home/IAIS/tschmude/anaconda3/envs/bert_env_remote/lib/python3.8/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
01/21/2021 11:07:30 - INFO - run_ner -
                                    precision    recall  f1-score   support
                             Datum       1.00      1.00      1.00         1
                    Gestaendnis_ja       1.00      1.00      1.00         1
                               Ort       0.50      1.00      0.67         1
                    Schadensbetrag       1.00      1.00      1.00         1
Strafe_Gesamtfreiheitsstrafe_Dauer       1.00      1.00      1.00         1
                 Strafe_Tatbestand       0.00      0.00      0.00         0
             Taeter_Drogenbezug_ja       1.00      1.00      1.00         1
                         micro avg       0.55      1.00      0.71         6
                         macro avg       0.79      0.86      0.81         6
                      weighted avg       0.92      1.00      0.94         6
/tmp/pycharm_project_44/src/transformers/trainer.py:1174: FutureWarning: This method is deprecated, use `Trainer.is_world_process_zero()` instead.
  warnings.warn("This method is deprecated, use `Trainer.is_world_process_zero()` instead.", FutureWarning)
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'Angeklagte'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'trägt'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'die'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'Kosten'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'des'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'Verfahrens'.
(and a lot more of these)
```
Config that the model was trained on:
```
Training Arguments:
	Data directory: ...ng_scripts/CrossVal_Files/Rotation/Train_file_swap
	Model: ...bert-base-german-cased
	Epochs: 8
	Seq length: 300
	Learning rate: 5e-05
	Batch size: 16
	Seed: 105
	Do Train: True
	Do Eval: True
	Do Test: True
```

## Expected behavior

That the predictions for the documents are correctly listed and written to file without a sequence length error or shifted indices.

Thank you for your help!