IndexError: index out of range in self while using longformers when i try to pass token_type_ids

- `transformers` version: 3.0.0
- Platform: windows
- Python version: 3.6.10 :: Anaconda, Inc.
- PyTorch version (GPU?): 1.7.0+cu101
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## To reproduce
import torch
from transformers import LongformerModel, LongformerTokenizer
model = LongformerModel.from_pretrained('allenai/longformer-base-4096')
tokenizer = LongformerTokenizer.from_pretrained('roberta-base')
SAMPLE_TEXT = ' '.join(['Hello world! '] * 100)  # long input document
input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1
attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device)
global_attention_mask = torch.zeros(input_ids.shape, dtype=torch.long, device=input_ids.device)
segment_ids = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device)
outputs = model(input_ids=input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask,token_type_ids=segment_ids)

## Error info
IndexError                                Traceback (most recent call last)
<ipython-input-357-c7bf5dc7cbc9> in <module>
----> 1 outputs = model(input_ids=input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask,token_type_ids=segment_ids)

~\.conda\envs\env\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\.conda\envs\env\lib\site-packages\transformers\modeling_longformer.py in forward(self, input_ids, attention_mask, global_attention_mask, token_type_ids, position_ids, inputs_embeds, output_attentions, output_hidden_states)
    675             encoder_attention_mask=None,
    676             output_attentions=output_attentions,
--> 677             output_hidden_states=output_hidden_states,
    678         )
    679

~\.conda\envs\env\lib\site-packages\transformers\modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states)
    751
    752         embedding_output = self.embeddings(
--> 753             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    754         )
    755         encoder_outputs = self.encoder(

~\.conda\envs\env\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\.conda\envs\env\lib\site-packages\transformers\modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     66
     67         return super().forward(
---> 68             input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
     69         )
     70

~\.conda\envs\env\lib\site-packages\transformers\modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    178             inputs_embeds = self.word_embeddings(input_ids)
    179         position_embeddings = self.position_embeddings(position_ids)
--> 180         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    181
    182         embeddings = inputs_embeds + position_embeddings + token_type_embeddings

~\.conda\envs\env\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\.conda\envs\env\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
    124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
--> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
    127
    128     def extra_repr(self) -> str:

~\.conda\envs\env\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853
   1854

IndexError: index out of range in self

## Expected behavior

If i am not passing a segment_id (token_type_ids) then model is executing or if i am passing segment_id's as zeros then its getting executed when i am passing segment_id's 1 or 0's and 1's then i am getting an error as index out of range in self.


Metric calculation across batches in seq2seq examples

### Who can help
@sshleifer, @patil-suraj

## Information

Currently, the seq2seq finetune script calculates the final metrics (bleu or rouge) per batch, and then simply averages these numbers across batches. Is it correct? For example, I just took a brief look at sacrebleu (which the script uses), and it seems that `corpus_bleu(preds, targets)` isn't necessarily equal to `(corpus_bleu(preds[:n], targets[:n] + corpus_bleu(preds[n:], targets[n:])/2`.


Trainer bug? Loss and logits are “nan” when fine-tuning NLI model (both RoBERTa/BART)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.1  (also reproduced the same issue with 3.5.1)
- Platform: Google Colab
- Python version: 3.6
- PyTorch version (GPU?): 1.7.0+cu101
- Tensorflow version (GPU?): no
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: (don't know. Probably not)

### Who can help
@sgugger

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information
The problem arises when using:
* [ x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ x] my own task or dataset: (give details below)

**Description:**
I’m trying to fine-tune a pre-trained NLI model (`ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli`) on a dataset of around 276.000 hypothesis-premise pairs. I’m following the instructions from the docs [here](https://huggingface.co/transformers/custom_datasets.html) and [here](https://huggingface.co/transformers/training.html). When I run the training, it seems like the fine-tuning works (it does the training and saves the checkpoints), but `trainer.train()` and `trainer.evaluate()` return "nan" as loss value.

**What I've tried:**
- I tried using both `ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli` and `facebook/bart-large-mnli` to make sure that it's not linked to specific model, but I get the issue for both models
- I tried following the advice in this [related github issue](https://github.com/huggingface/transformers/issues/1727), but adding `num_labels=3` to the config file does not solve the issue. (I think my issue is different because the models are already fine-tuned on NLI in my case)
- I tried many different ways of changing my input data because I suspected that there could be an issue with my input data, but I also couldn't solve it that way.
- **The probable source of the issue:** I inspected the prediction output from the model during training and the weird thing is that the prediction value always seems to be "0" (entailment) in 100% of cases (see printed output at the bottom of the code below). This cannot be right.
Even weirder: When I first run the model to predict a test sequences before running the trainer, I get normal logits as output. When I run the exact same code block again at the end after having run the trainer, I get `tensor([[nan, nan, nan]]` as output (see code below).
- I suspect that the source for the 'only 0 prediction output' is that the logits the model returns during training are possibly always `torch.tensor([[np.nan, np.nan, np.nan]])`. `torch.tensor([[np.nan, np.nan, np.nan]]).argmax(-1)` returns torch.tensor(0) without triggering an error. The big mystery for me is why the logits would become "nan", because the model does not do that when I use the same input data only outside of the trainer, but something changes once I've run the trainer.

=> I would be very thankful for any help on this! (I've been trying to solve this since two days now)
Thanks a lot in advance.

## To reproduce

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

### Here is my code:

    ### load model & tokenize
    from transformers import AutoTokenizer, AutoModelForSequenceClassification
    import torch

    max_length = 256
    hg_model_hub_name = "ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli"
    # also tried: hg_model_hub_name = "facebook/bart-large-mnli"
    tokenizer = AutoTokenizer.from_pretrained(hg_model_hub_name)
    model = AutoModelForSequenceClassification.from_pretrained(hg_model_hub_name)
    model.config

    device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"Device: {device}")
    if device == "cuda":
      model = model.half()
    model.to(device)
    model.train();

**Running a test inference with the model at this point works fine:**
```
test_enc = tokenizer(nli_train[0]["premise"], nli_train[0]["hypothesis"], return_tensors="pt", max_length=max_length,
                            return_token_type_ids=True, truncation=True, padding=True)
model.eval();
test_output_loss = model(test_enc["input_ids"].to(device), attention_mask=test_enc["attention_mask"].to(device), token_type_ids=test_enc["token_type_ids"].to(device), labels=torch.tensor(2).to(device))
print(test_output_loss)
#output: SequenceClassifierOutput(loss=tensor(2.2168, device='cuda:0', dtype=torch.float16, grad_fn=<NllLossBackward>), logits=tensor([[ 0.4075,  0.8511, -0.7549]], device='cuda:0', dtype=torch.float16,
       grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)
```

**Then I continue with preprocessing and training:**
    #... some data preprocessing

    encodings_train = tokenizer(premise_train, hypothesis_train, return_tensors="pt", max_length=max_length,
                                return_token_type_ids=True, truncation=False, padding=True)
    encodings_val = tokenizer(premise_val, hypothesis_val, return_tensors="pt", max_length=max_length,
                              return_token_type_ids=True, truncation=False, padding=True)
    encodings_test = tokenizer(premise_test, hypothesis_test, return_tensors="pt", max_length=max_length,
                               return_token_type_ids=True, truncation=False, padding=True)

    ### create pytorch dataset object
    class XDataset(torch.utils.data.Dataset):
        def __init__(self, encodings, labels):
            self.encodings = encodings
            self.labels = labels
        def __getitem__(self, idx):
            item = {key: torch.as_tensor(val[idx]) for key, val in self.encodings.items()}
            #item = {key: torch.as_tensor(val[idx]).to(device) for key, val in self.encodings.items()}
            item['labels'] = torch.as_tensor(self.labels[idx])
            #item['labels'] = self.labels[idx]
            return item
        def __len__(self):
            return len(self.labels)

    dataset_train = XDataset(encodings_train, label_train)
    dataset_val = XDataset(encodings_val, label_val)
    dataset_test = XDataset(encodings_test, label_test)

    # compute metrics with trainer
    from sklearn.metrics import accuracy_score, precision_recall_fscore_support
    def compute_metrics(pred):
        labels = pred.label_ids
        print(labels)
        preds = pred.predictions.argmax(-1)
        print(preds)
        precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary', pos_label=0)
        acc = accuracy_score(labels, preds)
        return {
            'accuracy': acc,
            'f1': f1,
            'precision': precision,
            'recall': recall
        }

    ## training
    from transformers import Trainer, TrainingArguments

    # https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments
    training_args = TrainingArguments(
        output_dir='./results',          # output directory
        num_train_epochs=1,              # total number of training epochs
        per_device_train_batch_size=8,  # batch size per device during training
        per_device_eval_batch_size=8,   # batch size for evaluation
        warmup_steps=500,                # number of warmup steps for learning rate scheduler
        weight_decay=0.01,               # strength of weight decay
        logging_dir='./logs',            # directory for storing logs
        logging_steps=100,
    )

    trainer = Trainer(
        model=model,                         # the instantiated 🤗 Transformers model to be trained
        args=training_args,                  # training arguments, defined above
        train_dataset=dataset_train,         # training dataset
        eval_dataset=dataset_val             # evaluation dataset
    )

    trainer.train()
    # output: TrainOutput(global_step=181, training_loss=nan)
    trainer.evaluate()
    # output:
    [2 2 2 0 0 2 2 2 0 2 0 0 2 2 2 2 0 2 0 2 2 2 2 0 2 0 2 0 0 2 0 0 2 0 0 0 2
     0 2 0 0 0 0 0 2 0 0 2 2 2 0 2 2 2 2 2 0 0 0 0 2 0 0 0 2 2 0 0 0 2 0 0 0 2
     2 0 2 0 0 2 2 2 0 2 2 0 0 0 0 0 0 0 2 0 0 0 0 2 0 2 2 0 2 0 0 2 2 2 2 2 2
     2 0 0 0 0 2 0 0 2 0 0 0 0 2 2 2 0 0 0 0 0 2 0 0 2 0 2 0 2 0 2 0 0 2 2 0 0
     2 2 2 2 2 2 0 0 2 2 2 2 0 2 0 0 2 2 2 0 0 2 0 2 0 2 0 0 0 0 0 0 2 0 0 2 2
     0 2 2 2 0 2 2 0 2 2 2 2 2 2 0 0 2 0 0 2 2 0 0 0 2 0 2 2 2 0 0 0 0 0 0 0 0
     2 0 2 2 2 0 2 0 0 2 0 2 2 0 0 0 0 2 2 2 0 0 0 2 2 2 2 0 2 0 2 2 2]
    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

    {'epoch': 1.0,
     'eval_accuracy': 0.5137254901960784,
     'eval_f1': 0.6787564766839378,
     'eval_loss': nan,
     'eval_precision': 0.5137254901960784,
     'eval_recall': 1.0}

**Test running the model again after training, returns `tensor([[nan, nan, nan]]` for some reason:**
```
test_enc = tokenizer(nli_train[0]["premise"], nli_train[0]["hypothesis"], return_tensors="pt", max_length=max_length,
                            return_token_type_ids=True, truncation=True, padding=True)
model.eval();
test_output_loss = model(test_enc["input_ids"].to(device), attention_mask=test_enc["attention_mask"].to(device), token_type_ids=test_enc["token_type_ids"].to(device), labels=torch.tensor(2).to(device))
print(test_output_loss)
#output: SequenceClassifierOutput(loss=tensor(nan, device='cuda:0', dtype=torch.float16, grad_fn=<NllLossBackward>), logits=tensor([[nan, nan, nan]], device='cuda:0', dtype=torch.float16,
       grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)
```

  [1]: https://huggingface.co/

## Expected behavior

Model should not return "nan" for logits and return a loss value.

<!-- A clear and concise description of what you would expect to happen. -->


Unified transformer interface

# 🚀 Feature request

As we're called `transformers`,  it would be nice if there's a `Transformer` class that is not associated with any pretrained model and that we can directly instantiate, like
```python
from transformers import Transformer, Seq2SeqTransformer
transformer = Transformer(n_layers=8, dim=512)
seq2seq_transformer = Seq2SeqTransformer(n_enc_layers=6, n_dec_layers=8, dim=512)
```
They could follow a modular design that makes swapping components very easy, e.g.
```python
class SelfAttention:
    ...

class Transformer:
    def __init__(self, self_attention_cls=SelfAttention, n_layers=8, ...):
        ...
```

## Motivation

1. Makes it easier for researchers to modify the transformer architecture and train it from scratch.
2. Allows great refactoring of the current `modeling_xxx.py` files. Often these files are highly identical minus a few minor differences. For example, the embeddings, self-attention layers, etc. are often identical across multiple models, but the current design does not allow much code re-use. With this change, I imagine a lot of the current `modeling_xxx.py` files can be reduced significantly.
3. A side-effect of this refactoring would be increased readability. Without much repetitive code, it would be much clearer to see the differences between architectures, which would be helpful for many users.


Getting a 404 error when loading 'model=facebook/bart-large-mnli' from pipeline('zero-shot-classification')

Getting a 404 when trying to load the model.
model='joeddav/bart-large-mnli-yahoo-answers' also not working.

`404 Client Error: Not Found for url: https://huggingface.co/facebook/bart-large-mnli/resolve/main/tf_model.h5
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    695                 # Load from URL or cache if already cached
--> 696                 resolved_archive_file = cached_path(
    697                     archive_file,

~/.local/lib/python3.8/site-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
    999         # URL, so get it from the cache (downloading if necessary)
-> 1000         output_path = get_from_cache(
   1001             url_or_filename,

~/.local/lib/python3.8/site-packages/transformers/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only)
   1127             r = requests.head(url, headers=headers, allow_redirects=False, proxies=proxies, timeout=etag_timeout)
-> 1128             r.raise_for_status()
   1129             etag = r.headers.get("X-Linked-Etag") or r.headers.get("ETag")

/usr/lib/python3/dist-packages/requests/models.py in raise_for_status(self)
    939         if http_error_msg:
--> 940             raise HTTPError(http_error_msg, response=self)
    941

HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/facebook/bart-large-mnli/resolve/main/tf_model.h5

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-48-a42b4974064c> in <module>
----> 1 classifier = pipeline('zero-shot-classification',
      2                       model='facebook/bart-large-mnli',
      3 #                      model='joeddav/bart-large-mnli-yahoo-answers',
      4 #                       model = 'bert-base-uncased'
      5 #                       model ='phiyodr/bart-large-finetuned-squad2',

~/.local/lib/python3.8/site-packages/transformers/pipelines.py in pipeline(task, model, config, tokenizer, framework, revision, use_fast, **kwargs)
   2934         model = get_default_model(targeted_task, framework, task_options)
   2935
-> 2936     framework = framework or get_framework(model)
   2937
   2938     task_class, model_class = targeted_task["impl"], targeted_task[framework]

~/.local/lib/python3.8/site-packages/transformers/pipelines.py in get_framework(model, revision)
    106             model = AutoModel.from_pretrained(model, revision=revision)
    107         elif is_tf_available() and not is_torch_available():
--> 108             model = TFAutoModel.from_pretrained(model, revision=revision)
    109         else:
    110             try:

~/.local/lib/python3.8/site-packages/transformers/models/auto/modeling_tf_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    558
    559         if type(config) in TF_MODEL_MAPPING.keys():
--> 560             return TF_MODEL_MAPPING[type(config)].from_pretrained(
    561                 pretrained_model_name_or_path, *model_args, config=config, **kwargs
    562             )

~/.local/lib/python3.8/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    709                     f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a file named one of {TF2_WEIGHTS_NAME}, {WEIGHTS_NAME}.\n\n"
    710                 )
--> 711                 raise EnvironmentError(msg)
    712             if resolved_archive_file == archive_file:
    713                 logger.info("loading weights file {}".format(archive_file))

OSError: Can't load weights for 'facebook/bart-large-mnli'. Make sure that:

- 'facebook/bart-large-mnli' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'facebook/bart-large-mnli' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.`


T5 checkpoint contains weights missing on current model.

## Environment info
Colab (16 december 2020), transformers 4.0.1

### Who can help
 T5: @patrickvonplaten

## Information
When loading the pre-trained T5 weights, directy with .from_pretrained, with the newest version, it returns the following warning:

>  Some weights of the model checkpoint at t5-small were not used when initializing T5ForConditionalGeneration: ['decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight']
> - This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
> - This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

This also caused old checkpoints of mine to not load due to the missing weight.

## To reproduce
Reproduced in Colab:
https://colab.research.google.com/drive/158OiSKHz80b0PQYaWB6c-AI9xKCzE1ns?usp=sharing

## Expected behavior

Loading T5 should not return warnings if i am loading the pre-trained weights from the library.


Sharded DDP training fails with seq2seq models

## Information

Model I am using (Bert, XLNet ...): T5/BART/mBART/Marian

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: seq2seq
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Run
```
python -m torch.distributed.launch --nproc_per_node=2  examples/seq2seq/finetune_trainer.py --model_name_or_path sshleifer/tiny-mbart --output_dir output_dir --adam_eps 1e-06 --data_dir ~/Downloads/wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size 4 --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500 --sharded_ddp
```
will fail with
```
Traceback (most recent call last):                                                                                File "examples/seq2seq/finetune_trainer.py", line 379, in <module>                                                                                main()                                                                                File "examples/seq2seq/finetune_trainer.py", line 316, in main                                                                                model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None                                                                                File "/home/sgugger/git/transformers/src/transformers/trainer.py", line 821, in train                                                                                self.optimizer.step()                                                                                File "/home/sgugger/.pyenv/versions/base/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper                                                                                return wrapped(*args, **kwargs)                                                                                File "/home/sgugger/git/fairscale/fairscale/optim/oss.py", line 210, in step                                                                                self._broadcast_params()                                                                                File "/home/sgugger/git/fairscale/fairscale/optim/oss.py", line 522, in _broadcast_params                                                                                if self.should_bucket_param[param]:                                                                                KeyError: Parameter containing:                                                                                tensor([[-0.0296,  0.0038],                                                                                [ 0.0000,  0.0000],                                                                                [ 0.0298,  0.0385],                                                                                ...,                                                                                [-0.0161, -0.0024],                                                                                [ 0.0022, -0.0576],                                                                                [ 0.0053,  0.0256]], device='cuda:1')                                                                                0%|
```

Using FP16 also fails.

## Expected behavior

The script should run to completion.


evaluate_during_training is not acceptable in newer version of the Transformer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:  4.0.0
- Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: <True>
- Using distributed or parallel set-up in script?: <False>

### Who can help
 Trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [* ] my own modified scripts: (give details below)
*
config=RobertaConfig(vocab_size=30_000, max_position_embedding=512, num_attention_heads=12, num_hidden_layers=12, hidden_dropout_prob=0.1, attention_dropout_prob=0.1, initializer_range=0.2, intermediate_size=3072, type_vocab_size=1)

tokenizer=RobertaTokenizerFast.from_pretrained("XXX", max_len=512)
model=RobertaForMaskedLM(config=config)
dataset=LineByLineTextDataset(tokenizer, file_path="XXX")
data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

training_args=TrainingArguments(output_dir="xxxx", overwrite_output_dir=True,
num_training_epochs=2, do_train=True, do_eval=True, **evaluate_during_training=True**,
per_gpu_train_batch_size=128, learning_rate=0.0004,
gradient_accumulation_steps=32,
logging_steps=2048,
warmup_steps=10000,
weight_decay=0.01,
eval_steps=2048,
save_steps=2048,
save_total_limit=2, prediction_loss_only=True)

trainer=Trainer(model=model, args=training_args,data_collator=data_collator, train_dataset=dataset, eval_dataset=dataseteval, **prediction_loss_only=True**)

trainer.train()

The tasks I am working on is:

* [* ] my own task or dataset: (give details below)

It is a dataset of several million lines of text.

## To reproduce

Steps to reproduce the behavior:

1. Having transformers installed
2. Add lines of code for accessing a training and evaluation dataset and run it

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

The following code was working fine with transformers 3.5 but since I updated the transformers version, it does not accept the following arguments:
evaluate_during_training=True
prediction_loss_only=True
 The error is invalid arguments for both of them.

## Expected behavior
I need to have them to be able to check the validation loss during the model training. By removing the following arguments, the model just reports the training loss and not the evaluation loss during the training.

<!-- A clear and concise description of what you would expect to happen. -->


AutoModelForTableQuestionAnswering

Adds the `AutoModelForTableQuestionAnswering`


BertForSequenceClassification and DistilBertForSequenceClassification use pooler output in different ways

Hi,
The `BertForSequenceClassification` includes a forward pass of the BertModel, and it takes the second element (index 1) from its output before moving forward, as shown [here ](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L1378 )
This is the return of `BertModel`
```
    return BaseModelOutputWithPoolingAndCrossAttentions(
            last_hidden_state=sequence_output,
            pooler_output=pooled_output,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
            cross_attentions=encoder_outputs.cross_attentions,
        )
```
hence `output[1]` is taking the pooler_output.

However, in `DistilBertForSequenceClassification`, it takes the first element (index 0) of the `DistilBertModel`'s forward pass, i.e. `distilbert_output[0]`, as shown [here](https://github.com/huggingface/transformers/blob/master/src/transformers/models/distilbert/modeling_distilbert.py#L625)

This is the last hidden state for all tokens.
Why is there this discrepancy between the two models? The behaviour of `DistilBertForSequenceClassification` makes more intuitive sense to me.
Why is `BertForSequenceClassification` using only the pooler_output from `BertModel` (i.e. the hidden state of the first token)? Why are all other hidden states of other tokens not needed here, but needed in the distilled version?

Thanks for the help!


Add message to documentation that longformer doesn't support token_type_ids

# What does this PR do?

Fixes #9111. This pull request adds a notice to the Longformer model documentation that it does not have `token_type_ids`, similarly to the message on the [RoBERTa documentation](https://huggingface.co/transformers/model_doc/roberta.html).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@sgugger (documentation)
@patrickvonplaten (longformer)


Added TF CTRL Sequence Classification

This PR implements Sequence classification for TF CTRL model.
TFCTRLForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. Transformer XL ,GPT-2) do.

Fixes #7623

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you write any new necessary tests?

@jplu @LysandreJik


Add flags to return scores, hidden states and / or attention weights in GenerationMixin

Core: Modeling

Internals of the library; Models.

Ex: Generation

Natural Language Generation

# What does this PR do?

Add flags and logic to return attention scores, hidden states, and/or logits when using a model in generation mode.

Fixes #9121

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Issue: #9121
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten

=> Check out forum post for more detail: https://discuss.huggingface.co/t/announcement-generationoutputs-scores-attentions-and-hidden-states-now-available-as-outputs-to-generate/3094

## UPDATE:

This PR is almost ready for merge. Here are the last things to take care of:

- [x] @patrickvonplaten, @SBrandeis, @LysandreJik, @sgugger - check if naming of return arguments is ok.
- [x] @patrickvonplaten add a correct `_check_outputs()` function for special models (Reformer, XLNet, XLM, TransfoXL, ...)
- [x] @SBrandeis We could think about a simple code snippet to advertise the new feature. Maybe use "Beam Search" for translation and show the different "sequence_scores" (probs) for 3 different translations.

## Future PR:

- [ ] Change the output in all notebooks, examples, tests to `return_dict_in_generate=True`.
- [ ] Even further in the future PR: Think about a way to deprecate the default usage of `return_dict_in_generate=False`.
- [ ] @patrickvonplaten Discuss with @sgugger if model documentation is ok. @sgugger - I think `generate()` might deserve its own "main" doc page maybe in "main classes" under "Models". I would add a small description at the top, then add all the generate functions and the new "Generation Outputs" there. Do you think this makes sense or would you rather keep "generate" as a subsection under "Models"? IMO, "generate" could be more visible in the docs.
- [ ] @patrickvonplaten, @LysandreJik, @sgugger - this PR adds a lot more generate testing to many models. Since `generate` quite an expensive method, it might be worth checking here that this PR does yield a significant slow down of the tests. Simon and I tried to make the generate test as short and light-weight as possible, but it might still be significant.


# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Log metrics along with hparams in TensorBoardCallback.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

It seems useless to log hparams with empty metric dict because the training arguments are already logged in the text section. Now users can see only a blank screen when they click on the HPARAMS of the TensorBoard. So I think it will be better to call `add_hparams` with evaluation metrics when available. Otherwise just don't call this function.


DistilBertForSequenceClassification

DistilBertForSequenceClassification

fix small shape error in comments


BertTokenizer.from_pretrained fails for local_files_only=True when added_tokens.json is missing

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.1
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.6.1810-Core
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@mfuntowicz
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): `google/bert_uncased_L-2_H-128_A-2`

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Run the following:
```
from transformers import BertTokenizer
BertTokenizer.from_pretrained('google/bert_uncased_L-2_H-128_A-2')
BertTokenizer.from_pretrained('google/bert_uncased_L-2_H-128_A-2', local_files_only=True)
```

In the Python interpreter, this produces the following error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/gscratch/cse/julianjm/anaconda3/lib/python3.7/site-packages/transformers-4.0.1-py3.8.egg/transformers/tokenization_utils_base.py", line 1747, in from_pretrained
  File "/gscratch/cse/julianjm/anaconda3/lib/python3.7/site-packages/transformers-4.0.1-py3.8.egg/transformers/file_utils.py", line 1007, in cached_path
  File "/gscratch/cse/julianjm/anaconda3/lib/python3.7/site-packages/transformers-4.0.1-py3.8.egg/transformers/file_utils.py", line 1171, in get_from_cache
ValueError: Cannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.
```

Looking more closely, I have isolated the issue to the logic [here](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L1774). In this case, the error is because the cached path for the url `https://huggingface.co/google/bert_uncased_L-2_H-128_A-2/resolve/main/added_tokens.json` cannot be found in the cache when `local_files_only=True`. This is because the URL 404s; i.e., the file does not exist.

When `local_files_only=False`, the GET returns a 404 and the tokenizer init code just ignores the missing file. However, when `local_files_only=True` and the file is not found, it throws a `ValueError` instead which is not caught.

What makes this non-trivial is that without making HTTP requests, there is no way of telling the difference between a file that doesn't exist and a file which exists but hasn't been downloaded. It seems to me that there are several potential ways of fixing the issue.

1. Ensure that all files exist. Don't let people upload incomplete sets of files (and fix the ones which are currently incomplete).
2. Recover from 404s by caching an "empty" file here. But this only works where there is a meaningful notion of "empty" file, like lists of tokens. I think this would not work for json files or serialized models.
3. Put a special kind of file in the cache which says "hey, this file isn't supposed to exist", and handle appropriately everywhere files are loaded. Potentially could throw a special error saying the file isn't supposed to exist; HTTP 404s could then be caught and re-thrown as this special error, so, the case could be handled uniformly.
4. Just log a warning for files that aren't in the cache, and treat them like 404s. Wild west, but at least if the code unexpectedly fails later the user will be able to guess the problem. Easy to implement, but will worsen the UX every time someone tries to use `local_files_only` without downloading the model first.

Option 3 seems the cleanest to me, while option 4 is what I'm shunting into my transformers egg for now so I can keep working.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

After downloading, I would expect any artifact to be loadable from cache and equivalent to the downloaded one.

<!-- A clear and concise description of what you would expect to happen. -->


Ray tune hyperparameters search error

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.0.dev0
- Platform: Linux-4.4.0-139-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

@sgugger

## Information

Model I am using (Bert, XLNet ...): Roberta-large

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: GLUE SST-2

## To reproduce

Steps to reproduce the behavior:

1. I wanted to do a hyperparameter search so I referred to https://huggingface.co/blog/ray-tune and modified the `examples/text-classification/run_glue.py` replacing the training part with
```
    def model_init():
        model = AutoModelForSequenceClassification.from_pretrained(
            model_args.model_name_or_path,
            from_tf=bool(".ckpt" in model_args.model_name_or_path),
            config=config,
            cache_dir=model_args.cache_dir,
        )
        return model
    trainer = Trainer(
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        compute_metrics=compute_metrics,
        tokenizer=tokenizer,
        # Data collator will default to DataCollatorWithPadding, so we change it if we already did the padding.
        data_collator=default_data_collator if data_args.pad_to_max_length else None,
        model_init=model_init,
    )
```
```
    # Training
    if training_args.do_train:
        from ray import tune
        import ray
        ray.init()
        best_trial = trainer.hyperparameter_search(
            hp_space=lambda _ : {"seed": tune.grid_search([31, 42, 53])},
            direction="maximize",
            backend="ray",
        )
        logger.info(" Best run %s" % str(best_trial))
```

2. Run `python run_glue.py --model_name_or_path roberta-large --do_train --do_eval --per_gpu_train_batch_size 8 --output_dir hypersearch-0 --task_name sst2 --evaluation_strategy steps --eval_steps 20 --logging_steps 10`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Then the script exited with exception:

```
Traceback (most recent call last):
  File "run_glue.py", line 428, in <module>
    main()
  File "run_glue.py", line 359, in main
    best_trial = trainer.hyperparameter_search(
  File "/data1/howard/transformers/src/transformers/trainer.py", line 1039, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/data1/howard/transformers/src/transformers/integrations.py", line 241, in run_hp_search_ray
    analysis = ray.tune.run(_objective, config=trainer.hp_space(None), num_samples=n_trials, **kwargs)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/tune.py", line 299, in run
    experiments[i] = Experiment(
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/experiment.py", line 138, in __init__
    self._run_identifier = Experiment.register_if_needed(run)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/experiment.py", line 276, in register_if_needed
    register_trainable(name, run_object)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/registry.py", line 71, in register_trainable
    _global_registry.register(TRAINABLE_CLASS, name, trainable)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/registry.py", line 124, in register
    self.flush_values()
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/registry.py", line 146, in flush_values
    _internal_kv_put(_make_key(category, key), value, overwrite=True)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/experimental/internal_kv.py", line 27, in _internal_kv_put
    updated = worker.redis_client.hset(key, "value", value)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/redis/client.py", line 3004, in hset
    return self.execute_command('HSET', name, key, value)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/redis/client.py", line 877, in execute_command
    conn.send_command(*args)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/redis/connection.py", line 720, in send_command
    self.send_packed_command(self.pack_command(*args),
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/redis/connection.py", line 712, in send_packed_command
    raise ConnectionError("Error %s while writing to socket. %s." %
redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

The script should run without errors.

## Related Issues

https://github.com/ray-project/ray/issues/2931

https://ray.readthedocs.io/en/latest/tune-usage.html#handling-large-datasets


TableQuestionAnsweringPipeline

## TableQuestionAnsweringPipeline

Introduces the `TableQuestionAnsweringPipeline` which will be used for the `TableQuestionAnswering` widget:

There are examples of usage within the documentation, but here are some others if you want to give it a spin:

### WTQ and aggregators

```py
tqa_pipeline = pipeline("table-question-answering")

data = {
    "Repository": ["Transformers", "Datasets", "Tokenizers"],
    "Stars": ["36542", "4512", "3934"],
    "Contributors": ["651", "77", "34"],
    "Programming language": ["Python", "Python", "Rust, Python and NodeJS"],
}
table = pd.DataFrame.from_dict(data)
queries = [
    "What repository has the largest number of stars?",
    "Given that the numbers of stars defines if a repository is active, what repository is the most active?",
    "What is the number of repositories?",
    "What is the average number of stars?",
    "What is the total amount of stars?"
]

outputs = tqa_pipeline(table, queries)
print(outputs)
```
This outputs the following (given that the aggregator setup is respected in the model configuration, which won't be the case until the configuration changes as proposed here are accepted):
```
[
    {'answer': 'Transformers', 'coordinates': [(0, 0)], 'cells': ['Transformers'], 'aggregator': 'NONE'},
    {'answer': 'Transformers', 'coordinates': [(0, 0)], 'cells': ['Transformers'], 'aggregator': 'NONE'},
    {'answer': 'COUNT > Transformers, Datasets, Tokenizers', 'coordinates': [(0, 0), (1, 0), (2, 0)], 'cells': ['Transformers', 'Datasets', 'Tokenizers'], 'aggregator': 'COUNT'},
    {'answer': 'AVERAGE > 36542, 4512, 3934', 'coordinates': [(0, 1), (1, 1), (2, 1)], 'cells': ['36542', '4512', '3934'], 'aggregator': 'AVERAGE'},
    {'answer': 'SUM > 36542, 4512, 3934', 'coordinates': [(0, 1), (1, 1), (2, 1)], 'cells': ['36542', '4512', '3934'], 'aggregator': 'SUM'}
]

```

Please note the aggregators, their presence in the answer when they exist, and their absence when they do not.

### SQA and sequential inference

```py
data = {'Actors': ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
        'Age': ["56", "45", "59"],
        'Number of movies': ["87", "53", "69"],
        'Date of birth': ["7 february 1967", "10 june 1996", "28 november 1967"]}
queries = ["How many movies has George Clooney played in?", "How old is he?", "What's his date of birth?"]
table = pd.DataFrame.from_dict(data)

tqa_pipeline = pipeline("table-question-answering", model="nielsr/tapas-base-finetuned-sqa", tokenizer="nielsr/tapas-base-finetuned-sqa")

outputs = tqa_pipeline(table, queries, sequential=True)
print(outputs)
```
This outputs the following:
```
[
    {'answer': '69', 'coordinates': [(2, 2)], 'cells': ['69']},
    {'answer': '59', 'coordinates': [(2, 1)], 'cells': ['59']},
    {'answer': '28 november 1967', 'coordinates': [(2, 3)], 'cells': ['28 november 1967']}
]
```
Please note the relationship between questions ("how old is he", who is "he"?) and the correct answers given by the model. One can try passing `sequential=False` and obtain vastly different results.

Here is the [documentation page](https://138478-155220641-gh.circle-artifacts.com/0/docs/_build/html/main_classes/pipelines.html#transformers.TableQuestionAnsweringPipeline).


Saving model errors

I am trying to fine-tune distillbert for a multilabel task using a V100 gpu and latest transformers from pip. When i try to save the model i get this error:
```
Traceback (most recent call last):
  File "script/fine_tune_distillbert.py", line 51, in <module>
    model.save_pretrained(ROOT_DIR)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_tf_utils.py", line 534, in save_pretrained
    self.save_weights(output_model_file)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2085, in save_weights
    hdf5_format.save_weights_to_hdf5_group(f, self.layers)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/saving/hdf5_format.py", line 640, in save_weights_to_hdf5_group
    param_dset = g.create_dataset(name, val.shape, dtype=val.dtype)
  File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/group.py", line 143, in create_dataset
    if '/' in name:
TypeError: a bytes-like object is required, not 'str'
```

The model init:

```
model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels= max_lab)
optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)
model.compile(optimizer=optimizer, loss=model.compute_loss, metrics=['accuracy']) # can also use any keras loss fn

model.fit(train_dataset.shuffle(1000).batch(256), epochs=3, batch_size=256,
          validation_data=val_dataset.shuffle(1000).batch(256))
```

The way i am saving after sucessfuly training the model is this :

```
import os
ROOT_DIR = os.path.abspath(os.curdir)
ROOT_DIR = ROOT_DIR + "/model"

tokenizer.save_pretrained(ROOT_DIR)
model.save_pretrained(ROOT_DIR)
```

Tokenizer is saved perfectly. I have tried to run the same code in a colab notebook ( with much less data) and it saves perfectly but when i use a service like `spell.ml` i get this error.

@LysandreJik


Pass kwargs to Pipeline's tokenizer call

# What does this PR do?
When calling a Pipeline, the `kwargs` argument is not passed to the tokenizer (it is actually not used at all).
I think the intended behavior is to pass it (as the base tokenizer's `__call__()` method already supports `kwargs`), and that's what this PR does.
[Related to #8180]

The call order is:

```Python3
SpecificPipeline.__call__(..., **kwargs)
# Which calls
Pipeline.__call__(..., **kwargs)
# Which calls
SpecificPipeline._parse_and_tokenize(..., **kwargs)
# Which in turn calls
self.tokenizer(...) # No kwargs in this call
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@sgugger @LysandreJik


RAGRetriever loads dataset in the default cache dir even if a different one is specified

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: latest
- Platform: any
- Python version: 3.8
- PyTorch version (GPU?): any
- Tensorflow version (GPU?): any
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 RAG: @patrickvonplaten, @lhoestq
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. load a `RagRetriever` model with `cache_dir='/mnt/.cache/huggingface/'
2. notice that the dataset is still downloaded to `'~/.cache'`

```{python
from transformers import RagRetriever
rag_retriever = RagRetriever.from_pretrained('facebook/rag-token-base', cache_dir='/mnt/.cache/huggingface')
```

## Expected behavior
The dataset is still downloaded in `'~/.cache'` even though we want it to download to the cache in '/mnt'

The reason this is happening is because, in `retrieval_rag.py`, `config.cache_dir` isn't passed through to `load_dataset` on line 273, for example


Support for private models from huggingface.co

Add a `use_auth_token` flag (or string) to all `from_pretrained` entry points, to specify token to use as Bearer authorization for remote files.

- if it's a string, use it
- If it's true, will get token from `~/.huggingface/token` (will throw if no token there)

You can test this with:

```python
model = AutoModelForMaskedLM.from_pretrained("pierric/hf-private", use_auth_token=True)
```

We'll add unit tests down the line but need to think about which environment those tests are going to hit.

⚠ For now, I decided against adding token by default to all calls if user is logged in. Let's discuss though!


Fix T5 Encoder model parallel tests


Experimental support for fairscale ShardedDDP

# What does this PR do?

This PR adds support for [FairScale](https://github.com/facebookresearch/fairscale)'s shared DDP training to save GPU memory when training distributed models. Initial tests see a nice reduction of GPU memory used indeed!

This follows the steps of the [main example](https://github.com/facebookresearch/fairscale/blob/master/benchmarks/oss.py) provided on the FairScale repo, integrating them in our Trainer API. To activate training with shared DDP, one must pass along the flag `--sharded_ddp` in a distributed launch command.

Benchmarks tried:
- a fine-tuning on MRPC with `bert_base_uncased` -> goes from 5GB per GPU to 4GB per GPU with no hurt on accuracy
- a fine-tuning on SQUAD v2 with `xlnet_large-cased` -> goes from 11.5GB per GPU to 8GB per GPU (didn't go until the end so didn't check if the accuracy was the same. Training loss seemed equivalent.)


adapting trainer.py for multiple optimizers

Hi
I was wondering if there can be an easy way to adapt trainer.py for multiple optimizers, where each optimizer is responsible for updating a part of model. thanks


Add possibility to switch between APEX and AMP in Trainer

# What does this PR do?

When PyTorch >= 1.6 is installed, Trainer is always using native AMP right now. This PR adds the option to switch between AMP and APEX, which can be useful:
- because of the memory leak in AMP fixed (fixed in 1.7.1 but present in 1.6)
- to benchmark APEX vs. AMP

It also simplifies a little bit the internal of Trainer with those.


Update notebook table and transformers intro notebook

# What does this PR do?

Update the examples table and the notebooks table to include all recent examples. Also fix the intro notebook to the transformers library, in particular, the image that was missing.

Fixes #9083


Fix Bart Shift

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Previous PR #9134 was still WIP and accidentally merged to quickly -> sorry for the many commits.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


[Bart] Correct wrong order in shift token to right in Bart

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

The previous PR #9131 implemented the replacement of -100 with pad_token after retrieving the eos_token_idx. However it should be done before to make sure the correct eos_token_id is found.

Thanks a lot @patil-suraj for spotting this.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


[Examples] Add automatic dataset splitting in language-modeling examples

# What does this PR do?

Currently, language-modeling examples support passing a HF-datasets dataset as training data. However, this dataset needs to have a `train` and `validation` split, which is not the case for many language-modeling datasets, which are just unstructured text. The updated scripts automatically partition the `train` split to create a `validation` split if it doesn't exist already, and adds `validation_split_percentage` argument to control the split ratio, set to 5% by default.


Fix typo in trainer_tf.py

# What does this PR do?
Fixes a typo in trainer_tf.py

Fixes #9053

@sgugger


[Bart] fix bart loss masking

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9123

Bart should be able to replace -100 tokens when prepping `decoder_input_ids`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Trainer: support iterable datasets for evaluation

The trainer seems to support passing iterable datasets as the `train_dataset`  (see #5829) but misses to support the same for the `eval_dataset`. I have implemented an iterable dataset for training and now I cannot use the same implementation for evaluation. This doesn't make much sense as evaluation could easily be done using the iterable dataset.

Currently the evaluation fails with the following exception:
```
ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.sampler.SequentialSampler object at 0x7f53900c50d0>
```

I suspect the below method should return None in case of an iterable dataset:

https://github.com/huggingface/transformers/blob/ef2d4cd4457a344b633173c14ca7789f18f75b59/src/transformers/trainer.py#L402-L408

Just like how it is handled for the `train_dataset`:

https://github.com/huggingface/transformers/blob/ef2d4cd4457a344b633173c14ca7789f18f75b59/src/transformers/trainer.py#L380-L400


Fix TF Transfo XL

# What does this PR do?

This PR fixes an issue in TFTransfoXL the last layer was added into the complete list of `hidden_states` while being already transposed. Then adding it at the end after all the other states have been transposed.


BartForCausalLM analogs to `ProphetNetForCausalLM`

# What does this PR do?

Implementing BartForCausalLM anologs for ProphetNetForCausalLM

Fixes #9066


[Flax] Bugfixes in `run_mlm_flax.py`

# What does this PR do?

This PR fixes a few bugs I have observed when using `run_mlm_flax.py`:
- As discussed with @mfuntowicz , `jnp.split` is a lot slower than `np.split` on the first iteration, outright hanging in my tests on simplewiki (~20MB). As this operation doesn't need to be traced. we can use `np.split` instead.
- When using a HF `datasets`, the text column was also passed to the model as input, causing a bug. The PR removes the text column in `dataset.map` to avoid this.
- Finally, using `warmup_steps = 0` (as is default) causes the Flax optimizer to output NaNs. We use 1 as a minimum value for the same warmup-less behaviour.


seq2seq finetuning scripts break before training (cannot import name ParallelMode)

File "finetune_trainer.py", line 24, in <module>
    from seq2seq_trainer import Seq2SeqTrainer
  File "/home/---/transformers/examples/seq2seq/seq2seq_trainer.py", line 35, in <module>
    from transformers.training_args import ParallelMode
ImportError: cannot import name 'ParallelMode' from 'transformers.training_args'


Predict single sentence for Glue Tasks

I have trained a custom binary classifier using `run_glue.py` and have the `pytorch_model.bin` file saved to a directory. Is there a way to predict for a given sentence and extract its label?

I know `trainer.predict(test_dataset)` does it. But I am having some trouble converting the string to the format that it takes.


Improve BERT-like models performance with better self attention

# What does this PR do?

This PR updates the way we implement the self attention layers in order to be aligned on the original BERT performance. Small breaking change, this improvement needs at least TF 2.3. This change has already been discussed with @thomwolf, and he agreed. But still needs the approval of @LysandreJik @patrickvonplaten and @sgugger

@patrickvonplaten I have removed the comment for `check_copies` in the Longformer model because I don't know enough this model to apply the proper changes, I will apply this update to one model by one model for the ones I know but can you take this one?

@jlei2 as I'm on Windows, unfortunately the GPU profiling is not yet available in WSL, can you clone this branch and be sure that everythings works like expected with your benchmark? Thanks!!

Fixes # (issue)

#6771


BART cannot accept -100 as ignored label

## Environment info

- `transformers` version: 4.0.1
- Platform: Linux
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

 Bart: @patrickvonplaten

## Information

I'm using ``BartForConditionalGeneration`` to do some natural language generation tasks. By the [doc](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration) I should be able to set -100 for some tokens to ignore. However, it would raise an out of index error.

## To reproduce

```python
from transformers import BartForConditionalGeneration, AutoTokenizer

b = BartForConditionalGeneration.from_pretrained("facebook/bart-base")
t = AutoTokenizer.from_pretrained("facebook/bart-base")

s1 = "hello hello hello hello world"
inputs = t(s1, return_tensors="pt")
label = inputs["input_ids"].clone()
label[0, 2:3] = -100

outputs = b(**inputs, labels=label)
```

Then it raise the following error:

> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 1032, in forward
>     return_dict=return_dict,
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 915, in forward
>     return_dict=return_dict,
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 564, in forward
>     x = self.embed_tokens(input_ids) * self.embed_scale
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
>     self.norm_type, self.scale_grad_by_freq, self.sparse)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
>     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
> IndexError: index out of range in self

Without giving -100 in the label, it can return the output correctly.

## Expected behavior

Should return the output correctly.


RobertaTokenizer fails to do_lower_case, different behavior between version 2 and 3

## Environment info

- `transformers` version: 3.4.0 / 2.8.0
- Platform: linux
- Python version: 3.8

### Who can help
@mfuntowicz

## Information

Tokenizer I am using: RobertaTokenizer
The tokenizer do not lower case the text even if I explicitly set do_lower_case=True. The behavior is different between version 2.8.0 and 3.4.0

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained("roberta-base", do_lower_case=True)
print(tokenizer.tokenize("Huggingface"))
```

## Expected behavior

Version 3.4.0 prints: ['Hug', 'ging', 'face']
Version 2.8.0 prints: ['h', 'ug', 'ging', 'face']


[Generation] Add generation outputs

# 🚀 Feature request

We've had multiple issues asking for the possibility to output the scores/probabilities of each token during generation, see:
https://github.com/huggingface/transformers/issues/7654
https://github.com/huggingface/transformers/issues/3891
https://github.com/huggingface/transformers/issues/8656

Also we should be able to output the models attentions `hidden_states` at each generation step, *a.k.a* make use of those model outputs:
https://github.com/huggingface/transformers/blob/c19d04623eacfbc2c452397a5eda0fde42db3fc5/src/transformers/models/bert/modeling_bert.py#L883 in generation as well.

To do so we should create a new generation output class for each "sub" generation function:

1) `GreeySearchDecoderOnlyOutput(output_ids,  logits, attentions, hidden_states)` for decoder-only models, where as `output_ids` are the current outputs of generate, `logits` will be the logit vectors at each generation step (so should be of shape `Tuple((logits_1,), ..., (logits_max_length,))`) and `attentions` and `hidden_states` should be of shape `Tuple((attentions_1,), ..., (attentions_max_length,))`. As before `attentions` and `hidden_states` will be output if a flag `output_attentions` or `output_hidden_states` iset to True and for the logits we should add a flag `output_scores`. Also we should have a `GreeySearchEncoderDecoderOutput(output_ids,  logits, encoder_attentions, decoder_attentions, encoder_hidden_states, decoder_hidden_states)` class with the respective enc and dec outputs.

2) `SampleDecoderOnlyOutput(output_ids,  probabilities, attentions, hidden_states)` -> the same outputs only that we replace the logits output with the softmax probabilities (of the same shape); same flags as in 1) and encoder-decoder class as well

3) `BeamSearchDecoderOnlyOutput(output_ids, probs, attentions, hidden_states)` -> `probs` should be this tensor: https://github.com/huggingface/transformers/blob/c19d04623eacfbc2c452397a5eda0fde42db3fc5/src/transformers/generation_utils.py#L1235 at each step same flags as in 1) and encoder-decoder class as well

Each output class should be derived from https://github.com/huggingface/transformers/blob/c19d04623eacfbc2c452397a5eda0fde42db3fc5/src/transformers/file_utils.py#L1306 just as the model output classes are in https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_outputs.py .

A PR should start with the "GreedySearchOutput" model classes and add this to `generation_utils.py` => then we should add the  three flags to both `generate()` and `greedy_search()`. Then `SampleOutput` and `BeamSerachOutput` should be added. The PR should also include good documentation for each of the outputs as it is the case for the current model outputs.

## Your contribution

I'm happy to help the contributor throughout the PR :-)


Fix tf2.4

# What does this PR do?

Fix the tests to make them compliant with the new TF 2.4


Which dataset is used for training GPT, GPT2 from scratch?

Hi,

I checked the model card of GPT and GPT2, but I can't find the dataset which was used for training.
Where can I find the datasets which is used for?

https://huggingface.co/openai-gpt
https://huggingface.co/gpt2


Different inference results of a keras including transformer model on TPU vs CPU?

transformers version: 4.0.0
Platform: Linux-4.9.0-11-amd64-x86_64-with-debian-9.11
Python version: 3.7.9
PyTorch version (GPU?): 1.6.0a0+bf2bbd9 (False)
Tensorflow version (GPU?): 2.3.1 (False)
Using GPU in script?: No
Using distributed or parallel set-up in script?: Distributed

I am building a Keras model which consists of a TFRobertaModel with 2 custom heads on top. One is a QA head which outputs span predictions and the other is a binary classification head. I train the model on TPUs and everything works fine with prediction and inference on the TPU with great model performance. The issue I am having is loading the saved model and/or weights and doing inference on CPU. I am getting completely different results compared to inference on the TPU.

I save the model using model.save and the weights as well with models.save_weights and it doesn't matter which one I load, I get the same results (using tf.keras.models.load_model). I do get a warning that there was an error saving the state of the optimiser which is initialized at random on model loading. I figure this is not an issue with keras since I compile the model with a tf.keras.optimizers.Adam which should be saved with model.save with a keras only model. I have also tried building the model from scratch and only loading the saved weights but I get the same results.

This is a convoluted problem to reproduce but I was wondering if you had any pointers on how to debug this or if this was a known problem. Here is a sample of the model output on TPU vs CPU:

TPU - the first 10 binary answer predictions:

array([[9.9994159e-01, 5.8382138e-05],
       [9.9990284e-01, 9.7181561e-05],
       [9.9995410e-01, 4.5917721e-05],
       [9.9996519e-01, 3.4784229e-05],
       [9.9975628e-01, 2.4374224e-04],
       [9.9997389e-01, 2.6103005e-05],
       [9.9995828e-01, 4.1662061e-05],
       [9.9998319e-01, 1.6824890e-05],
       [7.0182599e-02, 9.2981732e-01],
       [9.9993420e-01, 6.5814391e-05]], dtype=float32)

CPU - the first 10 binary answer predictions:

array([[0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ],
       [0.06942184, 0.9305781 ]], dtype=float32)

The model building and compilation:

```
dropout=False
def create_model():

    input_shape = (None,)
    model = TFRobertaForQuestionAnswering.from_pretrained(model_path, from_pt=True, trainable=True)

    input_ids = tf.keras.layers.Input(shape=input_shape, dtype=np.int32, name='input_ids')
    attention_mask = tf.keras.layers.Input(shape=input_shape, dtype=np.int32, name='attention_mask')
    token_type_ids = tf.keras.layers.Input(shape=input_shape, dtype=np.int32, name='token_type_ids')

    outputs = model.roberta(input_ids,
                            attention_mask=attention_mask,
                            token_type_ids=token_type_ids,
                            output_hidden_states=True)

    seq_output = outputs[0]
    logits = model.qa_outputs(seq_output)
    start_logits, end_logits = tf.split(logits, 2, axis=-1)
    start_logits = tf.squeeze(start_logits, axis=-1)
    end_logits = tf.squeeze(end_logits, axis=-1)

    #BINARY_ANSWER
    concat_hidden_layers = tf.concat(tuple([outputs.hidden_states[i] for i in [-4, -3, -2, -1]]), axis=-1)
    pooled_output = concat_hidden_layers[:, 0, :]

    binary_answer_logits = tf.keras.layers.Dense(768,
            kernel_initializer=tf.keras.initializers.truncated_normal(stddev=0.02),
            activation="tanh",
            name="dense_tanh")(pooled_output)

    if dropout:
        binary_answer_logits = tf.keras.layers.Dropout(0.1)(binary_answer_logits)

    binary_answer_probs = tf.keras.layers.Dense(2, activation='softmax', name="binary_answer")(binary_answer_logits)

    keras_model = Model(inputs={'input_ids':input_ids,
                                'attention_mask':attention_mask,
                                'token_type_ids':token_type_ids},
                        outputs={'start_logits':start_logits,
                                 'end_logits':end_logits,
                                 'binary_answer_probs':binary_answer_probs})

    return keras_model

with strategy.scope():
    keras_model = create_model()
    optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
    keras_model.compile(loss={'start_logits':compute_loss,
                              'end_logits':compute_loss,
                              'binary_answer_probs':tf.keras.losses.binary_crossentropy},
                                optimizer=optimizer)
```

I also get completely different results for span predictions on TPU vs CPU which is not surprising, seeing how different the binary prediction is. Any help or pointers are appreciated.

## Expected behavior

The same inference results on TPU vs CPU.


Tapas v4 (tres)

model card

Related to pretrained model cards

Here we are again, opening a new PR based on the former (#8988) which had some Github issues.
cc @LysandreJik


Roberta training crashing due to position_id embedding

I've been trying to work out why I keep getting a CUDA assert in a specific mini-batch when training RoBERTa from scratch. I finally tracked it down after switching to CPU.

I don't understand why `padding_idx` is added to `incremental_indices` below? - _edit: I do in that the embedding needs a padding mask but it I'm not sure it's the correct way to do it._

In my case padding_idx=3. And one of my input_ids rows was truncated. Say I have input_ids = [[4,5,6],[4,3,3]], this results in mask=[[1,1,1],[1,0,0]] and incremental index=[[1,2,3],[1,0,0]]. Adding padding_idx then produces [[4,5,6],[4,3,3]].

The issue is `self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)` so for any sequences which are truncated adding anything to the indices results in an index which is greater than the embedding dim.

Perhaps you can argue that max_position_embeddings is supposed to be larger than the largest possible sequence so this doesn't happen? There is a check in `run_mlm.py` that `data_args.max_seq_length > tokenizer.model_max_length` but it seems that in actual fact to avoid a very hard to track down error you must have tokenizer.model_max_length < data_args.max_seq_length

```
def create_position_ids_from_input_ids(input_ids, padding_idx):
    """
    Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols
    are ignored. This is modified from fairseq's `utils.make_positions`.

    Args:
        x: torch.Tensor x:

    Returns: torch.Tensor
    """
    # The series of casts and type-conversions here are carefully balanced to both work with ONNX export and XLA.
    mask = input_ids.ne(padding_idx).int()
    incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
    return incremental_indices.long() + padding_idx
```


[doc] pytorch native amp leak fix landed in 1.7.1

update README with good news that the leak fix has been applied to pytorch-1.7.1 and not just 1.8.

Reference: https://github.com/pytorch/pytorch/issues/48049#issuecomment-742790722

@LysandreJik


Fix stack overflow

Currently calling `n_sequences` on a `BatchEncoding` results in a stack overflow.


Some Models do not support gradient checkpointing

Thanks for this wonderful library.

I found some models do not support gradient_checkpointing, which I believe is a very important feature. For example,

ElectraModel: ElectraConfig has no gradient_checkpointing option but ElectraModel will use gradient_checkpointing if config.gradient_checkpointing = True
DistillBERT: DistillBertConfig has no gradient_checkpointing option and DistillBertModel does not support gradient_checkpointing.

I assume all transformer-based models should be able to support gradient_checkpointing.


Add BORT

Hi,

this PR adds the recently introduced BORT model from @adewynter and Daniel J. Perry from the Alexa team into Transformers.

----

BORT was introduced in the [Optimal Subarchitecture Extraction For BERT](https://arxiv.org/abs/2010.10499).

Details about BORT:

> We extract an optimal subset of architectural parameters for the BERT architecture from Devlin et al. (2018) by applying recent breakthroughs in algorithms for neural architecture search. This optimal subset, which we refer to as "Bort", is demonstrably smaller, having an effective (that is, not counting the embedding layer) size of 5.5% the original BERT-large architecture, and 16% of the net size. Bort is also able to be pretrained in 288 GPU hours, which is 1.2% of the time required to pretrain the highest-performing BERT parametric architectural variant, RoBERTa-large (Liu et al., 2019), and about 33% of that of the world-record, in GPU hours, required to train BERT-large on the same hardware. It is also 7.9x faster on a CPU, as well as being better performing than other compressed variants of the architecture, and some of the non-compressed variants: it obtains performance improvements of between 0.3% and 31%, absolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.

This should fix #8135 :hugs:

---

ToDo tasks:

* [x] Upload models (both PyTorch and TensorFlow model) to model hub
* [x] Add conversion script from Gluonnlp to Transformers
* [x] Enable unit tests (they are working and just wait for the model upload)