When encoding text to feature vectors - Would be awesome to be able to use the simplest tokenizer with a split on spaces

# 🚀 Feature request

When using the `DistilBertTokenizer` (or `BertTokenizer`) I would love to tokenize my text by simply splitting it on spaces:
```
['also', 'du', 'fängst', 'an', 'mit', 'der', 'Stadtrundfahrt']
```
instead of the default behavior, which is splitting it into sub-parts:
```
['also', 'du', 'f', '##ängst', 'an', 'mit', 'der', 'Stadt', '##rund', '##fahrt']
```

## Motivation
That's needed in order to have feature vector length to be the same as the number of words in the text, so that we can have 1-to-1 correspondence between words and their features

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

I have seen that such tokenization can be done using `tokenizer.basic_tokenizer` and I have tried to use it to encode the text:
```
from transformers import DistilBertModel, DistilBertTokenizer
import torch

text_str = "also du fängst an mit der Stadtrundfahrt"

# create DistilBERT tokenizer and model
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-german-cased')
model = DistilBertModel.from_pretrained('distilbert-base-german-cased')

# check if tokens are correct
tokens = tokenizer.basic_tokenizer.tokenize(text_str)
print("Tokens: ", tokens)

# Encode the curent text
input_ids = torch.tensor(tokenizer.basic_tokenizer.encode(text_str)).unsqueeze(0)
outputs = model(input_ids)
last_hidden_states = outputs[0]
```

But this code raises an error because `BasicTokenizer` does not have attribute 'encode' yet:
```
Traceback (most recent call last):
  File "/home/tarask/Desktop/Work/Code/Git/probabilistic-gesticulator/my_code/data_processing/annotations/feat/test.py", line 15, in <module>
    input_ids = torch.tensor(tokenizer.basic_tokenizer.encode(text_str)).unsqueeze(0)
AttributeError: 'BasicTokenizer' object has no attribute 'encode'
```


Fixed docs for the shape of `scores` in `generate()`

# What does this PR do?
Fixed the shape of `scores` from `generate()` outputs to be `max_length-1` for all output classes. The first token `decoder_start_token_id` is not generated and thus no scores.

[Scores in generate()](https://discuss.huggingface.co/t/scores-in-generate/3450)
[Generation Probabilities: How to compute probabilities of output scores for GPT2](https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175)

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Play around with mask-filling of original model

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Cannnot train Roberta: 2 different errors

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No (Single GPU) --> **COLAB**

### Who can help
I am not sure, since the error is very vague and untraceable
## Information

Model I am using (Bert, XLNet ...): Roberta

The problem arises when using:
* [ ] the official example scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

It is a private dataset, so I am not at liberty to share it. However, I can provide a clue as to how the `csv` looks like:-

,ID,Text,Label
......................

> I do not think there can be anything wrong with the DataFrame as I am taking data from specific columns and converting them to numpy arrays for the rest of the steps in the HF "Fine-tuning" guide.

## To reproduce

Steps to reproduce the behavior:

```
!git clone https://github.com/huggingface/transformers.git
!cd transformers
!pip install -e .

train_text = list(train['Text'].values)
train_label = list(train['Label'].values)

val_text = list(val['Text'].values)
val_label = list(val['Label'].values)

from transformers import RobertaTokenizer, TFRobertaForSequenceClassification
import tensorflow as tf

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = TFRobertaForSequenceClassification.from_pretrained('roberta-base')

train_encodings = tokenizer(train_text, truncation=True, padding=True)
val_encodings = tokenizer(val_text, truncation=True, padding=True)

train_dataset = tf.data.Dataset.from_tensor_slices((
    dict(train_encodings),
    train_label
))
val_dataset = tf.data.Dataset.from_tensor_slices((
    dict(val_encodings),
    val_label
))

```

All this code is common. Howver, now there is a difference in errors depending upon the training method.

###Training using `trainer`

Code:
```

from transformers import TFTrainingArguments, TFTrainer

training_args = TFTrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
)

with training_args.strategy.scope():
    model = TFRobertaForSequenceClassification.from_pretrained("roberta-base")

trainer = TFTrainer(
    model=model,                         # the instantiated Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset             # evaluation dataset
)

trainer.train()
```
**ERROR:-**

```
All model checkpoint layers were used when initializing TFRobertaForSequenceClassification.

Some layers of TFRobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-52-f86f69d7497b> in <module>()
     22 )
     23
---> 24 trainer.train()

10 frames

/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py in train(self)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    826     tracing_count = self.experimental_get_tracing_count()
    827     with trace.Trace(self._name) as tm:
--> 828       result = self._call(*args, **kwds)
    829       compiler = "xla" if self._experimental_compile else "nonXla"
    830       new_tracing_count = self.experimental_get_tracing_count()

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    869       # This is the first call of __call__, so we have to initialize.
    870       initializers = []
--> 871       self._initialize(args, kwds, add_initializers_to=initializers)
    872     finally:
    873       # At this point we know that the initialization is complete (or less

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    724     self._concrete_stateful_fn = (
    725         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 726             *args, **kwds))
    727
    728     def invalid_creator_scope(*unused_args, **unused_kwds):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2967       args, kwargs = None, None
   2968     with self._lock:
-> 2969       graph_function, _ = self._maybe_define_function(args, kwargs)
   2970     return graph_function
   2971

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3359
   3360           self._function_cache.missed.add(call_context_key)
-> 3361           graph_function = self._create_graph_function(args, kwargs)
   3362           self._function_cache.primary[cache_key] = graph_function
   3363

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3204             arg_names=arg_names,
   3205             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3206             capture_by_value=self._capture_by_value),
   3207         self._function_attributes,
   3208         function_spec=self.function_spec,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    988         _, original_func = tf_decorator.unwrap(python_func)
    989
--> 990       func_outputs = python_func(*func_args, **func_kwargs)
    991
    992       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    632             xla_context.Exit()
    633         else:
--> 634           out = weak_wrapped_fn().__wrapped__(*args, **kwds)
    635         return out
    636

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in bound_method_wrapper(*args, **kwargs)
   3885     # However, the replacer is still responsible for attaching self properly.
   3886     # TODO(mdan): Is it possible to do it here instead?
-> 3887     return wrapped_fn(*args, **kwargs)
   3888   weak_bound_method_wrapper = weakref.ref(bound_method_wrapper)
   3889

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    975           except Exception as e:  # pylint:disable=broad-except
    976             if hasattr(e, "ag_error_metadata"):
--> 977               raise e.ag_error_metadata.to_exception(e)
    978             else:
    979               raise

TypeError: in user code:

    /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:669 distributed_training_steps  *
        nb_instances_in_batch = self._compute_nb_instances(batch)
    /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:681 _compute_nb_instances  *
        nb_instances = tf.reduce_sum(tf.cast(labels != -100, dtype=tf.int32))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:1786 tensor_not_equals
        return gen_math_ops.not_equal(self, other, incompatible_shape_error=False)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_math_ops.py:6412 not_equal
        name=name)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:531 _apply_op_helper
        repr(values), type(values).__name__, err))

    TypeError: Expected string passed to parameter 'y' of op 'NotEqual', got -100 of type 'int' instead. Error: Expected string, got -100 of type 'int' instead.

```

###Using Native `Tensorflow` code (from official example)
CODE:

from transformers import TFRobertaForSequenceClassification
```

model = TFRobertaForSequenceClassification.from_pretrained('roberta-base')

optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)
model.compile(optimizer=optimizer, loss=model.compute_loss) # can also use any keras loss fn
model.fit(train_dataset.shuffle(1000).batch(16), validation_data=val.shuffle(1000).batch(16), epochs=3, batch_size=16)
```

**ERROR:-**
```
All model checkpoint layers were used when initializing TFRobertaForSequenceClassification.

Some layers of TFRobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-51-a13d177c752e> in <module>()
      5 optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)
      6 model.compile(optimizer=optimizer, loss=model.compute_loss) # can also use any keras loss fn
----> 7 model.fit(train_dataset.shuffle(1000).batch(16), validation_data=val.shuffle(1000).batch(16), epochs=3, batch_size=16)

/usr/local/lib/python3.6/dist-packages/pandas/core/generic.py in __getattr__(self, name)
   5139             if self._info_axis._can_hold_identifiers_and_holds_name(name):
   5140                 return self[name]
-> 5141             return object.__getattribute__(self, name)
   5142
   5143     def __setattr__(self, name: str, value) -> None:

AttributeError: 'DataFrame' object has no attribute 'shuffle'

```

This is very surprising since the error are pretty different and I can't find many fixes online. I tested the datatypes of the input data and it seems to check out.

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
 The model to start training on this `SequenceClassification` task and achieve good accuracy on it.


Error: "Transformers CLI tool: error: unrecognized arguments: kvantorium-small"  while deploying machine learning model to hugging face profile

 I work with hugging face on Google Colab
My process of training and tuning the model looks like this. The model is successfully trained and saved in the models folder, but when I am going to upload it to my huggingface repository, an error occurs

````
tokenizer = `PreTrainedTokenizerFast(tokenizer_file='/content/bert-datas7.json')
tokenizer.add_special_tokens({'pad_token': '[PAD]'})

dataset = LineByLineTextDataset(tokenizer=tokenizer, file_path="/content/For_ITMO (1).txt",block_size=128)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=mlm, mlm_probability=0.15
    )
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="models/kvantorium-small",
    overwrite_output_dir=True,
    num_train_epochs=1000,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset)

trainer.train()
````
#8480
When I am going to upload it to my huggingface repository, an error occurs, why is it displayed for me? And how to specify the correct documentation? My folder in the Google Colab directory is called "/ content / models / kvantorium-small" This is the folder where I save the model after training

````
!transformers-cli login
!transformers-cli upload  "/content/models/kvantorium-small"

2021-02-07 16:56:32.089450: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
usage: transformers-cli <command> [<args>]
Transformers CLI tool: error: unrecognized arguments: /content/models/kvantorium-small
````
Is this a google collab problem and how can I rewrite the request or fix the error?

Sorry in advance if my issue seems to be incorrect to you, I'm new to git.
I also attached my data and my tokenizer file
[For_ITMO (1).txt](https://github.com/huggingface/transformers/files/5939713/For_ITMO.1.txt)

[For_ITMO.txt-vocab (1) (1).txt](https://github.com/huggingface/transformers/files/5939719/For_ITMO.txt-vocab.1.1.txt)


Add CharacterBERT model [WIP]

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9061

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
*I'm still missing the updates in the general documentation.*

- [x] Did you write any new necessary tests?
*Some of the tests are currently failing. This is due to CharacterBERT having a different input shape than bert (batch size, seq length, token length) instead of (batch size, seq length). Also, there are some other tests that are related to reshaping the embedding which fail for the same reason. I did not fix these test as that would mean changing things in the way the common tests are currently working. For all other cases, I tried my best to implement tests that by adapting those from the BERT suite (these pass).*

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors which may be interested in your PR.

@LysandreJik please have a look at this PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


implementing tflxmertmodel integration test

# What does this PR do?
this PR implement an integration test for TFLxmertmodel as requested in #9954

@LysandreJik


[example] run_ner.py raised error: IndexError: Target 3 is out of bounds.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0 or 4.2.2
- Platform: MacOS
- Python version: 3.6
- PyTorch version (GPU?):   1.7.1
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## To reproduce

Steps to reproduce the behavior:

1. bash [run.sh](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run.sh) to [run_ner.py](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py)

## Error

```
[INFO|trainer.py:837] 2021-02-07 22:22:31,755 >> ***** Running training *****
[INFO|trainer.py:838] 2021-02-07 22:22:31,755 >>   Num examples = 14041
[INFO|trainer.py:839] 2021-02-07 22:22:31,755 >>   Num Epochs = 3
[INFO|trainer.py:840] 2021-02-07 22:22:31,755 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:841] 2021-02-07 22:22:31,755 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:842] 2021-02-07 22:22:31,755 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:843] 2021-02-07 22:22:31,755 >>   Total optimization steps = 5268
  0%|                                                                                | 0/5268 [00:00<?, ?it/s]Traceback (most recent call last):
  File "run_ner.py", line 443, in <module>
    main()
  File "run_ner.py", line 377, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/Users/bytedance/transformers/src/transformers/trainer.py", line 940, in train
    tr_loss += self.training_step(model, inputs)
  File "/Users/bytedance/transformers/src/transformers/trainer.py", line 1304, in training_step
    loss = self.compute_loss(model, inputs)
  File "/Users/bytedance/transformers/src/transformers/trainer.py", line 1334, in compute_loss
    outputs = model(**inputs)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/bytedance/transformers/src/transformers/models/bert/modeling_bert.py", line 1701, in forward
    loss = loss_fct(active_logits, active_labels)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 962, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 3 is out of bounds.
Exception ignored in: <bound method tqdm.__del__ of   0%|                                                                                | 0/5268 [00:01<?, ?it/s]>
Traceback (most recent call last):
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/tqdm/std.py", line 1086, in __del__
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/tqdm/std.py", line 1270, in close
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/tqdm/std.py", line 572, in _decr_instances
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/site-packages/tqdm/_monitor.py", line 51, in exit
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/threading.py", line 521, in set
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/threading.py", line 364, in notify_all
  File "/Users/bytedance/opt/anaconda3/lib/python3.6/threading.py", line 347, in notify
TypeError: 'NoneType' object is not callable

```


run_ner.py fails when loading a model/checkpoint from a directory

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-3.10.0-957.1.3.el7.x86_64-x86_64-with-centos-7.6.1810-Core
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 --> @sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): XLMRobertaModel

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behaviour:

The run_ner.py seems to be unable to load models from a directory. The following command works as expected:

```
python transformers/examples/token-classification/run_ner.py --dataset_name conll2003 --model_name_or_path xlm-roberta-base --output_dir output --num_train_epochs 10 --per_device_train_batch_size 32 --per_device_eval_batch_size 32 --learning_rate 5e-05 --seed 29 --save_steps 9223372036854775807 --do_train --do_eval --overwrite_cache --overwrite_output_dir  --fp16
```

However, if we load 'xlm-roberta-base', we save the model to a directory, and we try to run the script with the directory set as the model path, the script fails. (Same behaviour if we use XLMRobertaForTokenClassification instead of XLMRobertaModel in step 1)

1.

```
from transformers import XLMRobertaTokenizer,  XLMRobertaModel
tokenizer =  XLMRobertaTokenizer.from_pretrained('xlm-roberta-base')
model =  XLMRobertaModel.from_pretrained('xlm-roberta-base')
spath = "models/xlmroberta"
tokenizer.save_pretrained(spath)
model.save_pretrained(spath)
```

2.
```
python transformers/examples/token-classification/run_ner.py --dataset_name conll2003 --model_name_or_path models/xlmroberta --cache_dir models/xlmroberta --output_dir output --num_train_epochs 10 --per_device_train_batch_size 32 --per_device_eval_batch_size 32 --learning_rate 5e-05 --seed 29 --do_train --do_eval --overwrite_cache --overwrite_output_dir
```

Error message (Running the command in CPU to be able to see it):

```
INFO|tokenization_utils_base.py:1688] 2021-02-07 14:38:00,146 >> Model name 'models/xlmroberta' not found in model shortcut name list (xlm-roberta-base, xlm-roberta-large, xlm-roberta-large-finetuned-conll02-dutch, xlm-roberta-large-finetuned-conll02-spanish, xlm-roberta-large-finetuned-conll03-english, xlm-roberta-large-finetuned-conll03-german). Assuming 'models/xlmroberta' is a path, a model identifier, or url to a directory containing tokenizer files.
[INFO|tokenization_utils_base.py:1721] 2021-02-07 14:38:00,146 >> Didn't find file models/xlmroberta/tokenizer.json. We won't load it.
[INFO|tokenization_utils_base.py:1721] 2021-02-07 14:38:00,147 >> Didn't find file models/xlmroberta/added_tokens.json. We won't load it.
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,147 >> loading file models/xlmroberta/sentencepiece.bpe.model
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,148 >> loading file None
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,148 >> loading file None
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,148 >> loading file models/xlmroberta/special_tokens_map.json
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,148 >> loading file models/xlmroberta/tokenizer_config.json
[INFO|modeling_utils.py:1025] 2021-02-07 14:38:02,120 >> loading weights file models/xlmroberta/pytorch_model.bin
[INFO|modeling_utils.py:1143] 2021-02-07 14:38:13,379 >> All model checkpoint weights were used when initializing XLMRobertaForTokenClassification.

[WARNING|modeling_utils.py:1146] 2021-02-07 14:38:13,380 >> Some weights of XLMRobertaForTokenClassification were not initialized from the model checkpoint at models/xlmroberta and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
100%|###################################################################################################################################################################################################################################################################################################################################################################################################| 15/15 [00:01<00:00,  7.55ba/s]
100%|#####################################################################################################################################################################################################################################################################################################################################################################################################| 4/4 [00:00<00:00,  8.69ba/s]
100%|#####################################################################################################################################################################################################################################################################################################################################################################################################| 4/4 [00:00<00:00,  8.25ba/s]
[INFO|trainer.py:429] 2021-02-07 14:38:18,120 >> The following columns in the training set don't have a corresponding argument in `XLMRobertaForTokenClassification.forward` and have been ignored: tokens, ner_tags, chunk_tags, id, pos_tags.
[INFO|trainer.py:429] 2021-02-07 14:38:18,121 >> The following columns in the evaluation set don't have a corresponding argument in `XLMRobertaForTokenClassification.forward` and have been ignored: tokens, ner_tags, chunk_tags, id, pos_tags.
[INFO|trainer.py:721] 2021-02-07 14:38:18,122 >> Loading model from models/xlmroberta).
[INFO|configuration_utils.py:443] 2021-02-07 14:38:18,123 >> loading configuration file models/xlmroberta/config.json
[INFO|configuration_utils.py:481] 2021-02-07 14:38:18,125 >> Model config XLMRobertaConfig {
  "architectures": [
    "XLMRobertaModel"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "xlm-roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.3.0.dev0",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 250002
}

[INFO|modeling_utils.py:1025] 2021-02-07 14:38:18,126 >> loading weights file models/xlmroberta/pytorch_model.bin
[INFO|modeling_utils.py:1143] 2021-02-07 14:38:30,381 >> All model checkpoint weights were used when initializing XLMRobertaForTokenClassification.

[WARNING|modeling_utils.py:1146] 2021-02-07 14:38:30,381 >> Some weights of XLMRobertaForTokenClassification were not initialized from the model checkpoint at models/xlmroberta and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|trainer.py:832] 2021-02-07 14:38:30,394 >> ***** Running training *****
[INFO|trainer.py:833] 2021-02-07 14:38:30,394 >>   Num examples = 14041
[INFO|trainer.py:834] 2021-02-07 14:38:30,394 >>   Num Epochs = 10
[INFO|trainer.py:835] 2021-02-07 14:38:30,394 >>   Instantaneous batch size per device = 32
[INFO|trainer.py:836] 2021-02-07 14:38:30,394 >>   Total train batch size (w. parallel, distributed & accumulation) = 32
[INFO|trainer.py:837] 2021-02-07 14:38:30,395 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:838] 2021-02-07 14:38:30,395 >>   Total optimization steps = 4390
  0%|                                                                                | 0/4390 [00:00<?, ?it/s]Traceback (most recent call last):
  File "third_party/transformers/examples/token-classification/run_ner.py", line 454, in <module>
    main()
  File "third_party/transformers/examples/token-classification/run_ner.py", line 388, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/transformers/trainer.py", line 931, in train
    tr_loss += self.training_step(model, inputs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/transformers/trainer.py", line 1295, in training_step
    loss = self.compute_loss(model, inputs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/transformers/trainer.py", line 1325, in compute_loss
    outputs = model(**inputs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 1349, in forward
    loss = loss_fct(active_logits, active_labels)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 962, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 3 is out of bounds.
```


Installing tf2.0 in my env but still get ImportError in my code

Hi, I have installed tf2.0 in my env and I followed the readme which says if you have installed the tf2.0 you can just run pip install transformers. But I got Error: "ImportError: cannot import name 'TFBertForSequenceClassification' from 'transformers' (unknown location)"
My code:

from transformers import BertTokenizer, TFBertForSequenceClassification
import tensorflow as tf
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
inputs["labels"] = tf.reshape(tf.constant(1), (-1, 1))  # Batch size 1
outputs = model(inputs)
loss = outputs.loss
logits = outputs.logits

Picture(I have tf2.0 and transformers):
![image](https://user-images.githubusercontent.com/67748964/107148058-172b8c80-698c-11eb-803f-182ea99c23c1.png)

And I also used conda install -c huggingface transformers but it still doesn't work. Could you help me, Thanks!


I'm having an unusual issue on one computer and I'm hoping that someone out there has seen something like this before.  This issue does not exist on another computer.  Both computers are windows 10 machines, using python 3.6.4, virtualenv, and visual studio code as the ide

I have created a clean virtualenv and installed only datasets.  I get an import error when I try to import any of the built in functions, list_datasets, load_dataset, etc.

I have tried installing different versions of datasets and I have tried installing datasets from source instead of through pip with no success.

Has anyone seen anything like this?  Any suggestions for something I can try to help debug?

Here's the code:
import sys

```
print (sys.version)

from datasets import list_datasets

for i, ds in enumerate(list_datasets()):
  print (f"{i}: {ds}")
```

Here is the output:
3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)]
Traceback (most recent call last):
  File "c:\code\dataset_test\main.py", line 5, in <module>
    from datasets import list_datasets
ImportError: cannot import name 'list_datasets'

Here is the pip list:
Package            Version
------------------ ---------
certifi            2020.12.5
chardet            4.0.0
dataclasses        0.8
datasets           1.2.1
dill               0.3.3
idna               2.10
importlib-metadata 3.4.0
multiprocess       0.70.11.1
numpy              1.19.5
object-detection   0.1
pandas             1.1.5
pip                21.0.1
pyarrow            3.0.0
python-dateutil    2.8.1
pytz               2021.1
requests           2.25.1
setuptools         53.0.0
six                1.15.0
tqdm               4.49.0
typing-extensions  3.7.4.3
urllib3            1.26.3
wheel              0.36.2
xxhash             2.0.0
zipp               3.4.0


Can you give some suggestion about add features with input_ids to token-classification model ?

Hi, I want to add POS TAG labels as input with input_ids as auxiliary feature feature in NER model.
How can i get the entry in the forward function ?
And i review some implementations about model use bert model to perform NER  such as
https://github.com/monologg/JointBERT/blob/master/model/modeling_jointbert.py
It seems if i want to ad this feature (POS TAG) i must rewrite the forward function in BertModel
and seems i should train the edit model from scratch.
Can you give some suggestions about edit model structure and partial weights load from
huggingface pre-trained-models ?


[s2s examples] Replace -100 token ids with the tokenizer pad_id for compute_metrics

# What does this PR do?

This PR is a small fix that replaces the -100 token ids with the tokenizer pad_id when decoding sequences to compute metrics as was done in [this HF blog post](https://huggingface.co/blog/warm-starting-encoder-decoder)

## When does this problem occur?

When running `examples/seq2seq/finetune_trainer.py` with padding to the `max_seq_len`, an error is thrown at the evaluation step:

```
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1004, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 96, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1442, in evaluate
    output = self.prediction_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1601, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "/eai/transformers/examples/seq2seq/utils.py", line 98, in translation_metrics
    pred_str, label_str = decode_pred(pred)
  File "/eai/transformers/examples/seq2seq/utils.py", line 85, in decode_pred
    label_str = tokenizer.batch_decode(pred.label_ids, skip_special_tokens=True)
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3070, in batch_decode
    return [
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3071, in <listcomp>
    self.decode(
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3109, in decode
    return self._decode(
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 495, in _decode
    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
OverflowError: out of range integral type conversion attempted
```

This is because in the prediction loop, the labels will be padded with -100 if the prediction or labels have different sequence length https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L1637.


BertGenerationTokenizer provides an unexpected value for BertGenerationModel

- `transformers` version: 4.2.2
- PyTorch version (GPU?): 1.7.0+cu101
- tokenizers: @n1t0, @LysandreJik

## Information

in both models BertGenerationEncoder,  BertGenerationDecoder, there's no need for `token_type_ids` however the BertGenerationTokenizer provides it, this issue will be raised if you want to input the tokenizer results directly with `**`,

and if it meant to be like this, and the user should be aware of this behaviour, I think a change should be in the documentation.

Note: Another issue with BertGenerationTokenizer is the necessity of sentencepiece module, do you prefer that it should for the user to install it separately or it should be included in transformers dependencies.


[s2s examples] dataset porting

We need to port these to jsonlines and ideally make them part of dataset hub:
https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz
https://cdn-datasets.huggingface.co/translation/wmt_en_de.tgz
https://cdn-datasets.huggingface.co/summarization/cnn_dm_v2.tgz

The problem is that nobody knows how these came to be - they are pre-processed but it's unclear how.

@sshleifer, do you by chance remember how these were created? If we put those on the new datasets hub - it'd be good to explain how these are different from normal wmt datasets.

Also do you remember which wmtXX they came from?

Thank you!

----

The resolution is here: https://github.com/huggingface/transformers/issues/10044#issuecomment-779555741


[s2s examples] README.md fixes

This PR:
* fixes a cl arg typo
* clarifies that it's jsonlines format and not json that's expected
* adds a link explaining jsonlines

@sgugger, how do we apply the auto-re-wrapping to examples README.md files? Currently the new file is all very long lines. Thank you!

@patil-suraj, @sgugger


Pegasus ONNX format?

I tried to convert Pegasus to the ONNX format using [this ](https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb#scrollTo=foYlXrSksR_v) guide but it failed. Can Pegasus be converted to the ONNX format or is that not possible yet?


[s2s] Can't mix --fp16 and --device cpu

This PR fixes this user-side error:

```
RuntimeError: "threshold_cpu" not implemented for 'Half'
```
reported at https://github.com/huggingface/transformers/issues/10040

This combination `--fp16 --device cpu` is not possible, as explained here: https://github.com/pytorch/pytorch/issues/48245#issuecomment-730714723
and it's not really usable anyway - it takes minutes to run fp16 on cpu while it takes a split second on the gpu.

full trace:
```
export DATA_DIR=wmt_en_ro; PYTHONPATH=../../src ./run_eval.py t5-base     $DATA_DIR/val.source t5_val_generations.txt     --reference_path $DATA_DIR/val.target     --score_path enro_bleu.json     --task translation_en_to_ro     --n_obs 100     --device cpu     --fp16     --bs 32

Traceback (most recent call last):
  File "./run_eval.py", line 176, in <module>
    run_generate(verbose=True)
  File "./run_eval.py", line 137, in run_generate
    runtime_metrics = generate_summaries_or_translations(
  File "./run_eval.py", line 67, in generate_summaries_or_translations
    summaries = model.generate(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/generation_utils.py", line 847, in generate
    model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/generation_utils.py", line 379, in _prepare_encoder_decoder_kwargs_for_generation
    model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/models/t5/modeling_t5.py", line 946, in forward
    layer_outputs = layer_module(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/models/t5/modeling_t5.py", line 683, in forward
    hidden_states = self.layer[-1](hidden_states)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/models/t5/modeling_t5.py", line 299, in forward
    forwarded_states = self.DenseReluDense(forwarded_states)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/models/t5/modeling_t5.py", line 258, in forward
    hidden_states = F.relu(hidden_states)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/functional.py", line 1206, in relu
    result = torch.relu(input)
RuntimeError: "threshold_cpu" not implemented for 'Half'
```

@sgugger, @patil-suraj

Fixes: https://github.com/huggingface/transformers/issues/10040


seq2seq: fail gracefully when predicting using --device cpu and --fp16

When using the recommended seq2seq evaluation procedure in https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md :

```
export DATA_DIR=wmt_en_ro
./run_eval.py t5-base \
    $DATA_DIR/val.source t5_val_generations.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path enro_bleu.json \
    --task translation_en_to_ro \
    --n_obs 100 \
    --device cuda \
    --fp16 \
    --bs 32
```

If `--device cuda` is switched to `--device cpu` then it eventually fails with a pytorch error if `--fp16` is also enabled:

```
"threshold_cpu" not implemented for 'Half'
```

It seems that fp16 and cpu evaluation may currently be incompatible.


[trainer] deepspeed bug fixes and tests

This PR:
* fixes a bug with `model.no_sync()` which is not supported by DeepSpeed - we had the test but it's not running on CI
* fixes a bug with when `--train` is not used - have to `.to(device)` in that case - reported here (https://github.com/huggingface/transformers/issues/9996#issuecomment-774232901)
* splits the deepspeed tests into its own dedicated file and will slowly start to build it up - but it's good enough for now - especially since we are going to switch over `run_seq2seq.py`, except it's not fully ready yet for adoption.
* adds a new test which doesn't use `--train`

@sgugger


[examples s2s] run_seq2seq.py tweaks

Would it be possible to sync the new `run_seq2seq.py` with `./finetune_trainer.py` outputs?

Before:
```
2021-02-05 12:59:55 | INFO | __main__ | ***** train metrics *****
2021-02-05 12:59:55 | INFO | __main__ |   epoch = 1.0
2021-02-05 12:59:55 | INFO | __main__ |   train_n_objs = 60
2021-02-05 12:59:55 | INFO | __main__ |   train_runtime = 9.7768
2021-02-05 12:59:55 | INFO | __main__ |   train_samples_per_second = 6.137
2021-02-05 12:59:55 | INFO | __main__ | *** Evaluate ***
[INFO|trainer.py:1600] 2021-02-05 12:59:55,434 >> ***** Running Evaluation *****
[INFO|trainer.py:1601] 2021-02-05 12:59:55,434 >>   Num examples = 10
[INFO|trainer.py:1602] 2021-02-05 12:59:55,434 >>   Batch size = 1

2021-02-05 13:00:00 | INFO | __main__ | ***** val metrics *****
2021-02-05 13:00:00 | INFO | __main__ |   epoch = 1.0
2021-02-05 13:00:00 | INFO | __main__ |   val_bleu = 33.3125
2021-02-05 13:00:00 | INFO | __main__ |   val_gen_len = 50.1
2021-02-05 13:00:00 | INFO | __main__ |   val_loss = inf
2021-02-05 13:00:00 | INFO | __main__ |   val_n_objs = 10
2021-02-05 13:00:00 | INFO | __main__ |   val_runtime = 4.7266
2021-02-05 13:00:00 | INFO | __main__ |   val_samples_per_second = 2.116
2021-02-05 13:00:00 | INFO | __main__ | *** Predict ***

```
With the new script:
```
02/05/2021 13:00:41 - INFO - __main__ -   ***** Train results *****
02/05/2021 13:00:41 - INFO - __main__ -     epoch = 1.0
02/05/2021 13:00:41 - INFO - __main__ -     train_runtime = 1.33
02/05/2021 13:00:41 - INFO - __main__ -     train_samples_per_second = 3.008
02/05/2021 13:00:41 - INFO - __main__ -   *** Evaluate ***
***** Running Evaluation *****
  Num examples = 100
  Batch size = 32
02/05/2021 13:00:42 - INFO - __main__ -   ***** Eval results *****
02/05/2021 13:00:42 - INFO - __main__ -     epoch = 1.0
02/05/2021 13:00:42 - INFO - __main__ -     eval_bleu = 1.3059269919149237
02/05/2021 13:00:42 - INFO - __main__ -     eval_gen_len = 17.66
02/05/2021 13:00:42 - INFO - __main__ -     eval_loss = 5.084951400756836
02/05/2021 13:00:42 - INFO - __main__ -     eval_runtime = 0.7079
02/05/2021 13:00:42 - INFO - __main__ -     eval_samples_per_second = 141.261
```

As you can see:
1. the metrics numbers aren't rounded up
2. missing `*_n_obj` in metrics
3. logging is inconsistent.

   the old script has its own issues with log consistency, this one introduces its own inconsistencies.

   To make it easy to read the results - the user-targeted output should ideally be aligned on one side and not in 2 columns - i.e. left column date, etc. right column information.  In the new script `***** Running Evaluation *****` is missing the logger prefix.

Thank you!

@patil-suraj, @sgugger


[examples] make run scripts executable

For consistently and convenience of not needing to type `python` to run a script this PR adds to all `examples/*/run_*py` scripts a python shebang and make the scripts executable.

Was done with:
```
perl -0777 -pi -e 's|^|#!/usr/bin/env python\n|' */run_*.py
chmod a+x */run_*.py
```

@patil-suraj , @sgugger


[s2s examples] convert existing scripts to run_seq2seq.py from finetune_trainer.py

Examples

Which is related to examples in general

As `transformers` examples are evolving it seems that the good old `finetune_trainer.py` is going to be moved into unmaintained `examples/legacy/` area, and `run_seq2seq.py` is to be the new king, so let's automate this process

Assuming your cmd script is `process.txt` (and replace with the file names that you have (one or many), let's auto-adjust it:

1. Renames
```
# main name and args rename
perl -pi -e 's|finetune_trainer|run_seq2seq|g; s#--n_(train|val)#--max_$1_samples#g; \
s|--src_lang|--source_lang|g; s|--tgt_lang|--target_lang|g; s|--eval_beams|--num_beams|'  process.txt

# drop no longer supported args
perl -pi -e 's|--freeze_embeds||; s|--test_max_target_length[ =]+\d+||;' process.txt

```

2. t5 auto-adding prefix has been dropped, so you need to add it manually, e.g.:
```
--source_prefix "translate English to Romanian: "
```
otherwise the results would be terrible.

3. Datasets are different

a. need to convert the normal dataset into jsonlines (unless the data is already on datasets hub)
instructions are: https://huggingface.co/docs/datasets/loading_datasets.html#json-files

b. new arguments:

instead of
```
            --data_dir {data_dir}
```
now you need:
```
            --train_file {data_dir}/train.json
            --validation_file {data_dir}/val.json
```

Here's is an example conversion script for the `wmt_en_ro` dataset:

```
# convert.py
import io
import json
import re

src_lang, tgt_lang = ["en", "ro"]

for split in ["train", "val", "test"]:
    recs = []
    fout = f"{split}.json"
    with io.open(fout, "w", encoding="utf-8") as f:
        for type in ["source", "target"]:
            fin = f"{split}.{type}"
            recs.append([line.strip() for line in open(fin)])
        for src, tgt in zip(*recs):
            out = {"translation": { src_lang: src, tgt_lang: tgt } }
            x = json.dumps(out, indent=0, ensure_ascii=False)
            x = re.sub(r'\n', ' ', x, 0, re.M)
            f.write(x + "\n")
```

Or if you find an existing dataset in `datasets`, you can supply it instead of the `--data_dir` arg as following:

```
--dataset_name wmt16 --dataset_config_name ro-en
```

Here is the full conversion table from the previously recommended 4 datasets in the `examples/seq2seq` folder:
* [x] `--data_dir wmt_en_de` => `--dataset_name wmt14 --dataset_config "de-en"` or if you want the highest score use: `--dataset_name wmt14-en-de-pre-processed`
* [x] `--data_dir wmt_en_ro` => --dataset_name wmt16 --dataset_config "ro-en"`
* [x] `--data_dir cnn_dm` => `--dataset_name cnn_dailymail --dataset_config "3.0.0"`
* [x] `--data_dir xsum` => `--dataset_name xsum`

You will find more details [here](https://github.com/huggingface/transformers/issues/10044#issuecomment-779555741)

----

t5-specific changes: from https://github.com/huggingface/transformers/pull/10133#issuecomment-778071812

    1. Use the same dataset
    2. if using T5 manually pass the `prefix` argument,
    3. manually copy the `task_specific_parms` to `config`


Cannot import DataCollatorForSeq2Seq from Transformers library

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.2
- Platform: Ubuntu
- Python version: 3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 1.7.1 - GPU : T4
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:

### Who can help
@patil-suraj
@sgugger

## Information

Model I am using (Bert, XLNet ...): mT5

The problem arises when using:
* [ ] the official example scripts: run_seq2seq.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Translation
* [ ] my own task or dataset: My own json dataset

## To reproduce

Steps to reproduce the behavior:

```
python run_seq2seq.py \
    --model_name_or_path google/mt5-small \
    --do_train \
    --do_eval \
    --task translation_en_to_fa \
    --train_file Persian/seq2seq_train.json \
    --validation_file Persian/seq2seq_val.json \
    --output_dir translation_task_output \
    --overwrite_output_dir \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generate \
    --text_column text_column_name \
    --max_source_length 64 \
    --max_target_length 64 \
    --max_train_samples 10240 \
    --max_val_samples 512 \
    --source_lang en \
    --target_lang fa \
    --eval_beams 1 \
    --source_prefix "translate English to Persian: "
```

## Error:
```
File "run_seq2seq.py", line 31, in <module>
    from transformers import (
ImportError: cannot import name 'DataCollatorForSeq2Seq' from 'transformers' (unknown location)

```


Truncate max length if needed in all examples

# What does this PR do?

As pointed out in #10015, most examples will let the tokenzation and training run when `tokenizer.model_max_length < max_seq_length` and the default value is sometimes bigger than the max length for some models (like BertTweet). This PR addresses that.

Fixes #10015


A few fixes in the documentation

# What does this PR do?

This PR fixes a few things in the documentation mainly:
- version tags to reflect the latest patch-release (4.2.2
- documents decode and batch_decode in `PreTrainedTokenizer` and `PreTrainedTokenizerFast`.

Fixes #10019


generation length always equal to 20 when using run_seq2seq.py script

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.4.0-197-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten  @LysandreJik

Model I am using : any seq2seq model

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: cnn-dailymail / xsum
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
python examples/seq2seq/run_seq2seq.py     --model_name_or_path t5-small     --do_train     --do_eval     --task summarization     --dataset_name cnn_dailymail  --dataset_config_name 3.0.0     --output_dir tmp     --per_device_train_batch_size=4     --per_device_eval_batch_size=4     --overwrite_output_dir  --max_steps 250 --eval_steps 249  --save_steps 249  --max_val_samples 250  --max_target_length 100 --val_max_target_length 100 --predict_with_generate
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

```
02/05/2021 16:36:34 - INFO - __main__ -   ***** Eval results *****
02/05/2021 16:36:34 - INFO - __main__ -     epoch = 0.01
02/05/2021 16:36:34 - INFO - __main__ -     eval_gen_len = 19.0
02/05/2021 16:36:34 - INFO - __main__ -     eval_loss = 2.064879894256592
02/05/2021 16:36:34 - INFO - __main__ -     eval_rouge1 = 21.837379907220157
02/05/2021 16:36:34 - INFO - __main__ -     eval_rouge2 = 7.506564948396541
02/05/2021 16:36:34 - INFO - __main__ -     eval_rougeL = 18.074704390199546
02/05/2021 16:36:34 - INFO - __main__ -     eval_rougeLsum = 17.99211046381146
02/05/2021 16:36:34 - INFO - __main__ -     eval_runtime = 19.079
02/05/2021 16:36:34 - INFO - __main__ -     eval_samples_per_second = 13.103
```
`eval_gen_len` is never exceeding 20 for some reason. I tried with several models and datasets


## Environment info

- `transformers` version: 4.2.2
- Platform: Linux-5.8.0-40-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@sgugger @patil-suraj
Models:
-tuner007/pegasus_paraphrase

## Information

trying to import "PegasusForCausalLM" from transformers and I got AttributeError: module 'transformers' has no attribute 'PegasusForCausalLM'

## To reproduce

```python

from transformers import PegasusForCausalLM, AutoConfig
model_name = "tuner007/pegasus_paraphrase"
output_logits = True
output_hidden_states = True
p_config =AutoConfig.from_pretrained(
            model_name,
            output_hidden_states=output_hidden_states,
            output_logits=output_logits)
pegasus = PegasusForCausalLM.from_pretrained(model_name, config=p_config)

```
```
AttributeError: module transformers has no attribute PegasusForCausalLM
```


Check copies match full class/function names

# What does this PR do?

@LysandreJik pointed out a failure in the `check_copies` script that came from the fact that when looking for the code of an object, the script was matching the first line it found that begun with the name of the object. So it would match `DebertaLayer` with `DebertaLayerNorm` and compare those.

This PR fixes that.


Override Default Params on QnA Pipeline

How should we override the default params of QnA Model in Pipelines?

I tried the below but it throws an error?!

![image](https://user-images.githubusercontent.com/22409996/107048748-f586a580-67d1-11eb-88fe-f78901d99734.png)


custom JSON data breaks run_seq2seq.py

## Environment info
- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.19.0-11-amd64-x86_64-with-debian-10.6
- Python version: 3.7.3
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
 @sgugger, @patil-suraj

## Information

I'm trying to use mt5 model (but same happens with the example t5-small)

The problem arises when using:
* [ x ] the official example scripts: (give details below)

## To reproduce

When I try to run example/seq2se1/run_seq2seq.py with my own data files, the following error occurs:

```Downloading and preparing dataset json/default-f1b6ef8723ed4d49 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/xxxxxx/.cache/huggingface/datasets/json/default-f1b6ef8723ed4d49/0.0.0/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514...
Traceback (most recent call last):
  File "/home/xxxxx/.cache/huggingface/modules/datasets_modules/datasets/json/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514/json.py", line 82, in _generate_tables
    parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 247, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: JSON parse error: Column() changed from object to array in row 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "examples/seq2seq/run_seq2seq.py", line 537, in <module>
    main()
  File "examples/seq2seq/run_seq2seq.py", line 287, in main
    datasets = load_dataset(extension, data_files=data_files)
  File "/home/xxxxx/.local/lib/python3.7/site-packages/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/home/xxxxx/.local/lib/python3.7/site-packages/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/xxxxx/.local/lib/python3.7/site-packages/datasets/builder.py", line 604, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/xxxx/.local/lib/python3.7/site-packages/datasets/builder.py", line 959, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
  File "/home/xxxx/.local/lib/python3.7/site-packages/tqdm/std.py", line 1166, in __iter__
    for obj in iterable:
  File "/home/xxxxxxxx/.cache/huggingface/modules/datasets_modules/datasets/json/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514/json.py", line 88, in _generate_tables
    f"Not able to read records in the JSON file at {file}. "
AttributeError: 'list' object has no attribute 'keys'
```

I'm using the following json format:
```
[
  {
  "title": "summarised text",
   "body": "text to be summarized"
  },
{
   "title": "summarised text",
   "body": "text to be summarized"
  }
]
```

I pass the --text_column body --summary_column title to the script and I can't understand why it's breaking.


Bump minimum Jax requirement to 2.8.0

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

According to https://github.com/google/jax/issues/5374 this should fix #10017

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


T5 doubling training time per iteration from save_steps to save_steps (1st 100 steps 33s/it - then, 75s/it)

## Environment info

- `transformers` version: 4.3.0.dev0
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1 (YES)
- Tensorflow version (GPU?):
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?:

### Who can help
@patrickvonplaten @patil-suraj @sgugger

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [x] the official example scripts: (give details below)
The problem arises when using examples/seq2seq/run_seq2seq.py
* [ ] my own modified scripts: (give details below)

The problem arises when trying to fine-tune t5 in a text2text task using the official scripts (run_seq2seq.py). The first problem is that after the first checkpoint saved, the training becomes super slow, taking twice the time it took until that checkpoint. The second problem is that when you try to load a model from one of those checkpoints, it's like the model has increased in size (not directly, but when processing batches it uses much more memory). Let me explain myself. If you start with t5-large, in a P100 16GB gpu I can fit around 350 sequence length, 52 target length, 2 train batch size per device. However, if I start from one of the checkpoints saved (which are also t5-large, just a little bit more trained) I cannot even fit batch size 1, seq length 256, target length 50, and this is really strange.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
A summarization dataset

## To reproduce

Steps to reproduce the behavior:

1. Get any summarization or any text2text dataset.
2. Train a t5-large using examples/seq2seq/run_seq2seq.py and set save_steps to 100
3. Wait until the 100 first steps have been completed, you'll see that with a batch size of 64 (using gradient accumulation) it takes 33s/iter approx.
4. After that, when the first checkpoint is created, the training will re-start, but this time using 75s/iter, doubling its time. Over the course of the rest of the training, you'll see this same timing, it never goes back to 33s/iter.
5. Then, try to adapt this model to another dataset or to the same dataset itself, and appreciate that it's unfeasible to train with the previously specified parameters, which shouldn't happen because this model should use the same memory as the original t5. However, it's not that the model itself is bigger when loaded in the gpu, what happens is that when loading batches of tensors and processing them, memory requirements exceed significantly the ones required by the same training setup changing my trained model for t5-large (the only thing that changes is the model).

## Expected behavior

It's expected that training time (iterations /s) is approx. the same during the whole training process, it's not supposed to double due to (apparently) no reason. Moreover, it's expected that when you load a re-trained t5-large you can fine-tune it with the same training setup (batch size etc) as the one used for t5-large.


Check TF ops for ONNX compliance

# What does this PR do?

This PR aims to check if a model is compliant with ONNX Opset12 by adding a quick test and a script. The script is only for testing a saved model while the quick test aims to be run over a manually built graph. For now, only BERT is forced to be compliant with ONNX, but the test can be unlocked for any other model.

The logic can also be extended to any other framework/SDK we might think of, such as TFLite or NNAPI.


Datasets library not suitable for huge text datasets

## Environment info

- `transformers` version: 4.3.0.dev0
- Platform: Ubuntu 18
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@n1t0, @LysandreJik @patrickvonplaten @sgugger

## Information

Model I am using (Bert, XLNet ...):

BERT; but the problem arises before using it.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Get 187GB (or other big-sized) dataset
2. Try to tokenize it.
3. Wait until your whole server crashes due to memory or disk (probably this last one).

The code used to train the tokenizer is:

```{python}
from argparse import ArgumentParser

from datasets import load_dataset

from transformers import AutoTokenizer

# luego se puede cambiar esto con load_from_disk

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("--train_file", type=str, required=True, help="Train file with data.")
    parser.add_argument("--val_file", type=str, required=True, help="Val file with data.")
    parser.add_argument("--tokenizer_path", type=str, required=True, help="Path to tokenizer.")
    parser.add_argument("--num_workers", type=int, required=False, default=40, help="Number of workers for processing.")
    parser.add_argument("--save_path", type=str, required=True, help="Save path for the datasets.")
    args = parser.parse_args()
    data_files = {"train": args.train_file, "val": args.val_file}
    print("Loading dataset...")
    datasets = load_dataset("text", data_files=data_files)
    print("Loading tokenizer...")
    tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path)
    column_names = datasets["train"].column_names
    text_column_name = "text" if "text" in column_names else column_names[0]

    def tokenize_function(examples):
        # Remove empty lines
        examples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]
        return tokenizer(
            examples["text"],
            padding="max_length",
            truncation=True,
            max_length=512,
            # We use this option because DataCollatorForLanguageModeling (see below) is more efficient when it
            # receives the `special_tokens_mask`.
            return_special_tokens_mask=True,
        )
    print("Tokenizing dataset...")
    tokenized_datasets = datasets.map(
        tokenize_function,
        batched=True,
        num_proc=args.num_workers,
        remove_columns=[text_column_name],
        load_from_cache_file=False,
    )
    print("Saving to disk...")
    tokenized_datasets.save_to_disk(args.save_path)
```

## Expected behavior

It's expected that the tokenized texts occupy less space than pure texts, however it uses approx 2 orders of magnitude more disk, making it unfeasible to pre-train a model using Datasets library and therefore examples scripts from Transformers.


Accessing language modeling script checkpoint model and tokenizer for finetuning

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Red Hat Enterprise Linux Server 7.9 (Maipo)
- Python version:Python 3.7.9
- PyTorch version (GPU?):1.7.1
- Tensorflow version (GPU?):2.4.1
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. I was trying to further pretrain the xlm-roberta model on custom domain dataset using run_mlm.py. The checkpoints got saved in the checkpoint directory
2.  But when I try to access the tokenizer or model. I get the error message.
3. When i tried to find the solution for the tokenizer issue it was trying to find the config.json file in the checkpoint folder but only tokenizer_config.json was available and it had parameter "name_or_path" instead of "model_type"

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
Tokenizer error:
ValueError: Unrecognized model in /output_dir/checkpoint-1000/. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: led, blenderbot-small, retribert, mt5, t5, mobilebert, distilbert, albert, bert-generation, camembert, xlm-roberta, pegasus, marian, mbart, mpnet, bart, blenderbot, reformer, longformer, roberta, deberta, flaubert, fsmt, squeezebert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm-prophetnet, prophetnet, xlm, ctrl, electra, encoder-decoder, funnel, lxmert, dpr, layoutlm, rag, tapas

Model error:
RuntimeError: Error(s) in loading state_dict for XLMRobertaForSequenceClassification:
	size mismatch for roberta.embeddings.position_ids: copying a param with shape torch.Size([1, 514]) from checkpoint, the shape in current model is torch.Size([1, 512]).
	size mismatch for roberta.embeddings.word_embeddings.weight: copying a param with shape torch.Size([250002, 768]) from checkpoint, the shape in current model is torch.Size([30522, 768]).
	size mismatch for roberta.embeddings.position_embeddings.weight: copying a param with shape torch.Size([514, 768]) from checkpoint, the shape in current model is torch.Size([512, 768]).
	size mismatch for roberta.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([1, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The tokenizer and model should load from the saved checkpoint folder


Added integration tests for Pytorch implementation of the FlauBert model

Fixes #9950

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik


Clarify QA pipeline output based on character

Fixes https://github.com/huggingface/transformers/issues/10013


Protobuf

## Environment info
- `transformers` version: 4.2.2
- Platform: aws/codebuild/amazonlinux2-x86_64-standard:3.0 AND Windows-10-10.0.17763-SP0
- Python version: 3.8.3 AND 3.8.7
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help
@thomwolf @LysandreJik

Models:

- T-Systems-onsite/cross-en-de-roberta-sentence-transformer

Packages:
- pipenv
- sentence-transformers

## Information

Model I am using (Bert, XLNet ...): T-Systems-onsite/cross-en-de-roberta-sentence-transformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a new empty project with pipenv
2. Install sentence-transformers
3. Call SentenceTransformer('T-Systems-onsite/cross-en-de-roberta-sentence-transformer')

```
Traceback (most recent call last):
  File "C:/Source/pythonProject/main.py", line 4, in <module>
    SentenceTransformer('T-Systems-onsite/cross-en-de-roberta-sentence-transformer')
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\sentence_transformers\SentenceTransformer.py", line 87, in __init__
    transformer_model = Transformer(model_name_or_path)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\sentence_transformers\models\Transformer.py", line 31, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, cache_dir=cache_dir, **tokenizer_args)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 385, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\tokenization_utils_base.py", line 1768, in from_pretrained
    return cls._from_pretrained(
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\tokenization_utils_base.py", line 1841, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\models\xlm_roberta\tokenization_xlm_roberta_fast.py", line 133, in __init__
    super().__init__(
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\tokenization_utils_fast.py", line 89, in __init__
    fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\convert_slow_tokenizer.py", line 659, in convert_slow_tokenizer
    return converter_class(transformer_tokenizer).converted()
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\convert_slow_tokenizer.py", line 301, in __init__
    requires_protobuf(self)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\file_utils.py", line 467, in requires_protobuf
    raise ImportError(PROTOBUF_IMPORT_ERROR.format(name))
ImportError:
XLMRobertaConverter requires the protobuf library but it was not found in your environment. Checkout the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment.
```

## Expected behavior

Somehow the protobuf dependency doesn't get installed properly with Pipenv and when I try initializing a SentenceTransformer Object with the T-Systems-onsite/cross-en-de-roberta-sentence-transformer it crashes. It can be resolved by manually installing Protobuf. I saw, that it is in your dependencies. This might be a Pipenv or SentenceTransformer issue as well but I thought I would start with you folks.

The error occured on our Cloud instance as well as on my local windows machine. If you think the issue is related to another package please let me know, then I will contact them 😊

Thanks a lot


Tokenizer Batch decoding  of predictions obtained from model.generate in t5

How to do batch decoding of sequences obtained from model.generate in t5? Is there a function available for batch decoding in tokenizer `tokenizer.batch_decode_plus` similar to batch enocding `tokenizer.batch_encode_plus`?


Integrate DeBERTa v2(the 1.5B model surpassed human performance on Su…

# What does this PR do?
Integrate DeBERTa v2
1. Add DeBERTa XLarge model, DeBERTa v2 XLarge model, XXLarge model

|Model               | Parameters| MNLI-m/mm|
|----------------- |------------ | ---------------|
|Base                    |140M        |88.8/88.6       |
|Large                  |400M        |91.3/91.1        |
|[XLarge](https://huggingface.co/microsoft/deberta-xlarge)              |750M          |91.5/91.2        |
|[V2-XLarge](https://huggingface.co/microsoft/deberta-v2-xlarge)         |900M         |91.7/91.6        |
|**[V2-XXLarge](https://huggingface.co/microsoft/deberta-v2-xxlarge)**|1.5B            |**91.7/91.9**  |

The 1.5B XXLarge-V2 model is the model that surpass human performance and T5 11B on [SuperGLUE](https://super.gluebenchmark.com/leaderboard) leaderboard.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


python utils/check_repo.py fails

on master after making sure I got all the deps updated (from `make style/quality/fixup`)

```
No library .py files were modified
running deps_table_update
updating src/transformers/dependency_versions_table.py
python utils/check_copies.py
python utils/check_table.py
python utils/check_dummies.py
python utils/check_repo.py
Checking all models are properly tested.
2021-02-04 14:36:09.588141: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "utils/check_repo.py", line 487, in <module>
    check_repo_quality()
  File "utils/check_repo.py", line 479, in check_repo_quality
    check_all_models_are_tested()
  File "utils/check_repo.py", line 251, in check_all_models_are_tested
    modules = get_model_modules()
  File "utils/check_repo.py", line 165, in get_model_modules
    modeling_module = getattr(model_module, submodule)
  File "src/transformers/file_utils.py", line 1488, in __getattr__
    value = self._get_module(name)
  File "src/transformers/models/bert/__init__.py", line 134, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "src/transformers/models/bert/modeling_flax_bert.py", line 20, in <module>
    import flax.linen as nn
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/flax/__init__.py", line 36, in <module>
    from . import core
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/flax/core/__init__.py", line 15, in <module>
    from .frozen_dict import FrozenDict, freeze, unfreeze
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/flax/core/frozen_dict.py", line 19, in <module>
    import jax
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/jax/__init__.py", line 22, in <module>
    from .api import (
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/jax/api.py", line 37, in <module>
    from . import core
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/jax/core.py", line 31, in <module>
    from . import dtypes
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/jax/dtypes.py", line 31, in <module>
    from .lib import xla_client
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/jax/lib/__init__.py", line 60, in <module>
    from jaxlib import cusolver
ImportError: cannot import name 'cusolver' from 'jaxlib' (/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/jaxlib/__init__.py)
make: *** [Makefile:28: extra_quality_checks] Error 1
```


Feature-extraction pipeline to return Tensor

# 🚀 Feature request

Actually, to code of the feature-extraction pipeline
`transformers.pipelines.feature-extraction.FeatureExtractionPipeline l.82` return a `super().__call__(*args, **kwargs).tolist()`

Which gives a list[float] (or list[list[float]] if list[str] in input)

I guess it's to be framework agnostic, but we can specify `framework='pt'` in the pipeline config so I was expecting a `torch.tensor`.

Could we add some logic to return tensors ?

# Motivation
Features will be used as input of other models, so keeping them as tensors (even better on GPU) would be profitable.

Thanks in advance for the reply,

Have a great day.


Do not allow fine tuning with sequence size larger than during training

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

I just wasted some hours chasing a problem see #10010  that I think could be avoided with the following simple solution:

Do not allow `max_seq_length` to be higher than `max_position_embeddings`.

Most models are built with 512 and, as such, this problem doesn't happen too often. It so happens that BERTweet was trained with 130. The code allows it to run with `max_seq_length` higher than 130 and it ends up with cryptic cuda errors down the pipe.


Update doc for pre-release

# What does this PR do?

This PR puts the default version of the doc for the pre-release.


[Question] Pipeline QA start_index

I got a dictionary with 'score', 'start', 'end' and 'answer'. I want to use the 'start' and 'end' index. But the index of 'start' and 'end' are usually greater than length of encoding? Code:
text = text
question = quesiton
pipeline = pipeline('question-answering' , model = model)
answers = pipeline(context = data, question = questions)
print(answers)
{'score': 0.7909670472145081, 'start': 6192, 'end': 6195, 'answer': '111'}
but the length of context + question is smaller than 6000. How can I use 'start_index' and 'end_index' to verify that the result is '111'?