[Flax] Add other BERT classes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds the other BERT model classes for Flax.

Also the following checkpoints have been uploaded for Flax:
- https://huggingface.co/bert-base-cased
- https://huggingface.co/bert-large-cased
- https://huggingface.co/bert-base-uncased
- https://huggingface.co/bert-large-uncased
- https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Transformers QA Online Demo is not working: https://huggingface.co/qa/

I am trying to recreate ELI5 but unable to find enough information @yjernite Can you Please help.

Please let me know if I can help

Thanks


Merge trainers

# What does this PR do?

This PR merges the specific `SageMakerTrainer` into the main `Trainer` to make all the scripts work directly with model parallelism. In passing, a few internal breaking changes:
- `is_sagemaker_distributed_available` is renamed to `is_sagemaker_dp_enabled` since it's about data parallelism and not specifically distributed training, and it's True when the user has activated it, not when it's merely "available"
- in the ParallelMode enum, the case `SAGEMAKER_DISTRIBUTED` is renamed as well (but it wasn't used anywhere).

Both only concern the internals of the library and no public API is breaking.


Reproducing DistilRoBERTa

I've been trying to retrain DistilRoBERTa from the information given [here](https://huggingface.co/distilroberta-base) along with the example code/documentation [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation).

I'm a bit unclear on the exact configuration used to train the DistilRoBERTa model. I have been assuming it uses the same configuration as the DistilBERT model with minor changes, though some things, such as the loss coefficients are still a bit ambiguous.

**Would it be possible to share the exact command/configuration to train DistilRoBERTa?**

I've been able to replicate DistilRoBERTa to similar evaluation MLM perplexity but there still seems to be a small but statistically significant difference, I can share the full config if it's helpful.

Thank you!


accelerate scripts for question answering and qa with beam search

# What does this PR do?
Adding example script for question answering and question answering beam search using accelerate library .

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.  @sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Add more metadata to the user agent

# What does this PR do?

This PR adds a bit more metadata to the user agent to allow us to have more statistics on the usage, more precisely, it registers:
- the type of the file asked on the hub ("config", "tokenizer", "model" or "model_card")
- the framework used for the model ("pytorch", "tensorflow" or "flax"). Note that this is the framework actually used even in the case of a conversion (so if download the PyTorch checkpoint but use it to instantiate a Flax model, it will be "flax")
- for a tokenizer, whether it's fast or slow (like from the framework it checks the class used at the end, not the files downloaded)
- whether the Auto API was used or not
- if the instantiation came from a given pipeline or not
- if the instantiation came from the CI or not (by using a specific env variable)

There is no personal data collected but if a user wants to deactivate this behavior, the `DISABLE_TELEMETRY` env variable can be set to any truthy value and none of this will be shared.


added py7zr

# What does this PR do?

This PR adds `py7zr` to use `samsum` as a dataset.

```python
[1,14]<stdout>:    use_auth_token=use_auth_token,
[1,14]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 448, in prepare_module
[1,14]<stdout>:    f"To be able to use this {module_type}, you need to install the following dependencies"
[1,14]<stdout>:ImportError: To be able to use this dataset, you need to install the following dependencies['py7zr'] using 'pip install py7zr' for instance'
```


Fixed a bug where the `pipeline.framework` would actually contain a fully qualified model.

# What does this PR do?

We simply forgot to change the call for this one when this landed:
https://github.com/huggingface/transformers/pull/10888

It's odd that tests didn't catch that. Should we add some ?
(It's a pretty edgy test case, but it does run within the API).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[GPT Neo] defaults for max length and sampling

# What does this PR do?

Update defaults , `max_length=50` and `do_sample=True`


GPT Neo few fixes

# What does this PR do?

- update checkpoint names
- auto model


[BigBird] Fix big bird gpu test

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

`torch.randint(...)` does not seem to be reproducible across versions and devices

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


improved sagemaker documentation for git_config and examples

# What does this PR do?

This PR improves Amazon SageMaker documentation to make it more clear how `git_config` works with `examples/`. Related to #10957.


Gradient checkpointing in Wav2Vec2

Hi,
Has anyone managed to fine-tune a **Wav2Vec2** model on long audio recordings which cannot fit into a GPU even with `batch_size=1`?  I tried out to set `gradient_checkpointing=true`, but it didn't help to solve the _CUDA Out of Memory Error_. Could it mean that gradient checkpointing does not work properly with **Wav2Vec2** models or are there other tricks needed to be added to the fine-tuning script in addition to the gradient checkpointing?


pkg_resources' working_set caching breaks transformers import on google colab

## Environment info

- `transformers` version: 4.4.2
- Platform: google colab
- Python version: 3.7.10
- PyTorch version (GPU?): n/a
- Tensorflow version (GPU?):  n/a
- Using GPU in script?:  n/a
- Using distributed or parallel set-up in script?:  n/a

### Who can help

CC @stas00 as this was implemented in #8645

## To reproduce

You can find a complete example in [this google colab](https://colab.research.google.com/drive/1WT7GSd4uk9TLle9q9ftRNFy4BLZWCvZa?usp=sharing), also exported to [this gist](https://gist.github.com/konstin/e42d8f428fa11ba389e31be69cdc5646).

To reproduce, first [create a new google colab notebook](https://colab.research.google.com/#create=true). Let's install recent transformers and tqdm versions in it:

```
!pip install -U pip
!pip install -U "transformers<5.0.0,>=4.0.0" "tqdm<5.0.0,>=4.45.0"
```

This currently install transformers 4.4.2 and tqdm 4.59.0.

Surprisingly, now running `import transformers` fails. We get an error in pkg_resources, which is looking for the .dist-info of tqdm 4.41.1, when the installed version is 4.59.0:

```
[...]
/usr/local/lib/python3.7/dist-packages/pkg_resources/__init__.py in _get(self, path)
   1609
   1610     def _get(self, path):
-> 1611         with open(path, 'rb') as stream:
   1612             return stream.read()
   1613

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/dist-packages/tqdm-4.41.1.dist-info/METADATA'
```

The cause is that pkg_resources uses the cached [WorkingSet](https://setuptools.readthedocs.io/en/latest/pkg_resources.html#workingset-objects), which contains the state before the pip install. We can confirm this by recreating pkg_resources' cache manually:

```python
import pkg_resources

pkg_resources.working_set = pkg_resources.WorkingSet()
```

Afterwards, importing transformers works.

The above example is the minimized version of our real [notebooks examples](https://github.com/sacdallago/bio_embeddings/tree/develop/notebooks):

```python
!pip install -U pip
!pip install -U bio_embeddings[all]

from bio_embeddings.embed import SeqVecEmbedder # This line fails with the tqdm .dist-info not found error
```

## Expected behavior

transformers should use the actual installed versions for checking compatibility instead of pkg_resources cache. This could be achieved e.g. by using [importlib_metadata](https://github.com/python/importlib_metadata) instead of pkg_resources [here](https://github.com/huggingface/transformers/blob/1c06240e1b3477728129bb58e7b6c7734bb5074e/src/transformers/utils/versions.py#L80) or by recreating pkg_resources` cache with `pkg_resources.working_set = pkg_resources.WorkingSet()` before checking versions.

I've used the following snippet to check that importlib_metadata works, which prints `4.41.1` and `4.59.0`:

```python
import pkg_resources
import importlib_metadata

print(pkg_resources.get_distribution("tqdm").version)
print(importlib_metadata.version("tqdm"))
```

I can prepare a pull request for either solution.


compute perplexity using a custom metric function

Hello,

I am trying to replicate the "On the Cross-lingual Transferability of Monolingual Representations" paper from Artetxe et al. and I am using the code you're providing, specifically [run_mlm.py]
(https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py)

I wanted to log the perplexity to tensorboard during the evaluation step. I found out that the best option is to add a custom compute_metrics function in the trainer that uses the evaluation results (predictions and target) to compute perplexity. However, I didn't manage to do that because I couldn't understand what the output of predictions represents.

I am really new to NLP and your help is very much appreciated.


fix md file to avoid evaluation crash

# What does this PR do?

Fix the crash due to the memory usage in the instructions for model evaluation in `FINE_TUNE_XLSR_WAV2VEC2.md`.
The original version `test_dataset["speech"][:2]` load the whole speech array into memory which is too large.
Change it to `test_dataset[:2]["speech"]` runs smoothly and much faster.

## Before submitting
- [ ] This PR improves the docs

## Who can review?

@patrickvonplaten


Supporting `config_path` for `AutoModel`

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
Creating a model like this:
```python
AutoModel.from_pretrained('https://myserver/model/pytorch_model.bin', config='path/to/local/config.json')
```
## Motivation

For the moment, instantiating a model like above is possible with `PretrainedModel` but not with `AutoModel`. i.e.
```python
BertModel.from_pretrained('https://myserver/model/pytorch_model.bin', config='path/to/local/config.json') # possible
AutoModel.from_pretrained('https://myserver/model/pytorch_model.bin', config='path/to/local/config.json') # not possible
```

To elaborate, the optional `config` argument passed to **`AutoModel.from_pretrained`** method should be `PretrainedConfig`, while it could be either `PretrainedConfig` or 'a string or path valid as input to `PretrainedConfig.from_pretrained`' in the case of **`PretrainedModel.from_pretrained`**.

The difference comes from the lack of [this line](https://github.com/huggingface/transformers/blob/8780caa388c7b2aa937454ed96bcdd3f097f851d/src/transformers/modeling_utils.py#L974) in `AutoModel`.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution
I would like to submit a PR.
<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


What is the score of trainer.predict()?

I want to know the meaning of output of trainer.predict().

example:
`PredictionOutput(predictions=array([[-2.2704859,  2.442343 ]], dtype=float32), label_ids=array([1]), metrics={'eval_loss': 0.008939245715737343, 'eval_runtime': 0.0215, 'eval_samples_per_second': 46.56})`

What is this score? -> predictions=array([[-2.2704859,  2.442343 ]]

I use it for Sequence Classification.


Fix summarization notebook link

This PRs fixes the link to the new summarization notebook


Returning Confidence Score For Extractive QA Task When Using Non-Pipeline Approach

Feature request

Request for a new feature

# 🚀 Feature request

HF's Extract QA pipeline provides an excellent interface for start. It returns 4 values including a **probability score / confidence score**. Unfortunately same is not the case when using the non-pipeline approach i.e using model and tokenizer for question answering.
[Both methods are mentioned here, The pipeline one and the other](https://huggingface.co/transformers/task_summary.html#extractive-question-answering)

## Motivation
The confidence score will help a lot in various tasks. For example when I am developing a complete pipeline for QA, consisting of recall, retriever and some other models for entity matching and etc. I need to calculate scores of each models and then rank the final list of documents based on the weighted sum of score from each model. I believe this is a very common practice among NLP practitioners and not just for QA task. The point is confidence scores are usually a pretty standard requirement for each model output because we have to take further actions based on its score.

## Your contribution
I want to. but unfortunately I am not at the level where I can understand the code. I have went through the code and I believe its the "decode" function in "QuestionAnsweringPipeline" class which has the code the which generates the probability scores. If you guys can just provide an interface for it or provide docs for how to calculate this score using the model and tokenizer approach then that would be great too. And if you do decide to do this then please also add this addition to docs in the link mentioned at the top.

Thanks.


check_version not valid

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: AWS Sagemaker
- Python version: 3.6
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): 2.4.1 (kernal: conda_tensorflow2_p36)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
@LysandreJik, @sgugger, @patil-suraj

Models: Bert

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [X ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run sample Sagemaker notebook:    ./notebooks/sagemaker/02_getting_started_tensorflow/
2. Change the entry_point from "train.py" to "run_ner.py"
3. Copy run_ner.py from the examples for ner:  .\transformers\examples\token-classification\run_ner.py
3. Execute notebook
4. Line 47 executes a  check_min_version("4.5.0.dev0")  but that version does not exist. Newest version I see is 4.4.2.  This results in the following error message:

Lines 47-48:
   # Will error if the minimal version of Transformers is not installed. Remove at your own risks.
   check_min_version("4.5.0.dev0")

![image](https://user-images.githubusercontent.com/3164663/112927978-da565900-90e3-11eb-9672-d4c7c4f73c4f.png)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I am trying to get a baseline Sagemaker notebook working that fine tunes Bert for token classification.  (Using Colnn2003 or other.)   This should use the new Sagemaker Deep Learning Containers.  This is a first step for the project where we will next use custom data to fine tune the model.


[T5/MT5] resolve inf/nan under amp (mixed precision)

As reported in multiple issues t5/mt5 models produce loss of `nan` under mixed precision training, starting with t5-large and mt5-small and up. This PR is an attempt to fix this issue. This is crucial for DeepSpeed where it's always mixed precision training.

I spent some time with the debugger and the new `detect_overflow` helper util (added in this PR) and discovered that the best place to fix the whole problem is to not `T5LayerFF` in mixed precision. This slightly slows things down/consumes more gpu memory, but no longer requires clamping and running after ever overflowing `hidden_states`.

This PR:

* turns `autocast` off during `T5LayerFF` if run under amp
* removes the previous attempt to clamp the values as it now works without it
* introduces `debug_utils.py` with a helper function `detect_overflow` which is super-handy for tracking overflows automatically (as it's silent if all goes well). It also has some extra features, such as reporting a number of large elements - disabled by default.

Important:
* The fix is only for pytorch built-in amp. apex still has this problem since I haven't researched if the same could be done there, but it's probably a waste of time since apex is being phased out. And deepspeed doesn't use amp so it's till affected.

## Variations

Other possible variations to this solution:

1. to do the `autocast` disabling dynamically. That is trying with `autocast` and checking if any elements of output are `inf` (not sure of the overhead) and re-running this layer in full fp32 and setting a flag to continue in fp32 from then on. Here the main price will be paid by models that don't need this workaround, but they will gain but not having `autocast` turned off - so it might still be a beneficial solution to all
2. give users a switch to turn this feature on if they discover they need it - or have it on by default and allow users to turn it off if they "know what they are doing".

I am suggesting this since I don't know if all t5/mt5 models are impacted. Definitely t5-small doesn't need this.

## Questions:

* If this solution solves the problem at large and is accepted then we probably should document somewhere in t5/mt5 docs that it won't run AMP 100%?

* Test is needed: any suggestions to how we could write a test that is not too big and still gets nans prior to this PR? `t5-small` and `t5-base` don't have this problem (at least with a small sample), in my experiments the first model that gets `inf/nan` on the first batch is `mt5-small` (1.2GB), so my minimal test is:
```
rm -rf output_dir; CUDA_VISIBLE_DEVICES=0 USE_TF=0 PYTHONPATH=src python examples/seq2seq/run_translation.py \
--model_name_or_path google/mt5-small --do_train  --source_lang en --target_lang ro --dataset_name wmt16 \
--dataset_config_name ro-en --output_dir output_dir --per_device_train_batch_size=4  --logging_step 2 --save_steps 0 \
--fp16 --max_train_samples 10 --save_total_limit 0 --save_strategy no
```
We can then run this as a test and check for `nan` in loss reports.

But the 1.2GB download is somewhat big even for `@slow` tests.
**edit**: @LysandreJik says it's not a problem since we are now caching the models on the test machine.

If it ok I will just stick this with all the extended tests under `examples/tests/trainer/test_trainer_ext.py` where we have a setup for this type of full application-based tests.

* I also know some users mentioned that `inf` may happen much later in the game. I haven't run very long tests.

TODO:

* [ ] I left all the debug prints in place so that you could experiment with it easily - will remove when this is approved to be a good change

Fixes: https://github.com/huggingface/transformers/issues/10830
Fixes: https://github.com/huggingface/transformers/issues/10819

@patrickvonplaten, @patil-suraj, @LysandreJik


Input gets lost when converting mBART decoder to onnx

I'm trying to convert the mBART decoder to onnx and have the problem, that one of the inputs gets lost during the conversion, which leads to errors when trying to use the onnx model. (See code example below.)

I'm trying to understand why this is the case and how to circumvent this.

Thanks alot for any help!

## Environment info

- `transformers` version: 4.4.2
- Platform: Linux-5.4.0-65-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Who can help

@mfuntowicz @patil-suraj @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): mBART

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

## To reproduce

If you run the code below, you should see the following print output:

```
['input_ids', 'encoder_attention_mask', 'encoder_hidden_states']
```

Now, if we uncomment the commented line in `DecoderWithLMhead.forward` and pass the `past_key_values` to the decoder and run the code again, the additional inputs will be added, but `encoder_hidden_states` is not present as an input any longer.

If we run `torch.onnx.export` with `verbose=True`, `encoder_hidden_states` seems not to be part of the graph. Is there a condition in the mBART decoder implementation that excludes `encoder_hidden_states` from the graph, when `past_key_values` is given to the decoder?

Code to reproduce the issue (adapted from [FastT5](https://github.com/Ki6an/fastT5/blob/master/fastT5/onnx_exporter.py)):
```python
import functools
import operator
import os
import tempfile

from transformers import AutoTokenizer, MBartForConditionalGeneration, AutoConfig
from onnxruntime import InferenceSession
import torch

model_or_model_path = 'facebook/mbart-large-cc25'
model = MBartForConditionalGeneration.from_pretrained(model_or_model_path)
model_config = AutoConfig.from_pretrained(model_or_model_path)

class DecoderWithLMhead(torch.nn.Module):
    def __init__(self, decoder, lm_head, config):
        super().__init__()
        self.decoder = decoder
        self.lm_head = lm_head
        self.config = config

    def forward(self, *inputs):
        input_ids, attention_mask, encoder_hidden_states = inputs[:3]
        list_pkv = inputs[3:]
        past_key_values = tuple(list_pkv[i : i + 4] for i in range(0, len(list_pkv), 4))
        decoder_output = self.decoder(
            input_ids=input_ids,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=attention_mask,
            # past_key_values=past_key_values,
        )
        lm_head_out = self.lm_head(decoder_output[0] * (self.config.d_model ** -0.5))
        return lm_head_out, decoder_output[1]

decoder_with_lm_head = DecoderWithLMhead(
    decoder=model.get_decoder(),
    lm_head=model.get_output_embeddings(),
    config=model_config
)

batch_size = 5
sequence_length = 10

input_ids_dec = torch.ones((batch_size, 1), dtype=torch.int64)
attention_mask_dec = torch.ones((batch_size, sequence_length), dtype=torch.int64)
enc_out = torch.ones(
    (batch_size, sequence_length, model_config.d_model), dtype=torch.float32
)
head_dim = model_config.d_model // model_config.encoder_attention_heads
a = torch.ones((batch_size, model_config.decoder_attention_heads, sequence_length, head_dim), dtype=torch.float32)
attention_block = (a, a, a, a)
past_key_values = (attention_block,) * model_config.decoder_layers
flat_past_key_values = functools.reduce(operator.iconcat, past_key_values, [])
decoder_all_inputs = tuple(
    [input_ids_dec, attention_mask_dec, enc_out] + flat_past_key_values
)
num_of_inputs = 4 * model_config.decoder_layers

with torch.no_grad():
    decoder_inputs = [
        "input_ids",
        "encoder_attention_mask",
        "encoder_hidden_states",
    ]
    pkv_input_names = ["input_{}".format(i) for i in range(0, num_of_inputs)]
    decoder_input_names = decoder_inputs + pkv_input_names
    decoder_output_names = ["logits", "output_past_key_values"]
    dyn_axis = {
        "input_ids": {0: "batch", 1: "sequence"},
        "encoder_attention_mask": {0: "batch", 1: "sequence"},
        "encoder_hidden_states": {0: "batch", 1: "sequence"},
        "logits": {0: "batch", 1: "sequence"},
        "output_past_key_values": {0: "batch", 1: "sequence"},
    }
    dyn_pkv = {
        "input_{}".format(i): {0: "batch", 1: "n_head", 2: "seq_length", 3: "d_kv"}
        for i in range(0, num_of_inputs)
    }
    dyn_axis_params = {**dyn_axis, **dyn_pkv}

    temp_dir = tempfile.TemporaryDirectory()
    onnx_output_path = os.path.join(temp_dir.name, "decoder.onnx")

    torch.onnx.export(
        decoder_with_lm_head,
        decoder_all_inputs,
        onnx_output_path,
        export_params=True,
        do_constant_folding=True,
        opset_version=12,
        input_names=decoder_input_names,
        output_names=decoder_output_names,
        dynamic_axes=dyn_axis_params,
        use_external_data_format=True,
    )

session = InferenceSession(onnx_output_path)
print(list(map(lambda x: x.name, session.get_inputs())))  # encoder_hidden_states should be in here
temp_dir.cleanup()
```

## Expected behavior

All inputs passed to the onnx export function are present in the created onnx model.


[vulnerability] dep fix

Fixes https://github.com/huggingface/transformers/security/dependabot/examples/research_projects/lxmert/requirements.txt/Pygments/open

@LysandreJik


Use pre-computed lengths, if available, when grouping by length

# What does this PR do?

A new argument `length_column_name` has been added to
`TrainingArguments`, with default value `"length"`. If this column
exists and `group_by_length` is `True`, the train sampler will use
it for grouping rather than computing all lengths before training starts.

This is an optimization that allows the user to prepare data for fast
processing, preventing sequential access to the dataset as described in
issue #10909.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? [Discussion](https://discuss.huggingface.co/t/spanish-asr-fine-tuning-wav2vec2/4586/6), related issue #10909.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@sgugger, this is what we discussed during the fine-tuning week.


[Trainer] possible DDP memory regression

I think we may have created a memory regression somewhere recently.

I tried with pt-1.7 and pt-1.8 with the same results.

memory limit on this setup is 8gb

on `transformers` master:

This takes about 5.5GB/gpu:
```
PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0,1 python examples/seq2seq/run_translation.py --model_name_or_path google/mt5-small --do_train --do_eval --source_lang en --target_lang ro --dataset_name wmt16 --dataset_config_name ro-en --output_dir /tmp/test --per_device_train_batch_size=4 --per_device_eval_batch_size=4 --overwrite_output_dir --predict_with_generate --logging_step 10
```

(no need to run more than a few secs, we are just trying to see that the job can start training)

switching to DDP immediately OOMs:
```
PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0,1 python  -m torch.distributed.launch --nproc_per_node=2  examples/seq2seq/run_translation.py --model_name_or_path google/mt5-small --do_train --do_eval --source_lang en --target_lang ro --dataset_name wmt16 --dataset_config_name ro-en --output_dir /tmp/test --per_device_train_batch_size=4 --per_device_eval_batch_size=4 --overwrite_output_dir --predict_with_generate --logging_step 10
```
even if I reduce the bs from 4 to 1 it still goes over 8GB.

@sgugger, could you please confirm if you're seeing the same?


Fixes in the templates

# What does this PR do?

Fixes a few things I noticed from new models PR in the templates directly.


Add Vision Transformer and ViTFeatureExtractor

# What does this PR do?

Opening up a new PR based on #10513 which uses @sgugger's new `image_utils.py` instead of `torchvision` for the image transformations, and is up-to-date with master.

Things to do:

- [x] fix one integration test (currently `ViTFeatureExtractor` converts the numpy arrays into DoubleTensors, but the model expects FloatTensors)
- [x] fix styling (`make style` is not working as expected on my machine, see remaining comments in previous PR)
- [x] perhaps change pooler logic? Design (and updated conversion script) currently at branch "add_pooler_to_vit"

cc @LysandreJik


How to freeze Camembert model for Classification tasks?

# 📚 Migration

## Information

<!-- Important information -->

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## Details

<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):

## Checklist

- [ ] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.


[MarianMTModel] 'list' object has no attribute 'size'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Google Colab
- Python version: 3.7
- PyTorch version (GPU?): 1.8.1
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

```
!pip install transformers==4.1.1 sentencepiece==0.1.94
!pip install mosestokenizer==1.1.0

from transformers import MarianMTModel, MarianTokenizer

target_model_name = 'Helsinki-NLP/opus-mt-en-ROMANCE'
target_tokenizer = MarianTokenizer.from_pretrained(target_model_name)
target_model = MarianMTModel.from_pretrained(target_model_name)

en_model_name = 'Helsinki-NLP/opus-mt-ROMANCE-en'
en_tokenizer = MarianTokenizer.from_pretrained(en_model_name)
en_model = MarianMTModel.from_pretrained(en_model_name)

def translate(texts, model, tokenizer, language="fr"):
    # Prepare the text data into appropriate format for the model
    template = lambda text: f"{text}" if language == "en" else f">>{language}<< {text}"
    src_texts = [template(text) for text in texts]

    # Tokenize the texts
    encoded = tokenizer.prepare_seq2seq_batch(src_texts)

    # Generate translation using model
    translated = model.generate(**encoded)

    # Convert the generated tokens indices back into text
    translated_texts = tokenizer.batch_decode(translated, skip_special_tokens=True)

    return translated_texts

def back_translate(texts, source_lang="en", target_lang="vi"):
    # Translate from source to target language
    fr_texts = translate(texts, target_model, target_tokenizer,
                         language=target_lang)

    # Translate from target language back to source language
    back_translated_texts = translate(fr_texts, en_model, en_tokenizer,
                                      language=source_lang)

    return back_translated_texts

en_texts = ['This is so cool', 'I hated the food', 'They were very helpful']

aug_texts = back_translate(en_texts, source_lang="en", target_lang="es")
print(aug_texts)

```
The problem arises when using:
* [x] my own modified scripts:
```
<ipython-input-1-83d3425f13db> in back_translate(texts, source_lang, target_lang)
     36     # Translate from source to target language
     37     fr_texts = translate(texts, target_model, target_tokenizer,
---> 38                          language=target_lang)
     39
     40     # Translate from target language back to source language

<ipython-input-1-83d3425f13db> in translate(texts, model, tokenizer, language)
     26
     27     # Generate translation using model
---> 28     translated = model.generate(**encoded)
     29
     30     # Convert the generated tokens indices back into text

/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     25         def decorate_context(*args, **kwargs):
     26             with self.__class__():
---> 27                 return func(*args, **kwargs)
     28         return cast(F, decorate_context)
     29

/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, num_return_sequences, max_time, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, **model_kwargs)
    914         if self.config.is_encoder_decoder:
    915             # add encoder_outputs to model_kwargs
--> 916             model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
    917
    918             # set input_ids as decoder_input_ids

/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py in _prepare_encoder_decoder_kwargs_for_generation(self, input_ids, model_kwargs)
    409                 argument: value for argument, value in model_kwargs.items() if not argument.startswith("decoder_")
    410             }
--> 411             model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
    412         return model_kwargs
    413

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/marian/modeling_marian.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
    712             raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
    713         elif input_ids is not None:
--> 714             input_shape = input_ids.size()
    715             input_ids = input_ids.view(-1, input_shape[-1])
    716         elif inputs_embeds is not None:

AttributeError: 'list' object has no attribute 'size'

```

Thanks for your support


Save model error: list index out of range after pass input_processing call

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0
- Platform: Linux-4.19.0-14-cloud-amd64-x86_64-with-debian-10.8
- Python version: 3.7.10
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
Maybe @LysandreJik or @jplu
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using albert:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## Code for Tensorflow
* albert_zh: https://github.com/brightmart/albert_zh

```python
strategy = tf.distribute.MirroredStrategy()
max_seq_len = 128
cache_folder = '/tmp'
pretrain_model = 'voidful/albert_chinese_tiny'
albert_config = AlbertConfig.from_json_file(albert_zh / 'albert_config' / 'albert_config_tiny.json')

def sms_classifier_model(pretrain_model, config, max_seq_len, cache_folder):
    input_ids = tf.keras.layers.Input(shape=(max_seq_len, ), name='input_ids', dtype=tf.int32)
    input_token_type_ids = tf.keras.layers.Input(shape=(max_seq_len, ), name='token_type_ids', dtype=tf.int32)
    input_attention_mask = tf.keras.layers.Input(shape=(max_seq_len, ), name='attention_mask', dtype=tf.int32)

    albert_model = TFAlbertForSequenceClassification.from_pretrained(
        pretrain_model,
        config=config,
        from_pt=True,
        cache_dir=cache_folder)

    x = albert_model([input_ids, input_token_type_ids, input_attention_mask])
    output = tf.keras.activations.softmax(x[0])
    model = tf.keras.models.Model(
        inputs=[input_ids, input_token_type_ids, input_attention_mask],
        outputs={'target': output}, name='sms_classifier')
    return model

K.clear_session()
albert_config.hidden_act = 'gelu_new'
albert_config.num_labels = 4

with strategy.scope():
    albert_model = sms_classifier_model(pretrain_model, albert_config, 128, cached_pretarin_model_folder)

with strategy.scope():
    albert_model.compile(optimizer=tf.keras.optimizers.Adam(),
                         loss=tf.keras.losses.CategoricalCrossentropy(),
                         metrics=tf.keras.metrics.CategoricalAccuracy())

albert_model.fit(
        x=training_dataset,
        validation_data=validation_dataset,
        steps_per_epoch=200,
        validation_steps=100,
        epochs=2,
        verbose=1,
        use_multiprocessing=True)

albert_model.save('/tmp/albert_model')
```
## Error Message
```python
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-33-9fbc26d706ab> in <module>
      1 albert_model.save(
----> 2     str(saved_tf_model_folder / f'{run_id}')
      3 )

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in save(self, filepath, overwrite, include_optimizer, save_format, signatures, options, save_traces)
   2000     # pylint: enable=line-too-long
   2001     save.save_model(self, filepath, overwrite, include_optimizer, save_format,
-> 2002                     signatures, options, save_traces)
   2003
   2004   def save_weights(self,

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options, save_traces)
    155   else:
    156     saved_model_save.save(model, filepath, overwrite, include_optimizer,
--> 157                           signatures, options, save_traces)
    158
    159

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save.py in save(model, filepath, overwrite, include_optimizer, signatures, options, save_traces)
     87     with distribution_strategy_context._get_default_replica_context():  # pylint: disable=protected-access
     88       with utils.keras_option_scope(save_traces):
---> 89         save_lib.save(model, filepath, signatures, options)
     90
     91   if not include_optimizer:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py in save(obj, export_dir, signatures, options)
   1031
   1032   _, exported_graph, object_saver, asset_info = _build_meta_graph(
-> 1033       obj, signatures, options, meta_graph_def)
   1034   saved_model.saved_model_schema_version = constants.SAVED_MODEL_SCHEMA_VERSION
   1035

/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py in _build_meta_graph(obj, signatures, options, meta_graph_def)
   1196
   1197   with save_context.save_context(options):
-> 1198     return _build_meta_graph_impl(obj, signatures, options, meta_graph_def)

/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py in _build_meta_graph_impl(obj, signatures, options, meta_graph_def)
   1131   if signatures is None:
   1132     signatures = signature_serialization.find_function_to_export(
-> 1133         checkpoint_graph_view)
   1134
   1135   signatures, wrapped_functions = (

/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/signature_serialization.py in find_function_to_export(saveable_view)
     73   # If the user did not specify signatures, check the root object for a function
     74   # that can be made into a signature.
---> 75   functions = saveable_view.list_functions(saveable_view.root)
     76   signature = functions.get(DEFAULT_SIGNATURE_ATTR, None)
     77   if signature is not None:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py in list_functions(self, obj, extra_functions)
    149     if obj_functions is None:
    150       obj_functions = obj._list_functions_for_serialization(  # pylint: disable=protected-access
--> 151           self._serialization_cache)
    152       self._functions[obj] = obj_functions
    153     if extra_functions:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _list_functions_for_serialization(self, serialization_cache)
   2611     self.predict_function = None
   2612     functions = super(
-> 2613         Model, self)._list_functions_for_serialization(serialization_cache)
   2614     self.train_function = train_function
   2615     self.test_function = test_function

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _list_functions_for_serialization(self, serialization_cache)
   3085   def _list_functions_for_serialization(self, serialization_cache):
   3086     return (self._trackable_saved_model_saver
-> 3087             .list_functions_for_serialization(serialization_cache))
   3088
   3089   def __getstate__(self):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/base_serialization.py in list_functions_for_serialization(self, serialization_cache)
     92       return {}
     93
---> 94     fns = self.functions_to_serialize(serialization_cache)
     95
     96     # The parent AutoTrackable class saves all user-defined tf.functions, and

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py in functions_to_serialize(self, serialization_cache)
     77   def functions_to_serialize(self, serialization_cache):
     78     return (self._get_serialized_attributes(
---> 79         serialization_cache).functions_to_serialize)
     80
     81   def _get_serialized_attributes(self, serialization_cache):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py in _get_serialized_attributes(self, serialization_cache)
     93
     94     object_dict, function_dict = self._get_serialized_attributes_internal(
---> 95         serialization_cache)
     96
     97     serialized_attr.set_and_validate_objects(object_dict)

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/model_serialization.py in _get_serialized_attributes_internal(self, serialization_cache)
     55     objects, functions = (
     56         super(ModelSavedModelSaver, self)._get_serialized_attributes_internal(
---> 57             serialization_cache))
     58     functions['_default_save_signature'] = default_signature
     59     return objects, functions

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py in _get_serialized_attributes_internal(self, serialization_cache)
    102     """Returns dictionary of serialized attributes."""
    103     objects = save_impl.wrap_layer_objects(self.obj, serialization_cache)
--> 104     functions = save_impl.wrap_layer_functions(self.obj, serialization_cache)
    105     # Attribute validator requires that the default save signature is added to
    106     # function dict, even if the value is None.

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in wrap_layer_functions(layer, serialization_cache)
    163   call_fn_with_losses = call_collection.add_function(
    164       _wrap_call_and_conditional_losses(layer),
--> 165       '{}_layer_call_and_return_conditional_losses'.format(layer.name))
    166   call_fn = call_collection.add_function(
    167       _extract_outputs_from_fn(layer, call_fn_with_losses),

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in add_function(self, call_fn, name)
    503       # Manually add traces for layers that have keyword arguments and have
    504       # a fully defined input signature.
--> 505       self.add_trace(*self._input_signature)
    506     return fn
    507

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in add_trace(self, *args, **kwargs)
    418             fn.get_concrete_function(*args, **kwargs)
    419
--> 420         trace_with_training(True)
    421         trace_with_training(False)
    422       else:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in trace_with_training(value, fn)
    416           utils.set_training_arg(value, self._training_arg_index, args, kwargs)
    417           with K.deprecated_internal_learning_phase_scope(value):
--> 418             fn.get_concrete_function(*args, **kwargs)
    419
    420         trace_with_training(True)

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in get_concrete_function(self, *args, **kwargs)
    548     if not self.call_collection.tracing:
    549       self.call_collection.add_trace(*args, **kwargs)
--> 550     return super(LayerCall, self).get_concrete_function(*args, **kwargs)
    551
    552

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in get_concrete_function(self, *args, **kwargs)
   1297       ValueError: if this object has not yet been called on concrete values.
   1298     """
-> 1299     concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
   1300     concrete._garbage_collector.release()  # pylint: disable=protected-access
   1301     return concrete

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _get_concrete_function_garbage_collected(self, *args, **kwargs)
   1203       if self._stateful_fn is None:
   1204         initializers = []
-> 1205         self._initialize(args, kwargs, add_initializers_to=initializers)
   1206         self._initialize_uninitialized_variables(initializers)
   1207

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    724     self._concrete_stateful_fn = (
    725         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 726             *args, **kwds))
    727
    728     def invalid_creator_scope(*unused_args, **unused_kwds):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2967       args, kwargs = None, None
   2968     with self._lock:
-> 2969       graph_function, _ = self._maybe_define_function(args, kwargs)
   2970     return graph_function
   2971

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3359
   3360           self._function_cache.missed.add(call_context_key)
-> 3361           graph_function = self._create_graph_function(args, kwargs)
   3362           self._function_cache.primary[cache_key] = graph_function
   3363

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3204             arg_names=arg_names,
   3205             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3206             capture_by_value=self._capture_by_value),
   3207         self._function_attributes,
   3208         function_spec=self.function_spec,

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    988         _, original_func = tf_decorator.unwrap(python_func)
    989
--> 990       func_outputs = python_func(*func_args, **func_kwargs)
    991
    992       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    632             xla_context.Exit()
    633         else:
--> 634           out = weak_wrapped_fn().__wrapped__(*args, **kwds)
    635         return out
    636

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in wrapper(*args, **kwargs)
    525       with autocast_variable.enable_auto_cast_variables(
    526           layer._compute_dtype_object):  # pylint: disable=protected-access
--> 527         ret = method(*args, **kwargs)
    528     _restore_layer_losses(original_losses)
    529     return ret

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in wrap_with_training_arg(*args, **kwargs)
    169     return control_flow_util.smart_cond(
    170         training, lambda: replace_training_and_call(True),
--> 171         lambda: replace_training_and_call(False))
    172
    173   # Create arg spec for decorated function. If 'training' is not defined in the

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/control_flow_util.py in smart_cond(pred, true_fn, false_fn, name)
    113         pred, true_fn=true_fn, false_fn=false_fn, name=name)
    114   return smart_module.smart_cond(
--> 115       pred, true_fn=true_fn, false_fn=false_fn, name=name)
    116
    117

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/smart_cond.py in smart_cond(pred, true_fn, false_fn, name)
     52   if pred_value is not None:
     53     if pred_value:
---> 54       return true_fn()
     55     else:
     56       return false_fn()

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in <lambda>()
    168
    169     return control_flow_util.smart_cond(
--> 170         training, lambda: replace_training_and_call(True),
    171         lambda: replace_training_and_call(False))
    172

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in replace_training_and_call(training)
    165     def replace_training_and_call(training):
    166       set_training_arg(training, training_arg_index, args, kwargs)
--> 167       return wrapped_call(*args, **kwargs)
    168
    169     return control_flow_util.smart_cond(

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in call_and_return_conditional_losses(inputs, *args, **kwargs)
    568   def call_and_return_conditional_losses(inputs, *args, **kwargs):
    569     """Returns layer (call_output, conditional losses) tuple."""
--> 570     call_output = layer_call(inputs, *args, **kwargs)
    571     if version_utils.is_v1_layer_or_model(layer):
    572       conditional_losses = layer.get_losses_for(inputs)

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py in call(self, inputs, training, mask)
    423     """
    424     return self._run_internal_graph(
--> 425         inputs, training=training, mask=mask)
    426
    427   def compute_output_shape(self, input_shape):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py in _run_internal_graph(self, inputs, training, mask)
    558
    559         args, kwargs = node.map_arguments(tensor_dict)
--> 560         outputs = node.layer(*args, **kwargs)
    561
    562         # Update tensor_dict.

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
   1010         with autocast_variable.enable_auto_cast_variables(
   1011             self._compute_dtype_object):
-> 1012           outputs = call_fn(inputs, *args, **kwargs)
   1013
   1014         if self._activity_regularizer:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in return_outputs_and_add_losses(*args, **kwargs)
     71     inputs = args[inputs_arg_index]
     72     args = args[inputs_arg_index + 1:]
---> 73     outputs, losses = fn(inputs, *args, **kwargs)
     74     layer.add_loss(losses, inputs=inputs)
     75

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in wrap_with_training_arg(*args, **kwargs)
    169     return control_flow_util.smart_cond(
    170         training, lambda: replace_training_and_call(True),
--> 171         lambda: replace_training_and_call(False))
    172
    173   # Create arg spec for decorated function. If 'training' is not defined in the

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/control_flow_util.py in smart_cond(pred, true_fn, false_fn, name)
    113         pred, true_fn=true_fn, false_fn=false_fn, name=name)
    114   return smart_module.smart_cond(
--> 115       pred, true_fn=true_fn, false_fn=false_fn, name=name)
    116
    117

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/smart_cond.py in smart_cond(pred, true_fn, false_fn, name)
     52   if pred_value is not None:
     53     if pred_value:
---> 54       return true_fn()
     55     else:
     56       return false_fn()

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in <lambda>()
    168
    169     return control_flow_util.smart_cond(
--> 170         training, lambda: replace_training_and_call(True),
    171         lambda: replace_training_and_call(False))
    172

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in replace_training_and_call(training)
    165     def replace_training_and_call(training):
    166       set_training_arg(training, training_arg_index, args, kwargs)
--> 167       return wrapped_call(*args, **kwargs)
    168
    169     return control_flow_util.smart_cond(

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in __call__(self, *args, **kwargs)
    542   def __call__(self, *args, **kwargs):
    543     if not self.call_collection.tracing:
--> 544       self.call_collection.add_trace(*args, **kwargs)
    545     return super(LayerCall, self).__call__(*args, **kwargs)
    546

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in add_trace(self, *args, **kwargs)
    418             fn.get_concrete_function(*args, **kwargs)
    419
--> 420         trace_with_training(True)
    421         trace_with_training(False)
    422       else:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in trace_with_training(value, fn)
    416           utils.set_training_arg(value, self._training_arg_index, args, kwargs)
    417           with K.deprecated_internal_learning_phase_scope(value):
--> 418             fn.get_concrete_function(*args, **kwargs)
    419
    420         trace_with_training(True)

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in get_concrete_function(self, *args, **kwargs)
    548     if not self.call_collection.tracing:
    549       self.call_collection.add_trace(*args, **kwargs)
--> 550     return super(LayerCall, self).get_concrete_function(*args, **kwargs)
    551
    552

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in get_concrete_function(self, *args, **kwargs)
   1297       ValueError: if this object has not yet been called on concrete values.
   1298     """
-> 1299     concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
   1300     concrete._garbage_collector.release()  # pylint: disable=protected-access
   1301     return concrete

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _get_concrete_function_garbage_collected(self, *args, **kwargs)
   1215       # run the first trace but we should fail if variables are created.
   1216       concrete = self._stateful_fn._get_concrete_function_garbage_collected(  # pylint: disable=protected-access
-> 1217           *args, **kwargs)
   1218       if self._created_variables:
   1219         raise ValueError("Creating variables on a non-first call to a function"

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_garbage_collected(self, *args, **kwargs)
   3017       args, kwargs = None, None
   3018     with self._lock:
-> 3019       graph_function, _ = self._maybe_define_function(args, kwargs)
   3020       seen_names = set()
   3021       captured = object_identity.ObjectIdentitySet(

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3359
   3360           self._function_cache.missed.add(call_context_key)
-> 3361           graph_function = self._create_graph_function(args, kwargs)
   3362           self._function_cache.primary[cache_key] = graph_function
   3363

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3204             arg_names=arg_names,
   3205             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3206             capture_by_value=self._capture_by_value),
   3207         self._function_attributes,
   3208         function_spec=self.function_spec,

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    988         _, original_func = tf_decorator.unwrap(python_func)
    989
--> 990       func_outputs = python_func(*func_args, **func_kwargs)
    991
    992       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    632             xla_context.Exit()
    633         else:
--> 634           out = weak_wrapped_fn().__wrapped__(*args, **kwds)
    635         return out
    636

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in wrapper(*args, **kwargs)
    525       with autocast_variable.enable_auto_cast_variables(
    526           layer._compute_dtype_object):  # pylint: disable=protected-access
--> 527         ret = method(*args, **kwargs)
    528     _restore_layer_losses(original_losses)
    529     return ret

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in wrap_with_training_arg(*args, **kwargs)
    169     return control_flow_util.smart_cond(
    170         training, lambda: replace_training_and_call(True),
--> 171         lambda: replace_training_and_call(False))
    172
    173   # Create arg spec for decorated function. If 'training' is not defined in the

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/control_flow_util.py in smart_cond(pred, true_fn, false_fn, name)
    113         pred, true_fn=true_fn, false_fn=false_fn, name=name)
    114   return smart_module.smart_cond(
--> 115       pred, true_fn=true_fn, false_fn=false_fn, name=name)
    116
    117

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/smart_cond.py in smart_cond(pred, true_fn, false_fn, name)
     52   if pred_value is not None:
     53     if pred_value:
---> 54       return true_fn()
     55     else:
     56       return false_fn()

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in <lambda>()
    168
    169     return control_flow_util.smart_cond(
--> 170         training, lambda: replace_training_and_call(True),
    171         lambda: replace_training_and_call(False))
    172

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/utils.py in replace_training_and_call(training)
    165     def replace_training_and_call(training):
    166       set_training_arg(training, training_arg_index, args, kwargs)
--> 167       return wrapped_call(*args, **kwargs)
    168
    169     return control_flow_util.smart_cond(

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py in call_and_return_conditional_losses(inputs, *args, **kwargs)
    568   def call_and_return_conditional_losses(inputs, *args, **kwargs):
    569     """Returns layer (call_output, conditional losses) tuple."""
--> 570     call_output = layer_call(inputs, *args, **kwargs)
    571     if version_utils.is_v1_layer_or_model(layer):
    572       conditional_losses = layer.get_losses_for(inputs)

/opt/conda/lib/python3.7/site-packages/transformers/models/albert/modeling_tf_albert.py in call(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict, labels, training, **kwargs)
   1144             labels=labels,
   1145             training=training,
-> 1146             kwargs_call=kwargs,
   1147         )
   1148         outputs = self.albert(

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in input_processing(func, config, input_ids, **kwargs)
    372                     output[tensor_name] = input
    373                 else:
--> 374                     output[parameter_names[i]] = input
    375             elif isinstance(input, allowed_types) or input is None:
    376                 output[parameter_names[i]] = input

IndexError: list index out of range
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

In version <= 4.0.0 use save method was without the error, but after `input_processing` function comment that the error happened.
Does any advice to fix the problem?


[Feature] Add a new tiny feature for self-attention analysis

# What does this PR do?
To implement the integral in [this paper](https://arxiv.org/abs/2004.11207), users need to take two items out: Output and Attention weights. However, the "Output" here is somewhat special; the attention weights of each layer should be multiplied by a scalar "alpha". I try to put this "alpha" in and set default to 1.0 (which is the same as the original BERT).

@LysandreJik

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


We’ve been pretraining a T5-small model from scratch with using DeepSpeed v0.3.10.
We’ve found that cpu memory was increasing over time. (We’ve trained about 150 hours)
Are there memory leaks when using DeepSpeed on training T5?


Please implement DUMA: Reading Comprehension with Transposition Thinking

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
This one is on the race leaderborad top, will you guys consider implement this?

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Converting marian tatoeba models

## Environment info

- `transformers` version: 4.4.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help

- marian: @patrickvonplaten, @patil-suraj

## Information

Model I am using (Bert, XLNet...): marian

The problem arises when using:
* [x] the official example scripts: Tatoeba models converting script
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: machine translation
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

All steps are the same as in [official script](https://github.com/huggingface/transformers/blob/master/scripts/tatoeba/README.md) for converting marian tatoeba models to pytorch.

Error log:
```
Traceback (most recent call last):
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 1267, in <module>
    resolver = TatoebaConverter(save_dir=args.save_dir)
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 80, in __init__
    released.columns = released_cols
  File "/usr/local/lib/python3.7/dist-packages/pandas/core/generic.py", line 5154, in __setattr__
    return object.__setattr__(self, name, value)
  File "pandas/_libs/properties.pyx", line 66, in pandas._libs.properties.AxisProperty.__set__
  File "/usr/local/lib/python3.7/dist-packages/pandas/core/generic.py", line 564, in _set_axis
    self._mgr.set_axis(axis, labels)
  File "/usr/local/lib/python3.7/dist-packages/pandas/core/internals/managers.py", line 227, in set_axis
    f"Length mismatch: Expected axis has {old_len} elements, new "
ValueError: Length mismatch: Expected axis has 7 elements, new values have 9 elements
```

## Expected behavior

IMO, main problem is in changes in fields of file [Tatoeba-Challenge/models/released-models.txt](https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/models/released-models.txt).
I'm expecting clean conversion of model for choosed language pair.


Wav2Vec2CTCTokenizer does not take the vocabulary into account when identifying tokens in a sentence

## Environment info
- `transformers` version: 4.4.0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.9
- PyTorch version (GPU?): 1.8.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten @LysandreJik

## Information

Model I am using (Bert, XLNet ...): wav2vec2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

We are trying to train an automatic phonemic transcription system for a low-resource language using the instructions [here](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2 )

I created a `Wav2Vec2CTCTokenizer` tokenizer as follows:
```python
from transformers import Wav2Vec2CTCTokenizer

# example of a phonemic transcription
sent = "ʈʰ æ æ̃ ˧ kʰ"
# phonemes are separated by spaces in the transcription
vocab = {phoneme for phoneme in sent.split()}
vocab_dict = {k: v for v, k in enumerate(vocab)}

# <space> will be our phoneme separator
vocab_dict[" "] = len(vocab_dict)
vocab_dict["[UNK]"] = len(vocab_dict)
vocab_dict["[PAD]"] = len(vocab_dict)

import json
with open('vocab.json', 'w') as vocab_file:
    json.dump(vocab_dict, vocab_file)

tokenizer = Wav2Vec2CTCTokenizer("./vocab.json",
                                 unk_token="[UNK]",
                                 pad_token="[PAD]",
                                 word_delimiter_token=" ")

```

The result of the sentence tokenization is:
```
>>> tokenizer(sent)

{'input_ids': [6, 6, 5, 2, 5, 2, 6, 5, 4, 5, 6, 6], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

```
Where 6 is the id of the <unk> token. The vocabulary is :
`{'æ̃': 0, 'kʰ': 1, 'æ': 2, 'ʈʰ': 3, '˧': 4, ' ': 5, '[UNK]': 6, '[PAD]': 7}`

It appears that phonemes made of several characters (e.g. ʈʰ) are not recognized as a whole but rather taken separately (ʈ and than ʰ) each token being mapped to a separate id (here `<unk>` as the separated characters are not in the vocabulary).

The tokenization output results from the `Wav2Vec2CTCTokenizer._tokenize` function being called before looking in the dictionary representing the vocabulary to map tokens into IDs. This function converts the string representing the sentence to tokenize into a list without taking into account the tokens defined in the vocabulary.

I do not know if this is the intended behaviour or if we are not using the tokenizer correctly (in which case the documentation might be improved)


Added documentation for data collator.

# What does this PR do?

This PR aims to improve  coverage of the documentation for the Data Collators.

Fixes #9035

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Addition of  SequenceClassification config specific documentation to XModelForSequenceClassification.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Windows 10
- Python version: 3.6.12
- Tensorflow version (GPU?): 2.3.1
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help: @sgugger
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (DistilBert, Longformer):

The tasks I am working on is:
* [ ] my own task or dataset:
A document classification task with 2 or more custom classes.

## To reproduce

Steps to reproduce the behavior:

1. Navigate to the documentation of any [transformers.XForSequenceClassification](https://huggingface.co/transformers/model_doc/longformer.html#transformers.LongformerForSequenceClassification)
2. You will notice an absence of documentation for setting of any Sequence Classification related configs.
3. For example: `id2label, label2id, num_labels`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Documentation for SequenceClassification specific settings like: `id2label, label2id, num_labels` in the documentation page for  [transformers.XForSequenceClassification](https://huggingface.co/transformers/model_doc/longformer.html#transformers.LongformerForSequenceClassification)
Additionally in the documentation of the  `from_pretrained` method, when loading models fine-tuned on non `SequenceClassification` tasks.


[Example] Fixed finename for Saving null_odds in the evaluation stage in QA Examples

# What does this PR do?

Earlier file `eval_null_odds_eval.json` because of a typo in code now it will save it like `eval_null_odds.json` for squadv2 dataset Saving null_odds in the evaluation stage

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? I mentioned it in #10482
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@stas00 @sgugger


saving pretrained models that were obtained from another model

I am trying to save a pretrained model that I created from RobertaForQuestionAnswering by changing some layers.
However, when I load the model with from_pretrained, my new layers disappear.. It makes sense since binary seems to save only the model weights, but I wonder if there's a way to work around this?


[trainer metrics] fix cpu mem metrics; reformat runtime metric

This PR improves and fixes trainer metrics:

* reworks the general ram memory tracking replacing `tracemalloc`, with "sampling" via `psutil` - in particular for peak tracking using a thread. `tracemalloc` proved to not track anything but python memory allocations, so we were missing most of the general RAM from reports. Now we are reporting much more. (other than swapped out memory).
* adds important details to memory metrics docs
* moves `psutil` dependency from just-for-tests to the core. I tried to find a built-in python equivalent, but the closest that I found is `resource.getrusage(resource.RUSAGE_SELF).ru_maxrss` which doesn't report what we need and it's not cross-platform.
* reformats secs to be in `hh:mm:ss.msec` format so it's much easier to read the runtime metric

Discovered the `tracemalloc` limitation while tracking a huge memory leak in DeepSpeed when re-using deepspeed in the same process. My tests were consuming hundreds of MBs of general RAM and the metrics were reporting nothing.

before:
```
BS=4; PYTHONPATH=src USE_TF=0 python examples/seq2seq/run_translation.py --model_name_or_path \
t5-small --output_dir /tmp/zero3 --overwrite_output_dir --max_train_samples 64 --max_val_samples 64 \
--max_source_length 128 --max_target_length 128 --val_max_target_length 128 --do_train \
--num_train_epochs 1 --per_device_train_batch_size $BS --per_device_eval_batch_size $BS \
--learning_rate 3e-3 --warmup_steps 500 --predict_with_generate --logging_steps 0 --save_steps 0 \
--eval_steps 0 --group_by_length --adafactor --dataset_name wmt16 --dataset_config ro-en \
--source_lang en --target_lang ro --source_prefix "translate English to Romanian: "

***** train metrics *****
  epoch                      =    1.0
  init_mem_cpu_alloc_delta   =    3MB
  init_mem_cpu_peaked_delta  =    0MB
  init_mem_gpu_alloc_delta   =  230MB
  init_mem_gpu_peaked_delta  =    0MB
  train_mem_cpu_alloc_delta  =   60MB
  train_mem_cpu_peaked_delta =    0MB
  train_mem_gpu_alloc_delta  =  232MB
  train_mem_gpu_peaked_delta =  472MB
  train_runtime              = 5.5261
  train_samples              =     64
  train_samples_per_second   =  1.448
```

after this PR:

```
***** train metrics *****
  epoch                      =        1.0
  init_mem_cpu_alloc_delta   =     1298MB
  init_mem_cpu_peaked_delta  =      154MB
  init_mem_gpu_alloc_delta   =      230MB
  init_mem_gpu_peaked_delta  =        0MB
  train_mem_cpu_alloc_delta  =     3446MB
  train_mem_cpu_peaked_delta =        0MB
  train_mem_gpu_alloc_delta  =      232MB
  train_mem_gpu_peaked_delta =      472MB
  train_runtime              = 0:00:05.66
  train_samples              =         64
  train_samples_per_second   =      1.412
```

@sgugger, @LysandreJik


Fix initializing BertJapaneseTokenizer with AutoTokenizers

# What does this PR do?

This PR fixes a bug in loading some kinds of tokenizers using `AutoTokenizers.from_pretrained()`.
This issue is discussed in https://github.com/cl-tohoku/bert-japanese/issues/25.

When `sentencepiece` is not installed, the initialization of several tokenizers such as `BertJapaneseTokenizer`, `BarthezTokenizer`, and `MBart50Tokenizer` fails.
The exception is raised in `tokenizer_class_from_name()` when iterating over tokenizer classes which are `NoneType` objects.
Such tokenizer classes are set to `None` in `is_sentencepiece_available()` if `sentencepiece` is not available.

This error affects the initialization of `BertJapaneseTokenizer` even though it does not depend on `sentencepiece`.

```sh
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese")
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 479/479 [00:00<00:00, 143kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/m-suzuki/.pyenv/versions/py3.7/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 370, in from_pretrained
    tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
  File "/Users/m-suzuki/.pyenv/versions/py3.7/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 273, in tokenizer_class_from_name
    if c.__name__ == class_name:
AttributeError: 'NoneType' object has no attribute '__name__'
```

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@LysandreJik