Repository: huggingface/transformers
Pull request #15058 (2022-01-06 17:01:36)

Model summary doc page horizontal banners
# What does this PR do?
Horizontally align model summary page banners

To achieve this results, there are 2 options:
1. Write html directly (as done in this PR)
OR
2. Create a specific svelte component & use it (something like `import SomeComponent.svelte` & `<SomeComponent banners={...}>`)

I think option2 is an overkill for this use case since option1 is a very simple html. @sgugger wdyt

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
As mentioned in https://github.com/huggingface/doc-builder/issues/73#issuecomment-1006423852

compare "Original GPT" banners vs "GPT-2" banners

<img width="785" alt="Screenshot 2022-01-06 at 17 46 00" src="https://user-images.githubusercontent.com/11827707/148421166-f613b7d9-b4ed-4ef7-925b-35b9f647bba5.png">



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


mishig25: cc: @AK391 

Patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 097a6dea80..12f13381f2 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -99,7 +99,7 @@ jobs:
                   path: ~/transformers/tests_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_tests_torch_and_tf_all:
         working_directory: ~/transformers
         docker:
@@ -169,7 +169,7 @@ jobs:
                   path: ~/transformers/tests_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_tests_torch_and_flax_all:
         working_directory: ~/transformers
         docker:
@@ -237,7 +237,7 @@ jobs:
                   path: ~/transformers/tests_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_tests_torch_all:
         working_directory: ~/transformers
         docker:
@@ -304,7 +304,7 @@ jobs:
                   path: ~/transformers/tests_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_tests_tf_all:
         working_directory: ~/transformers
         docker:
@@ -370,7 +370,7 @@ jobs:
                   path: ~/transformers/tests_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_tests_flax_all:
         working_directory: ~/transformers
         docker:
@@ -437,7 +437,7 @@ jobs:
                   path: ~/transformers/tests_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_tests_pipelines_torch_all:
         working_directory: ~/transformers
         docker:
@@ -549,7 +549,7 @@ jobs:
                       - v0.4-custom_tokenizers-{{ checksum "setup.py" }}
                       - v0.4-{{ checksum "setup.py" }}
             - run: pip install --upgrade pip
-            - run: pip install .[ja,testing,sentencepiece,jieba]
+            - run: pip install .[ja,testing,sentencepiece,jieba,spacy,ftfy]
             - run: python -m unidic download
             - save_cache:
                   key: v0.4-custom_tokenizers-{{ checksum "setup.py" }}
@@ -557,7 +557,7 @@ jobs:
                       - '~/.cache/pip'
             - run: |
                   if [ -f test_list.txt ]; then
-                    python -m pytest -s --make-reports=tests_custom_tokenizers ./tests/test_tokenization_bert_japanese.py | tee tests_output.txt
+                    python -m pytest -s --make-reports=tests_custom_tokenizers ./tests/test_tokenization_bert_japanese.py ./tests/test_tokenization_openai.py | tee tests_output.txt
                   fi
             - store_artifacts:
                   path: ~/transformers/tests_output.txt
@@ -662,7 +662,7 @@ jobs:
                   path: ~/transformers/flax_examples_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_examples_flax_all:
         working_directory: ~/transformers
         docker:
@@ -729,7 +729,7 @@ jobs:
                   path: ~/transformers/tests_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_tests_hub_all:
         working_directory: ~/transformers
         docker:
@@ -795,7 +795,7 @@ jobs:
                   path: ~/transformers/tests_output.txt
             - store_artifacts:
                   path: ~/transformers/reports
-    
+
     run_tests_onnxruntime_all:
         working_directory: ~/transformers
         docker:
diff --git a/.github/ISSUE_TEMPLATE/bug-report.md b/.github/ISSUE_TEMPLATE/bug-report.md
index d63dc6830f..943ca51c37 100644
--- a/.github/ISSUE_TEMPLATE/bug-report.md
+++ b/.github/ISSUE_TEMPLATE/bug-report.md
@@ -49,7 +49,7 @@ Library:
 - Deepspeed: @stas00
 - Ray/raytune: @richardliaw, @amogkam
 - Text generation: @patrickvonplaten @narsil
-- Tokenizers: @LysandreJik
+- Tokenizers: @SaulLu
 - Trainer: @sgugger
 - Pipelines: @Narsil
 - Speech: @patrickvonplaten, @anton-l
diff --git a/docs/source/index.mdx b/docs/source/index.mdx
index b18487a081..eb57155a26 100644
--- a/docs/source/index.mdx
+++ b/docs/source/index.mdx
@@ -190,7 +190,7 @@ Flax), PyTorch, and/or TensorFlow.
 <!--This table is updated automatically from the auto modules with _make fix-copies_. Do not update manually!-->
 
 |            Model            | Tokenizer slow | Tokenizer fast | PyTorch support | TensorFlow support | Flax Support |
-|-----------------------------|----------------|----------------|-----------------|--------------------|--------------|
+|:---------------------------:|:--------------:|:--------------:|:---------------:|:------------------:|:------------:|
 |           ALBERT            |       ✅       |       ✅       |       ✅        |         ✅         |      ✅      |
 |            BART             |       ✅       |       ✅       |       ✅        |         ✅         |      ✅      |
 |            BEiT             |       ❌       |       ❌       |       ✅        |         ❌         |      ✅      |
diff --git a/docs/source/model_doc/trocr.mdx b/docs/source/model_doc/trocr.mdx
index b8cdbaeb6f..494895c1a8 100644
--- a/docs/source/model_doc/trocr.mdx
+++ b/docs/source/model_doc/trocr.mdx
@@ -55,9 +55,9 @@ Tips:
 TrOCR's [`VisionEncoderDecoder`] model accepts images as input and makes use of
 [`~generation_utils.GenerationMixin.generate`] to autoregressively generate text given the input image.
 
-The [`ViTFeatureExtractor`] class is responsible for preprocessing the input image and
-[`RobertaTokenizer`] decodes the generated target tokens to the target string. The
-[`TrOCRProcessor`] wraps [`ViTFeatureExtractor`] and [`RobertaTokenizer`]
+The [`ViTFeatureExtractor`/`DeiTFeatureExtractor`] class is responsible for preprocessing the input image and
+[`RobertaTokenizer`/`XLMRobertaTokenizer`] decodes the generated target tokens to the target string. The
+[`TrOCRProcessor`] wraps [`ViTFeatureExtractor`/`DeiTFeatureExtractor`] and [`RobertaTokenizer`/`XLMRobertaTokenizer`]
 into a single instance to both extract the input features and decode the predicted token ids.
 
 - Step-by-step Optical Character Recognition (OCR)
diff --git a/docs/source/model_summary.mdx b/docs/source/model_summary.mdx
index 2966771cc0..c6a99b8957 100644
--- a/docs/source/model_summary.mdx
+++ b/docs/source/model_summary.mdx
@@ -63,12 +63,14 @@ that at each position, the model can only look at the tokens before the attentio
 
 ### Original GPT
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=openai-gpt">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-openai--gpt-blueviolet">
 </a>
 <a href="model_doc/openai-gpt">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-openai--gpt-blueviolet">
 </a>
+</div>
 
 [Improving Language Understanding by Generative Pre-Training](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf), Alec Radford et al.
 
@@ -79,12 +81,14 @@ classification.
 
 ### GPT-2
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=gpt2">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-gpt2-blueviolet">
 </a>
 <a href="model_doc/gpt2">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-gpt2-blueviolet">
 </a>
+</div>
 
 [Language Models are Unsupervised Multitask Learners](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf),
 Alec Radford et al.
@@ -97,12 +101,14 @@ classification.
 
 ### CTRL
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=ctrl">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-ctrl-blueviolet">
 </a>
 <a href="model_doc/ctrl">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-ctrl-blueviolet">
 </a>
+</div>
 
 [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858),
 Nitish Shirish Keskar et al.
@@ -115,12 +121,14 @@ The library provides a version of the model for language modeling only.
 
 ### Transformer-XL
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=transfo-xl">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-transfo--xl-blueviolet">
 </a>
 <a href="model_doc/transfo-xl">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-transfo--xl-blueviolet">
 </a>
+</div>
 
 [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860), Zihang
 Dai et al.
@@ -143,12 +151,14 @@ The library provides a version of the model for language modeling only.
 
 ### Reformer
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=reformer">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-reformer-blueviolet">
 </a>
 <a href="model_doc/reformer">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-reformer-blueviolet">
 </a>
+</div>
 
 [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451), Nikita Kitaev et al .
 
@@ -178,12 +188,14 @@ The library provides a version of the model for language modeling only.
 
 ### XLNet
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=xlnet">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-xlnet-blueviolet">
 </a>
 <a href="model_doc/xlnet">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-xlnet-blueviolet">
 </a>
+</div>
 
 [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237), Zhilin
 Yang et al.
@@ -210,12 +222,14 @@ corrupted versions.
 
 ### BERT
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=bert">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-bert-blueviolet">
 </a>
 <a href="model_doc/bert">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-bert-blueviolet">
 </a>
+</div>
 
 [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805),
 Jacob Devlin et al.
@@ -236,12 +250,14 @@ token classification, sentence classification, multiple choice classification an
 
 ### ALBERT
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=albert">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-albert-blueviolet">
 </a>
 <a href="model_doc/albert">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-albert-blueviolet">
 </a>
+</div>
 
 [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942),
 Zhenzhong Lan et al.
@@ -262,12 +278,14 @@ classification, multiple choice classification and question answering.
 
 ### RoBERTa
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=roberta">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-roberta-blueviolet">
 </a>
 <a href="model_doc/roberta">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-roberta-blueviolet">
 </a>
+</div>
 
 [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692), Yinhan Liu et al.
 
@@ -284,12 +302,14 @@ classification, multiple choice classification and question answering.
 
 ### DistilBERT
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=distilbert">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-distilbert-blueviolet">
 </a>
 <a href="model_doc/distilbert">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-distilbert-blueviolet">
 </a>
+</div>
 
 [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108),
 Victor Sanh et al.
@@ -306,12 +326,14 @@ and question answering.
 
 ### ConvBERT
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=convbert">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-convbert-blueviolet">
 </a>
 <a href="model_doc/convbert">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-convbert-blueviolet">
 </a>
+</div>
 
 [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://arxiv.org/abs/2008.02496), Zihang Jiang,
 Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, Shuicheng Yan.
@@ -333,12 +355,14 @@ and question answering.
 
 ### XLM
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=xlm">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-xlm-blueviolet">
 </a>
 <a href="model_doc/xlm">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-xlm-blueviolet">
 </a>
+</div>
 
 [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291), Guillaume Lample and Alexis Conneau
 
@@ -364,12 +388,14 @@ question answering.
 
 ### XLM-RoBERTa
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=xlm-roberta">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-xlm--roberta-blueviolet">
 </a>
 <a href="model_doc/xlm-roberta">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-xlm--roberta-blueviolet">
 </a>
+</div>
 
 [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116), Alexis Conneau et
 al.
@@ -383,12 +409,14 @@ classification, multiple choice classification and question answering.
 
 ### FlauBERT
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=flaubert">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-flaubert-blueviolet">
 </a>
 <a href="model_doc/flaubert">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-flaubert-blueviolet">
 </a>
+</div>
 
 [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372), Hang Le et al.
 
@@ -398,12 +426,14 @@ The library provides a version of the model for language modeling and sentence c
 
 ### ELECTRA
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=electra">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-electra-blueviolet">
 </a>
 <a href="model_doc/electra">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-electra-blueviolet">
 </a>
+</div>
 
 [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://arxiv.org/abs/2003.10555),
 Kevin Clark et al.
@@ -419,12 +449,14 @@ classification.
 
 ### Funnel Transformer
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=funnel">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-funnel-blueviolet">
 </a>
 <a href="model_doc/funnel">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-funnel-blueviolet">
 </a>
+</div>
 
 [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236), Zihang Dai et al.
 
@@ -449,12 +481,14 @@ classification, multiple choice classification and question answering.
 
 ### Longformer
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=longformer">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-longformer-blueviolet">
 </a>
 <a href="model_doc/longformer">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-longformer-blueviolet">
 </a>
+</div>
 
 [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150), Iz Beltagy et al.
 
@@ -485,12 +519,14 @@ As mentioned before, these models keep both the encoder and the decoder of the o
 
 ### BART
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=bart">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-bart-blueviolet">
 </a>
 <a href="model_doc/bart">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-bart-blueviolet">
 </a>
+</div>
 
 [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/abs/1910.13461), Mike Lewis et al.
 
@@ -508,12 +544,14 @@ The library provides a version of this model for conditional generation and sequ
 
 ### Pegasus
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=pegasus">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-pegasus-blueviolet">
 </a>
 <a href="model_doc/pegasus">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-pegasus-blueviolet">
 </a>
+</div>
 
 [PEGASUS: Pre-training with Extracted Gap-sentences forAbstractive Summarization](https://arxiv.org/pdf/1912.08777.pdf), Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019.
 
@@ -535,12 +573,14 @@ The library provides a version of this model for conditional generation, which s
 
 ### MarianMT
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=marian">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-marian-blueviolet">
 </a>
 <a href="model_doc/marian">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-marian-blueviolet">
 </a>
+</div>
 
 [Marian: Fast Neural Machine Translation in C++](https://arxiv.org/abs/1804.00344), Marcin Junczys-Dowmunt et al.
 
@@ -551,12 +591,14 @@ The library provides a version of this model for conditional generation.
 
 ### T5
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=t5">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-t5-blueviolet">
 </a>
 <a href="model_doc/t5">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-t5-blueviolet">
 </a>
+</div>
 
 [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683), Colin Raffel et al.
 
@@ -580,12 +622,14 @@ The library provides a version of this model for conditional generation.
 
 ### MT5
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=mt5">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-mt5-blueviolet">
 </a>
 <a href="model_doc/mt5">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-mt5-blueviolet">
 </a>
+</div>
 
 [mT5: A massively multilingual pre-trained text-to-text transformer](https://arxiv.org/abs/2010.11934), Linting Xue
 et al.
@@ -598,12 +642,14 @@ The library provides a version of this model for conditional generation.
 
 ### MBart
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=mbart">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-mbart-blueviolet">
 </a>
 <a href="model_doc/mbart">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-mbart-blueviolet">
 </a>
+</div>
 
 [Multilingual Denoising Pre-training for Neural Machine Translation](https://arxiv.org/abs/2001.08210) by Yinhan Liu,
 Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
@@ -624,12 +670,14 @@ finetuning.
 
 ### ProphetNet
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=prophetnet">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-prophetnet-blueviolet">
 </a>
 <a href="model_doc/prophetnet">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-prophetnet-blueviolet">
 </a>
+</div>
 
 [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,](https://arxiv.org/abs/2001.04063) by
 Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou.
@@ -646,12 +694,14 @@ summarization.
 
 ### XLM-ProphetNet
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=xprophetnet">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-xprophetnet-blueviolet">
 </a>
 <a href="model_doc/xlm-prophetnet">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-xprophetnet-blueviolet">
 </a>
+</div>
 
 [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,](https://arxiv.org/abs/2001.04063) by
 Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou.
@@ -696,12 +746,14 @@ Some models use documents retrieval during (pre)training and inference for open-
 
 ### DPR
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=dpr">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-dpr-blueviolet">
 </a>
 <a href="model_doc/dpr">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-dpr-blueviolet">
 </a>
+</div>
 
 [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906), Vladimir Karpukhin et
 al.
@@ -722,12 +774,14 @@ then it calls the reader with the question and the retrieved documents to get th
 
 ### RAG
 
+<div class="flex flex-wrap space-x-1">
 <a href="https://huggingface.co/models?filter=rag">
 <img alt="Models" src="https://img.shields.io/badge/All_model_pages-rag-blueviolet">
 </a>
 <a href="model_doc/rag">
 <img alt="Doc" src="https://img.shields.io/badge/Model_documentation-rag-blueviolet">
 </a>
+</div>
 
 [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401), Patrick Lewis,
 Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau
diff --git a/docs/source/serialization.mdx b/docs/source/serialization.mdx
index 0d667fc070..091eac083b 100644
--- a/docs/source/serialization.mdx
+++ b/docs/source/serialization.mdx
@@ -436,3 +436,67 @@ Using the traced model for inference is as simple as using its `__call__` dunder
 ```python
 traced_model(tokens_tensor, segments_tensors)
 ```
+
+### Deploying HuggingFace TorchScript models on AWS using the Neuron SDK
+
+AWS introduced the [Amazon EC2 Inf1](https://aws.amazon.com/ec2/instance-types/inf1/) 
+instance family for low cost, high performance machine learning inference in the cloud. 
+The Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware accelerator, 
+specializing in deep learning inferencing workloads. 
+[AWS Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/#) 
+is the SDK for Inferentia that supports tracing and optimizing transformers models for 
+deployment on Inf1. The Neuron SDK provides:
+
+
+1. Easy-to-use API with one line of code change to trace and optimize a TorchScript model for inference in the cloud.
+2. Out of the box performance optimizations for [improved cost-performance](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/benchmark/>)
+3. Support for HuggingFace transformers models built with either [PyTorch](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert.html)
+   or [TensorFlow](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/huggingface_bert/huggingface_bert.html).
+
+#### Implications
+
+Transformers Models based on the [BERT (Bidirectional Encoder Representations from Transformers)](https://huggingface.co/docs/transformers/master/model_doc/bert) 
+architecture, or its variants such as [distilBERT](https://huggingface.co/docs/transformers/master/model_doc/distilbert)
+ and [roBERTa](https://huggingface.co/docs/transformers/master/model_doc/roberta) 
+ will run best on Inf1 for non-generative tasks such as Extractive Question Answering, 
+ Sequence Classification, Token Classification. Alternatively, text generation
+tasks can be adapted to run on Inf1, according to this [AWS Neuron MarianMT tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/transformers-marianmt.html). 
+More information about models that can be converted out of the box on Inferentia can be 
+found in the [Model Architecture Fit section of the Neuron documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/models/models-inferentia.html#models-inferentia).
+
+#### Dependencies
+
+Using AWS Neuron to convert models requires the following dependencies and environment:
+
+* A [Neuron SDK environment](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-frameworks/pytorch-neuron/index.html#installation-guide),
+  which comes pre-configured on [AWS Deep Learning AMI](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-inferentia-launching.html).
+
+#### Converting a Model for AWS Neuron
+
+Using the same script as in [Using TorchScript in Python](https://huggingface.co/docs/transformers/master/en/serialization#using-torchscript-in-python) 
+to trace a "BertModel", you import `torch.neuron` framework extension to access 
+the components of the Neuron SDK through a Python API.
+
+```python
+from transformers import BertModel, BertTokenizer, BertConfig
+import torch
+import torch.neuron
+```
+And only modify the tracing line of code
+
+from:
+
+```python
+torch.jit.trace(model, [tokens_tensor, segments_tensors])
+```
+
+to:
+
+```python
+torch.neuron.trace(model, [token_tensor, segments_tensors])
+```
+
+This change enables Neuron SDK to trace the model and optimize it to run in Inf1 instances.
+
+To learn more about AWS Neuron SDK features, tools, example tutorials and latest updates, 
+please see the [AWS NeuronSDK documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/index.html).
diff --git a/examples/pytorch/speech-recognition/README.md b/examples/pytorch/speech-recognition/README.md
index 8eeb8efc87..7db7811ba2 100644
--- a/examples/pytorch/speech-recognition/README.md
+++ b/examples/pytorch/speech-recognition/README.md
@@ -74,6 +74,7 @@ python run_speech_recognition_ctc.py \
 	--warmup_steps="500" \
 	--evaluation_strategy="steps" \
 	--text_column_name="sentence" \
+	--length_column_name="input_length" \
 	--save_steps="400" \
 	--eval_steps="100" \
 	--layerdrop="0.0" \
@@ -108,6 +109,7 @@ python -m torch.distributed.launch \
 	--warmup_steps="500" \
 	--evaluation_strategy="steps" \
 	--text_column_name="sentence" \
+	--length_column_name="input_length" \
 	--save_steps="400" \
 	--eval_steps="100" \
 	--logging_steps="1" \
diff --git a/examples/pytorch/speech-recognition/requirements.txt b/examples/pytorch/speech-recognition/requirements.txt
index b5c58171d3..b8c475e40a 100644
--- a/examples/pytorch/speech-recognition/requirements.txt
+++ b/examples/pytorch/speech-recognition/requirements.txt
@@ -2,3 +2,4 @@ datasets >= 1.13.3
 torch >= 1.5
 torchaudio
 librosa
+jiwer
diff --git a/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py b/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py
index 1784b613fa..0c460c3f02 100755
--- a/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py
+++ b/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py
@@ -396,7 +396,10 @@ def main():
 
     if training_args.do_train:
         raw_datasets["train"] = load_dataset(
-            data_args.dataset_name, data_args.dataset_config_name, split=data_args.train_split_name
+            data_args.dataset_name,
+            data_args.dataset_config_name,
+            split=data_args.train_split_name,
+            use_auth_token=data_args.use_auth_token,
         )
 
         if data_args.audio_column_name not in raw_datasets["train"].column_names:
@@ -418,7 +421,10 @@ def main():
 
     if training_args.do_eval:
         raw_datasets["eval"] = load_dataset(
-            data_args.dataset_name, data_args.dataset_config_name, split=data_args.eval_split_name
+            data_args.dataset_name,
+            data_args.dataset_config_name,
+            split=data_args.eval_split_name,
+            use_auth_token=data_args.use_auth_token,
         )
 
         if data_args.max_eval_samples is not None:
diff --git a/examples/research_projects/wav2vec2/README.md b/examples/research_projects/wav2vec2/README.md
index 235284840c..9561eb252a 100644
--- a/examples/research_projects/wav2vec2/README.md
+++ b/examples/research_projects/wav2vec2/README.md
@@ -1,12 +1,12 @@
+**NOTE**: This example is outdated and is not longer actively maintained. Please 
+follow the new instructions of fine-tuning Wav2Vec2 [here](https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/README.md)
+
 ## Fine-tuning Wav2Vec2
 
 The `run_asr.py` script allows one to fine-tune pretrained Wav2Vec2 models that can be found [here](https://huggingface.co/models?search=facebook/wav2vec2).
 
 This finetuning script can also be run as a google colab [TODO: here]( ).
 
-The script is actively maintained by [Patrick von Platen](https://github.com/patrickvonplaten).
-Feel free to ask a question on the [Forum](https://discuss.huggingface.co/) or post an issue on [GitHub](https://github.com/huggingface/transformers/issues/new/choose) and adding `@patrickvonplaten` as a tag.
-
 ### Fine-Tuning with TIMIT
 Let's take a look at the [script](./finetune_base_timit_asr.sh) used to fine-tune [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base)
 with the [TIMIT dataset](https://huggingface.co/datasets/timit_asr):
diff --git a/examples/research_projects/xls_r/README.md b/examples/research_projects/xls_r/README.md
new file mode 100644
index 0000000000..7c1cc5cc21
--- /dev/null
+++ b/examples/research_projects/xls_r/README.md
@@ -0,0 +1,395 @@
+# Speech recognition community week - version 2 🤗
+
+Welcome to the 2nd version of the speech recognition community event🎙️ !
+The goal of this event is to build **robust**, **real-world** speech recognition (ASR) models in as many languages as possible🌏🌍🌎.
+
+If necessary and available, free access to a V100 32 GB GPU will kindly be provided by the [OVH team](https://us.ovhcloud.com/) 🚀.
+
+This document summarizes all the relevant information required for the speech community event📋.
+
+Don't forget to sign up [here](TODO: Create google from)🤗.
+
+## Table of Contents
+
+- [Organization](#organization)
+- [Important dates](#important-dates)
+- [How to install pytorch, transformers, datasets](#how-to-install-relevant-libraries)
+- [How to fine-tune a speech recognition model](#how-to-finetune-a-model)
+- [Talks](#talks)
+- [Project evaluation](#project-evaluation)
+- [General Tips & Tricks](#general-tips-and-tricks)
+- [FAQ](#faq)
+
+## Organization
+
+Participants are encouraged to leverage pre-trained speech recognition checkpoints,
+preferably [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53), to train a speech recognition system in a language of their 
+choice.
+
+Participants can make use of whatever data they think is useful to build a 
+**robust** speech recognition system for **real-world** audio data. We strongly 
+recommend making use of [Mozilla's diverse Common Voice dataset](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) when training the model.
+Please do **not** use the `"test"` split of the Common Voice datasets for training
+as we will likely use this split for the final evaluation of your model.
+We kindly ask you to make sure that the dataset that you are using for training 
+has the appropriate licensing - see [here](TODO: ) for more information.
+
+During the event, the fine-tuned models will regularly be tested on a **development 
+dataset** provided by the Hugging Face team and at the end of the event, all models 
+will be tested on a **test dataset**. For each language, 
+the best performing model will receive a prize 🏆 - more information regarding 
+the testing [here](TODO: ) and prizes [here](TODO: ). We believe that framing the 
+event as a competition is more fun, but at the core, we strongly encourage 
+participants to work together by helping each other to solve bugs, share important findings, etc...🤗
+
+If possible it is encouraged to fine-tune the models on local GPU machines, but 
+if those are not available, the OVH cloud team kindly provides a limited 
+number of GPUs for the event. For more information on how to get access to the GPU - see [here](TODO: ).
+
+
+**Please note**:
+All important announcements will be made on discord. Please make sure that 
+you've joined the following discord server: TODO: fill out.
+Please make sure that you have been added to the [Speech Event Organization](https://huggingface.co/speech-recognition-community-v2). You should have received an 
+invite by email. If you didn't receive an invite, please contact the organizers, *e.g.* Anton, Patrick, or Omar on discord.
+
+
+## Important dates
+
+- **12.01.** Official announcement of the community event. Make sure to sign-up in [this google form](TODO: )
+- **12.01. - 19.01.** Participants sign up for the event.
+- **19.01.** Release of all relevant guides and training scripts.
+- **24.01.** Start of the community week! OVH & Hugging Face gives access to GPUs.
+- **24.01. - 07.02.** The OVH & Hugging Face team will be available for any questions, problems the participants might have.
+- **07.02.** Access to GPU is deactivated and community week officially ends.
+
+## How to install relevant libraries
+
+The following libraries are required to fine-tune a speech model with 🤗 Transformers and 🤗 Datasets in PyTorch.
+
+- [PyTorch](https://pytorch.org/)
+- [Transformers](https://github.com/huggingface/transformers)
+- [Datasets](https://github.com/huggingface/datasets)
+
+We recommend installing the above libraries in a [virtual environment](https://docs.python.org/3/library/venv.html). 
+If you're unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). Create a virtual environment with the version of Python you're going
+to use and activate it.
+
+You should be able to run the command:
+
+```bash
+python3 -m venv <your-venv-name>
+```
+
+You can activate your venv by running
+
+```bash
+source ~/<your-venv-name>/bin/activate
+```
+
+To begin with please make sure you have PyTorch and CUDA correctly installed. 
+The following command should return ``True``:
+
+```bash
+python -c "import torch; print(torch.cuda.is_available())"
+```
+
+If the above command doesn't print ``True``, in a first step, please follow the
+instructions [here](https://pytorch.org/) to install PyTorch with CUDA.
+
+We strongly recommend making use of the provided PyTorch examples scripts in [transformers/examples/pytorch/speech-recognition](https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition) to train your speech recognition
+system.
+In all likelihood, you will adjust one of the example scripts, so we recommend forking and cloning the 🤗 Transformers repository as follows. 
+
+1. Fork the [repository](https://github.com/huggingface/transformers) by
+   clicking on the 'Fork' button on the repository's page. This creates a copy of the code
+   under your GitHub user account.
+
+2. Clone your fork to your local disk, and add the base repository as a remote:
+
+   ```bash
+   $ git clone https://github.com/<your Github handle>/transformers.git
+   $ cd transformers
+   $ git remote add upstream https://github.com/huggingface/transformers.git
+   ```
+
+3. Create a new branch to hold your development changes. This is especially useful to share code changes with your team:
+
+   ```bash
+   $ git checkout -b a-descriptive-name-for-my-project
+   ```
+
+4. Set up a PyTorch environment by running the following command your virtual environment:
+
+   ```bash
+   $ pip install -e ".[torch-speech]"
+   ```
+
+   (If transformers was already installed in the virtual environment, remove
+   it with `pip uninstall transformers` before reinstalling it in editable
+   mode with the `-e` flag.)
+
+   If you have already cloned that repo, you might need to `git pull` to get the most recent changes in the `transformers`
+   library.
+
+   Running this command will automatically install `pytorch` and the most relevant 
+   libraries required for fine-tuning a speech recognition system.
+
+Next, you should also install the 🤗 Datasets library. We strongly recommend installing the 
+library from source to profit from the most current additions during the community week.
+
+Simply run the following steps:
+
+```
+$ cd ~/
+$ git clone https://github.com/huggingface/datasets.git
+$ cd datasets
+$ pip install -e ".[streaming]"
+```
+
+If you plan on contributing a specific dataset during 
+the community week, please fork the datasets repository and follow the instructions 
+[here](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-create-a-pull-request).
+
+To verify that all libraries are correctly installed, you can run the following command in a Python shell.
+It verifies that both `transformers` and `datasets` have been correclty installed.
+
+```python
+from transformers import AutoModelForCTC, AutoProcessor
+from datasets import load_dataset
+
+dummy_dataset = load_dataset("common_voice", "ab", split="test")
+
+model = AutoModelForCTC.from_pretrained("hf-internal-testing/tiny-random-wav2vec2")
+model.to("cuda")
+
+processor = AutoProcessor.from_pretrained("hf-internal-testing/tiny-random-wav2vec2")
+
+input_values = processor(dummy_dataset[0]["audio"]["array"], return_tensors="pt", sampling_rate=16_000).input_values
+input_values = input_values.to("cuda")
+
+logits = model(input_values).logits
+
+assert logits.shape[-1] == 32
+```
+
+## How to finetune a model
+
+In this section, we show you how to fine-tune a pre-trained [XLS-R Model](https://huggingface.co/docs/transformers/model_doc/xls_r) on the [Common Voice 7 dataset](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0). 
+
+We recommend fine-tuning one of the following pre-trained XLS-R checkpoints:
+
+- [300M parameters version](https://huggingface.co/facebook/wav2vec2-xls-r-300m)
+- [1B parameters version](https://huggingface.co/facebook/wav2vec2-xls-r-1b)
+- [2B parameters version](https://huggingface.co/facebook/wav2vec2-xls-r-2b)
+
+To begin with, please note that to use the Common Voice dataset, you 
+have to accept that **your email address** and **username** are shared with the 
+mozilla-foundation. To get access to the dataset please click on "*Access repository*" [here](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0).
+
+Next, we recommended that you get familiar with the XLS-R model and its capabilities.
+In collaboration with [Fairseq's Wav2Vec2 team](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec), 
+we've written ["Fine-tuning XLS-R for Multi-Lingual ASR with 🤗 Transformers"](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) which gives an in-detail explanation of how XLS-R functions and how it can be fine-tuned.
+
+The blog can also be opened and directly fine-tuned in a google colab notebook.
+In this section, we will explain how to fine-tune the model on a local machine.
+
+1. **Log in**
+
+To begin with you should check that you are correctly logged in and that you have `git-lfs` installed so that your fine-tuned model can automatically be uploaded.
+
+Run:
+
+```bash
+huggingface-cli login
+```
+
+to login. It is recommend to login with your personal access token that can be found under your hugging face profile (icon in the top right corner on [hf.co](http://hf.co/), then Settings -> Access Tokens -> User Access Tokens -> New Token (if haven't generated one already)
+
+You can then copy-paste this token to log in locally.
+
+2. **Create your model repository**
+
+First, let's make sure that `git-lfs` is correctly installed. To so, simply run:
+
+```bash
+git-lfs -v
+```
+
+The output should show something like `git-lfs/2.13.2 (GitHub; linux amd64; go 1.15.4)`. If your console states that the `git-lfs` command was not found, please make
+sure to install it [here](https://git-lfs.github.com/) or simply via: 
+
+```bash
+sudo apt-get install git-lfs
+```
+
+Now you can create your model repository which will contain all relevant files to 
+reproduce your training. You can either directly create the model repository on the 
+Hub (Settings -> New Model) or via the CLI. Here we choose to use the CLI instead.
+
+Assuming that we want to call our model repository *xls-r-ab-test*, we can run the 
+following command:
+
+```bash
+huggingface-cli repo create xls-r-ab-test
+```
+
+You can now see the model on the Hub, *e.g.* under https://huggingface.co/hf-test/xls-r-ab-test .
+
+Let's clone the repository so that we can define our training script inside.
+
+```bash
+git lfs install
+git clone https://huggingface.co/hf-test/xls-r-ab-test
+```
+
+3. **Add your training script and `run`-command to the repository**
+
+We encourage participants to add all relevant files for training directly to the 
+directory so that everything is fully reproducible.
+
+Let's first copy-paste the official training script from our clone 
+of `transformers` to our just created directory:
+
+```bash
+cp ~/transformers/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py ./
+```
+
+Next, we'll create a bash file to define the hyper-parameters and configurations 
+for training. More detailed information on different settings (single-GPU vs. multi-GPU) can be found [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition#connectionist-temporal-classification).
+
+For demonstration purposes, we will use a dummy XLS-R model `model_name_or_path="hf-test/xls-r-dummy"` on the very low-resource language of "Abkhaz" of [Common Voice 7](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0): `dataset_config_name="ab"` for just a single epoch.
+
+Before starting to train, let's make sure we have installed all the required libraries. You might want to run:
+
+```bash
+pip install -r ~/transformers/examples/pytorch/speech-recognition/requirements.txt
+```
+
+Alright, finally we can define the training script. We'll simply use some 
+dummy hyper-parameters and configurations for demonstration purposes.
+
+Note that we add the flag `--use_auth_token` so that datasets requiring access, 
+such as [Common Voice 7](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) can be downloaded. In addition, we add the `--push_to_hub` flag to make use of the 
+[Trainers `push_to-hub` functionality](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer.push_to_hub) so that your model will be automatically uploaded to the Hub.
+
+Let's copy the following code snippet in a file called `run.sh`
+
+```bash
+echo '''python run_speech_recognition_ctc.py \
+	--dataset_name="mozilla-foundation/common_voice_7_0" \
+	--model_name_or_path="hf-test/xls-r-dummy" \
+	--dataset_config_name="ab" \
+	--output_dir="./" \
+	--overwrite_output_dir \
+	--max_steps="10" \
+	--per_device_train_batch_size="2" \
+	--learning_rate="3e-4" \
+	--save_total_limit="1" \
+	--evaluation_strategy="steps" \
+	--text_column_name="sentence" \
+	--length_column_name="input_length" \
+	--save_steps="5" \
+	--layerdrop="0.0" \
+	--freeze_feature_encoder \
+	--gradient_checkpointing \
+	--fp16 \
+	--group_by_length \
+	--push_to_hub \
+	--use_auth_token \
+	--do_train --do_eval''' > run.sh
+```
+
+4. **Start training**
+
+Now all that is left to do is to start training the model by executing the 
+run file.
+
+```bash
+bash run.sh
+```
+
+The training should not take more than a couple of minutes. 
+During the training intermediate saved checkpoints are automatically uploaded to
+your model repository as can be seen [on this commit](https://huggingface.co/hf-test/xls-r-ab-test/commit/0eb19a0fca4d7d163997b59663d98cd856022aa6) . 
+
+At the end of the training, the [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer) automatically creates a nice model card and all 
+relevant files are uploaded.
+
+5. **Tips for real model training**
+
+The above steps illustrate how a model can technically be fine-tuned.
+However as you can see on the model card [hf-test/xls-r-ab-test](https://huggingface.co/hf-test/xls-r-ab-test), our demonstration has a very poor performance which is
+not surprising given that we trained for just 10 steps on a randomly initialized
+model.
+
+For a real model training, one of the actual pre-trained XLS-R models should be used:
+
+- [300M parameters version](https://huggingface.co/facebook/wav2vec2-xls-r-300m)
+- [1B parameters version](https://huggingface.co/facebook/wav2vec2-xls-r-1b)
+- [2B parameters version](https://huggingface.co/facebook/wav2vec2-xls-r-2b)
+
+Also, the hyper-parameters should be carefully chosen depending on the dataset.
+As an example, we will fine-tune the 300M parameters model on Swedish on a single 
+TITAN RTX 24GB GPU.
+
+The model will be called `"xls-r-300m-sv"`. 
+Following the above steps we first create the model:
+
+```bash
+huggingface-cli repo create xls-r-300m-sv
+```
+
+and then clone it locally:
+
+```bash
+
+
+and we define the following 
+
+hyperparameters for training
+
+```bash
+
+echo '''python run_speech_recognition_ctc.py \
+	--dataset_name="mozilla-foundation/common_voice_7_0" \
+	--model_name_or_path="facebook/wav2vec2-xls-r-300m" \
+	--dataset_config_name="sv-SE" \
+	--output_dir="./" \
+	--overwrite_output_dir \
+	--num_train_epochs="50" \
+	--per_device_train_batch_size="8" \
+	--per_device_eval_batch_size="8" \
+	--gradient_accumulation_steps="4" \
+	--learning_rate="7.5e-5" \
+	--warmup_steps="2000" \
+	--length_column_name="input_length" \
+	--evaluation_strategy="steps" \
+	--text_column_name="sentence" \
+	--chars_to_ignore , ? . ! \- \; \: \" “ % ‘ ” � — ’ … – \
+	--save_steps="500" \
+	--eval_steps="500" \
+	--logging_steps="100" \
+	--layerdrop="0.0" \
+	--activation_dropout="0.1" \
+	--save_total_limit="3" \
+	--freeze_feature_encoder \
+	--feat_proj_dropout="0.0" \
+	--mask_time_prob="0.75" \
+	--mask_time_length="10" \
+	--mask_feature_prob="0.25" \
+	--mask_feature_length="64" \
+	--gradient_checkpointing \
+	--use_auth_token \
+	--fp16 \
+	--group_by_length \
+	--do_train --do_eval \
+	--push_to_hub''' > run.sh
+```
+
+The training takes *ca.* 7 hours and yields a reasonable test word 
+error rate of 27% as can be seen on the automatically generated [model card](https://huggingface.co/hf-test/xls-r-300m-sv).
+
+The above-chosen hyperparameters probably work quite well on a range of different 
+datasets and languages, but are by no means optimal. It is up to you to find a good set of 
+hyperparameters.
diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py
index 8a809a92e1..aa910fdf83 100644
--- a/src/transformers/file_utils.py
+++ b/src/transformers/file_utils.py
@@ -512,6 +512,14 @@ def is_pytesseract_available():
     return importlib.util.find_spec("pytesseract") is not None
 
 
+def is_spacy_available():
+    return importlib.util.find_spec("spacy") is not None
+
+
+def is_ftfy_available():
+    return importlib.util.find_spec("ftfy") is not None
+
+
 def is_in_notebook():
     try:
         # Test adapted from tqdm.autonotebook: https://github.com/tqdm/tqdm/blob/master/tqdm/autonotebook.py
diff --git a/src/transformers/models/megatron_bert/convert_megatron_bert_checkpoint.py b/src/transformers/models/megatron_bert/convert_megatron_bert_checkpoint.py
index 2a31c0aeec..719b53ba1b 100644
--- a/src/transformers/models/megatron_bert/convert_megatron_bert_checkpoint.py
+++ b/src/transformers/models/megatron_bert/convert_megatron_bert_checkpoint.py
@@ -33,13 +33,14 @@
 #
 
 import argparse
-import json
 import os
 import re
 import zipfile
 
 import torch
 
+from transformers import MegatronBertConfig
+
 
 ####################################################################################################
 
@@ -64,13 +65,62 @@ def recursive_print(name, val, spaces=0):
         print(msg, ":", val)
 
 
+def fix_query_key_value_ordering(param, checkpoint_version, num_splits, num_heads, hidden_size):
+    # Permutes layout of param tensor to [num_splits * num_heads * hidden_size, :]
+    # for compatibility with later versions of NVIDIA Megatron-LM.
+    # The inverse operation is performed inside Megatron-LM to read checkpoints:
+    # https://github.com/NVIDIA/Megatron-LM/blob/v2.4/megatron/checkpointing.py#L209
+    # If param is the weight tensor of the self-attention block, the returned tensor
+    # will have to be transposed one more time to be read by HuggingFace BERT.
+    input_shape = param.size()
+    if checkpoint_version == 1.0:
+        # version 1.0 stores [num_heads * hidden_size * num_splits, :]
+        saved_shape = (num_heads, hidden_size, num_splits) + input_shape[1:]
+        param = param.view(*saved_shape)
+        param = param.transpose(0, 2)
+        param = param.transpose(1, 2).contiguous()
+    elif checkpoint_version >= 2.0:
+        # other versions store [num_heads * num_splits * hidden_size, :]
+        saved_shape = (num_heads, num_splits, hidden_size) + input_shape[1:]
+        param = param.view(*saved_shape)
+        param = param.transpose(0, 1).contiguous()
+    param = param.view(*input_shape)
+    return param
+
+
 ####################################################################################################
 
 
-def convert_megatron_checkpoint(args, input_state_dict):
+def convert_megatron_checkpoint(args, input_state_dict, config):
     # The converted output model.
     output_state_dict = {}
 
+    # old versions did not store training args
+    ds_args = input_state_dict.get("args", None)
+    if ds_args is not None:
+        # do not make the user write a config file when the exact dimensions/sizes are already in the checkpoint
+        # from pprint import pprint
+        # pprint(vars(ds_args))
+
+        config.tokenizer_type = ds_args.tokenizer_type
+        config.vocab_size = ds_args.padded_vocab_size
+        config.max_position_embeddings = ds_args.max_position_embeddings
+        config.hidden_size = ds_args.hidden_size
+        config.num_hidden_layers = ds_args.num_layers
+        config.num_attention_heads = ds_args.num_attention_heads
+        config.intermediate_size = ds_args.get("ffn_hidden_size", 4 * ds_args.hidden_size)
+        # pprint(config)
+
+    # The number of heads.
+    heads = config.num_attention_heads
+    # The hidden_size per head.
+    hidden_size_per_head = config.hidden_size // heads
+    # Megatron-LM checkpoint version
+    if "checkpoint_version" in input_state_dict.keys():
+        checkpoint_version = input_state_dict["checkpoint_version"]
+    else:
+        checkpoint_version = 0.0
+
     # The model.
     model = input_state_dict["model"]
     # The language model.
@@ -80,13 +130,14 @@ def convert_megatron_checkpoint(args, input_state_dict):
 
     # The word embeddings.
     word_embeddings = embeddings["word_embeddings"]["weight"]
+    # Truncate the embedding table to vocab_size rows.
+    word_embeddings = word_embeddings[: config.vocab_size, :]
     # Store the word embeddings.
     output_state_dict["bert.embeddings.word_embeddings.weight"] = word_embeddings
 
     # The position embeddings.
     pos_embeddings = embeddings["position_embeddings"]["weight"]
-    # Trained for 512 x 1024.
-    assert pos_embeddings.size(0) == 512 and pos_embeddings.size(1) == 1024
+    assert pos_embeddings.size(0) == config.max_position_embeddings and pos_embeddings.size(1) == config.hidden_size
     # Store the position embeddings.
     output_state_dict["bert.embeddings.position_embeddings.weight"] = pos_embeddings
 
@@ -96,7 +147,7 @@ def convert_megatron_checkpoint(args, input_state_dict):
     output_state_dict["bert.embeddings.token_type_embeddings.weight"] = tokentype_embeddings
 
     # The transformer.
-    transformer = lm["transformer"]
+    transformer = lm["transformer"] if "transformer" in lm.keys() else lm["encoder"]
 
     # The regex to extract layer names.
     layer_re = re.compile("layers\.(\d+)\.([a-z0-9_.]+)\.([a-z]+)")
@@ -142,8 +193,9 @@ def convert_megatron_checkpoint(args, input_state_dict):
             # Make sure the QKV pointer is nil.
             assert attention_qkv_weight is None, ""
 
+            out_val = fix_query_key_value_ordering(val, checkpoint_version, 3, heads, hidden_size_per_head)
             # Store the tensor as we need the bias as well to interleave QKV and biases.
-            attention_qkv_weight = val
+            attention_qkv_weight = out_val
 
         # Transpose the bias.
         elif op_name == "attention.query_key_value" and weight_or_bias == "bias":
@@ -152,14 +204,15 @@ def convert_megatron_checkpoint(args, input_state_dict):
             assert attention_qkv_weight is not None, ""
 
             # Split the QKV matrix into Q, K and V. Megatron stores Q,K,V interleaved.
-            q = attention_qkv_weight[0 * 1024 : 1 * 1024, :]
-            k = attention_qkv_weight[1 * 1024 : 2 * 1024, :]
-            v = attention_qkv_weight[2 * 1024 : 3 * 1024, :]
+            q = attention_qkv_weight[0 * config.hidden_size : 1 * config.hidden_size, :]
+            k = attention_qkv_weight[1 * config.hidden_size : 2 * config.hidden_size, :]
+            v = attention_qkv_weight[2 * config.hidden_size : 3 * config.hidden_size, :]
 
+            out_val = fix_query_key_value_ordering(val, checkpoint_version, 3, heads, hidden_size_per_head)
             # Split the bias.
-            q_bias = val[0 * 1024 : 1 * 1024]
-            k_bias = val[1 * 1024 : 2 * 1024]
-            v_bias = val[2 * 1024 : 3 * 1024]
+            q_bias = out_val[0 * config.hidden_size : 1 * config.hidden_size]
+            k_bias = out_val[1 * config.hidden_size : 2 * config.hidden_size]
+            v_bias = out_val[2 * config.hidden_size : 3 * config.hidden_size]
 
             # Store.
             output_state_dict[f"{layer_name}.attention.self.query.weight"] = q
@@ -182,24 +235,6 @@ def convert_megatron_checkpoint(args, input_state_dict):
     output_state_dict["bert.encoder.ln.weight"] = transformer["final_layernorm.weight"]
     output_state_dict["bert.encoder.ln.bias"] = transformer["final_layernorm.bias"]
 
-    # The config.
-    output_config = {
-        "vocab_size": word_embeddings.size(0),
-        "hidden_size": 1024,
-        "num_hidden_layers": 24,
-        "num_attention_heads": 16,
-        "hidden_act": "gelu_new",
-        "intermediate_size": 4096,
-        "hidden_dropout_prob": 0.1,
-        "attention_probs_dropout_prob": 0.1,
-        "max_position_embeddings": 512,
-        "type_vocab_size": 2,
-        "initializer_range": 0.2,
-        "layer_norm_eps": 1e-12,
-        "position_embedding_type": "absolute",
-        "use_cache": False,
-    }
-
     # The pooler.
     pooler = lm["pooler"]
 
@@ -230,7 +265,7 @@ def convert_megatron_checkpoint(args, input_state_dict):
     output_state_dict["cls.seq_relationship.bias"] = binary_head["bias"]
 
     # It should be done!
-    return output_state_dict, output_config
+    return output_state_dict
 
 
 ####################################################################################################
@@ -241,30 +276,44 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument("--print-checkpoint-structure", action="store_true")
     parser.add_argument("path_to_checkpoint", type=str, help="Path to the ZIP file containing the checkpoint")
+    parser.add_argument(
+        "--config_file",
+        default="",
+        type=str,
+        help="An optional config json file describing the pre-trained model.",
+    )
     args = parser.parse_args()
 
     # Extract the basename.
     basename = os.path.dirname(args.path_to_checkpoint)
 
     # Load the model.
+    # the .zip is very optional, let's keep it for backward compatibility
     print(f'Extracting PyTorch state dictionary from "{args.path_to_checkpoint}"')
-    with zipfile.ZipFile(args.path_to_checkpoint, "r") as checkpoint:
-        with checkpoint.open("release/mp_rank_00/model_optim_rng.pt") as pytorch_dict:
-            input_state_dict = torch.load(pytorch_dict, map_location="cpu")
+    if args.path_to_checkpoint.endswith(".zip"):
+        with zipfile.ZipFile(args.path_to_checkpoint, "r") as checkpoint:
+            with checkpoint.open("release/mp_rank_00/model_optim_rng.pt") as pytorch_dict:
+                input_state_dict = torch.load(pytorch_dict, map_location="cpu")
+    else:
+        input_state_dict = torch.load(args.path_to_checkpoint, map_location="cpu")
+
+    if args.config_file == "":
+        # Default config of megatron-bert 345m
+        config = MegatronBertConfig()
+    else:
+        config = MegatronBertConfig.from_json_file(args.config_file)
 
     # Convert.
     print("Converting")
-    output_state_dict, output_config = convert_megatron_checkpoint(args, input_state_dict)
+    output_state_dict = convert_megatron_checkpoint(args, input_state_dict, config)
 
     # Print the structure of converted state dict.
     if args.print_checkpoint_structure:
         recursive_print(None, output_state_dict)
 
     # Store the config to file.
-    output_config_file = os.path.join(basename, "config.json")
-    print(f'Saving config to "{output_config_file}"')
-    with open(output_config_file, "w") as f:
-        json.dump(output_config, f)
+    print("Saving config")
+    config.save_pretrained(basename)
 
     # Store the state_dict to file.
     output_checkpoint_file = os.path.join(basename, "pytorch_model.bin")
diff --git a/src/transformers/models/openai/tokenization_openai.py b/src/transformers/models/openai/tokenization_openai.py
index af96fb06a9..ca21943a23 100644
--- a/src/transformers/models/openai/tokenization_openai.py
+++ b/src/transformers/models/openai/tokenization_openai.py
@@ -104,7 +104,7 @@ class OpenAIGPTTokenizer(PreTrainedTokenizer):
             from spacy.lang.en import English
 
             _nlp = English()
-            self.nlp = _nlp.Defaults.create_tokenizer(_nlp)
+            self.nlp = _nlp.tokenizer
             self.fix_text = ftfy.fix_text
         except ImportError:
             logger.warning("ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.")
diff --git a/src/transformers/models/rag/modeling_rag.py b/src/transformers/models/rag/modeling_rag.py
index b4d14f936d..dc2de04b01 100644
--- a/src/transformers/models/rag/modeling_rag.py
+++ b/src/transformers/models/rag/modeling_rag.py
@@ -305,7 +305,7 @@ class RagPreTrainedModel(PreTrainedModel):
         >>> from transformers import RagModel
 
         >>> # initialize a RAG from two pretrained models.
-        >>> model = RagModel.from_question_encoder_generator_pretrained(
+        >>> model = RagModel.from_pretrained_question_encoder_generator(
         ...     "facebook/dpr-question_encoder-single-nq-base", "t5-small"
         ... )
         >>> # saving model after fine-tuning
diff --git a/src/transformers/models/segformer/modeling_segformer.py b/src/transformers/models/segformer/modeling_segformer.py
index afa6d8cde8..1fa4f73266 100755
--- a/src/transformers/models/segformer/modeling_segformer.py
+++ b/src/transformers/models/segformer/modeling_segformer.py
@@ -490,8 +490,8 @@ class SegformerModel(SegformerPreTrainedModel):
         >>> from PIL import Image
         >>> import requests
 
-        >>> feature_extractor = SegformerFeatureExtractor.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512")
-        >>> model = SegformerModel("nvidia/segformer-b0-finetuned-ade-512-512")
+        >>> feature_extractor = SegformerFeatureExtractor.from_pretrained("nvidia/mit-b0")
+        >>> model = SegformerModel.from_pretrained("nvidia/mit-b0")
 
         >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
         >>> image = Image.open(requests.get(url, stream=True).raw)
diff --git a/src/transformers/models/trocr/processing_trocr.py b/src/transformers/models/trocr/processing_trocr.py
index 3166cbae20..a74dc0b38e 100644
--- a/src/transformers/models/trocr/processing_trocr.py
+++ b/src/transformers/models/trocr/processing_trocr.py
@@ -17,25 +17,27 @@ Processor class for TrOCR.
 """
 from contextlib import contextmanager
 
+from transformers import AutoFeatureExtractor, AutoTokenizer
 from transformers.feature_extraction_utils import FeatureExtractionMixin
 from transformers.models.roberta.tokenization_roberta import RobertaTokenizer
 from transformers.models.roberta.tokenization_roberta_fast import RobertaTokenizerFast
-
-from ..auto.feature_extraction_auto import AutoFeatureExtractor
+from transformers.models.xlm_roberta.tokenization_xlm_roberta import XLMRobertaTokenizer
+from transformers.models.xlm_roberta.tokenization_xlm_roberta_fast import XLMRobertaTokenizerFast
 
 
 class TrOCRProcessor:
     r"""
     Constructs a TrOCR processor which wraps a vision feature extractor and a TrOCR tokenizer into a single processor.
 
-    [`TrOCRProcessor`] offers all the functionalities of [`AutoFeatureExtractor`] and [`RobertaTokenizer`]. See the
-    [`~TrOCRProcessor.__call__`] and [`~TrOCRProcessor.decode`] for more information.
+    [`TrOCRProcessor`] offers all the functionalities of [`ViTFeatureExtractor`/`DeiTFeatureExtractor`] and
+    [`RobertaTokenizer`/`XLMRobertaTokenizer`]. See the [`~TrOCRProcessor.__call__`] and [`~TrOCRProcessor.decode`] for
+    more information.
 
     Args:
-        feature_extractor ([`AutoFeatureExtractor`]):
-            An instance of [`AutoFeatureExtractor`]. The feature extractor is a required input.
-        tokenizer ([`RobertaTokenizer`]):
-            An instance of [`RobertaTokenizer`]. The tokenizer is a required input.
+        feature_extractor ([`ViTFeatureExtractor`/`DeiTFeatureExtractor`]):
+            An instance of [`ViTFeatureExtractor`/`DeiTFeatureExtractor`]. The feature extractor is a required input.
+        tokenizer ([`RobertaTokenizer`/`XLMRobertaTokenizer`]):
+            An instance of [`RobertaTokenizer`/`XLMRobertaTokenizer`]. The tokenizer is a required input.
     """
 
     def __init__(self, feature_extractor, tokenizer):
@@ -43,9 +45,11 @@ class TrOCRProcessor:
             raise ValueError(
                 f"`feature_extractor` has to be of type {FeatureExtractionMixin.__class__}, but is {type(feature_extractor)}"
             )
-        if not (isinstance(tokenizer, RobertaTokenizer) or (isinstance(tokenizer, RobertaTokenizerFast))):
+        if not isinstance(
+            tokenizer, (RobertaTokenizer, RobertaTokenizerFast, XLMRobertaTokenizer, XLMRobertaTokenizerFast)
+        ):
             raise ValueError(
-                f"`tokenizer` has to be of type {RobertaTokenizer.__class__} or {RobertaTokenizerFast.__class__}, but is {type(tokenizer)}"
+                f"`tokenizer` has to be of type {RobertaTokenizer.__class__} or {RobertaTokenizerFast.__class__} or {XLMRobertaTokenizer.__class__} or {XLMRobertaTokenizerFast.__class__}, but is {type(tokenizer)}"
             )
 
         self.feature_extractor = feature_extractor
@@ -103,7 +107,7 @@ class TrOCRProcessor:
                 [`PreTrainedTokenizer`]
         """
         feature_extractor = AutoFeatureExtractor.from_pretrained(pretrained_model_name_or_path, **kwargs)
-        tokenizer = RobertaTokenizer.from_pretrained(pretrained_model_name_or_path, **kwargs)
+        tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, **kwargs)
 
         return cls(feature_extractor=feature_extractor, tokenizer=tokenizer)
 
diff --git a/src/transformers/models/vision_text_dual_encoder/modeling_vision_text_dual_encoder.py b/src/transformers/models/vision_text_dual_encoder/modeling_vision_text_dual_encoder.py
index 927012e42e..2112afc1e4 100755
--- a/src/transformers/models/vision_text_dual_encoder/modeling_vision_text_dual_encoder.py
+++ b/src/transformers/models/vision_text_dual_encoder/modeling_vision_text_dual_encoder.py
@@ -214,6 +214,7 @@ class VisionTextDualEncoderModel(PreTrainedModel):
         input_ids=None,
         attention_mask=None,
         position_ids=None,
+        token_type_ids=None,
         output_attentions=None,
         output_hidden_states=None,
         return_dict=None,
@@ -238,6 +239,7 @@ class VisionTextDualEncoderModel(PreTrainedModel):
             input_ids=input_ids,
             attention_mask=attention_mask,
             position_ids=position_ids,
+            token_type_ids=token_type_ids,
             output_attentions=output_attentions,
             output_hidden_states=output_hidden_states,
             return_dict=return_dict,
diff --git a/utils/check_repo.py b/utils/check_repo.py
index 47ff835414..121ea3e755 100644
--- a/utils/check_repo.py
+++ b/utils/check_repo.py
@@ -665,7 +665,7 @@ def check_docstrings_are_in_md():
         raise ValueError(
             "The following files have docstrings written in rst:\n"
             + "\n".join([f"- {f}" for f in files_with_rst])
-            + "To fix this run `doc-builder convert path_to_py_file` after installing `doc-builder`\n"
+            + "\nTo fix this run `doc-builder convert path_to_py_file` after installing `doc-builder`\n"
             "(`pip install git+https://github.com/huggingface/doc-builder`)"
         )
 
diff --git a/utils/check_table.py b/utils/check_table.py
index b0f0ed5f9e..17ab62e4aa 100644
--- a/utils/check_table.py
+++ b/utils/check_table.py
@@ -140,7 +140,8 @@ def get_model_table_from_auto_modules():
 
     # Build the table per se
     table = "|" + "|".join([_center_text(c, w) for c, w in zip(columns, widths)]) + "|\n"
-    table += "|" + "|".join(["-" * w for w in widths]) + "|\n"
+    # Use ":-----:" format to center-aligned table cell texts
+    table += "|" + "|".join([":" + "-" * (w - 2) + ":" for w in widths]) + "|\n"
 
     check = {True: "✅", False: "❌"}
     for name in model_names:


Test patch:
diff --git a/src/transformers/testing_utils.py b/src/transformers/testing_utils.py
index 60521acbfc..a0654bd3f4 100644
--- a/src/transformers/testing_utils.py
+++ b/src/transformers/testing_utils.py
@@ -34,6 +34,7 @@ from .file_utils import (
     is_detectron2_available,
     is_faiss_available,
     is_flax_available,
+    is_ftfy_available,
     is_keras2onnx_available,
     is_librosa_available,
     is_onnx_available,
@@ -46,6 +47,7 @@ from .file_utils import (
     is_scatter_available,
     is_sentencepiece_available,
     is_soundfile_availble,
+    is_spacy_available,
     is_tensorflow_probability_available,
     is_tf_available,
     is_timm_available,
@@ -412,6 +414,26 @@ def require_vision(test_case):
         return test_case
 
 
+def require_ftfy(test_case):
+    """
+    Decorator marking a test that requires ftfy. These tests are skipped when ftfy isn't installed.
+    """
+    if not is_ftfy_available():
+        return unittest.skip("test requires ftfy")(test_case)
+    else:
+        return test_case
+
+
+def require_spacy(test_case):
+    """
+    Decorator marking a test that requires SpaCy. These tests are skipped when SpaCy isn't installed.
+    """
+    if not is_spacy_available():
+        return unittest.skip("test requires spacy")(test_case)
+    else:
+        return test_case
+
+
 def require_torch_multi_gpu(test_case):
     """
     Decorator marking a test that requires a multi-GPU setup (in PyTorch). These tests are skipped on a machine without
diff --git a/tests/test_tokenization_openai.py b/tests/test_tokenization_openai.py
index 1a7568aa5a..6d3730cebc 100644
--- a/tests/test_tokenization_openai.py
+++ b/tests/test_tokenization_openai.py
@@ -20,13 +20,14 @@ import unittest
 
 from transformers import OpenAIGPTTokenizer, OpenAIGPTTokenizerFast
 from transformers.models.openai.tokenization_openai import VOCAB_FILES_NAMES
-from transformers.testing_utils import require_tokenizers
+from transformers.testing_utils import require_ftfy, require_spacy, require_tokenizers
 
 from .test_tokenization_common import TokenizerTesterMixin
 
 
 @require_tokenizers
 class OpenAIGPTTokenizationTest(TokenizerTesterMixin, unittest.TestCase):
+    """Tests OpenAIGPTTokenizer that uses BERT BasicTokenizer."""
 
     tokenizer_class = OpenAIGPTTokenizer
     rust_tokenizer_class = OpenAIGPTTokenizerFast
@@ -132,3 +133,12 @@ class OpenAIGPTTokenizationTest(TokenizerTesterMixin, unittest.TestCase):
     # tokenizer has no padding token
     def test_padding_different_model_input_name(self):
         pass
+
+
+@require_ftfy
+@require_spacy
+@require_tokenizers
+class OpenAIGPTTokenizationTestWithSpacy(OpenAIGPTTokenizationTest):
+    """Tests OpenAIGPTTokenizer that uses SpaCy and ftfy."""
+
+    pass