[Community Event] Doc Tests Sprint
Author: patrickvonplaten
Pull request: no
Labels: name: Good First Issue

### This issue is part of our **Doc Test Sprint**. If you're interested in helping out come [join us on Discord](https://discord.gg/J8bW9u5abB) and talk with other contributors!

Docstring examples are often the first point of contact when trying out a new library! So far we haven't done a very good job at ensuring that all docstring examples work correctly in 🤗 Transformers - but we're now very dedicated to ensure that all documentation examples work correctly by testing each documentation example via Python's doctest (https://docs.python.org/3/library/doctest.html) on a daily basis.

In short we should do the following for all models for both PyTorch and Tensorflow:
1. - Check the current doc examples will run without failure
2. - Check whether the current doc example of the forward method is a sensible example to better understand the model or whether it can be improved. E.g. is the example of https://huggingface.co/docs/transformers/v4.17.0/en/model_doc/bert#transformers.BertForQuestionAnswering.forward a good example of the model? Could it be improved?
3. -  Add an expected output to the doc example and test it via Python's doc test (see **Guide to contributing** below)

Adding a documentation test for a model is a great way to better understand how the model works, a simple (possibly first) contribution to Transformers and most importantly a very important contribution to the Transformers community 🔥 

If you're interested in adding a documentation test, please read through the **Guide to contributing** below.

This issue is a call for contributors, to make sure docstring exmaples of existing model architectures work correctly. If you wish to contribute, reply in this thread which architectures you'd like to take :)

### Guide to contributing:
1. Ensure you've read our contributing [guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) 📜 
2. Claim your architecture(s) in this thread (confirm no one is working on it) 🎯 
3. Implement the changes as in https://github.com/huggingface/transformers/pull/15987 (see the diff on the model architectures for a few examples) 💪 
    - The file you want to look at is in `src/transformers/models/[model_name]/modeling_[model_name].py`, `src/transformers/models/[model_name]/modeling_tf_[model_name].py` or `src/transformers/doc_utils.py` or `src/transformes/file_utils.py`
    - Make sure to run the doc example doc test locally as described in https://github.com/huggingface/transformers/tree/master/docs#for-python-files
    - Optionally, change the example docstring to a more sensible example that gives a better suited result
    - Make the test pass
    - Add the file name to https://github.com/huggingface/transformers/blob/master/utils/documentation_tests.txt (making sure the file stays in alphabetical order)
    - Run the doc example test again locally

    In addition, there are a few things we can also improve, for example : 
    - Fix some style issues: for example, change **``decoder_input_ids```** to **\`decoder_input_ids\`**.
    - Using a small model checkpoint instead of a large one: for example, change **"facebook/bart-large"** to **"facebook/bart-base"** (and adjust the expected outputs if any)
4. Open the PR and tag me @patrickvonplaten @ydshieh or @patil-suraj (don't forget to run `make fixup` before your final commit) 🎊 
    - Note that some code is copied across our codebase. If you see a line like `# Copied from transformers.models.bert...`, this means that the code is copied from that source, and our scripts will automatically keep that in sync. If you see that, you should not edit the copied method! Instead, edit the original method it's copied from, and run make fixup to synchronize that across all the copies. Be sure you installed the development dependencies with `pip install -e ".[dev]"`, as described in the contributor guidelines above, to ensure that the code quality tools in `make fixup` can run.

### PyTorch Model Examples added to tests:
- [ ] **ALBERT** (@vumichien)
- [x] **BART** (@abdouaziz)
- [x] BEiT
- [ ] **BERT** (@vumichien)
- [ ] Bert
- [ ] BigBird (@vumichien)
- [x] BigBirdPegasus
- [x] Blenderbot
- [x] BlenderbotSmall
- [ ] CamemBERT  (@abdouaziz)
- [ ] Canine (@NielsRogge)
- [ ] **CLIP** (@Aanisha)
- [ ] ConvBERT (@simonzli)
- [x] ConvNext
- [ ] CTRL (@jeremyadamsfisher)
- [x] Data2VecAudio
- [ ] Data2VecText
- [ ] DeBERTa (@Tegzes)
- [ ] **DeBERTa-v2** (@Tegzes)
- [x] DeiT
- [ ] DETR
- [ ] **DistilBERT** (@jmwoloso)
- [ ] DPR
- [ ] **ELECTRA** (@bhadreshpsavani)
- [ ] Encoder
- [ ] FairSeq
- [ ] FlauBERT (@abdouaziz)
- [ ] FNet
- [ ] Funnel
- [ ] **GPT2** (@ArEnSc)
- [ ] GPT-J (@ArEnSc)
- [x] Hubert
- [ ] I-BERT (@abdouaziz)
- [ ] ImageGPT
- [ ] LayoutLM (chiefchiefling @ discord)
- [ ] LayoutLMv2
- [ ] LED
- [x] **Longformer** (@KMFODA)
- [ ] LUKE (@Tegzes)
- [ ] LXMERT
- [ ] M2M100
- [x] **Marian**
- [x] MaskFormer (@reichenbch)
- [x] **mBART**
- [ ] MegatronBert
- [ ] MobileBERT (@vumichien)
- [ ] MPNet
- [ ] mT5
- [ ] Nystromformer
- [ ] OpenAI
- [ ] OpenAI
- [x] Pegasus
- [ ] Perceiver
- [x] PLBart
- [x] PoolFormer
- [ ] ProphetNet
- [ ] QDQBert
- [ ] RAG
- [ ] Realm
- [ ] **Reformer**
- [x] ResNet
- [ ] RemBERT
- [ ] RetriBERT
- [ ] **RoBERTa** (@patrickvonplaten )
- [ ] RoFormer
- [x] SegFormer
- [x] SEW
- [x] SEW-D
- [x] SpeechEncoderDecoder
- [x] Speech2Text
- [x] Speech2Text2
- [ ] Splinter
- [ ] SqueezeBERT
- [x] Swin
- [ ] **T5** (@MarkusSagen)
- [ ] TAPAS (@NielsRogge)
- [ ] Transformer-XL (@simonzli)
- [ ] TrOCR (@arnaudstiegler)
- [x] UniSpeech
- [x] UniSpeechSat
- [x] Van
- [x] ViLT
- [x] VisionEncoderDecoder
- [ ] VisionTextDualEncoder
- [ ] VisualBert
- [x] **ViT**
- [x] ViTMAE
- [x] **Wav2Vec2**
- [x] WavLM
- [ ] XGLM
- [ ] **XLM**
- [ ] **XLM-RoBERTa** (@AbinayaM02)
- [ ] XLM-RoBERTa-XL
- [ ] XLMProphetNet
- [ ] **XLNet**
- [ ] YOSO


### Tensorflow Model Examples added to tests:
- [ ] **ALBERT** (@vumichien)
- [ ] **BART**
- [ ] BEiT
- [ ] **BERT** (@vumichien)
- [ ] Bert
- [ ] BigBird (@vumichien)
- [ ] BigBirdPegasus
- [ ] Blenderbot
- [ ] BlenderbotSmall
- [ ] CamemBERT
- [ ] Canine
- [ ] **CLIP** (@Aanisha)
- [ ] ConvBERT (@simonzli)
- [ ] ConvNext
- [ ] CTRL
- [ ] Data2VecAudio
- [ ] Data2VecText
- [ ] DeBERTa
- [ ] **DeBERTa-v2**
- [ ] DeiT
- [ ] DETR
- [ ] **DistilBERT** (@jmwoloso)
- [ ] DPR
- [ ] **ELECTRA** (@bhadreshpsavani)
- [ ] Encoder
- [ ] FairSeq
- [ ] FlauBERT
- [ ] FNet
- [ ] Funnel
- [ ] **GPT2** (@cakiki)
- [ ] GPT-J (@cakiki)
- [ ] Hubert
- [ ] I-BERT
- [ ] ImageGPT
- [ ] LayoutLM
- [ ] LayoutLMv2
- [ ] LED
- [x] **Longformer** (@KMFODA)
- [ ] LUKE
- [ ] LXMERT
- [ ] M2M100
- [ ] **Marian**
- [x] MaskFormer (@reichenbch)
- [ ] **mBART**
- [ ] MegatronBert
- [ ] MobileBERT (@vumichien)
- [ ] MPNet
- [ ] mT5
- [ ] Nystromformer
- [ ] OpenAI
- [ ] OpenAI
- [ ] Pegasus
- [ ] Perceiver
- [ ] PLBart
- [ ] PoolFormer
- [ ] ProphetNet
- [ ] QDQBert
- [ ] RAG
- [ ] Realm
- [ ] **Reformer**
- [ ] ResNet
- [ ] RemBERT
- [ ] RetriBERT
- [ ] **RoBERTa** (@patrickvonplaten)
- [ ] RoFormer
- [ ] SegFormer
- [ ] SEW
- [ ] SEW-D
- [ ] SpeechEncoderDecoder
- [ ] Speech2Text
- [ ] Speech2Text2
- [ ] Splinter
- [ ] SqueezeBERT
- [ ] Swin (@johko)
- [ ] **T5** (@MarkusSagen)
- [ ] TAPAS
- [ ] Transformer-XL (@simonzli)
- [ ] TrOCR (@arnaudstiegler)
- [ ] UniSpeech
- [ ] UniSpeechSat
- [ ] Van
- [ ] ViLT
- [ ] VisionEncoderDecoder
- [ ] VisionTextDualEncoder
- [ ] VisualBert
- [ ] **ViT** (@johko)
- [ ] ViTMAE
- [ ] **Wav2Vec2**
- [ ] WavLM
- [ ] XGLM
- [ ] **XLM**
- [ ] **XLM-RoBERTa** (@AbinayaM02)
- [ ] XLM-RoBERTa-XL
- [ ] XLMProphetNet
- [ ] **XLNet**
- [ ] YOSO

Comments:
@patrickvonplaten I would like to start with Maskformer for Tensorflow/Pytorch. Catch up with how the event goes.

Awesome! Let me know if you have any questions :-)

Hello! I'd like to take on Longformer for Tensorflow/Pytorch please.

@patrickvonplaten I would like to start with T5 for pytorch and tensorflow

Sounds great!

LayoutLM is also taken as mentioned by a contributor on Discord!

@patrickvonplaten I would take GPT and GPT-J (TensorFlow editions) if those are still available.

I'm guessing GPT is GPT2?

I will take Bert, Albert, and Bigbird for both Tensorflow/Pytorch

I'll take Swin and ViT for Tensorflow

I'd like DistilBERT for both TF and PT please

> @patrickvonplaten I would take GPT and GPT-J (TensorFlow editions) if those are still available.
> 
> I'm guessing GPT is GPT2?

@cakiki You can go for GPT2 (I updated the name in the test)

Can I try GPT2 and GPTJ for Pytorch? if @ydshieh  you are not doing so?

I would like to try CLIP for Tensorflow and PyTorch.

I'll take CANINE and TAPAS.

> Can I try GPT2 and GPTJ for Pytorch? if @ydshieh you are not doing so?

@ArEnSc
No, you can work on these 2 models :-) Thank you!

@ydshieh Since the MobileBertForSequenceClassification is the copy of BertForSequenceClassification, so I think I will do check doc-test of MobileBert as well to overcome the error from `make fixup`

I'll take FlauBERT  and CamemBERT.

@abdouaziz Awesome! Do you plan to work on both PyTorch and TensorFlow versions, or only one of them?


I would like to work on LUKE model for both TF and PT

@Tegzes you're lucky because there's no LUKE in TF ;) the list above actually just duplicates all models, but many models aren't available yet in TF.

In this case, I will also take  DeBERTa and DeBERTa-v2 for PyTorch

> @ydshieh 

I plan to work only with PyTorch

> @Tegzes you're lucky because there's no LUKE in TF ;) the list above actually just duplicates all models, but many models aren't available yet in TF.

True - sorry I've been lazy at creating this list! 

Happy to work on TrOCR (pytorch and TF)

I take RoBERTa in PT and TF

I would like to pick up  XLM-RoBERTa in PT and TF.

I can work on `ELECTRA`  for PT and TF

Hey guys, 

We've just merged the first template for Roberta-like model doc tests: https://github.com/huggingface/transformers/pull/16363 :-)
Lots of models like `ELETRA`, `XLM-RoBERTa`, `DeBERTa`, `BERT` are very similar in spirit, it would be great if you could try to rebase your PR to the change done in https://github.com/huggingface/transformers/pull/16363 . Usually all you need to do is to add the correct `{expected_outputs}`, `{expected_loss}` and `{checkpoint}` to the docstring of each model (ideally giving sensible results :-)) until it passes locally and then the file can be added to the tester :-) 

Also if you have open PRs and need help, feel free to ping me or @ydshieh and link the PR here so that we can nicely gather everything :-)

One of the most difficult tasks here might be to actually find a well-working model. As a tip what you can do:

1. Find all models of your architecture as it's always stated in the modeling files here: https://github.com/huggingface/transformers/blob/77c5a805366af9f6e8b7a9d4006a3d97b6d139a2/src/transformers/models/roberta/modeling_roberta.py#L67 e.g. for ELECTRA: https://huggingface.co/models?filter=electra
2. Now click on the task (in left sidebar) your working on, e.g. say you work on `ForSequenceClassification` of a text model go under this task metric: https://huggingface.co/models?other=electra&pipeline_tag=text-classification&sort=downloads
3. Finally, click on the framework metric  (in left sidebar)  you're working with: e.g. for TF: https://huggingface.co/models?library=tf&other=electra&pipeline_tag=text-classification&sort=downloads . If you see too few or too many not well performing models in TF you might also want to think about converting a good PT model to TF under your Hub name and to use this one instead :-)

I'll take a shot with the PyTorch implementation of CTRL

Here the mirror of RoBERTa for Tensorflow: https://github.com/huggingface/transformers/pull/16370

Hi, contributors, thank you very much for participating this sprint ❤️.

Here is one tip that might reduce some issues:

Considering the following 2 facts:

  - A previous file `file_utils.py` contains some code regarding documentation. It was recently refactorized to different files. It might be a good idea (necessary in some case) to update your working branch in your local clone.
 - The file `transformers/utils/documentation_tests.txt` will be updated frequently by different contributors during this event.

Some testing issues could be resolved as:

```bash
git checkout main  # or `master`, depends on your local clone
git fetch upstream
git pull upstream main  # Hugging Face `transformers` renamed the default branch to `main` recently
git checkout your_working_branch_for_this_sprint
git rebase main  # or `master`
```

Don't hesitate if you encounter any problem. Enjoy~

I take BART and IBERT for  PT

I'd like to take a crack on Transformer-XL and ConvBert

Hi, contributors!

For the model(s) you work with for this sprint, if you could not find any checkpoint for a downstream task, say `XXXModelForTokenClassification` model, but there is a checkpoint for the base model, what you could do is:

```python
model = XXXModelForTokenClassification.from_pretrained(base_model_checkpoint_name)
model.save_pretrained(local_path)
```
Then you can upload this new saved checkpoint to Hugging Face Hub, and you can use this uploaded model for the docstring example.

The head part of the model will have randomly initialized weights, and the result is likely to be imperfect, but it is fine for this sprint :-)

> I'd like to take a crack on Transformer-XL and ConvBert

@simonzli, great :-).  Do you plan to work with the PyTorch or TensorFlow version, or both?

> > I'd like to take a crack on Transformer-XL and ConvBert
> 
> 
> 
> @simonzli, great :-).  Do you plan to work with the PyTorch or TensorFlow version, or both?

I'll work on both PyTorch and TensorFlow😊

@patrickvonplaten: I chose XLM-RoBERTa and it's a sub-class of RoBERTa. The comments in the file for both PyTorch and TF  suggests that the superclass should be referred for the appropriate documentation alongside usage examples (XLM-RoBERTa documentations shows RoBERTa examples). Should I still be adding examples for XLM-RoBERTa or should I pick some other model?




@AbinayaM02 : 

Could you show me which line you see `suggests that the superclass should be referred for the appropriate documentation` in the `XLM-RoBERTa` model file, please? Thank you :-)


> @AbinayaM02 :
> 
> Could you show me which line you see `suggests that the superclass should be referred for the appropriate documentation` in the `XLM-RoBERTa` model file, please? Thank you :-)

Hi @ydshieh: Here are the files https://github.com/huggingface/transformers/blob/main/src/transformers/models/xlm_roberta/modeling_xlm_roberta.py
https://github.com/huggingface/transformers/blob/main/src/transformers/models/xlm_roberta/modeling_tf_xlm_roberta.py

Snippet for some classes:

```python
@add_start_docstrings(
    "The bare XLM-RoBERTa Model transformer outputting raw hidden-states without any specific head on top.",
    XLM_ROBERTA_START_DOCSTRING,
)
class XLMRobertaModel(RobertaModel):
    """
    This class overrides [`RobertaModel`]. Please check the superclass for the appropriate documentation alongside
    usage examples.
    """

    config_class = XLMRobertaConfig


@add_start_docstrings(
    "XLM-RoBERTa Model with a `language modeling` head on top for CLM fine-tuning.",
    XLM_ROBERTA_START_DOCSTRING,
)
class XLMRobertaForCausalLM(RobertaForCausalLM):
    """
    This class overrides [`RobertaForCausalLM`]. Please check the superclass for the appropriate documentation
    alongside usage examples.
    """

    config_class = XLMRobertaConfig

```



@AbinayaM02 

Thank you. You can leave `XLM-RoBERTa` as it is.

We should prepare the model list for this sprint in a better way, sorry for this inconvenience.

Would you like to look into another architecture? You can try to check the models whose names are in bold font first,
but other models are also welcomed (they **might** have fewer checkpoints available though).



Sure @ydshieh. I'll pick up XLM for both PyTorch and TF then!

Hi again, contributors:

In a previous comment, I mentioned uploading a checkpoint with random head - if no checkpoint for a specific model + downstream task could be found on the Hub. After some internal discussion, we think there should be a better approach.

Actually, it would be a good idea to check some checkpoints in [hf-internal-testing](https://huggingface.co/hf-internal-testing). In this page, you don't need to check the task type, just check the model architecture.

If you could not find any checkpoint for the model you work with at that page, we encourage you to work with other models for which you could find checkpoints**.

I update the model list to use bold font to indicate those models that are likely to have checkpoints.

> Hi, contributors!
> 
> For the model(s) you work with for this sprint, if you could not find any checkpoint for a downstream task, say `XXXModelForTokenClassification` model, but there is a checkpoint for the base model, what you could do is:
> 
> ```python
> model = XXXModelForTokenClassification.from_pretrained(base_model_checkpoint_name)
> model.save_pretrained(local_path)
> ```
> 
> Then you can upload this new saved checkpoint to Hugging Face Hub, and you can use this uploaded model for the docstring example.
> 
> The head part of the model will have randomly initialized weights, and the result is likely to be imperfect, but it is fine for this sprint :-)



@ydshieh It throws this error when I tried to load one of [hf-internal-testing](https://huggingface.co/hf-internal-testing).
```---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-239-5c839d6ee084>](https://localhost:8080/#) in <module>()
      3 from transformers import AlbertTokenizer, BertTokenizer, BigBirdTokenizer, MobileBertTokenizer
      4 checkpoint = "hf-internal-testing/tiny-random-big_bird"
----> 5 tokenizer = BigBirdTokenizer.from_pretrained(f"{checkpoint}")
      6 model = BigBirdForSequenceClassification.from_pretrained(f"{checkpoint}" )
      7 inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

4 frames
[/usr/local/lib/python3.7/dist-packages/sentencepiece/__init__.py](https://localhost:8080/#) in LoadFromFile(self, arg)
    169 
    170     def LoadFromFile(self, arg):
--> 171         return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
    172 
    173     def DecodeIdsWithCheck(self, ids):

TypeError: not a string
```

Thank you. I think it is because we didn't upload the necessary tokenizer file.
I will talk to the team members. Thank you for spotting this!

I would like to take XLNet for PT and TF.

I'd like to work on DistilBERT for PT and TF with my coworker @jmwoloso 

I just wanted to start on the doc tests for TF Swin, turns out it doesn't exist, only the PyTorch version. So I suppose that can be seen as done ;)

> I just wanted to start on the doc tests for TF Swin, turns out it doesn't exist, only the PyTorch version. So I suppose that can be seen as done ;)

Sure, thank you for the feedback :-). We should have better prepared the model list.
And thank you for the work on TFViT - I will keep you updated after some discussion with our team!

Hi, I'd like to work on Reformer PyTorch.

hey @ydshieh / @patrickvonplaten, should we avoid having `expected_outputs` & `expected_loss` for models that are randomly initialised?

> hey @ydshieh / @patrickvonplaten, should we avoid having `expected_outputs` & `expected_loss` for models that are randomly initialised?

Hi, @KMFODA

It's ok to add them, but as in a previous comment [here](https://github.com/huggingface/transformers/issues/16292#issuecomment-1078999353), we decide that we would like to **use the tiny random models in [hf-internal-testing](https://huggingface.co/hf-internal-testing)**, instead of creating big models with random weights in their heads.

Could you find a checkpoint in `hf-internal-testing` for the model your are working with?

I couldn't find my model no. I'm working with Longformer.

I'd like to try working on the Perceiver

> I couldn't find my model no. I'm working with Longformer.

@KMFODA I have put your GitHub user name after the model name in the model list (in this page).

> hey @ydshieh / @patrickvonplaten, should we avoid having `expected_outputs` & `expected_loss` for models that are randomly initialised?

No, please try to add them. We know this is not ideal, but as long as the checkpoint name contains `random`, it should be fine for now :)

But it would be really appreciated that if you could first try to check if there are existing checkpoint on the Hub for the model + the task. ❤️

@ydshieh 
I got the same error with [hf-internal-testing](https://huggingface.co/hf-internal-testing/tiny-random-reformer) working on Reformer. And I couldn't find any checkpoints without randomly initialized.
https://github.com/huggingface/transformers/issues/16292#issuecomment-1079036359

> @ydshieh I got the same error with [hf-internal-testing](https://huggingface.co/hf-internal-testing/tiny-random-reformer) working on Reformer. And I couldn't find any checkpoints without randomly initialized. [#16292 (comment)](https://github.com/huggingface/transformers/issues/16292#issuecomment-1079036359)

Hi, @hiromu166 

Looks like we didn't upload a tokenizer for `hf-internal-testing/tiny-random-reformer`. These tiny models are created for some internal testing purpose, and we didn't have the intention to use it for doctest until now.

I will upload a tokenizer today! Thank you very much for pointing out this.

@hiromu166 

I uploaded the tokenizer file **spiece.model** to [hf-internal-testing/tiny-random-reformer](https://huggingface.co/hf-internal-testing/tiny-random-reformer/tree/main).

Let me know if you have any problem using it in the doctest 🙏, thanks!



@ydshieh 
Thank you for dealing with it quickly!!
I'll check it soon.

Hi @ydshieh,

For TFElectra i am getting answer index [issue](https://colab.research.google.com/drive/1buzYMKgm3b8kfTcyVvnToJCFLqdnZDxM?usp=sharing)

`doc.py` file `TF_QUESTION_ANSWERING_SAMPLE` has target index at `14` and  `15`, while i am getting  my prediction at `10`, `12` index

@ydshieh quick question. @jessecambon and I are working on the DistilBert model but we're seeing many of the examples are actually drawn and built from the examples in `src/transformers/utils/doc.py` which seems to be a global location, meaning it seems multiple models get their doc tests from this single file. So my question is whether someone else has edited those already or if we're free to edit them as part of our work, which will then flow out to other model architectures.

Hi @bhadreshpsavani , Yes there is  an issue regarding the target indices. Please follow this discussion

https://github.com/huggingface/transformers/pull/16523#issuecomment-1088635272
https://github.com/huggingface/transformers/pull/16523#issuecomment-1090346578

I will discuss the team.

> Hi @ydshieh,
> 
> For TFElectra i am getting answer index [issue](https://colab.research.google.com/drive/1buzYMKgm3b8kfTcyVvnToJCFLqdnZDxM?usp=sharing)
> 
> `doc.py` file `TF_QUESTION_ANSWERING_SAMPLE` has target index at `14` and `15`, while i am getting my prediction at `10`, `12` index



Hi, @jmwoloso & @jessecambon 

`doc.py` is not meant to be modified by the sprint contributor :-)

In order to customize, you can use `add_code_sample_docstrings`, and provide it with `expected_output=...`, `expected_loss=...`, `checkpoint=...`, etc.

You can looks [this change on Roberta](https://github.com/huggingface/transformers/pull/16363/files#diff-5707805d290617078f996faf1138de197fa813f78c0aa5ea497e73b5228f1103) as a reference. 

Let me know if you have any difficulty using this approach.

> @ydshieh quick question. @jessecambon and I are working on the DistilBert model but we're seeing many of the examples are actually drawn and built from the examples in `src/transformers/utils/doc.py` which seems to be a global location, meaning it seems multiple models get their doc tests from this single file. So my question is whether someone else has edited those already or if we're free to edit them as part of our work, which will then flow out to other model architectures.



> Hi @bhadreshpsavani , Yes there is an issue regarding the target indices. Please follow this discussion
> 
> [#16523 (comment)](https://github.com/huggingface/transformers/pull/16523#issuecomment-1088635272) [#16523 (comment)](https://github.com/huggingface/transformers/pull/16523#issuecomment-1090346578)
> 
> I will discuss the team.
> 
> > Hi @ydshieh,
> > For TFElectra i am getting answer index [issue](https://colab.research.google.com/drive/1buzYMKgm3b8kfTcyVvnToJCFLqdnZDxM?usp=sharing)
> > `doc.py` file `TF_QUESTION_ANSWERING_SAMPLE` has target index at `14` and `15`, while i am getting my prediction at `10`, `12` index

By the way, you don't have this issue for `Electra` (PyTorch version)? Do you use different checkpoint for PyTorch/TensorFlow Electra?

> > Hi @bhadreshpsavani , Yes there is an issue regarding the target indices. Please follow this discussion
> > [#16523 (comment)](https://github.com/huggingface/transformers/pull/16523#issuecomment-1088635272) [#16523 (comment)](https://github.com/huggingface/transformers/pull/16523#issuecomment-1090346578)
> > I will discuss the team.
> > > Hi @ydshieh,
> > > For TFElectra i am getting answer index [issue](https://colab.research.google.com/drive/1buzYMKgm3b8kfTcyVvnToJCFLqdnZDxM?usp=sharing)
> > > `doc.py` file `TF_QUESTION_ANSWERING_SAMPLE` has target index at `14` and `15`, while i am getting my prediction at `10`, `12` index
> 
> By the way, you don't have this issue for `Electra` (PyTorch version)? Do you use different checkpoint for PyTorch/TensorFlow Electra?

Hi @ydshieh,
Actually, I am using the same checkpoint.

> Hi @ydshieh, Actually, I am using the same checkpoint. In python, we use `where` to find index but here its hardcoded i guess

What I saw in `PT_QUESTION_ANSWERING_SAMPLE`

```
    ```python
    >>> # target is "nice puppet"
    >>> target_start_index, target_end_index = torch.tensor([14]), torch.tensor([15])

    >>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
    >>> loss = outputs.loss
    >>> round(loss.item(), 2)
    {expected_loss}
```
So if you use the same checkpoint for PT/TF Electra, the PyTorch should have the same issue I think.
Could you try run the doc test (for PT Electra) and see if it pass?

Also, let's move this discussion to your PR.

@ydshieh ok, thank you for the clarification. so is the goal then to move away from doc.py and have individual doc tests in the specific model architecture files?

Hi, @jmwoloso not exactly. We still use `doc.py`, but in the individual model file(s), we tried to use a method `add_code_sample_docstrings` and provide some arguments.

The method `add_code_sample_docstrings` itself will use the example in `doc.py` and customize the doc examples using the arguments you provide.

ok, awesome. thanks @ydshieh!

Hi,

Is anyone working on BART TF version? If not I can try and contribute WRT that model.
Else can you suggest some model from above list(with slightly lower complexity as this will be my first attempt at contribution to transformers library :) ) which can be taken up?

Thanks 

Not sure if someone is working on TF Bart. You can search the Pull request list.

Otherwise, TF Wav2Vec2 should be a good one to try. Should be quite easy with the recent change in 

https://github.com/huggingface/transformers/pull/16494#issue-1186617056

Since no one seems to have taken layoutlmv2 and the last activity on this thread was 25 days ago, I’m going to start working on layoutlmv2 :)

Edit: The PR is ready for review now

@patrickvonplaten I would like to start with `RoFormer` for TF and PT.

@nandwalritik - that's great! Do you want to open a PR for this?

Hi @patrickvonplaten , I am interested in working on data2VecText.  Thanks!

Hi @patrickvonplaten, I would like to work on DETR for TF and PT.

That's awesome! Noted! Also cc @ydshieh 

Great, @ti-ginkgo, looking forward to review your PR!

@patrickvonplaten I would love to work on SqueezeBert for TF and PT

I would like to try to work on DETR (I did a PR already...sorry about that !)

@ydshieh please reassign DETR task to @qherreros 
I will take another one

I'll work on `perceiver`.

hello, I would love to contribute to encoder for PT and TF
Thankyou @sgugger 

I'd like to try Reformer for PyTorch and Tensorflow ☕ 

I would to try for Data2VecText @patrickvonplaten 

@SauravMaheshkar It looks PyTorch Reformer is already done, see [here](https://github.com/huggingface/transformers/blob/10100979ed0594d4cfe1982cdfac9642a68e473e/utils/documentation_tests.txt#L68)

Or do you mean `docs/source/en/model_doc/reformer.mdx`?

@patrickvonplaten I would like to work on Marian for TensorFlow please. Thank You

I would like to work on OpenAI for Pytorch and Tensorflow @ydshieh 

@ydshieh I am working on clip model


@patrickvonplaten I'd like to try working on the CamemBERT model for TensorFlow

@patrickvonplaten I'd like to work on ImageGPT for pytorch

The list in this thread is one year old and outdated, as well as some guidelines. I will have to make some update.

@patrickvonplaten I want to work on FNet

could you please assign this issue to me

@ydshieh Can I work on the CLIP model for PT, if no one is working on it.

Hi @asarthaks  Thank your for the interest on this.

The list is outdated, and CLIP might likely no longer require the changes.

If you find anything in it needs an update, go ahead :-)

Want to work on the RAG , can you assign it to me

I would like to work on the TensorFlow models for CamemBERT and Canine.