Add Doc Test for BERT
Author: vumichien
Pull request: yes

# What does this PR do?

Add doc tests for BERT, a part of issue #16292 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten, @ydshieh 
Documentation: @sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi, @vumichien, 

Thank you for this PR. I currently only have a quick look, and overall it is good 😊.

I saw that your use different checkpoints for PyTorch & TensorFlow BERT models in some cases.
I understand it very well, because some checkpoints only exist in PyTorch version.

After an internal discussion, although we think it is totally fine, we would prefer to use the same checkpoints for the PyTorch & TensorFlow models. Therefore, I will try to convert some PyTorch checkpoints you used in this PR, and upload them to Hugging Face Hub. Once this is done, we can change the checkpoints in this PR, and update the expected values (which should be just about copying the values from the PyTorch side).

I will keep you updated 😊!

@ydshieh Thank you very much. I am looking for your update.
Besides that, could you please take a look at `modeling_mobilebert.py`?  When I run `make fixup`, it asks me to run `make fix-copies` . If I follow, the model checkpoint, expected_output, expected_loss,.. will be copied from `bert` to `mobilebert`, which leads to unwanted results.
I think it is due to these hashtags: 
https://github.com/huggingface/transformers/blob/9de70f213eb234522095cc9af7b2fac53afc2d87/src/transformers/models/mobilebert/modeling_mobilebert.py#L1315-L1318
https://github.com/huggingface/transformers/blob/9de70f213eb234522095cc9af7b2fac53afc2d87/src/transformers/models/mobilebert/modeling_mobilebert.py#L1419-L1422
https://github.com/huggingface/transformers/blob/9de70f213eb234522095cc9af7b2fac53afc2d87/src/transformers/models/mobilebert/modeling_mobilebert.py#L1517-L1520
So what should I do to avoid this problem?




> # Copied from transformers.models.bert.modeling_bert.BertForQuestionAnswering with Bert->MobileBert all-casing

Hi, this is very tricky, and I need some discussion with the team.
This `# Copied from` thing is used to keep track the origin of some code blocks, and it's much better to keep it as many as possible. Thank you for pointing out this issue!



@vumichien 

I uploaded the TensorFlow checkpoint here

For `TFBertForSequenceClassification`:

    "ydshieh/bert-base-uncased-yelp-polarity"

For `TFBertForQuestionAnswering`:

    "ydshieh/bert-base-cased-squad2"

Whenever you have time, could you change the correspondence places in `modeling_tf_bert.py` to use these checkpoint?
Ping me if you encounter any difficulty.

Thanks!


@ydshieh could you please check again your check point `ydshieh/bert-base-cased-squad2` for `TFBertForQuestionAnswering`?Every time I run, the output is different  (maybe the weight of head is still random)

> @ydshieh could you please check again your check point `ydshieh/bert-base-cased-squad2` for `TFBertForQuestionAnswering`?Every time I run, the output is different (maybe the weight of head is still random)

~~OK, I will take a look today.~~

@vumichien 

I checked it, and it turns out that I loaded the PyTorch (QA) checkpoint into a TF model for another task type!
Sorry about this, ~~I will upload the corrected TF checkpoint today~~, and keep you updated!

Uploaded the correct TF checkpoint 🙂 and tested it myself. The result now is always the same (and same as the PT result).

Hi, @vumichien ,

regarding [fix-copies](https://github.com/huggingface/transformers/pull/16523#issuecomment-1085865707) you mentioned in a previous comment: after some discussion, we think the best way is:

- In `Bert` model file:
  - at the beginning part of the model file, set something like
    - EXPECTED_OUTPUT_FOR_TOKEN_CLASSIFICATION = ...
    - EXPECTED_LOSS_FOR_TOKEN_CLASSIFICATION = ...
  - In `add_code_sample_docstrings` for`BertForTokenClassification`, use
    -  expected_output=EXPECTED_OUTPUT_FOR_TOKEN_CLASSIFICATION
    - expected_loss=EXPECTED_LOSS


- In `MobileBert` model file:
  - at the beginning part of the model file, set something like
    - EXPECTED_OUTPUT_FOR_TOKEN_CLASSIFICATION = ... `(set to empty string if you don't want to work on doctest for MobileBert)`
    - EXPECTED_LOSS_FOR_TOKEN_CLASSIFICATION = ... `(set to empty string if you don't want to work on doctest for MobileBert)`
  - In `add_code_sample_docstrings` for`MobileBertForTokenClassification`, use
    -  expected_output=EXPECTED_OUTPUT_FOR_TOKEN_CLASSIFICATION
    - expected_loss=EXPECTED_LOSS

And if you don't provide the values for `MobileBert`, just don't add it to `documentation_tests.txt`.

This should avoid the issue coming from `copies`. Let us know if you encounter any difficulty, thanks!


Thank you very much @ydshieh. I will update the docs test following your instructions 

Hi @ydshieh 
- I have followed your instructions [define-expected-output](https://github.com/huggingface/transformers/pull/16523#issuecomment-1087875387) but the problem doesn't solve. Please help me check if I did something wrong.
- In this PR, I think it's better if I do both `bert` and `mobilebert` at the same time but it still requires [fix-copies](https://github.com/huggingface/transformers/pull/16523#issuecomment-1085865707). 
- I am still waiting for your update for `The loss is always 0.0 (or nan) because the shape of input_ids is [1, 14] `
 with `QuestionAnswering ` task 🙂. It leads to the problem with `MobileBertForQuestionAnswering` and `MobileBertForQuestionAnswering` as well

Hi @vumichien Let me check, both the fix-copies & loss 0.0 things.


@vumichien

Regarding `fix-copies`, I probably should be more thorough in the previous comment.
One reason for the current PR version not working is:

In `BertForSequenceClassification`:  you have

```python
checkpoint="textattack/bert-base-uncased-yelp-polarity",
```

however, in `MobileBertForSequenceClassification`, you have

```python
checkpoint="lordtt13/emo-mobilebert",
```

And this difference will be detected by `make fix-copies`.

In general, these situations could be solved using the same method:

  - set `_CHECKPOINT_FOR_SEQUENCE_CLASSIFICATION = ...` at the beginning of the model file
  - set `checkpoint=_CHECKPOINT_FOR_SEQUENCE_CLASSIFICATION ` in `xxxForSequenceClassification` model.

The same approach applies to other models like `xxxForQuestionAnswering`.

Could you try it and let me know if you still have problem regarding this, please?

In the meantime, I am going to check the loss 0.0.

@patrickvonplaten 

I think we need to update `PT_QUESTION_ANSWERING_SAMPLE`.

The indices (hard-coded) below
```python
>>> target_start_index, target_end_index = torch.tensor([14]), torch.tensor([15])
```
actually depends on the different tokenizers. See the code snippet below for `Albert` v.s. `Roberta`.
Let me know your thought on this, thanks!

## Code Snippet

```python
from transformers import AlbertTokenizer, AlbertForQuestionAnswering, RobertaTokenizer
import torch

albert_checkpoint = "twmkn9/albert-base-v2-squad2"
roberta_checkpoint = "deepset/roberta-base-squad2"

albert_tokenizer = AlbertTokenizer.from_pretrained(f"{albert_checkpoint}")
roberta_tokenizer = RobertaTokenizer.from_pretrained(f"{roberta_checkpoint}")

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"

albert_input_ids = albert_tokenizer(question, text, return_tensors="pt").input_ids.numpy().tolist()[0]
roberta_input_ids = roberta_tokenizer(question, text, return_tensors="pt").input_ids.numpy().tolist()[0]

albert_decoded_tokens = albert_tokenizer.convert_ids_to_tokens(albert_input_ids)
roberta_decoded_tokens = roberta_tokenizer.convert_ids_to_tokens(roberta_input_ids)

# Albert
print(f"Albert: tokens = {albert_decoded_tokens}")
print(f"Albert: num_tokens = {len(albert_decoded_tokens)}")
print(f"Albert: position of `_nice`: {albert_decoded_tokens.index('▁nice')}\n")

# Roberta
print(f"Roberta: tokens = {roberta_decoded_tokens}")
print(f"Roberta: num_tokens = {len(roberta_decoded_tokens)}")
print(f"Roberta: position of `Ġnice`: {roberta_decoded_tokens.index('Ġnice')}")
```

### Outputs

```
Albert: tokens = ['[CLS]', '▁who', '▁was', '▁jim', '▁henson', '?', '[SEP]', '▁jim', '▁henson', '▁was', '▁a', '▁nice', '▁puppet', '[SEP]']
Albert: num_tokens = 14
Albert: position of `_nice`: 11

Roberta: tokens = ['
', 'Who', 'Ġwas', 'ĠJim', 'ĠH', 'enson', '?', '
', '
', 'Jim', 'ĠH', 'enson', 'Ġwas', 'Ġa', 'Ġnice', 'Ġpuppet', '
']
Roberta: num_tokens = 17
Roberta: position of `Ġnice`: 14
```

@ydshieh Thank you very much for your clear explanation. Now I understand why we should do it to get over the problem with `fix-copies` 🙏 

Regarding the issue mentioned [here](https://github.com/huggingface/transformers/pull/16523#issuecomment-1088635272), one solution is to pass `target_start_index` and  `target_end_index` to the sample defined in `doc.py` (we need to update the sample though to accept these).

I don't think there is a super good heuristic to determine these targets in the sample directly. Even with these 2 tokenizers, we already have `'▁nice'` v.s. `'Ġnice'`.

Let's wait Patrick's response.


Hey @ydshieh and @vumichien, 

IMO the best thing we can and should do here is to let the user pass the label idx.

@vumichien  You can ignore the failed test ` run_tests_hub `. I will check the PR later

@ydshieh Thank you very much

All good :-) I merge it now, thank you again ❤️ @vumichien 

@ydshieh You, too. Thanks a lot for helping with this PR 🙏