Make TF pt-tf equivalence test more aggressive
Author: ydshieh
Pull request: yes

# What does this PR do?

Make TF pt-tf equivalence test more aggressive.

After a series of fixes done so far, I think it is a good time to include this more aggressive testing in `master` branch.
(Otherwise, the new models added might have undetected issues. For example, the recent `TFConvNextModel` would have `hidden_states` not transposed to match Pytorch version issue - I tested it on my local branch and informed the author to fix it).

There are still 3 categories of PT/TF inconsistency to address, but they are less urgent in my opinion. See below.
Currently, the test makes a few exception to not test these 3 cases (in order to get green test) - I add `TODO` comments in the code.

TF: @Rocketknight1 @gante 
Test: @LysandreJik  @sgugger 

## TODO in separate PRs

- failing due to the difference of large negative values (used for attn mask) between PT/TF:
  - https://circleci.com/api/v1.1/project/github/huggingface/transformers/379126/output/112/0?file=true&allocation-id=621b51ac72134246517cac0c-0-build%2F6E4C1534
  - albert, convbert, speech_to_text, t5, tapas 
- failing due to cache format difference between PT/TF:
  - https://app.circleci.com/pipelines/github/huggingface/transformers/35203/workflows/d0c0fbb4-661b-4a61-96e7-2fd4a3249f44/jobs/379098/parallel-runs/0/steps/0-112
  - gpt2, t5, speech_to_text
- failing due to outputting loss or not between PT/TF:
  - https://app.circleci.com/pipelines/github/huggingface/transformers/35206/workflows/d2533f38-6877-4db5-9eb6-da92e6421e11/jobs/379144/parallel-runs/0/steps/0-112
  - flaubert, funnel, transfo_xl, xlm

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15839). All of your documentation changes will be reflected on that endpoint.

> Thank you for your time spent making this better! Two questions:
> 
>     * Does it run on GPU?

For now, I haven't tested it with GPU. I can run it on the office's  GPU machine this week (a good chance to learn how to connect to those machine!)

>     * How long does it take to run? Our test suite is already taking a significant time to run, so aiming for common tests that run as fast as possible is important.

Let me measure the timing of this test with the current master and with this PR. Will report it.

> 
> 
> There's quite a bit of model-specific logic which I'm not particularly enthusiastic about (pre-training models + convnext), but I understand why it's rigorous to do it like that here.

(Yeah, once we fix all the inconsistency, we can remove all these exceptional conditions.)

Good news! Testing on single GPU with the small tolerance `1e-5` still works! All models pass `test_pt_tf_model_equivalence`.
(I forgot to install CUDA driver - and I'm glad I double-checked and fixed this silly mistake :-) )

I will address a few style review suggestions. After that, I think it's ready to merge ..?

It's weird that it took more time when you expected it to take less, no? Can you try running the test suite with `--durations=0` to see all the tests and the time it took for them to run? 

After a thorough verification and a few fixes, this PR is ready (again) from my side.

I would love @sgugger  (and @LysandreJik  when he is back) to check it again, and @gante  & @Rocketknight1  if they want (no particular TF-related change compared to the last commit).

The following summary might save some review (again) time

- **Main fixes** (since the last review):

  - In the last commit, the following line forgot to use `to_tuple()`, and the method `check_output` (previous version) only deal with `tuple` or `tensor`. So half test cases passed just because no check was performed at all. (my bad ...).
https://github.com/huggingface/transformers/blob/6f2025054dc49a42dc3de82cbea22f2a5913f122/tests/test_modeling_tf_common.py#L447
This is fixed now, and also `check_output` now raises error if a result is not tested (if its type is not in `[tuple, list, tensor]`).

  - In the last commit, I accidentally deleted the following test case (`check the results after save/load checkpoints`, which exists on the current `master`).
https://github.com/huggingface/transformers/blob/6f2025054dc49a42dc3de82cbea22f2a5913f122/tests/test_modeling_tf_common.py#L576
This is added back now.

  - Make the test also run on GPU (put models/inputs on the correct device)

- **Style changes**

  - Since I have to add back the accidentally deleted test cases , I added `check_pt_tf_models` to avoid duplicated large code block.
  - `tfo` -> `tf_outputs` and `pto` -> `pt_outputs`
  - no need to use `np.copy` in [here](https://github.com/huggingface/transformers/blob/e9555ea9c75c3fb2741e27a1be7c2240ce8395f2/tests/test_modeling_tf_common.py#L406), which was discussed in [a comment](https://github.com/huggingface/transformers/pull/15256#discussion_r791942224).

I run this new version of test, and with the small tolerance `1e-5`, it pass both on CPU and GPU :tada:!

In order to be super sure, I also ran it on CPU/GPU for 100 times (very aggressive :fire: :fire: )! All models passed for this test, except for:

  - `TFSpeech2TextModel`: this is addressed in #15952
  - (`TFHubertModel`): There is 1 among 300 runs on CPU where I got a diff > 1e-5. I ran it 10000 times again, and still got only 1 such occurrence. (Guess it's OK to set `1e-5` in this case.)
   
Regarding the running time, I will measure it and post the results in the next comment.

In terms of running time:

- Circle CI
  - current : [56.11s](https://circleci.com/api/v1.1/project/github/huggingface/transformers/384755/output/112/0?file=true&allocation-id=62238c1d4f148c1110a1efa8-0-build%2F1610CB69)
  - this PR : [61.21s](https://circleci.com/api/v1.1/project/github/huggingface/transformers/384731/output/112/0?file=true&allocation-id=622385347ccc3f3894ec7697-0-build%2F351FF724)

- GCP VM (with `-n 1` + CPU)

  - current : 178.75s
  - this PR : 200.81s

These suggest an increase by roughly 10%.

I think I made a mistake that ~~TF 2.8 doesn't work with CUDA 11.4 (installed on my GCP VM machine)~~ K80 GPU doesn't work well with Ubuntu 20.04 (regarding the drivers and CUDA), and it fallbacks to CPU instead. I will fix this and re-run the tests.

After using Tesla T4 GPU (same as for the CI GPU testings), I confirmed that this aggressive PT/TF equivalence test passes with `1e-5` when running on GPU!

I ran this test **1000** times (on GPU) for each TF models.

Think it's ready if @LysandreJik is happy with the [slightly(?) increased running time](https://github.com/huggingface/transformers/pull/15839#issuecomment-1059791405).

(I saved all the differences. I can share the values if you would like to see them!)