[CI] remove run_tests_torch_and_tf
Author: sshleifer
Pull request: yes

Problem:
This job is rerunning all tests, after `run_tests_torch` and `run_tests_tf` with both packages installed. Meaning that it ~doubles the runtime of CI and doesn't find very many bugs.

Solution: This changes `run_tests_torch_and_tf` to only run the tests that require both TF and torch

Comments:
(fixed git history locally for future PRs)

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4266?src=pr&el=h1) Report
> Merging [#4266](https://codecov.io/gh/huggingface/transformers/pull/4266?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/3487be75ef8f0d51c51208f266644fc04a947085&el=desc) will **decrease** coverage by `0.84%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4266/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4266?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4266      +/-   ##
==========================================
- Coverage   78.39%   77.55%   -0.85%     
==========================================
  Files         120      120              
  Lines       19932    19932              
==========================================
- Hits        15626    15458     -168     
- Misses       4306     4474     +168     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4266?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4266/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0.00%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/4266/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.71% <0.00%> (-10.00%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/4266/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `95.19% <0.00%> (-2.63%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/4266/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.44% <0.00%> (-2.30%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4266/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.41% <0.00%> (-1.38%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4266/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `40.51% <0.00%> (-0.58%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4266/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.93% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4266?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4266?src=pr&el=footer). Last update [3487be7...4aab0f5](https://codecov.io/gh/huggingface/transformers/pull/4266?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I found two common tests that require both packages:
```
test_batch_encode_plus_tensors
test_pt_tf_model_equivalence
```
So I run test_pt_tf_model_equivalence for all test_modeling_tf* files, and
test_batch_encode_plus_tensors for all files. 
Still saves quite a bit of time.


Kinda reluctant to do this change because then we need to manually maintain over time a whitelist of tests, which complicates the (already convoluted) CI setup. I'm also worried we might miss some side-effects of having both frameworks installed.

What time do those tests take?

6 mins / 13 mins, at least once per PR.
I think it's a good tradeoff.
- we are losing 0 coverage.
- people rarely add new tests that check equivalence of torch and tf.  We would rarely need to update the whitelist.
- For most bugs, we would all be able to reproduce it locally where we have both frameworks installed.

Don't feel too strongly, of course.
- Also happy to move the old job to less often then every push.

What does 6 mins / 13 mins mean?

Before this change `run_test_torch_and_tf` takes 6 mins, and the whole CircleCI suite takes 13mins.

How do you measure the whole CircleCI suite, given that the tests are (AFAIK) parallelized?

By looking at circleci output like [this](https://app.circleci.com/pipelines/github/huggingface/transformers/5174/workflows/3aecb0e9-14f0-4762-ad6a-73b60db2d626) and guessing


[This](https://app.circleci.com/pipelines/github/huggingface/transformers/6581/workflows/b114b306-2553-42cf-bb67-ba507e051929) is after pipelines improvements, so I may be overestimating the benefit of the change. Although it seems ridiculous that `check_code_quality` took 3 mins.

@julien-c seems like we don't want to do this? Happy to abandon.