Test fetch v2
Author: sgugger
Pull request: yes

# What does this PR do?

This PR rewrites the test fetcher util to be more accurate in the tests collection, and also comes with a restriction on the tests run when a large amount of tests are picked when modifying a core file (like modeling_utils).

The code that extracts the dependencies of a given module now inspects the inits to pinpoint the exact location of imported objects. So for instance if a test file has an import `from transformers import BertModel`, this new version will detect a dependency on `transformers/models/bert/modeling_bert.py`. As a comparison, the previous version stopped at `transformers/__init__.py`. This removes the need for all the complex logic that tried to match a given file with its corresponding tests, we now just look at the dependencies of the test file.

The second change is that when a given file is seen to trigger too many model tests (current trigger is set at half the models, it can evolve), it will only keep the tests relative to a given list of important models. If a PR changes many modeling files, all the tests for those models will still run, but if a PR only changes modeling_utils (for instance), this will trigger the core model tests only. The list of important models is built using:
- the most downloaded models in the last 30 days
- making sure each pipeline has a model in that list

To bypass this rule, one can add a special command in a commit message (circleCI does not have access to labels, so I can't rely on that):
- Including [skip ci] or [ci skip] or [circleci skip] or [skip circleci] or any variants with - or _ instead of a space will skip all tests
- Including [test all models] or any variant with the words in another order and/or with - or _ instead of a space will run all tests found without filtering on important models.
- Including [test all] or [all test] or any variants with - or _ instead of a space will run all tests.

A couple of adjustments to Transformers should be done (in follow-up PRs) to have the test fetcher be more accurate and more efficient:
- make sure all inits don't define any objects. Most of our inits only import all the stuff, and the test fetcher assumes they are all like that. Some inits (like `pipeline/__init__.py`) define real objects, it would be best to move them to a submodule.
- make sure test files test one thing: for instance `test_modeling_common.py` contains both the common tests and the test of the modeling_utils module. It would be best to split those in two files.

Lastly, this PR adds lots of tests to make sure future work doesn't break the test fetcher :-)

To see how the test fetcher behaves on some examples:
- for a modification in modeling_opt.py: only test_modeling_opt is run  [[fetch summary](https://app.circleci.com/pipelines/github/huggingface/transformers/60906/workflows/79cfaf18-d0da-4a5d-8bc6-fd8599b63468/jobs/747209/artifacts)] [[job page](https://app.circleci.com/pipelines/github/huggingface/transformers/60710/workflows/f43d1235-3337-484f-b79a-9260e24e3664)]
- for a modification in modeling_bert.py (which is imported in all the tests basically) all tests using BERT are run, but filtered to the list of important models [[fetch summary](https://app.circleci.com/pipelines/github/huggingface/transformers/60906/workflows/79cfaf18-d0da-4a5d-8bc6-fd8599b63468/jobs/747209/artifacts)] [[job page](https://app.circleci.com/pipelines/github/huggingface/transformers/60906/workflows/f85bd255-a01a-49af-a852-b7a00e13aad3)]
- for a modification in a pipeline file: all model tests are run, filtered to the list of important models [[fetch summary](https://app.circleci.com/pipelines/github/huggingface/transformers/60915/workflows/ab1dc699-5ef7-4b33-bcab-62f76688e9f4/jobs/747362/artifacts)] [[job page](https://app.circleci.com/pipelines/github/huggingface/transformers/60910/workflows/3f9df353-e3b0-490c-9724-f6ef59df5599)]
- for a modification in the main `__init__.py` all tests are run, but filtered to the list of important models [[fetch summary](https://app.circleci.com/pipelines/github/huggingface/transformers/60910/workflows/1f8039a1-af57-477d-bd20-43324d574549/jobs/747284/artifacts)] [[job page](https://app.circleci.com/pipelines/github/huggingface/transformers/60915/workflows/9615653e-b1e5-4c74-9668-ca7983bc3b68)]
- for a modification in the `setup.py` all tests are run [[fetch summary](https://app.circleci.com/pipelines/github/huggingface/transformers/60913/workflows/fd3a907c-98a3-40b0-8423-d1346b23498c/jobs/747326/artifacts)] [[job page](https://app.circleci.com/pipelines/github/huggingface/transformers/60913/workflows/809e8564-7df0-4321-8d7c-4e68c1d123a7)]

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I guess it's ready for a review?

No I haven't finished this PR yet.

Hi @sgugger . Thank you a lot for working on this important task! I feel it's better for me to look this work in depth, and I tried to play with the test fetcher (on `main` and on this PR) to understand it better. 

However, the first thing I tried (by following some sentences you mentioned) makes me a somehow confused. Here is what I saw:

- On the two branches `main` (or a new branch from it) and `test_fetch_v2`, do the following steps:
  - change the test file `tests/models/bert/test_modeling_bert.py`  (simply adding some dummy line like `foo = 1`)
  - commit the change
    ```bash
    git add tests/models/bert/test_modeling_bert.py
    git commit -m "dummy commit"
     ```
  - run the test fetcher against the previous commit
    ```bash
    python utils/tests_fetcher.py  --diff_with_last_commit
    ```
- Now, the results:
  TL;DR: `test_modeling_bert.py` is not included by the new version of test fetcher. But I think it should be included.

  - on `main`
  (`tests/models/bert/test_modeling_bert.py` is in `TEST TO RUN` and in the file `test_list.txt`)
    ```
    ### DIFF ###

    ### MODIFIED FILES ###
    - tests/models/bert/test_modeling_bert.py

    ### IMPACTED FILES ###
    - tests/models/auto/test_modeling_auto.py
    - tests/models/auto/test_modeling_tf_auto.py
    - tests/models/bert/test_modeling_bert.py
    - tests/models/encoder_decoder/test_modeling_encoder_decoder.py
    - tests/models/speech_encoder_decoder/test_modeling_speech_encoder_decoder.py
    - tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py
    - tests/models/vision_text_dual_encoder/test_modeling_vision_text_dual_encoder.py

    ### TEST TO RUN ###
    - tests/models/auto/test_modeling_auto.py
    - tests/models/auto/test_modeling_tf_auto.py
    - tests/models/bert/test_modeling_bert.py
    - tests/models/encoder_decoder/test_modeling_encoder_decoder.py
    - tests/models/speech_encoder_decoder/test_modeling_speech_encoder_decoder.py
    - tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py
    - tests/models/vision_text_dual_encoder/test_modeling_vision_text_dual_encoder.py
    ```

  - on `test_fetch_v2`
    (`tests/models/bert/test_modeling_bert.py` is **NEITHER** in `TEST TO RUN`, **NOR** in the file `test_list.txt`)
    ```
    ### MODIFIED FILES ###
    - tests/models/bert/test_modeling_bert.py

    ### IMPACTED FILES ###
    - tests/models/auto/test_modeling_auto.py
    - tests/models/auto/test_modeling_tf_auto.py
    - tests/models/bert/test_modeling_bert.py
    - tests/models/encoder_decoder/test_modeling_encoder_decoder.py
    - tests/models/speech_encoder_decoder/test_modeling_speech_encoder_decoder.py
    - tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py
    - tests/models/vision_text_dual_encoder/test_modeling_vision_text_dual_encoder.py

    ### TEST TO RUN ###
    - tests/models/auto/test_modeling_auto.py
    - tests/models/auto/test_modeling_tf_auto.py
    - tests/models/encoder_decoder/test_modeling_encoder_decoder.py
    - tests/models/speech_encoder_decoder/test_modeling_speech_encoder_decoder.py
    - tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py
    - tests/models/vision_text_dual_encoder/test_modeling_vision_text_dual_encoder.py
    ``` 
 

> As a comparison, the previous version stopped at transformers/__init__.py.

Is the following block (on `main`) what you mentioned by the above sentence?
```
    # We ignore the main init import as it's only for the __version__ that it's done
    # and it would add everything as a dependency.
    if not imported_module.endswith("transformers/__init__.py"):
        ...
```
 
----------------------------
[Not question - just to record something so I won't forget later]
I tried to change `src/transformers/models/bert/modeling_bert.py`, and I can see 
- `src/transformers/__init__.py` is given as impacted in both versions
- `src/transformers/models/gpt2/xxx` is given as impacted in the version on `main` but not the version on this PR
- `tests/models/gpt2/xxx` is NOT given as impacted in the version on `main` but given in the version on this PR.
  - but its in tests to run in bother version 


Well,  at least, when `src/transformers/models/bert/modeling_bert.py` is changed, the test file `tests/models/bert/test_modeling_bert.py` included 👍 . So the dependency detection seems to work well, and the above situation is just an edge case (to including self)

@ydshieh, good catch on a modified test file missing from the tests launched. I have only put the dependencies and forgot those. Will fix.

@ydshieh did you want to review more or is it good to merge?

> @ydshieh did you want to review more or is it good to merge?

Hi @sgugger If you feel urgent to merge, go ahead (I can leave comments afterward anyway). Otherwise, I would love to continue the review process despite I am slow.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22367). All of your documentation changes will be reflected on that endpoint.