Update self-push workflow
Author: ydshieh
Pull request: yes

# What does this PR do?

Update self-push CI workflow file:

- `tests_fetcher.py` is updated to output a json file, containing a dictionary mapping test categories to the identified test files (which is used by the updated push CI below)
- Reorganize the tests into models (e.g. `models/bert`, `models/gpt2`, etc.) and modeling categories (`pipeline`, `tokenization`), same as in scheduled CI
- `notification_service.py` and `self-scheduled.yml` are updated to use `[single/multi]-gpu` as artifact name prefixes (i.e. no more `-docker` at the end): with this minimal change, `notification_service.py` could be reused

Some workflow runs:
- [push CI](https://github.com/huggingface/transformers/actions/runs/2306332297)
- [scheduled CI](https://github.com/huggingface/transformers/actions/runs/2306421236)

Some tests failed intentionally (to verify their reports). The reports could be found on `transformers-ci-feedback-tests` channel.

### TODO:
  - create new report channel and add the channel ID to the workflow file

**I added some reviews that contain some of my questions.**

@sgugger Maybe you could have a look for the changes in `test_fetcher.py`?
@stas00 Maybe for the changes regarding DeepSpeed and multi-gpu configurations?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> I find it difficult to make sense of some of the changes, due to diff being hard to follow

OK @stas00 , the big diff is probably because I removed unused blocks. No real big change - I just copied from scheduled CI.
See below if you would like to have a quick look. 

My only questions are

  - why we use `options: --gpus 0` previously in `run_tests_torch_cuda_extensions_multi_gpu`
  - could we use `options: --gpus all` for single gpu case as well as multi gpu?

prev.
```
image: nvcr.io/nvidia/pytorch:21.03-py3
options: --gpus 0
```
now.
```
huggingface/transformers-pytorch-deepspeed-latest-gpu
options: --gpus all
```

and

prev.
```
      - name: Install dependencies
        run: |
          apt -y update && apt install -y libaio-dev
          pip install --upgrade pip
          pip install .[deepspeed-testing]
```
now
```
      - name: Re-compile DeepSpeed
        working-directory: /workspace
        run: |
          pip install deepspeed # installs the deps correctly
          rm -rf DeepSpeed
          git clone https://github.com/microsoft/DeepSpeed && cd DeepSpeed && rm -rf build
          DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 python3 -m pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check
```



thank you for highlighting the changes @ydshieh - that's super helpful.

1. So for gpus:

- multi-gpu should be `--gpus all` (needs 2 gpus)
- single-gpu should be `--gpus 0` (must have only one gpu)

so looking at the diff the original seems to be correct. perhaps not everywhere?

2. For dependencies the original is correct.

Have a look at what it signifies:
```
extras["deepspeed-testing"] = extras["deepspeed"] + extras["testing"] + extras["optuna"]
```

so the change is missing important dependencies install.

and the new instructions aren't correct.

We only want the bleed edge (your now) install only for nightly build. self-push should use the released `deepspeed` version, that `pip install .[deepspeed-testing]` takes care of (but which of course can be moved into the docker if it's running via the docker image). If it's already there, then there is no need for that last pip call either.

Bottom line - no change from the original in either case logically.

If I missed something please let me know.


> so looking at the diff the original seems to be correct. perhaps not everywhere?

The current main branch has a job `run_tests_torch_cuda_extensions_multi_gpu` in `self-push.yml` which has `--gpus 0`.
In the latest commit in this PR, I reverted to the  original version regarding DeepSpeed parts, but set `--gpus all` for multi-gpu job.

Remark 1: some places in `self-scheduled.yml` have to be fixed.)

Remarks 2: I checked this doc [expose-gpus-for-use](https://docs.docker.com/config/containers/resource_constraints/#expose-gpus-for-use), and think we can still use `--gpus all` even if the host machine has only 1 GPU. `--gpus 0` is necessary only if the host has multiple GPUs but we want to use only 1 of them. 

> 2. For dependencies the original is correct.
> self-push should use the released `deepspeed` version, that `pip install .[deepspeed-testing]` 

~~I will change back to the original version for this part~~ (Done), thank you.


As long as the tests are run with `CUDA_VISIBLE_DEVICES=0` for `run_tests_single_gpu` jobs it indeed doesn't matter if more than 1 gpu is available.

But it's critical we ensure that it is set correctly, otherwise tests requiring a single gpu will get skipped. 

Thank you for fixing where the setting are incorrect, @ydshieh!

> Before merging it, could you do a test run when modifying the `setup.py` to ensure that all tests are run correctly? Thank you!

I had to fix a bug (i.e. when the test list is `tests`, i.e. when `setup.py` is changed).
A full test workflow run is [here](https://github.com/huggingface/transformers/actions/runs/2318670278).
After looking some failures, I am convinced that this PR is ready to be merged (the failures are the same as in scheduled CI runs).

Thank you for the reviews!