[Test refactor 1/5] Per-folder tests reorganization
Author: LysandreJik
Pull request: yes

This PR separates the test files in independant folders so that testing may be broken down in smaller parts. The test files for each model are moved in their corresponding folder (e.g., `bert`, `bart`, `plbart`, ...), and others are moved to descriptive directories so that no test file is at the root of the test folder except files to be reused in other test files (mixins, common tests).

Such directories are:
-  `benchmark`
-  `deepspeed`
-  `extended`
-  `fixtures`
-  `generation`
-  `onnx`
-  `optimization`
-  `pipelines`
-  `sagemaker`
-  `trainer`
-  `utils`

Additionally, this PR does the following:

- It moves the `conftest.py` file so that it is at the root of the `transformers` repository. This is because we're expecting it to be shared across the `tests` folder and the `examples` folder as both make use of the `make-reports` utility. However, it cannot be done if `conftest.py` is in the `tests` folder as `examples` is not in its tree/children trees, as `conftest.py` is a [per-directory plugin](https://docs.pytest.org/en/latest/how-to/writing_plugins.html#conftest-py-local-per-directory-plugins).
- The files `test_examples.py` are renamed to contain the name of the framework within them as [no two files may have the same name in a `pytest` run, even if these are in held in different folders](https://github.com/pytest-dev/pytest/issues/3151).
- The `TF_FORCE_GPU_ALLOW_GROWTH` variable is set to `1` in the test so as to prevent TensorFlow from taking up the entire GPU memory. TensorFlow and PyTorch tests may run alongside each other in this setup.
- Each folder is now a submodule, as most of them require access to files that are only in the parent folder. Having each folder be a submodule (having an `__init__.py` file) allows that.

Comments:
conftest needs edit if you are moving it, this part:
```
git_repo_path = abspath(join(dirname(dirname(__file__)), "src"))
sys.path.insert(1, git_repo_path)
```
it is looking for the root dir there.

the new code should be:

```
git_repo_path = abspath(join(dirname(__file__), "src"))
sys.path.insert(1, git_repo_path)
```

I have just tested that it still works if moved outside of `tests`. I didn't know it did. Cool.
It's a bit an unintuitive placement but if you're saying it has to move it has to move then. 



I noticed a few days ago our transformers + DS integration tests were failing in our CI. I bisected the failure to this PR.

https://github.com/microsoft/DeepSpeed/runs/5365118440?check_suite_focus=true

I am able to quickly reproduce the error with this single test, but it happens with several of them: `TORCH_EXTENSIONS_DIR=./torch-extensions RUN_SLOW=1 pytest -s --color=yes --durations=0 --verbose tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_clm_0_zero2`

![image](https://user-images.githubusercontent.com/645595/156273424-677bc5ea-0b71-4f94-80a9-6acb1538daa7.png)

I think the python environment in the sub-process is getting messed up? @stas00 I know you are super busy with big-science right now but when you have a moment can you take a look?


Thank you for the heads up, Jeff.

Our CI is getting a complete revamp to support ever-growing number of models and I think the deepspeed tests were left behind as I wasn't quite around.

I can reproduce the problem, let me see how I can fix it.


Thanks Stas, that's great. I have done some further digging and i think i know the core of the issue. I think there's a name conflict between the `deepspeed` folder at `transformers/tests/deepspeed` and the real `deepspeed` package. I changed the test slightly to run my own test file and printed the deepspeed path and see this:

```
Running:  python /root/foo.py
stdout: <module 'deepspeed' from '/workspace/DeepSpeed/transformers/tests/deepspeed/__init__.py'>
```

the contents of foo.py are:
```python
import deepspeed
print(deepspeed)
```

The correct path of `deepspeed` on this box if you run `foo.py` outside this test environment is: 
```
<module 'deepspeed' from '/opt/conda/lib/python3.7/site-packages/deepspeed/__init__.py'>
```

Yes, something has changed in the paths.

I can run the training command just fine alone from the console, but it fails when run from inside `execute_subprocess_async` 

I'm on top of it, Jeff. Will report back as soon as I have something working.

I think it's because `tests/deepspeed/__init__.py` was added. now it thinks it's the package.

I think there are a lot more issues there to fix as files got moved around, but the first one should get fixed by
https://github.com/huggingface/transformers/pull/15881

I will see what other problems are meanwhile.

OK, I think I got them all fixed. At least they all now pass on my machine.

I will merge as soon as CI has completed, then we can re-check your side.

The CI complains about something unrelated to my PR, but I don't want to break master, so waiting for someone who knows to review. In interim we can merge this if it works: https://github.com/microsoft/DeepSpeed/pull/1803