[tests] where to put deepspeed + fairscale tests
Author: stas00
Pull request: no

As a split off from this comment https://github.com/huggingface/transformers/pull/10039#pullrequestreview-585482462 we need to find a new home for deepspeed + fairscale tests. 

Currently there are under `examples/seq2seq` because they rely on `finetune_trainer.py` ( `run_seq2seq.py` once the transition is over).

@sgugger suggests to keep the `seq2seq` folder as simple as possible. We also have `ds_config.json` there that could be moved too.

Seeing what's happening in the fairscale land - I think we will need a bunch of various tests there in the future too.

So where should we put the deepspeed + fairscale tests? 

Ideally they should be put under main `tests`, since they are part of the trainer core, but I'm not sure whether reaching across the tests suite is a clean approach.

My fantasy is that one day transformers will have a few essential tools that aren't examples, and those will then leave somewhere in the main tree, perhaps `src/transformers/apps` and then it'd be easy to have such tests under `tests`.

So suggestions for now:

1. create `examples/deepspeed` and `examples/fairscale`
2. create `examples/distributed` and perhaps have all those extensions tested in one folder
3. create a new 3rd test suite for integrations
4. create `tests/deepspeed` - but as voiced earlier I'm not sure how reaching across a test suite will work - need to try - also this proposes to change the current flat structure of `tests`.

Perhaps you have other ideas.

@sgugger, @patrickvonplaten, @LysandreJik, @patil-suraj 

Comments:
The examples are not tests, so `examples/deepspeed` should not be created to host some deepspeed tests. I'm wondering why we would need an `examples/deepspeed` since the base concept of having deepspeed integrating in our `Trainer` is to have it work out of the box for **all** our examples.

My suggestion was to create an `examples/tests` folder where all tests should go (so `test_examples`, and all `seq2seq/test_xxx`), to keep the examples folder themselves clean so that user can easily use them.

> The examples are not tests, so `examples/deepspeed` should not be created to host some deepspeed tests. I'm wondering why we would need an `examples/deepspeed` since the base concept of having deepspeed integrating in our `Trainer` is to have it work out of the box for **all** our examples.

I only meant it as a grouping, same as we did for models. first it was all flat and then we grouped them together under `models`.

> My suggestion was to create an `examples/tests` folder where all tests should go (so `test_examples`, and all `seq2seq/test_xxx`), to keep the examples folder themselves clean so that user can easily use them.

Sure, as long as you feel that it's OK that we test core integrations under `examples` (as it is now) that works for me.

Could you pelase clarify, do you prefer most/all of the  `examples/tests` to be flat, or would grouping make things easier to make sense of - I'm asking since some tests come with extra files (as is the case with ds_config files) - so `examples/tests/deepspeed`, ...


I agree with Sylvain, fairscale/deepspeed is supposed to work with all of our existing examples, so IMO we shouldn’t add `examples/deepspeed`.

 `examples/tests` makes sense to me.

> Could you pelase clarify, do you prefer most/all of the examples/tests to be flat, or would grouping make things easier to make sense of - I'm asking since some tests come with extra files (as is the case with ds_config files) - so examples/tests/deepspeed, ...

We can certainly have several files once they are all together in one folder. I'd just like the examples subfolders to be clean, our internal testing should be setup so it's the easiest for us to understand/debug.

Thank you for the clarification, @sgugger. I will start working on that transition.