[pip/setup.py] target management and updates
Author: stas00
Pull request: yes

This PR 

* [x] does a major revamp to how targets are specified and merged into groups. it lists all the dependencies vertically and allows you to write comments next to them, if needed - or completely comment them out
* [x] adds `docs` to `dev`, since we need to have the tools to run `make docs`
* [x] adds `flax` to `dev`, since we need to have the libs to run flax tests - except when on windows - it skips it then
* [x] brings `all` up-to-date

note, I removed the hardcoded `+ ["scikit-learn", "tensorflow", "torch", "sentencepiece!=0.1.92"]` and replaced with the full targets for `tf`, `torch`, etc., which include other deps. I'm not sure why we don't want all of them.

@LysandreJik, @sgugger, @thomwolf 

Comments:
Also what does `extras["all"]`  stand for? Is it a vestige of something?
https://github.com/huggingface/transformers/blob/master/setup.py#L94

I'd put it last and really put everything into `all` unless I'm missing a special purpose here.

Does `transformers` really work `torch==1.0` or would we realistically need to set some higher 1.x minimum? I'm curious whether anybody tested this. But I guess there is no need to waste time on this - someone will flag this in time if it's a problem.


I think I prefer the long version because I can understand it without thinking, this one... not so much.

I was also going to take a stab at the setup because we have a recurring complaint from Windows user they can't make a dev install (some of those dependencies like `faiss` should be dropped if on Windows).

Also `flax` is still experimental. Not sure it should be in dev just yet, especially since I'm doubtful about its Windows support. It shouldn't prevent people from developing and making PRs to the library.

Well, we could make it into a wrapper function so it'd be easier to read, but either way works.

Then at the very least can we add `docs` to `dev`?

with hardcoded targets in `dev` - let's have just one definition where we put numerical requirements (==, !=, etc.)

Yes adding `docs` to `dev` is definitely useful.

would this be easier to read:
```
def combine_targets(names):
    return list(chain(*map(extras.get, names)))

extras["dev"] = combine_targets("testing quality docs ja sklearn flax tf torch sentencepiece".split())
# or:
extras["dev"] = combine_targets(["testing", "quality", "docs", "ja", "sklearn", "flax", "tf", "torch", "sentencepiece"])
```
or you'd rather keep:
```
extras["dev"] = extras["testing"] + extras["quality"] + extras["docs"] + extras["flax"] + extras["ja"] + \
 extras["sklearn"]  + extras["tf"]  + extras["torch"]  + extras["sentencepiece"] 
```


I agree with @sgugger that the `list(chain(*map(...` is :dizzy_face:.
The way it's currently setup is fine by me, but your proposed fix:
```py
extras["dev"] = combine_targets(["testing", "quality", "docs", "ja", "sklearn", "flax", "tf", "torch", "sentencepiece"])
```
is also fine by me.

@LysandreJik, please have another look - we discussed this with @sgugger on slack and expanded this further to be even more flexible - easy to read vertical listing plus built-in comments are now supported.

If it looks too busy we can merge the base groups into a dict, so it'll look less busy and will be more compact:

So instead of this:
```
extras["serving"] = to_list("""
    fastapi
    pydantic
    starlette
    uvicorn
""")

extras["sentencepiece"] = to_list("""
    sentencepiece!=0.1.92
""")

extras["retrieval"] = to_list("""
    datasets
    faiss-cpu
""")

extras["testing-base"] = to_list("""
    parameterized
    psutil
    pytest
    pytest-xdist
    timeout-decorator
""")

```
it'd be:
```
extras = dict(
    serving=to_list("""
        fastapi
        pydantic
        starlette # some explanation
        uvicorn
"""),
    sentencepiece=to_list("""
        sentencepiece!=0.1.92
"""),
    retrieval=to_list("""
        datasets
        faiss-cpu
"""),
    testing-base=to_list("""
        parameterized
        psutil
        pytest
        pytest-xdist # some comment
        timeout-decorator
"""),
)
```
Actually, if we decide to go the dict way we can do all the processing later, why repeat the same function all the time, so it'd just leave:

```
extras = dict(
    serving="""
        fastapi
        pydantic
        starlette # some explanation
        uvicorn
""",
    sentencepiece="""
        sentencepiece!=0.1.92
""",
    retrieval="""
        datasets
        faiss-cpu
""",
    testing-base="""
        parameterized
        psutil
        pytest
        pytest-xdist # some comment
        timeout-decorator
""",
)
extras = process(extras) # not written yet.
```

It's different, but it's consistent. You never need to read everything at once - you only would care about reading one entry - a group or a subgroup - this is not code but a table of definitions - like a spreadsheet. You can always squash the vertical entries into a horizontal line, by losing the readability and functionality offered by the spreadsheet-type of data. 

I proposed here a much more compact way: https://github.com/huggingface/transformers/pull/7955#issuecomment-714845771

Also the idea is to have just one base definition with the specific version if any and a comment why it is so if needed including the non-optional requirements. Otherwise it's too easy to forget to update multiple definitions of the same.

These are just different suggestions, please feel free to cherry pick some, all or none and close this PR as well. No hard feelings.

As I personally don't see any improvements regarding readability in the offered solutions, I would vote to keep the original approach. It's a personal preference choice so I'm willing to compromise if others disagree!

All the other changes in the PR look good to me.

Thank you for indicating that the proposed change is not fitting, @LysandreJik and @sgugger.