[testing] USE_CUDA default and intuitive skip decorators
Author: stas00
Pull request: no

This library's primarily use is for gpu work, and currently many tests won't run even if gpu is available, since the current setup wants env var `USE_CUDA` to be true for anything to happen.  It's easy to forget to manually add this env var to pytest command line.

To maximize the testing potential I propose that only if `USE_CUDA=False` then the test is skipped (for CI jobs that need to test library's work on cpu), otherwise if `if torch.cuda.is_available()` cuda tests can be run.

In a brief discussion @julien-c suggested that:

> The original thinking was that we wanted to make sure that when we wanted to run on GPU it actually ran on GPU. i.e. it should even fail if you do `USE_CUDA`  and there's no GPU, to prevent silent failures on GPU

and the discussion stopped there. This ticket was opened to complete this discussion.

@julien-c, could you please share a specific scenario based on the design intention you shared?

also `CUDA_VISIBLE_DEVICES=""` could be used to easily emulate a non-gpu environment if need be, w/o introducing new env vars. i.e. it'd be a built-in equivalent of `USE_CUDA=False`.

Further, `USE_CUDA` is currently only used for skip decorators. This setting cannot currently be respected from within a test. e.g. in a test I'm currently working on I have:

```
        if torch.cuda.is_available():
            testargs += ['--fp16', '--gpus=1']
```
so it'll ignore  `USE_CUDA`, as the test must always run whether there is a gpu or not, so no skip decorator was used. This ignoring conceptually won't do the right thing then as it'll run the test on gpu even if `USE_CUDA==False` (or unset). So if `USE_CUDA`-functionalty remains, there is a need for an accessor that is not a [skip decorator](https://github.com/huggingface/transformers/blob/master/src/transformers/testing_utils.py#L126).

`require_multigpu` also currently ignores `USE_CUDA`


Comments:
Perhaps, while we are at it, it'd be good to discuss the related skip decorators. **Please let me know whether this is different enough and should be in its own issue.**

These are the torch-related skip decorators we [currently have](https://github.com/huggingface/transformers/blob/master/src/transformers/testing_utils.py#L73):

* `require_torch`
* `require_multigpu`
* `require_torch_and_cuda`
* `require_torch_tpu`

Currently there is `require_multigpu`, but no `require_gpu` - I tried using `require_torch_and_cuda` but it only works if USE_CUDA is set. And `require_torch_and_cuda` name is non-intuitive/inconsistent next to `require_multigpu`.

And `require_multigpu` should behave like `require_torch_and_cuda`, except require `gpus>1` - i.e. whatever the USE_CUDA discussion outcome will be - it should behave the same. Currently it **does not** respect the `USE_CUDA ` setting!

The `require_torch` decorator name is somewhat ambiguous - is it asking for just having `torch` installed or choosing torch vs tf?

Finally `require_torch_tpu` is again weirdly mismatching other decorator naming - should all of them have `_torch_` in the name or some of them?  `require_multigpu` is torch-specific.

My thinking is perhaps we need:
1. `require_torch` - this test will run only under torch
2. `require_torch_gpu` - as `require_torch` plus at least 1 gpu
3. `require_torch_multigpu` - as `require_torch` plus at least 2 gpus
4. `require_torch_tpu` - as `require_torch` plus at least 1 tpu

that's if we sort out `USE_CUDA` to not need `require_torch_and_cuda`.

And perhaps there might be a situation where we want: `gpu|tpu` - that is skip this test unless at least 1 gpu or 1 tpu is available, as perhaps it'd be too slow on cpu. `require_torch_gpu_or_tpu` or `require_torch_non_cpu`? Is there a common name/term for an environment that has either gpu or tpu?

And then `require_torch_cpu_only` - skip this test if either gpu or tpu is available? i.e. this test needs to be run under cpu.

So 2 more:

5. `require_torch_non_cpu` - as `require_torch` plus at least 1 gpu or 1 tpu
6. `require_torch_cpu_only`- as `require_torch` plus must have neither gpus nor tpus

And as discussed at the end of the comment above, in addition to the skip decorators we will find a good use for `has_` accessors with the same names (e.g. `has_torch_gpu`), so that a test could potentially behave differently depending on the environment, which could be changed globally by `USE_CUDA` or `CUDA_VISIBLE_DEVICES`.

I think life would be marginally better if we used `CUDA_VISIBLE_DEVICES` and your first 4 `@require` decorators. Basically just delete `USE_CUDA`. But @julien-c has more context.


@julien-c suggested we check in with @LysandreJik, @patrickvonplaten, @sgugger and @thomwolf so there is a full agreement, before we make the change.


I agree with this change. Not sure we need decorators 5 and 6 though. I'd wait for an occasion to see if they are needed.

Ok for clarifying this and making it more robust. I'm also not opposed to changing the `USE_CUDA` flag to `True` by default either.

I agree with @sgugger here

@thomwolf, @julien-c asked to confirm that you're in agreement with this proposal. Thank you! 

I think you have waited long enough to PR this @stas00 .
Apologies in advance if there is already a PR that I have not seen.

Thank you for affirming that, @sshleifer. 

I agree with @sshleifer, feel free to open a PR @stas00!

Thank you, @LysandreJik. I will work on that once I finish sorting out the fsmt nuances.