Enable pytorch nightly CI
Author: ydshieh
Pull request: yes

# What does this PR do?

- Make necessary changes to build `huggingface/transformers-pytorch-nightly-gpu`
- Update `self-nightly-scheduled.yml` to run PyTorch nightly build CI (almost a copy from scheduled CI)

[test workflow run](https://github.com/huggingface/transformers/actions/runs/2354046716) here.
[docker build run](https://github.com/huggingface/transformers/actions/runs/2353866659)

#### Print versions
<img width="389" alt="Screenshot 2022-05-19 214304" src="https://user-images.githubusercontent.com/2521628/169389521-328f6972-6ae4-40dc-9356-e4eb59319c6b.png">




Comments:
_The documentation is not available anymore as the PR was closed or merged._

@stas00, @LysandreJik 

I run the docker image `transformers-pytorch-deepspeed-latest-gpu` and found it's `PyTroch` has version `1.9.0`.
This image is based on `nvcr.io/nvidia/pytorch:21.03-py3`, which is used in the job `Test Torch CUDA Extensions` for both daily scheduled CI and push CI.

We can discuss about this. But so far I won't include DeepSpeed test with nightly-built PyTorch.

```
>>> import torch
>>> torch.__version__
'1.9.0a0+df837d0'
>>> exit()
```




> I run the docker image transformers-pytorch-deepspeed-latest-gpu and found it's PyTroch has version 1.9.0.
This image is based on nvcr.io/nvidia/pytorch:21.03-py3, which is used in the job Test Torch CUDA Extensions for both daily scheduled CI and push CI.

But that's not nightly. I don't think you can rely on any pre-made docker to run nightly. It has to be manually installed since it is a new version every day. You probably want to switch to [22.04](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_22-04.html#rel_22-04) (latest at the moment) and then update it to the actual nightly. Does it make sense?

I also don't understand why deepspeed tests were removed. It's critical that we run deepspeed tests on nightly. 
 

@stas00 

The **non** `DeepSpeed`  tests added in this PR use pytorch-nightly. You can verify in this run page https://github.com/huggingface/transformers/runs/6507879699?check_suite_focus=true and click `Echo versions`, and you will see `Pytorch Version: 1.12.0.dev20220519+cu102`. (It will change everyday).

## DeepSpeed

Regarding this, we have something to discuss:
- For the `push` and `scheuled` CIs currently running, the **deepspeed** tests are run with PyTorch `1.9.0.`
  - Before going to the `nightly pytorch` with `deepspeed`, it might be better to decide what should we test for `push` and `scheduled` CI for `deepspeed` test. Should we use the latest stable Pytorch instead?

- I don't mean to remove DeepSpeed tests with nightly PyTorch. The reason is that I am not able to make a docker image with `PyTorch Nightly + DeepSpeed`. I even tried with  `PyTorch Stable + DeepSpeed` and the docker image also fails.

### Details
In the Dockerfile `transformers-pytorch-deepspeed-latest-gpu`: 

```
RUN python3 -m pip install --no-cache-dir -e ./transformers[deepspeed-testing]

# This fail with if we install Pytorch Nightly or Stable above.
RUN git clone https://github.com/microsoft/DeepSpeed && cd DeepSpeed && rm -rf build && \
    DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 python3 -m pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check 2>&1
```


> Should we use the latest stable Pytorch instead?

Yes, please.

I think we should use the latest stable pytorch for all our tests unless we explicitly test older pytorch versions every few days or so. And to ensure we update to the new stable once it gets released.

> I even tried with PyTorch Stable + DeepSpeed and the docker image also fails.

Could you please point me to the actual issues and I will help to sort them out? 

We can discuss it on slack if it's easier.

I don't think we really care about nightly flax right now, so I would keep the following:
- tests with nightly torch + latest TF
- tests with latest torch + nightly TF

The rest sounds good to me!

Looks good to me, @ydshieh! Thank you!

A full workflow run is [here](https://github.com/huggingface/transformers/actions/runs/2490446459).