Use latest stable PyTorch/DeepSpeed for Push & Scheduled CI
Author: ydshieh
Pull request: yes

# What does this PR do?

(**As there are a few PRs waiting for review, feel free to postpone the review for this PR if things get confusing at this moment**)

Currently:
- scheduled CI uses latest stable PyTorch (OK) + nightly DeepSpeed (Not OK)
- push CI uses PyTorch 1.9 (Not OK)  + latest stable DeepSpeed (OK)

This PR fix it by using latest stable PyTorch + DeepSpeed, for both push/scheduled CIs

(**Let me run a dummy test before merge** 🙏 )

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@stas00 

Regarding `transformers-pytorch-gpu`:

https://github.com/huggingface/transformers/blob/7e46ec71371b5e705522821e741d7c0dac910859/docker/transformers-pytorch-gpu/Dockerfile#L17

Here we only upgrade `torch`, but not `torchvision` and `torcuaudio`. Those two are installed in a previous step

https://github.com/huggingface/transformers/blob/7e46ec71371b5e705522821e741d7c0dac910859/docker/transformers-pytorch-gpu/Dockerfile#L12

I am not sure if we need to upgrade all 3 modules at the same time. PyTorch installation instructions always install these 3 at the same time, so I guess yes ??


Yes, usually it's the easiest to always handle all 3 packages as a single package to avoid incompatibility conflicts down the road. 

I first tried to "optimize" and only install the other 2 when it was needed, but later I switched to always installing the 3 together. the other 2 are tiny compared to the main package.

> Yes, usually it's the easiest to always handle all 3 packages as a single package to avoid incompatibility conflicts down the road.
> 
> I first tried to "optimize" and only install the other 2 when it was needed, but later I switched to always installing the 3 together. the other 2 are tiny compared to the main package.

OK, I will try to get it done. A bit not easy as here we have using an argument `$PYTORCH` so we can specify torch version.
(although we only use the default value to get the latest stable version for now.)

[Just FYI] an update:

I am running the tests before merge. So far only a subset of tests is run. I got some issues

PyTorch pipelines (single-gpu)]
https://github.com/huggingface/transformers/runs/6728867682?check_suite_focus=true

Torch CUDA test (multi GPUs)
https://github.com/huggingface/transformers/runs/6728868779?check_suite_focus=true

I will try to re-run them later, also will wait the scheduled CIs during this weekend in order to compare.


Feel free to also spawn dummy machines to help you out if it helps getting it merged quicker

I am going to merge now.

----------------------------------

@stas00 

- `intel_extension_for_pytorch` is added
  - the version still uses the approach so far `$(python3 -c "from torch import version; print(version.__version__.split('+')[0])")` 
  - I have to remove `pip uninstall -y` that was in your original patch. The whole line wasn't working.

For the versions, let's discuss in #17586

----------------------------------
@stas00

I observed with the latest stable PyTorch/DeepSpeed, `test_can_resume_training_normal_zero2_fp16` takes quite long to run the first time: (On push CI, it will timeout)

First Run
```
63.46s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16
10.97s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16
```
Second Run
```
18.17s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16
11.01s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16
```

With previous setting (PyTorch 1.9 + DeepSpeed Recompiled)
```
6.16s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16
2.83s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16
```


> I observed with the latest stable PyTorch/DeepSpeed, test_can_resume_training_normal_zero2_fp16 takes quite long to run the first time: (On push CI, it will timeout)

The very first deepspeed test using deepspeed JIT install will have the overhead of building deepspeed, which takes about 1min - depending on the hardware.

This doesn't happen if deepspeed was prebuilt before tests were run.

Is it possible that this test happens to be the first one to run?

@stas00 

(This is for the stable release of `DeepSpeed` + `PyTorch`)

Here is the job run page
https://github.com/huggingface/transformers/runs/6761224960?check_suite_focus=true

The tests are ran in the following order 

```
# The following 3 are OK
tests/deepspeed/test_deepspeed.py::CoreIntegrationDeepSpeed::test_init_zero3_fp16 
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_errors_zero2_fp16 
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_errors_zero3_fp16 

# This one timed out
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16

# This one is OK
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16 

tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_config_object 
...
```

Dumb question: Is stable release of `DeepSpeed` == pre-built ?

You can see that it was indeed building deepspeed during that test's run, see: https://github.com/huggingface/transformers/runs/6761224960?check_suite_focus=true#step:6:334

so need to either have a longer timeout or always prebuild deepspeed.

> Dumb question: Is stable release of DeepSpeed == pre-built ?

Hope the following makes the whole situation loud and clear:

### What is being built:

* stable release install: `pip install deepspeed==0.6.5`
* bleed/master install: `pip install git+https://github.com/microsoft/DeepSpeed` (or `git clone ...; pip install -e .`)

### How is it being built:

* `pip install deepspeed` JIT build - this will build deepspeed the first time it's used. pip just installs the source files here.
* `DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1  DS_BUILD_UTILS=1 pip install -e . --global-option="build_ext" --global-option="-j8"` - this is the prebuiding - so that the first time it's used it's already ready to use

Thank you @stas00 , thankfully I get better understanding of the terminology now!

I will pre-build `DeepSpeed` so it will be indeed ready to be speedy!