Add deepspeed test to amd scheduled CI
Author: echarlaix
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27633). All of your documentation changes will be reflected on that endpoint.

There is a PR merged before this one

https://github.com/huggingface/transformers/pull/27743

Don't hesitate if you need my help on resolving the conflict :-)

Running the tests, all tests passing on A100 pass as well on MI250. Two failing ones (that seem unrelated to deepspeed: https://github.com/huggingface/transformers/blob/e0d2e695827594c6a95a68612f32c65b2686985e/tests/extended/test_trainer_ext.py#L125, for both MI250/A100) are:

```
FAILED tests/extended/test_trainer_ext.py::TestTrainerExt::test_trainer_log_level_replica_1_low - AssertionError: 8 != 2
FAILED tests/extended/test_trainer_ext.py::TestTrainerExt::test_trainer_log_level_replica_2_high - AssertionError: 7 != 1
```

which I'll leave to an other PR.

One issue with the image `rocm/pytorch:rocm5.6_ubuntu20.04_py3.8_pytorch_2.0.1` is that it has `numba==0.49.0` (that is imported [here](https://github.com/huggingface/transformers/blob/e0d2e695827594c6a95a68612f32c65b2686985e/tests/models/wav2vec2/test_modeling_flax_wav2vec2.py#L67) through librosa) installed, which plays not nicely with transformers dependency resolution as it uses outdated `np.long`.

I have the warning `cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!` as well with `rocm/pytorch:rocm5.6_ubuntu20.04_py3.8_pytorch_2.0.1` (for example `pytest tests/ -k "test_can_resume_training_errors_zero2_fp16" -s -vvvvv`), so I moved to use `rocm/dev-ubuntu-22.04:5.6` instead. Issue open in DeepSpeed about it: https://github.com/microsoft/DeepSpeed/issues/4768

@ydshieh Should be in good shape. There are many jobs though, but I guess it is the way transformers CI is designed? https://github.com/huggingface/transformers/actions/runs/7089122062/job/19293182925

We should not trigger that many jobs in a PR. We usually just removed unrelated jobs, although this is a bit time consuming.

I don't mind keep it running this way (for this time).

Well, it probably will affect tomorrow's scheduled CI.

Let's cancel it. I will show you how we usually do the experimentation tomorrow.

Sorry!

Hi @fxmarty Let me know if you have any question or need help regarding my above comments.

Thanks @fxmarty @ydshieh for the updates while I was sick, let me push the new image manually, just launching all the tests locally to verify everything is working with the updated image before pushing

Hi @echarlaix You don't need to run all the tests. Just make sure 

- the image could build
- the deepspeed build step works 
- the workflow contain no bug - i.e. it could be trigger and run on GH actions
- the deepspeed tests could be launched (we don't really care how many failing tests it has for now)
 

There is 11 failing [tests](https://github.com/huggingface/transformers/actions/runs/7114222566/job/19367866854?pr=27633) for AMD vs 7 for the current [CI](https://github.com/huggingface/transformers/actions/runs/7095377457), these 4 tests are bf16 variant of already failing tests.


Failing test current CI :

```
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_do_eval_no_train
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_non_distributed_zero2_fp16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_non_distributed_zero3_fp16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_resume_train_not_from_ds_checkpoint_zero2_fp16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_resume_train_not_from_ds_checkpoint_zero3_fp16
tests/deepspeed/test_model_zoo.py::TestDeepSpeedModelZoo::test_zero_to_fp32_zero3_qa_led
tests/deepspeed/test_model_zoo.py::TestDeepSpeedModelZoo::test_zero_to_fp32_zero3_trans_fsmt
```

Failing tests for AMD :
```
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_do_eval_no_train
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_non_distributed_zero2_bf16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_non_distributed_zero2_fp16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_non_distributed_zero3_bf16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_non_distributed_zero3_fp16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_resume_train_not_from_ds_checkpoint_zero2_bf16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_resume_train_not_from_ds_checkpoint_zero2_fp16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_resume_train_not_from_ds_checkpoint_zero3_bf16
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_resume_train_not_from_ds_checkpoint_zero3_fp16
tests/deepspeed/test_model_zoo.py::TestDeepSpeedModelZoo::test_zero_to_fp32_zero3_trans_t5_v1
tests/deepspeed/test_model_zoo.py::TestDeepSpeedModelZoo::test_zero_to_fp32_zero3_trans_fsmt
```






I will merge after a fix #27951 being merged.