CI: update to ROCm 6.0.2 and test MI300
Author: fxmarty
Pull request: yes

As per title

A few tests are failing due to `torch<2.2` and will be fixed once we bump to torch 2.3 + rocm6.0
```
FAILED tests/test_pipeline_mixin.py::FillMaskPipelineTests::test_fp16_casting - RuntimeError: "softmax_lastdim_kernel_impl" not implemented for 'Half'
FAILED tests/pipelines/test_pipelines_fill_mask.py::FillMaskPipelineTests::test_fp16_casting - RuntimeError: "softmax_lastdim_kernel_impl" not implemented for 'Half'
FAILED tests/models/mamba/test_modeling_mamba.py::MambaIntegrationTests::test_simple_generate_1_cpu - RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
FAILED tests/models/mamba/test_modeling_mamba.py::MambaIntegrationTests::test_simple_generate_cuda_kernels_tiny_1_cpu - RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
```

This PR requires a self hosted runner on MI300 first.

Report callbacks are skipped from the trainer tests, as codecarbon is not supported on ROCm and report callbacks are anyway tested independently (in `test_trainer_callback.py`).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30266). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> As per title
> 
> A few tests are failing due to `torch<2.2` and will be fixed once we bump to torch 2.3 + rocm6.0
> 
> ```
> FAILED tests/test_pipeline_mixin.py::FillMaskPipelineTests::test_fp16_casting - RuntimeError: "softmax_lastdim_kernel_impl" not implemented for 'Half'
> FAILED tests/pipelines/test_pipelines_fill_mask.py::FillMaskPipelineTests::test_fp16_casting - RuntimeError: "softmax_lastdim_kernel_impl" not implemented for 'Half'
> FAILED tests/models/mamba/test_modeling_mamba.py::MambaIntegrationTests::test_simple_generate_1_cpu - RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
> FAILED tests/models/mamba/test_modeling_mamba.py::MambaIntegrationTests::test_simple_generate_cuda_kernels_tiny_1_cpu - RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
> ```
> 
> This PR requires a self hosted runner on MI300 first.
> 
> Report callbacks are skipped from the trainer tests, as codecarbon is not supported on ROCm and report callbacks are anyway tested independently (in `test_trainer_callback.py`).

With torch==2.3 RC, from above 4 the following tests passed:

```
tests/test_pipeline_mixin.py::FillMaskPipelineTests::test_fp16_casting 
tests/pipelines/test_pipelines_fill_mask.py::FillMaskPipelineTests::test_fp16_casting
tests/models/mamba/test_modeling_mamba.py::MambaIntegrationTests::test_simple_generate_cuda_kernels_tiny_1_cpu
```

The third test failed with:

```
FAILED tests/models/mamba/test_modeling_mamba.py::MambaIntegrationTests::test_simple_generate_1_cpu - AssertionError: Tensor-likes are not close!

```

I will review tomorrow.

@fxmarty Do we already have a MI300 runner?

Also, could you trigger the workflow that would build the image whose dockerfile is modified in this PR?

(don't forget to comment out other jobs in the docker image build workflow)

conflicts resolved

Hi @mht-sharma 

Is anything I should wait before I merge this PR?

> Hi @mht-sharma
> 
> Is anything I should wait before I merge this PR?

As discussed on slack, all good from my side!

I have removed the `TODO: remove this`.

I will merge as the PR is currently. They will need some updates however to match the recent changes on the nvidia daily CI workflow files.

Thank you for all the work you have done @fxmarty and @mht-sharma