Fix `PhimoeIntegrationTest`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `PhimoeIntegrationTest`. See the comments.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46539). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Imo looks good, but would really wait on @Cyrilvallez here to double check since he does/did the most work regarding offloading. Maybe we can also not use accelerate, do we maybe have some util for that?

Just a bit more context:

Before this PR, on our scheduled CI: device map only has some in GPU and some in CPU, but nothing not on disk. This causes GPU OOM when the model running on inputs.

> device_map=OrderedDict([('model.embed_tokens', 0), ('model.layers.0', 0), ('model.layers.1', 0), ('model.layers.2', 0), ('model.layers.3', 0), ('model.layers.4', 0), ('model.layers.5', 0), ('model.layers.6', 0), ('model.layers.7', 1), ('model.layers.8', 1), ('model.layers.9', 1), ('model.layers.10', 1), ('model.layers.11', 1), ('model.layers.12', 1), ('model.layers.13', 1), ('model.layers.14', 1), ('model.layers.15', 1), ('model.layers.16', 'cpu'), ('model.layers.17', 'cpu'), ('model.layers.18', 'cpu'), ('model.layers.19', 'cpu'), ('model.layers.20', 'cpu'), ('model.layers.21', 'cpu'), ('model.layers.22', 'cpu'), ('model.layers.23', 'cpu'), ('model.layers.24', 'cpu'), ('model.layers.25', 'cpu'), ('model.layers.26', 'cpu'), ('model.layers.27', 'cpu'), ('model.layers.28', 'cpu'), ('model.layers.29', 'cpu'), ('model.layers.30', 'cpu'), ('model.layers.31', 'cpu'), ('model.norm', 'cpu'), ('model.rotary_emb', 'cpu'), ('lm_head', 'cpu')])

After this PR: device map only has some in GPU and some in CPU, and some on disk. The model could run on inputs and get outputs, no GPU OOM.

> device_map=OrderedDict([('model.embed_tokens', 0), ('model.layers.0', 0), ('model.layers.1', 0), ('model.layers.2', 0), ('model.layers.3', 0), ('model.layers.4', 0), ('model.layers.5', 0), ('model.layers.6', 0), ('model.layers.7', 'cpu'), ('model.layers.8', 'cpu'), ('model.layers.9', 'cpu'), ('model.layers.10', 'cpu'), ('model.layers.11', 'cpu'), ('model.layers.12', 'cpu'), ('model.layers.13', 'cpu'), ('model.layers.14', 'cpu'), ('model.layers.15', 'cpu'), ('model.layers.16', 'cpu'), ('model.layers.17', 'cpu'), ('model.layers.18', 'cpu'), ('model.layers.19', 'cpu'), ('model.layers.20', 'cpu'), ('model.layers.21', 'cpu'), ('model.layers.22', 'cpu'), ('model.layers.23', 'cpu'), ('model.layers.24', 'cpu'), ('model.layers.25', 'cpu'), ('model.layers.26', 'cpu'), ('model.layers.27', 'cpu'), ('model.layers.28', 'disk'), ('model.layers.29', 'disk'), ('model.layers.30', 'disk'), ('model.layers.31', 'disk'), ('model.norm', 'disk'), ('model.rotary_emb', 'disk'), ('lm_head', 'disk')])

It's unclear to me why the first case  will cause GPU OOM while the second case is OK.

I will wait until Friday evening. Otherwise I will simply merge (after implementing a reusable helper patching method).

Will merge on Sunday. Don't want to have a surprisingly sad Saturday!

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46539)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: phimoe