BarkModel.enable_cpu_offload() strands submodules on CPU → device mismatch at generate() (transformers 4.57.6 + accelerate 1.13.0)
Author: psyiode
Pull request: no

### System Info

- `transformers` version: 4.57.6
- `accelerate` version: 1.13.0
- `torch` version: 2.12.1 (CUDA 13.0), CUDA available: yes
- Platform: Linux, Python 3.11
- GPU: single CUDA device

### Who can help?

Text-to-speech / Bark maintainers. (Feel free to reassign — the root cause is a `BarkModel.enable_cpu_offload()` × `accelerate` CpuOffload interaction.)

### Reproduction

`BarkModel.enable_cpu_offload()` returns without error but leaves the model in an inconsistent device state, so the very next `generate()` crashes with a device mismatch. A single CUDA GPU is required.

```python
import torch
from transformers import AutoProcessor, BarkModel

model_id = "suno/bark-small"
model = BarkModel.from_pretrained(model_id)      # loads on CPU
model.enable_cpu_offload()                        # documented CPU-offload path; does NOT raise

processor = AutoProcessor.from_pretrained(model_id)
inputs = processor("Hello, my name is Suno.", return_tensors="pt")

model.generate(**inputs, do_sample=True)
# RuntimeError: Expected all tensors to be on the same device,
# but found at least two devices, cuda:0 and cpu!
```

This happens on **any** call order — it is not a "you moved the model yourself first" mistake; the model is loaded on CPU and only `enable_cpu_offload()` is called, exactly as the docs show.

### Expected behavior

`enable_cpu_offload()` followed by `generate()` should produce audio, keeping every tensor consistent across devices (as it does for other offloaded models).

### Root cause (source-traced)

The interaction is between how `BarkSemanticModel` is offloaded and accelerate's "first-parameter" device heuristic:

1. `BarkModel.enable_cpu_offload()` gives `semantic.input_embeds_layer` **its own** accelerate hook, installed *before* the hook on `semantic` itself (in `modeling_bark.py`, around the `enable_cpu_offload` setup — `semantic.input_embeds_layer` is hooked separately so it can be called outside `semantic`'s own `forward`).
2. `BarkSemanticModel.generate()` calls `input_embeds_layer(...)` **directly** on its first pass (bypassing `semantic.forward`). Its hook fires and moves only that one parameter (`input_embeds_layer.weight`) to `cuda`.
3. That single moved parameter is the one `next(module.parameters())` returns for `semantic`, so it now fools two independent device checks that both use the "first parameter" heuristic:
   - `BarkModel.device` (via `get_parameter_device()`) reports `cuda`, so generation scratch tensors are allocated on `cuda`; and
   - accelerate's `CpuOffload.pre_forward` short-circuits its "is `semantic` already on the execution device?" check and **skips** moving the rest of `semantic`.
4. Net effect: `semantic`'s remaining submodules (notably `position_embeds_layer`) stay on **CPU** while inputs/scratch land on **cuda** → `RuntimeError: Expected all tensors to be on the same device`.

### Sanity-check that isolates the state (no full generation needed)

```python
import torch
semantic = model.semantic
semantic.input_embeds_layer(torch.zeros((1, 1), dtype=torch.long))  # fire the early hook
print("first param:", next(semantic.parameters()).device)            # cuda:0
print("position_embeds_layer:", semantic.position_embeds_layer.weight.device)  # cpu  <-- stranded
```

### Suggested fix direction

The `pre_forward` "already on the execution device?" check being satisfied by a single hooked sub-layer's parameter is the crux — a submodule that has one child with its own offload hook can report the wrong aggregate device via `next(module.parameters())`. Options: (a) exclude separately-hooked children (like `input_embeds_layer`) from the parameter used for `semantic`'s device determination, or (b) don't rely on the first-parameter heuristic for a module that contains independently-hooked children.

### Workaround (for anyone hitting this today)

Detect the stranded state right after `enable_cpu_offload()` and fall back to full-GPU residency: `accelerate.hooks.remove_hook_from_module(model, recurse=True)` then `model.to("cuda")`. Bark fits comfortably in ~5 GB VRAM, so full residency is a fine fallback when offload is broken.


Comments:
We can't fix bugs on v4, I'm afraid! Please check if the bug still exists on the latest version of `v5`.

/assign