Implement VibeVoice 
Author: pengzhiliang
Pull request: yes
Labels: name: New model

name: Audio

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=40546)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=40546)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Merge the model from https://github.com/microsoft/VibeVoice/tree/main

HF: 
https://huggingface.co/microsoft/VibeVoice-1.5B

Comments:
Hopefully this PR can get merged, since the deletion of the VibeVoice repo not sure the original authors can continue contributing to this PR

Glad to see that someone has picked this PR up, thanks 🤗

run-slow: vibevoice, vibevoice_acoustic_tokenizer, vibevoice_semantic_tokenizer

This comment contains `run-slow`, running the specified jobs: 

models: ["models/vibevoice", "models/vibevoice_acoustic_tokenizer", "models/vibevoice_semantic_tokenizer"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19963357914)

### Model CI Report

#### ❌ Failed tests

- [vibevoice_acoustic_tokenizer](https://github.com/huggingface/transformers/actions/runs/19963357914/job/57249423332):
    tests/models/vibevoice_acoustic_tokenizer/test_modeling_vibevoice_acoustic_tokenizer.py::VibeVoiceAcousticTokenizerIntegrationTest::test_batch_integration

- [vibevoice_semantic_tokenizer](https://github.com/huggingface/transformers/actions/runs/19963357914/job/57249423326):
    tests/models/vibevoice_semantic_tokenizer/test_modeling_vibevoice_semantic_tokenizer.py::VibeVoiceSemanticTokenizerIntegrationTest::test_batch_integration



run-slow: vibevoice, vibevoice_acoustic_tokenizer, vibevoice_semantic_tokenizer

This comment contains `run-slow`, running the specified jobs: 

models: ["models/vibevoice", "models/vibevoice_acoustic_tokenizer", "models/vibevoice_semantic_tokenizer"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19965713824)

✅ No failing test specific to this PR 🎉 !


View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=40546&sha=116c3a

[Related Hub issue](https://github.com/huggingface/huggingface.js/issues/1960) (architecture not supported yet) so snippets don't work.

run-slow: vibevoice

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22668912565)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/vibevoice"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22668912565)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [70e4b7f8](https://github.com/huggingface/transformers/commit/70e4b7f849fc51740e9246789e9d993a69ba305b) | workflow commit (merge commit) |
| PR | [6c2526d9](https://github.com/pengzhiliang/transformers/commit/6c2526d9d70f7b7de44f00a073de28c3c65f1f02) | branch commit (from PR) |
| main | [7235d442](https://github.com/huggingface/transformers/commit/7235d44257e7e4765317d475622e2085a09c9e3b) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


run-slow: vibevoice, vibevoice_acoustic_tokenizer

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22675065040)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/vibevoice", "models/vibevoice_acoustic_tokenizer"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22675065040)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [4b9d46a1](https://github.com/huggingface/transformers/commit/4b9d46a1c8cb2ac75469a4858a2af6818b7a0138) | workflow commit (merge commit) |
| PR | [6f1fbf61](https://github.com/pengzhiliang/transformers/commit/6f1fbf6178d2b837f3b1cf23425507d2081813ea) | branch commit (from PR) |
| main | [30c48016](https://github.com/huggingface/transformers/commit/30c480166a95342fefdd9d727fc2a163bea7b2b1) | base commit (on `main`) |

### Model CI Report

❌ **[2 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/58593ae3e73f4149b92cc9c4e410a8713717765f/2026-03-04/runs/30178-22675065040/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [vibevoice](https://github.com/huggingface/transformers/actions/runs/22675065040/job/65730256450):
    tests/models/vibevoice/test_modeling_vibevoice.py::VibeVoiceForConditionalGenerationIntegrationTest::test_1b5_inference (✅ ⟹ ❌)
    tests/models/vibevoice/test_modeling_vibevoice.py::VibeVoiceForConditionalGenerationIntegrationTest::test_1b5_inference_no_voice (✅ ⟹ ❌)



run-slow: vibevoice

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22677526511)💔 This comment contains `run-slow`, but unknown error occurred and [the workflow run](https://github.com/huggingface/transformers/actions/runs/22677526511) aborted!

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22677551045)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/vibevoice"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22677551045)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [45934fcb](https://github.com/huggingface/transformers/commit/45934fcbf987ff38fe2ace56d3a3bd1ee4f12921) | workflow commit (merge commit) |
| PR | [7ded0321](https://github.com/pengzhiliang/transformers/commit/7ded03219731cfc7d22385db7f387f3d0084c95d) | branch commit (from PR) |
| main | [fd6bc380](https://github.com/huggingface/transformers/commit/fd6bc380c8854a370fbc9f68a157895d84dce7d7) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


run-slow: vibevoice

run-slow: vibevoice

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22759586400)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/vibevoice"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/22759586400)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [3670f8f5](https://github.com/huggingface/transformers/commit/3670f8f594b5e8bcddf8de8c31549a9cc976e3aa) | workflow commit (merge commit) |
| PR | [9501e9f6](https://github.com/pengzhiliang/transformers/commit/9501e9f6b07ca7773f59950d704669aba2e29b9d) | branch commit (from PR) |
| main | [4f91111b](https://github.com/huggingface/transformers/commit/4f91111b8ef37bd227f33c7facb92c41aa77604d) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


Are there still plans to get this merged? Would be really cool to get direct support for VibeVoice models, especially 7B!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40546). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: vibevoice

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28460389952)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/vibevoice"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28460389952)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [36208e7a](https://github.com/huggingface/transformers/commit/36208e7a6f25b5ece8798119e8011e484f20e511) | workflow commit (merge commit) |
| PR | [bbe259f6](https://github.com/pengzhiliang/transformers/commit/bbe259f657c281fa713cc1e5a9b08dad759b0d21) | branch commit (from PR) |
| main | [181beb3b](https://github.com/huggingface/transformers/commit/181beb3ba4c47098ed8cbc97ee250d1d45ae0107) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/28649276854) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/28650085367) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/28650161345) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/28650439093) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/28652794444) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/28673016086) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29434378594) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29480761383) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29481051653) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=40546)
**Latest run:** [28458306538:2](https://github.com/huggingface/transformers/actions/runs/29481052670)
**Result:** `failure` | **Jobs:** 13 | **Tests:** 25,697 | **Failures:** 0 | **Duration:** 1h 59m

> **Code quality check failed**: test jobs were skipped. Fix the code quality issues and push again to run tests.

<!-- ci-dashboard-recap:end -->