[models] Add AudioFlamingo3 integration
Author: lashahub
Pull request: yes
Labels: name: New model

name: Audio

This PR adds support for **AudioFlamingo3 (AF3)** — NVIDIA’s open large audio language model capable of reasoning over speech, sounds, and music.

- **Paper**: [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://arxiv.org/abs/2507.08128)
- **Model Weights**: [nvidia/audio-flamingo-3 on Hugging Face](https://huggingface.co/nvidia/audio-flamingo-3)
- **Original Code**: [NVIDIA Audio Flamingo GitHub (branch: audio_flamingo_3)](https://github.com/NVIDIA/audio-flamingo/tree/audio_flamingo_3)

It introduces the following components:

- `AudioFlamingo3` model class
- `AudioFlamingo3Processor` for preprocessing text + audio
- Configuration, modeling, and processing utilities
- Example usage

With this integration, AF3 can be loaded directly from the Hugging Face Hub:

```python
from transformers import AudioFlamingo3Processor, AudioFlamingo3

processor = AudioFlamingo3Processor.from_pretrained("nvidia/audio-flamingo-3")
model = AudioFlamingo3.from_pretrained("nvidia/audio-flamingo-3")

prompt = "What is happening in the audio?"
audio = "clap.wav"

input_ids, media, media_meta = processor(prompt, audio)
output_ids = model.generate(
    input_ids=input_ids,
    media=media,
    media_meta=media_meta,
    generation_config=model.default_generation_config,
)
print(processor.decode(output_ids))
# Example output: "A crowd is applauding and cheering."
```

Comments:
**Update (ready for another pass) - [Qwen2Audio](https://github.com/huggingface/transformers/tree/main/src/transformers/models/qwen2_audio) style**

* **HF-style split**: `AudioFlamingo3ForConditionalGeneration` (LLM + projector) and a lean `AudioFlamingo3Encoder` (audio backbone). No Llava wrappers; no tokenizer in modeling.
* **Processor = arrays in**: accepts NumPy audio arrays, does 30s windowing, builds Whisper-compatible masks, and inserts `<sound>` tokens. No file I/O deps.
* **Configs**: `AudioFlamingo3Config` (+ small `AudioFlamingo3EncoderConfig`). Media tokens, padding/eos/bos IDs, `model_max_length`, etc., read from the config and used in modeling.
* **Long audio**: attention masks + windowing restore support beyond 30s (Whisper stride logic kept), deterministic gen by default (`do_sample=False`).
* **Cleaner surfaces**: removed bare `except`, no custom `from_pretrained` in processor, minimal projector inlined.
* **Weights (temporary)**: `lashahub/audio-flamingo-3` (will switch to NVIDIA on approval).

**Example (arrays, no paths)**

```python
from transformers import AudioFlamingo3Processor, AudioFlamingo3ForConditionalGeneration
import librosa, torch, numpy as np

def load_audio(path, sr=16000):
    y, _ = librosa.load(path, sr=sr, mono=True)
    if y.size == 0: return y
    mn, mx = float(y.min()), float(y.max())
    m = (abs(mx) or 1.0) if mn >= 0 else (max(abs(mx), abs(mn)) or 1.0)
    y = (2 * y / m - 1.0) if mn >= 0 else (y / m)
    return y.astype("float64")

device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AudioFlamingo3Processor.from_pretrained("lashahub/audio-flamingo-3")
model = AudioFlamingo3ForConditionalGeneration.from_pretrained("lashahub/audio-flamingo-3").to(device)

audio = load_audio("skill.wav")
input_ids, media, media_meta = processor("Transcribe the input speech.", audio, device=device)
out = model.generate(input_ids=input_ids, media=media, media_meta=media_meta, generation_config=model.default_generation_config)
print(processor.decode(out))
```

**Open question**

* Prefer keeping the small `AudioFlamingo3EncoderConfig`, or flatten encoder fields into a single `AudioFlamingo3Config`?

**Next**

* Add conversion script (processor/feature\_extractor/tokenizer configs), unit tests (processor + generate), and docs.
* On approval, move weights to `nvidia/audio-flamingo-3`.


@lashahub another tip with regards to code style/formatting. You can run `make fixup` for running repo consistency checks, and `make style` for formatting.

If it passes locally, it should also pass the  [setup_and_quality](https://github.com/huggingface/transformers/pull/40290/checks?check_run_id=49308922594) CI tests.

Current usage:
```python
from transformers import AudioFlamingo3Processor, AudioFlamingo3ForConditionalGeneration
import librosa
import torch


def load_audio(path, sr=16000):
    y, _ = librosa.load(path, sr=sr, mono=True)
    if y.size == 0:
        return y
    dmin, dmax = y.min(), y.max()
    return (2 * y / (abs(dmax) or 1.0) - 1.0 if dmin >= 0 else y / (max(abs(dmax), abs(dmin)) or 1.0)).astype("float64")


device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AudioFlamingo3Processor.from_pretrained("lashahub/audio-flamingo-3")
model = AudioFlamingo3ForConditionalGeneration.from_pretrained("lashahub/audio-flamingo-3").to(device)

# Load one audio and run two prompts (batch size = 2)
audio = load_audio("audio.wav")

inputs = processor(
    ["Transcribe the input speech."] * 2,
    [audio] * 2,
)

inputs = inputs.to(device)
output_ids = model.generate(**inputs, generation_config=model.default_generation_config)
texts = processor.batch_decode(output_ids)
print(texts)
```

- Dropped the legacy `_merge_input_ids_with_audio_features` method
- Perform sound token expansion in the processor
- Switched from `generate()` to `forward()`
- Added the model conversion script
- Added the markdown doc
- _Writing tests in progress_

Updated usage:
```python
from transformers import AudioFlamingo3Processor, AudioFlamingo3ForConditionalGeneration
import librosa, torch


def load_audio(p, sr=16000):
    y, _ = librosa.load(p, sr=sr, mono=True)
    if y.size:
        dmin, dmax = y.min(), y.max()
        y = (2 * y / (abs(dmax) or 1.0) - 1.0) if dmin >= 0 else y / (max(abs(dmax), abs(dmin)) or 1.0)
    return y.astype("float64")


device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AudioFlamingo3Processor.from_pretrained("lashahub/audio-flamingo-3")
model = AudioFlamingo3ForConditionalGeneration.from_pretrained("lashahub/audio-flamingo-3").to(device).eval()

texts = ["Transcribe the input speech.", "Describe the music in details."]
audios = [load_audio("audio_1.wav"), load_audio("audio_2.wav")]
inputs = processor(texts, audios, padding_side="left", tensor_type="pt").to(device)

output_ids = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
texts = processor.batch_decode(output_ids, skip_special_tokens=True)
texts = [text.split("\nassistant\n")[-1] for text in texts]
print(texts)
```

@ebezzam Thanks for the review! I’ve updated the code based on your comments:

- The converted HF model is available [here](https://huggingface.co/nvidia/audio-flamingo-3-hf). The [previous repo](https://huggingface.co/nvidia/audio-flamingo-3) keeps the original weights.
- Added usage examples in the [docs](https://github.com/jsalt-2025/transformers/blob/audioflamingo3/docs/source/en/model_doc/audioflamingo3.md) for loading directly from URLs (audio+text, multi-turn, text-only, audio-only, batched).
- Small patch to align the converted MM projector weight keys.
- Cleaned up `generation_config.json`: correct `bos_token_id`, `eos_token_id`, and `pad_token_id`; default to greedy decoding.
- Use `self.embed_positions.weight` directly in `AudioFlamingo3Encoder.forward()`. We probably can’t reuse `Qwen2AudioEncoder.forward()` as-is, it enforces a fixed mel length, doesn’t slice positions, handles `LayerDrop` differently, and doesn’t expose the pre-pool hidden state we need.
- Added tests comparing generated responses and IDs to the [original model outputs](https://github.com/jsalt-2025/transformers/tree/audioflamingo3/tests/fixtures/audioflamingo3), produced by this [script](https://gist.github.com/lashahub/185b080ca87fe82fd397bc621e74efcb). Some values may differ, as the original model sometimes uses `.half()`.

run-slow: audioflamingo3

This comment contains run-slow, running the specified jobs: 

models: ['models/audioflamingo3']
quantizations: [] ...

Hi @ebezzam thank you for your help!

Changes:
- Added a convenience method `apply_transcription_request` to `AudioFlamingo3Processor`.
- Updated fixtures to include new examples for both single and batch inference.

All tests are passing locally except for:
- `test_eager_matches_fa2_generate`
- `test_flash_attn_2_can_dispatch_composite_models`

But those tests are also failing for `Qwen2Audio` and `Voxtral`. Everything else appears to be working as expected on my end.

run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19031008789)

✅ No failing test specific to this PR 🎉 !


run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19031773446)

✅ No failing test specific to this PR 🎉 !


run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19034559917)

✅ No failing test specific to this PR 🎉 !


run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19039178434)

✅ No failing test specific to this PR 🎉 !


run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19042546216)

✅ No failing test specific to this PR 🎉 !


Hi @ebezzam, I’ve made minor updates to the converter and modified `AudioFlamingo3MultiModalProjector.__init__()` so that the weights remain in BF16.

run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19059173040)

### Model CI Report

#### ❌ Failed tests

- [audioflamingo3](https://github.com/huggingface/transformers/actions/runs/19059173040/job/54435669823):
    tests/models/audioflamingo3/test_modeling_audioflamingo3.py::AudioFlamingo3ForConditionalGenerationIntegrationTest::test_fixture_batched_matches
    tests/models/audioflamingo3/test_modeling_audioflamingo3.py::AudioFlamingo3ForConditionalGenerationIntegrationTest::test_fixture_single_matches



@lashahub thanks! The model now loads on the CIs without memory issues. 

However, could you try recomputing the expected outputs in BF16? (maybe setting dtype in your reproducer [here](https://gist.github.com/lashahub/185b080ca87fe82fd397bc621e74efcb#file-reproducer-py-L77)) 

I'm wondering if that's what's causing the output differences (see [here](https://github.com/huggingface/transformers/actions/runs/19059173040/job/54435669823#step:14:682))

@ebezzam I wasn’t able to reproduce the test failures locally. I reinstalled the latest requirements using `uv pip install -e ".[dev-torch]"`, and both tests are passing on my end. Could you please try running them locally on your side to confirm?

run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19079455343)

### Model CI Report

#### ❌ Failed tests

- [audioflamingo3](https://github.com/huggingface/transformers/actions/runs/19079455343/job/54504394379):
    tests/models/audioflamingo3/test_modeling_audioflamingo3.py::AudioFlamingo3ForConditionalGenerationIntegrationTest::test_fixture_single_matches



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

Hi @ydshieh could you run the slow tests? Thanks

run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19095821991)

✅ No failing test specific to this PR 🎉 !


@lashahub all passes! let's go with these audio

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

run-slow: audioflamingo3

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19098619461)

✅ No failing test specific to this PR 🎉 !


Hi @ebezzam, I’ve updated the [model page](https://huggingface.co/nvidia/audio-flamingo-3-hf) with usage examples and added a `strip_prefix` parameter to `AudioFlamingo3Processor.batch_decode` (set to `False` by default) for ASR.

@ArthurZucker could we get your review on this audio-text-to-text model? (Whisper encoder to Qwen2 LLM)
Model page: https://huggingface.co/nvidia/audio-flamingo-3-hf

Similar models which are used in modular:
- Voxtral ([model](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507), [code](https://github.com/huggingface/transformers/tree/main/src/transformers/models/voxtral))
- Qwen2Audio ([model](https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct), [code](https://github.com/huggingface/transformers/tree/main/src/transformers/models/qwen2_audio))

A difference you'll see in modular (which is a nice aspect here), this model accounts for the attention mask in the encoder. See discussion [here](https://github.com/huggingface/transformers/pull/40290#discussion_r2473767582) with @eustlb 

Hi @lashahub , thanks to the open-source model and code support!

However, after installing transformers using pip
```
pip install --upgrade git+https://github.com/huggingface/transformers
```
I found that the current version does not support AudioFlamingo3ForConditionalGeneration.
```
>>> from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'AudioFlamingo3ForConditionalGeneration' from 'transformers' (/opt/conda/lib/python3.11/site-packages/transformers/__init__.py)
```
Does it require a specific branch?

Hi @Dong-mel , it hasn’t been merged into the main branch yet. You can currently use the [audioflamingo3 branch](https://github.com/jsalt-2025/transformers/tree/audioflamingo3) from the JSALT repo instead.
Hopefully, it’ll be merged within the next few days.

> Hi @Dong-mel , it hasn’t been merged into the main branch yet. You can currently use the [audioflamingo3 branch](https://github.com/jsalt-2025/transformers/tree/audioflamingo3) from the JSALT repo instead. Hopefully, it’ll be merged within the next few days.

Got it, thank you!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19274337788)

✅ No failing test specific to this PR 🎉 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

@eustlb thanks for the review and standardization commits!

@lashahub can you look into the changes [here](https://github.com/huggingface/transformers/pull/40290/commits/e63e295c47cc4dde98778e0b9435a3d3730320b8)? I think something might have introduced an error in the integration tests (see [here](https://github.com/huggingface/transformers/actions/runs/19274337788/job/55111004193#step:14:558)) (or similar issue as before and can try different test examples)

Otherwise I can take a look tomorrow

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

@ebezzam The tests were converting the batch to `bf16` before generation, so I removed all those conversions, it’s working fine on my end now. I also replaced one of the audio files, even though the previous one would’ve worked fine.

run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19290969625)

✅ No failing test specific to this PR 🎉 !


@lashahub we still need to keep the models in bf16 for the tests, otherwise they won't load properly (see [here](https://github.com/huggingface/transformers/actions/runs/19290969625/job/55161919802#step:14:767))

run-slow: audioflamingo3

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

@lashahub could you also add this training snippet: https://github.com/huggingface/transformers/pull/40290/commits/8bf40fa607c6b3e920ed6563893e723b80f18ea4

to the model page: https://huggingface.co/nvidia/audio-flamingo-3-hf

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19292791308)

✅ No failing test specific to this PR 🎉 !


run-slow: audioflamingo3

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19298444302)

✅ No failing test specific to this PR 🎉 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40290). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.