Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
Authors: Sreyan Ghosh
Arushi Goel
Jaehyeon Kim
Sonal Kumar
Zhifeng Kong
Sang-gil Lee
Chao-Han Huck Yang
Ramani Duraiswami
Dinesh Manocha
Rafael Valle
Bryan Catanzaro
Venue: NeurIPS 2025 spotlight ()
Keywords: Large Audio Language Models; Audio and Music Understanding; Multi-Dialogue Chat; Reasoning
Primary area: applications
TL;DR: A fully-open large audio-language model with SOTA performance across 20+ benchmarks.

Abstract: We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities of speech, sound, and music; (ii) flexible, on-demand thinking, allowing the model to do chain-of-thought-type reasoning before answering; (iii) multi-turn, multi-audio chat; (iv) long audio understanding and reasoning (including speech) up to 10 minutes; and (v) voice-to-voice interaction. To enable these capabilities, we propose several large-scale training datasets curated using novel strategies, including AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat, and train AF3 with a novel five-stage curriculum-based training strategy. Trained on only open-source audio data, AF3 achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open-weight and closed-source models trained on much larger datasets.

Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
Sreyan Ghosh Arushi Goel*1, JJehyeon Kim', Sonal Kumar², Zhifeng Kong', Sang-gil Lee', Chao-Han Huck Yang'. Ramani Duraiswami", Dinesh Manocha', Rafael Valle', Bryan Catanzaro'
NVIDIA, USA!, University of Maryland, College Park, USA
* Equally contributed and led the project. Work done by Sreyan during an internship at NVIDIA.
Project: arushig@nvidia.com
Correspondence:
Abstract
We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF- AF-Whisper, unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities speech, sound, and music; (ii) flexible, on-demand thinking, allowing the model to do chain-of-thouuhtt-ype reasoning before answering: (iii) multi-turn, multi- audio chat; (iv) long audio understanding and reasoning (including speech) up to 10 minutes; and (v) voice-to-voice interaction. To enable these capabilities, we propose several large-scale training datasets curated using novel strategies, including AudioSkills-XL. LongAudio-XL, AF-Think, and AF-Chat, and train AF3 with a novel five-stage aurricuuum-base training strategy. Trained on only open-source audio data, AF3 achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open- weeght and closed- source models trained on much larger datasets.
Introduction
speech, sounds, and music-is is cen-
tral to human perception and interaction. It enables us understand our surroundings, engage in conver- sations, express emotions, interpret videos, and enjoy music. For AI systems to approach artificial general intelligence (AGI) [88], they must similarly develop the ability comprehend and reason over diverse audio sig- nals. While Large Language Models (LLMs) excel at language-based reasoning, their audio comprehension remains limited - both in accessibility and capabil- ity (54, 106). Extending L.MSs to process and reason over audio is essential for building truly context-aware, intelligent agents.
HIEE
SEN MAU
Figure 1: AF3 prior SOTA LALMs (values normalized and WER=100-WER) AF3 outper- forms most ppenssourcewweight (e.g., Qwen2.5. Omni) and closed (e.g., Gemini 2.5 Pro) LALMs while being fully open.
Audio Ladio-Language Models (ALMs) extend the capabil- ities of LMs to the auditory domain. Early works such as CLAP [33] align audio and text in a shared embed- ding space, enabling them with tasks like retrieval [89].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Mudels
Audio Understanding
Voice
Multi-turn Chat
Long Audio (2:30 secs)
Open-Source
Sound
Music
Speech
Out
Single
Multiple
Speech
Sound
Music
Model
Data
Cade
LTU
x
X
LTU-AS
6
<
X
X
X
X
X
v
GAMA
4
/
X
X
X
X
x
X
V
4
SALMONN
/
4
X
x
4
/
MaLLaMa
x
V
X
X
V
V
Phi-4-mm
V
<
V
V
J
Qwea-Audio
Qwen2-Audio
X
Qwen2.5-Omni
4
V
J
GPT-4o Audio
V
V
4
Gemini 2.0/2.5
4
V
V
4
Audio Flamingo
Audio Flamingo
Audio Flamingo
Table 1: Comparison of various LALMS in terms of capabilities and openness. AF3 stands as the most capable and open model to date, achieving SOTA results across benchmarks (A Chat stands for Audio). *Voice-ou is powered by our novel streaming TTS implementation. which is also applicable to other LALMs.
More recently, the emergence of Large ALMs ( decoder-only language models augmented with audio understanding [20, 19, 105] has unlocked powerful capabilities, including open-ended audio arrestion-answeriigg (AQA) that demands both reasoning and world knowledge [101]. These
capabilities have further enabled tasks like audio analysis [32, 601, conversational assistants [24), etc.
However, existing models still fall short in key areas critical to AGI, such as expert-level reasoning [88, 101), multi-turn and multi-audio dialogue [44], and long audio understanding [40]. We identify two core limitations: (i) most L.LLSs are trained primarily on short audio for recognition tasks rather than ones that require deliberate reasoning; and (ii) in turn, they lack exposure to the skill sets required for complex tasks. Additionally, most LALMs that support all three modalities of speech, sound, and music are closed-source: while some have publicly released models weights [20, 19, 11. they offer limited to no information about their data, code, or recipes (more details in Table 1).
Main Contributions. To address these issues, we introduce Audio Flamingo 3 (AF3), fully open-source' LALM with state-of-the-art performance in audio understanding and reasoning across 20+ benchmarks. In addition, AF3 brings several novel capabilities, including multi-turn, multi-audio chat, on-demand thinking, voice-to-voice interaction, and long-context audio reasoning (up to 10 minutes). We propose three core innovations to enable these capabilities: (i) Data: We focus on curating high-quality data at scale and propose (a) Audio XL: large-scale dataset of 8.M diverse AQA pairs, (b) LongAudio-XL XL: large-scale dataset of 1 1.25M diverse audio QA pairs for long audio reasoning; (c) AF-Chtt: a multi-turn multi-audio chat dataset curated using a novel algorithm with 75k instances and (d) AF-Think: dataset with 250k+ AQA pairs with short length prefixes to encourage CoT-type reasoning before arriving at the answer (ii) AF- Ai--Wiisee: We train AF- AF-Whisper, unified audio encoder pretrained using novel strategy
large-scale audio-caption pairs, capable representations across speech, sounds, and music; and (iii) Learning Curriculum: We train AF3 with five-stage curriculum-based training strategy that progressively increases context length and task complexity. In summary, our main contributions are:
of learning
We introduce Audio Flamingo (AF3), the most open and capable foundational LALM to date. AF3 introduces key capabilities including: (i) long-context audio QA (extending beyond sounds as in [40] and including speech), and (ii) flcxible, on-demand thinking, enabling the model to generate concise, CoT-style reasoning steps when prompted. AF3 achieves sfate-of-the-art performance on 20+ audio understanding and reasoning benchmarks.
We also present AF3-Chat, a fine-tuned variant of AF3 designed for multi-turn, multi-audio chat and voice-to-voice interaction.
We propose novelties in data curation, audio encoder representation learning, and training strate- gies. Being fully open, we release our code, training recipes, and 4 new datasets to promote research this space.
By fully open, we mean that the model's weights, training data, and code will be publicly released, with full transparency about the training methodology. Due to the licensing and scope of the training data used in the work, all releases will be under research-only license.

---

2 Related Work
Audio Language Models. The rapid progress of LLMs has catalyzed the development of multimodal LLMs (MLLMs) capable of understanding and reasoning across diverse data modalities, including audio. Within this space, ALMs specifically target reasoning over auditory inputs such as speech, sounds, and music. ALMs typically follow two main architectural paradigms: (i) Encoder-only ALMs, which learn joint embedding space for audio and text, enabling tasks like cross-modal retrieval. Representative models include CLAP [333, Wav2CIIIP [112], and AudioCLIP [48]. (ii) Encoder- decoder ALMM,, also referred to as LALMs, which use decoder-only LLMS augmented with an audio encoder. Notable examples include LTU [46]. LTU-AS [45]. SALMONN [104], Pengi [27], Audio
Flamingo [65], Audio Flamingo 2 [40], AudioGPT [53], GAMA [41]. Qwen-Audio [20], and Qwen2- Audio [9]. These LALMs have significantly improved performance on core audio understanding tasks such as automatic speech recognition (ASR) [96], audio captioning [60], and acoustic scene classification [16]. More importantly, they have enabled new capabilities such as open-ended audio question answering, which requires complex reasoning and external world knowledge.
Despite these advancements, current LALMs fall short in supporting various capabilities, including multi-turn, multi-audio chat, long-context audio comprehension, etc. Moreover, most LAL.Ms are limited specific audio types, lacking the ability unify understanding across speech, sounds, and music. Finally, the most advanced LALMs remain only partially open, releasing model checkpoints without accompanying training code or data. This lack of transparency limits reproducibility and impedes scientific progress by obscuring the development process.
Reasoning and Long-Contex: Understanding. Recent progress in LLMs has increasingly empha- sized long-context understanding. In the vision- vision-language space, substantial strides have been made in modeling long videos [17]. In the audio domain, AF2 marked the first step toward long-context audio comprehension, though it is limited to sounds and music.
Parallel efforts have aimed to enhance reasoning in LLMs and MLL.Ms through improved reasoning datasets [101, 110], advancements in multimodal perception [116, 105), and emerging paradigms like chain-of-thougle (CoT) prompting [8Q]. which encourages models to "think before answering." In developing AF3, we combine these iwwances-intterratie controlled reasoning supervision, long- context training, and modality diversity-to equip the model with strong reasoning capabilities and long-context comprehension, including speech.
3 Methodology
3.1 Audio Flamingo 3 Architecture
In this section, we discuss our proposed architecture for Audio Flamingo 3 as shown in Figure AF3 consists of i) AF-Whisper: an audio encoder with sliding window feature extraction, ii) audio projector, iii) an LLM, and iv) a streaming TTS. We provide details of each component below.
AF-Whisper Audio Encoder. Prior work in audio representation learning typically treats speech, sounds, and music as separate modalities, and LALMs often rely on distinct encoders for each [104, 41]. Using separate encoders for LALMs increases model complexity, introduces frame- rate mismatches, and can lead training instability. To address this, we propose AF-Whisper, a unified audio encoder trained with simple yet effective representation learning strategy to model all three audio types.
As illustrated in Figure 2, we start with the pre-trained Whisper large-v3 encoder [96], attach it to a standard Transformer decoder, and train using the audio captioning task with the next-token- -eeeeiteen-rreiiction objective. To achieve this, we generate a natural language caption for each audio, describing its speech, sound, and music content. First, we pool several datasets and then prompt GPT-4.1 generate the audio caption. For prompting, we use available metadata for each sample, which includes transcripts, ambient sound descriptions, and music attributes. For samples lacking any of the metadata, we synthesize it using AF2 [40] or Whisper-Large-
ASR [96]. All datasets used for training are detailed Section A.2. We choose Whisper as the backbone due to its existing speech understanding capabilities and its dense, high-resolution audio features, which are more informative than those from models like CLAP [33]. We connect it with a Transformer decoder using aross-attention (similar to RECAP[77] and AF2 [40]) with 24 layers, 8 attention heads, and 1024 hidden size.
3

---

Chappt
Shaainn TTS
Large Language Model
Ellaary
Whippe
Dosells
- fe hoo
Andle
Aore Che ove aeaeittte deuedlr
Audio Adggee
Specck Masko
Figure 2: Overview of Audio Flamingo 3, AF-Whisper training, and five-stage curriculum training.
Feature Extraction. Given audio input A, we first resample it to 16kHz mono. The raw waveform then transformed into a 128-channel mel-spectrogram using window size of 25ms and a hop size of 10ms. This mel-spectrogram is processed by AF- AF-Whisper, producing hidden representations, denoted as h, = (A). where ha E Nx As shown Figure 2, each audio is processed in 30-second chunks of non-overlapping sliding windows, and N or the temporal resolution depends on the length of the audio and the maximum number of sliding windows (which varies according to the stage of training). AF- Whisper produces audio features a frame rate of 50Hz, and further apply a pooling layer with a stride of two similar to [19]. denotes the hidden dimension, which is 1280.
To align the audio modality with the text embedding space of the LLM, we introduce audio adaptor layers, denoted by A(.). Specifically, the encoded hidden representations ha from AF- Whisper are passed through these adaptor layers (2-layer MLP with GeLU layer) to produce embeddings: a = A(h.). These resulting embeddings serve as prompts to the LLM, alongside textual instruction.
Audio Adaptor.
Large Language Model (LLM).. We employ Qwen-2.5-7B [118) as backbone, decoder-only causal LLM with 7B parameters, 36 hidden layers, and 16 attention heads.
employ TTS module for streaming speech generation, supporting streaming inputs and outputs. Our TTS module employs a decoder- only transformer architecture: it predicts the subsequent audio token conditioned incoming subword text tokens from the LLM and the history previously generated audio tokens. Similar streaming TTS techniques have been explored with LLMs [115] (for voice-out LLM outputs), but not in the context of LALMs (which we define as models designed to perceive and reason over diverse audio inputs). Since not core novelty of our work, we provide more details, including training and architecture, in Appendix I.
Streaming TTS. To enable voice-to-voiee interaction,
4 Audio Flamingo 3 Training Data
We present detailed statistics for all datasets used to train AF3 in Table 11. AF3 has total stages of training, where each stage employs unique combination of datasets with unique weights (number of passes over that dataset for that particular stage). For Stages I and 2, we use open-source, recognition-focusee
foundational datasets converted to QA format. In the following sub-sections, we introduce our four novel skill-focused and unique datasets, each accompanied by custom data curation strategies, used in Stages 3, 3.5, and 4, which form core contribution of this work.
4.1 AudioSkills-XI. XI.: Expanding AudioSkills with Ceasoning-Focusse QAs
Audio QA pairs derived from foundational benchmarks focused on recognition tasks (e.g.. ASR, acoustic event classification) are insufficient for training models expert-level reasoning [101]. Therefore, in Stage 3 fine-tuning, we prioritize the development of reasoning and problem-solving abilities by curating large-scale, high-quality Audio QA data. Inspired by AF2, we limit this stage to short audio clips (<30s) and defer long audio reasoning to later stages. We expand the AudioSkills

---

dataset [40] by 4.5M new Audio QA pairs (majorly multiple-choice questions (MCQ)-based) to create AudioSkills a high quality corpus containing 8M Audio QA pairs, using two strategies:
(1) We expand coverage of existing reasoning skills and introduce new ones using additional audio sources, increasing the dataset by 3.5M QA pairs: (a) For sounds, we incorporate data from YouTube8M and synthetic sources. (b) For music, we include Music4All [102] and the Million Song Dataset [8]. For YouTube8M, we adapt captions from AudioSeiCaps [6] and generate QA using GPT-4.1 with general reasoning prompts from AF2. Additionally, we introduce new reasoning skills and design corresponding prompts to support them. For music, we generate data for novel skills (as AudioSkills was focused more on sounds; details in Table 6) and go beyond captions we leverage metadata such as song titles, artist names, album names, etc (see Fig. 4 for full list) to generate more complex, reasoning- focused QAs. We also use this metadata to generate rich music captions for Stage 1 and 2 pre-training (see Fig. 4), demonstrating how text-hased knowledge can enhance audio understanding, particularly in snowledge-driver domains like music. This method can be seen as synthetic knowledge generation, where we leverage text-based knowledge to enrich audio understanding and enable models to acquire domain-specific knowledge from unlabeled audios in the wild. Our analysis shows that LLMs like GPT-4. I hold substantial world knowledge about music, and that metadata improves QA quality significantly.
(2) We augment AudioSkills with 1M speech QA samples using YouTube8M (2), LibriSpeech [92] (read speech), GigaSpeech [14] (conversational), and VoxCeleb2 [21] (interviews). From YouTube8M, we introduce new task: Speech-in- Srreechin--ounne -ounnd QA, where the model must reason over both speech content and ambient sounds to understand complex auditory scenes. To create these QAs, we create Bnrech-in-SSund--C Caps, a new dataset with R22 speech-aware auditory scene captions from You'll M.. To curate this, we first filter the dataset for English speech (using AF2) and transcribe the spoken content with Whisper-Large-v3. We then generate two types of descriptions: one capturing sound events and another summarizing speech characteristics such as tone, emotion, and pitch (both using AF2 and custom prompts; see Appendix 26). Finally, we prompt GPT-4, I to synthesize a speech-aware scene caption. These captions significantly improve the quality of final audio captions (compared to only using sound information) by providing a more holistic representation of the audio. For LibriSpeech and GigaSpeech, selecting informattiondeens segments filtered by prompting an LLM. To move beyond basic spoken content understanding common in most current datasets [121], we design five distinct types of speech QA that require diverse reasoning skills (explained in the next subsection).
concatenate shorter segments into clips of 15-30 seconds,
4.2 LongAudio-XI.: Expanding LongAudio with Long Speech QA
To our knowledge, Long Speech QA (i.e., audio 2 30 seconds) has not been explored in prior work, despite its relevance real-world applications such as long-form conversation understanding, meeting summarization, and narrative comprehension. To bridge this gap, we extend the existing LodgAudio dataset [40] (focused on sounds and music) by incorporating over 1M reasoning-focused QA examples from long-form speech (30s-10min). We curate audios from diverse sources including:
Single-speaker speech: LibriSpeech (audiobooks) [92], EuroParl [62]. VoxPopuli (parliamentary
debates) [107] and Multi-speaker conversations: Spotify Podcasts [23]. Switchboard [43], Fisher
(dyadic calls) [22], MELD [94), DailyTalk [71], MMDialog (natural dialogues) [35]. We merge
consecutive short segments in chronological order to construct longer, coherent audios. We construct
QAs across a wide range of skills, as illustrated in Figure 3:
1. Sarcasm Identification: Inferring sarcasm by analyzing content, tone, and emotional cues.
2. Emotional State Reasoning: i) Identification: Determine the speaker's emotion at specific utterance. ii) Causal Reasoning: Identify the reason behind a speaker's 's emotional state using conversational context. iii) Emotion Flip: Explain shifts in a speaker's emotional state during the conversation.
3. Topic Relationship Reasoning: Understand how two ideas or topics are related within the overall discourse.
4. Information Extraction (LE): i) Needle QA: Targeted QA on specific utterances or parts of the speech (e.g., entity fact extraction, general knowledge linkage). ii) Causal QA: Identify causes for a particular utterance in context. iii) Respouse QA: Extract how one speaker responds to another's statement. iv) Topic Q4: Identify the main topic of the speech or conversation.
5. Summarization: Generate concise summary of the speech content.
5

---

Speedh-is Strnnd
Eantiinn State
Tayle Relatienshp Rewsning
Order
AF-Chat
Ever (amm fj- covinda Til
dagzate, dinely soo "Thukk you prid macling orthook pagas and occauzeal traghtr.
Caption: Á make vikoe acunskng
Mestiiicatin:
wher
penorol
Question: ssuting
Temporril Order: distoss
spbest, cssgotic place fids like
coald power inmo ganee levd. BiT
Segolice lets, wlat was the guakel's
profosiscal engagennat,
lepics is spook LAi The youker
led woold you angget
ertetiasdl aas wlile sckapwisiggng their encrggstt
peroin they stangf?
introdaces Bee last crino it
sencial liring.
shou the qpakkr's cultund contzc
Quastiea:
What can be infered
connerattion deai speake
Camaal Reasening In the ipp
Informatioon Extraction
Saca of spoach ovchee over ane?
Traporal Attrihaee Row doe the
inso a clill. lodf tack statiiyy diviing dowe the sirpo and
Al:
chtyuus
from his statemeer and the background sodio'
Neodle (AA What spanticc
cuample gir highligh
loga laine coallics.
stsphii the dlydaao
Narcasm Mentiiictton
lan decs the speakei cratioeal
Konation the crnveroties,
spocaliiius with esphocantaaiss
dilference betwees inveiing and
crilial
(8) autato optimoti and lecesses
recsricataaln lacd
(Jamn Tt rooud Baaliudiinnn
vilee
yoer
miiht the suggnator "horh eron
Queetion:: concoration, why
Grislesf consiaasd
edatorabis!
whee dasoninng for sclicco of arether sence reattd ao pad
sour shift froe anger mustrssn
peogrant?
Respasse OAi Rive does the speaker
Camall Que casud ouz cyeker
boouse fisstetted about lig
when inked shout ther
rood
repictti (A) vey heghening,
and facuees rslanie other
Trepporal Weteering: When doas the
spoker discas the secivll ofills
radiodic idess the
thi kind
the eackells books .o.
Al: A Aborserly Yoo could minserper
Uver (oum dp World tormonte
litteing style
AF-Thisk
Vemporral Grounding whai poiat
the sperch dase the speacer
structrr the retur seed change
ti6 could Al: Vou could keep
Question: Wlich Joss best docilo the scege Cheose one snncg dae fellowing options (A) Electoms (B) Jacz (C)
Pay rock (D) Country Plone think and rensse about the inget music beforr you reageed.
dos be der gpciffs seting of dee koi
crmadt fellowing options (A)A1
cornect ortire
meladie unitton hit acapling the hummery to ridee, more comppe en daadd would keb il feel loues in as...
accemiii bizncing diemons beth popater more and rock cntmmenttion the moe friing geree amony optern
haower Haod on the prosence eleatre gurare droms with driving hest, and vani style molodic and
baginning,
NC) Paprook
Figure 3: Examples from AudioSkill-XL, LongAAuio-XL. AF-Think and AF-Chat. We include addi- tional examples in Appendix B and C, featuring novel music reasoning QAs mentioned in detail in Section B.1.2.
6. Order: i) Temporal Order: Understanding the sequential order of topics in the speech; ii) Temporal Attribute: Understanding how topics change over time: iii) Temporal Referring: Resolve references to specific time points (e.g., "at the end") iv) Temporal Grounding: Identify when in the audio specific topic was discussed.
4.3 AF-Think: Towards flexible, on-demand reasoning
Recent studies show that making an LLM "think", similar to chain-of-thougle (CoT) prompting [111],
can improve reasoning performance in LLMs [47], especially for complex tasks like coding and math
(e.g., DeepSeek-RI. OpenAl-ol). Visual MLLMS have also benefited from this paradigm [116, 109]. In the audio domain, early attempts such as Audio-CoT [80], Audio-Reasoner [1]4], and R1-
AQA [73] have explored CoT--ttyle reasoning, but often yield limited gains and involve complex or inefficient training procedures. Moreover, consistent with findings in [73], we observe that deep, explicit thinking does not always improve performance in audio understanding tasks.
In AF3, we adopt a lightweight thinking mechanism with two key modifications: (i) We create AF-Think, a dataset of 250k MCQ-based QAs with short, controlled thought preceding the answer. This additional thinking serve as a prefix to the answer and are limited to an average of approximately 40 words, providing concise yet effective context for audio QA (example in Figure 3). (ii) Instead of explicitly post-training for COT, we add special suffix to QA prompts (highlighted in Figure 3). We include AF-Think in the Stage 3.5 training mixture, upweighted relative to standard QA data. This allows AF3 to think only when prompted, fleibbe on- on-demand additional reasoning.
To generate Ah-Think, we first sample a subset of multiple-choice reasoning QAs from AudioSkills- XL and LongAudio-XL (originally with just the correct option as the answer). Next, we prompt Gemini 2.0 Flash with the input audio, the question, and the answer to generate short thinking prefixes. We found Gemini to hallucinate less and generate more accurate reasoning when guided by the ground-truth truth answer, rather than producing CoT from scratch. We restrict this process to only high-quality datasets and filter out noisy instances.
4.4 AF-Chat: Multi-turn Multi-audio Chat Data
While single-turn sungle-audio QA training equips LALMs to reason over individual audio inputs, enabling free-form, multi-turn, multi-audio conversations requires a dedicated chat alignment tuning stage, akin to the instruction-tuning phases used for LLMs [122], Chat becomes significantly more complex when multiple audio inputs must be integrated across turns, requiring the model to track context, reason over relationships between past and current inputs, and generate coherent follow- ups. Despite its importance and chat being the most used application of LLMs, this capability remains underexplored in LALMs primarily due to the absence of open, high-quality training data.
To address this gap, we introduce AF-Chut, a high-quality fine-tuning dataset consisting of 75k multi-turn, multi-audio chat instances. On average, each dialogue includes 4.6 audio clips and 6.2 dialogue turns, with range of 2-8 audio clips and 2-10 turns. To construct this dataset, we draw

---

from Speech-in-Soune Caps (for speech and sounds), and Music4All and MSD (for music). We follow a two-step curation process: First, for each seed audio, we identify its top 8 most semantically similar and dissimilar clips using combination of captions, NV-Embed-v2 [68] embeddings, and FAISS-based clustering [31] (details in Appendix F.2). For every dialogue, we restrict the audios to this pool. This targeted clustering yields significantly higher-quality dialogues than random audio selection by ensuring each instance is grounded in a diverse yet semantically coherent audio pool.
Next, we prompt GPT-4.1 using carefully designed expert exemplars (Fig. 36 and 35) to generate natural, multi-tum chat sessions under the following constraints: (i) the model may choose any subset of the
audios (up to 10 turns,, prioritizing conversation quality; (ii) not all turns require a new audio-follow-up and clarification questions are encouraged; and (iii) later turns may refer back to earlier audios or responses to simulate real conversational grounding. The design of AF-Chat informed by extensive internal human studies to reflect how users naturally interact with mudio-language models. As result, it provides rich, diverse supervision for aligning LALMs to handle complex, contextual, and naturalistic audio conversations. Finally, we select 200 high-quality samples for the test set, known as AF-Chat-test, and ensure that the audios in these instances have audio clips that were not seen during training.
5 Audio Flamingo 3 Training Strategy
AF3 is trained using five-stage strategy designed to progressively enhance its capabilities by increasing audio context length, improving data quality, and diversifying tasks. A full list of datasets used at each stage is provided in Appendix 11.
Stage 1: Alignment pre-training. For this stage, we train only the audio adaptor layers while keeping the audio encoder and LLM frozen. This step aligns encoder representations with the language model. Stage 2: Encoder Tuning. The main purpose of this stage is to adapt AF-Whisper to diverse datasets and broaden and improve its audio understanding capabilities. We fine-tune both the audio encoder and adaptor while keeping the LLM frozen. both Stages and 2, the audio context length is limited to 30 seconds, and training uses recognition focusse datasets (e.g., classification, captioning, and ASR). Stage 3: Full Fine-Tunine Tuning. The primary purpose of this stage is to emphasize reasoning and skill acquisition by the LALM. As mentioned earlier, since skill-specific data is easy to scale on short audios, we still stick to short audios in this stage and use high-quality foundational and QA datasets and our proposed AudioSkills-XL. However, we increase the audio context length up 2.5 minutes now to accommodate the moderately long audios in AudioSkills. The resulting model at the end of Stage 3.5 is referred to as AF3. Stage 3.5: Context Extension and Thinking. This stage focuses on extending context length and encouraging CoT-style reasoning. In addition to the Stage 3 data mixture, we incorporate Long and AF-Think. We adopt LoRA-based training (51l-similar to LTU and GAMA-b. freezing the model's 's original weights and training LoRA adapters for the LLM. This approach allows end-users to flexibly enhance the model's reasoning and long-context understanding capabilities on demand. Stage 4: Chat and Voice Fine-Tunine on enabling multi-turn, interactive, and voice-based dialogue. We fine-tune the entire model on our proposed AF-Chat dataset to equip AF3 with conversational audio understanding and response generation capabilities. The resulting model at the end of Stage is referred to as AF3-Chat.
This stage focuses
6 Experiments
Experimental Setup. We train AF3 on 128 NVIDIA 1 GPUs, each with 80GB of memory.
Details about batch size, learning rates, and optimizers for each stage of training are in Appendix H.
Baselines. We evaluate our model against recent SOTA LALMs, including GAMA [41]. Audio
Flamingo [65], Audio Flamingo [40], Qwen-A(udio) [20], Qwen2-A(udio) [19]. Qwen2-A(udio)
(Inst)ruct, Qwen2.5-0n (mmii) [117], R1 R1-AQA [73]. Pengi [27), Phi-4-mm [1]. Baichun Audio [75].
Step-Audio-Chal Chat (52), LTU [46]. LTU-AS [45], SALMONN [104], AudioGPT [53], and Gemini (2.0
Flash, 1.5 Pro, 2.5 Flash and 2.5 Pro) [105] (note we do not evaluate Gemini on ASR benchmarks due to low rate limits), as well as GPT-4o-audio [54]. For LongAudioBench, for models that do
not support longer audio, we follow the cascaded approach for evaluation proposed by [40]. For Table 3, we only compare against open LALMS. All results reported in the tables correspond to the best-performing model. Evaluation for voice-to-voice capabilities is beyond our scope.
7

---

Table Comparison of AF3 with other LALMs on various benchmarks (WER & (Word Error Rate), ACC (Accuracy), and GPT4o f (GPT evaluation)). We report scores for only the lop-performing prior LALM. + +Think refers to AF3 with additional thinking. We highlight closed source, open weights, and open source models.
Task
Dataset
Prior SOTA
Metrics
Results
0M..-v05.15.25
Qwen2.5-O
76.77 66.33 68.90 17.00
Audio Flamingo 3 +Think
ACC
75.83 ( 74.47 66.97 72.42
Sound Music I Speeck Avg
75.27
73.16
MMAU-v0S.1S.22 (test-mini) Sound I Music I Speech I Avg
Qwen2.5 Audio Flamingo 3 +Think
78. 10 65.90
70.60 17..01
ACC
79.58 73.95 66.37 73.30
79.88 76.55
66.37 77.26
Qwen2.5-O Audio Flamingo 3 +Think
56,7
MMAR
ACC
58.5
60.1
Gemini-1.5 1,5-Pro
60.7
MMSU
Audio Flamingo -Think
ACC
61.4
62,3
ClothoAQA
Qwen2.5 Qive22.5.0
1.2.
ACC
inanimous non-binary
Audio Flamingo 3
91.1 56,2
Audio Captioning
Audio Flamingo 21 Audio Flamingo
0.46 I 10.58
CIDBr
Clotho-y2 AudioCaps
Audio Flamingo 3
0,50 0.70
Audio
Audio Entailment
Audio Flamingo 2 Audio Flamingo
92.5 193.3
Understanding
Clotho AudioCaps
Audio Flamingo 3
ACC
93.3 | 95.0
and Reasoning
Qwen2-A-Inst
59.2
IEMOCAP
ACC
Audio Flamingo 3
63.8
CochlScene
Pengi Audio Flamingo 3
ACC
91.6
93.2
NonSpeech7k
Audio Flamingo 2
ACC
84.3
Audio Flamingo 3
85.9
Gemini 2.5
82.0
CMM Hallucination
ACC
Audio Flamingo 3
86.5
Audio Flamingo 2
96.4
Comp.A-Reext
Audin Flamingo 3
ACC
98.0
MusicAVQA
Audio Flamingo
Qwen2.5.0
ACC1
73.4
76.7
NSyuth Source Instrament
Pengi Qwen-A
62.0178.8
Audio Flamingo 3
ACC
65.5 78.9
Music Instruct
Flamingo 2
90.2
Long
Audio Flamingo 3
ACC
92.7
Qwen2-A-Inst
46.2
MuchoMusic
Audio Flamingo 3 +Think
ACC
47.4
47.6
Gemini 2.5 Pro Audio Flamingo 3
8.7
LibriSQA
GPT4o
8.7
Bencg
Gemini 2.5 Pro
GPT4o
60.4
Flamingo
68.6
+Speech (ours)
Gemini 2.5 Pro
GPT4o
66.2
Audio Flamingo 3
72,9
LibriSpeech (en)
Phi-4-mmi Qwen2.5:0
1.67
WER L
test-cleam zest-other
Audin Flamingo 3
1,57 33
SPGISpeech (en)
Qwen2 A Inst
WER 1
3.0
Automatic Speech Recognition (ASR)
Audio Flamingo 3
1.86
Phi-4-mm
2.9
TEDLIUM (en)
WER
Audio Flamingo 3
3.5
GigaSpeech (en)
Phi-4-mm
WER 1
9.78
Audio Flamingo 3
10.27
Phi-4-mm Audio Flamingo 3
7.61
Common Voice 15 (en)
WER 1
7.4
Phi-4-mm Audio Flamingo 3
5.91
VoxPopuli (en)
WER 1
5.55
Evaluation Datasets. We evaluate AF3 on variety of tasks and benchmarks, including audio classifi-
cation (CochlScene [57], NSynth (Source and Instrument) [34], NonSpeech7k [99], IEMOCAP [1L]), audio QA (ClothoAQA [76]. MusicAVQA [74], Music Instruct [26), LibriSQA [121]), reasoning- focused audio QA (MMAU [101] (v05. 15.25), MuchoMMssic (perceptual version) [120, 110], MMAR [81], MMSU [108], CompA-R- R-tsst [42), Audio Entailment [29]), multimodal hallucination
8

---

Table Comparison of AF3 with open LALMs on AF-Chat, voice-text and TTS benchmarks. WER (Word Error Rate), SIM T (Similarity). Human f (Human evaluation) and GPT40 1 (GPT evaluation) indicate metrics and whether lower or higher is better.
Task
Dataset
Model
Metries
Results
AF-Chat-test
Qwen2,5-O
2.412.713.2
Multi-audio chat
Factuality Usefiduess Depuh
AF3-Cbot
Human"
3,61
OpenAudioBench
Qwen2-A-Inst
57.19 669677 40.400 30
alpara-eval I Homa-quessions
Qwen2.5.Q O
GPT40 1
72.76 75.33 57.06
Voice-Text
trivia-qa
AF3-Chat
76.26 80.33 53.05
VoiceBench
Qwen2-A-Inst
3.691 198. 85 149.01 3,.40
AlpocaFuual | AdvRRenkk
Qwen2,5-O
GPT4o f
4.33 99.62 179.12 3.84
OpenBookQA Commonevall
AF3-Chat
4.19 988 26 66.81 3.44
SEED (test-en)
Qwen2.5-O
2.72 10.63 14.62s (1.26s)
Speech Generation
Content Cons. Speaker Sim. I Inf. Time
AF3 Chat
WER JISIM 11 TTime 4
2.02 1 0.61 5.94s (0.02s)
detection (CMM [72)), audio captioning (Clotho-v2 [32], AudioCaps [60]) ASR (Librispeech (clean and other) [92]. SPGISpeech [90], TEDLIUM [100, 49]. GigaSpeech (Large) [14], Common Voice 15 [5] and Voxpopuli [107]) and long audio captioning and QA augment with 2.5k human- human-annoter annotated Long-speech QA instances). For evaluating chat capabilities, we conduct a human study of model outputs on AF-Chat-test (more details in Appendix E) and compare only with Qwen2-Audio. Fach annotator is asked to rate the response of the model for every turn on a scale of 1-5 for factuality, usefulness, and depth. We report results averaged across all instances across all turns. Furthermore, evaluate the voice-text capabilities of our AF3-Chat model on two datasets, OpenAudioBench [75] and VoiceBench [18]. These benchmarks consist of voice queries (synthetically generated speech from text queries) and assess aspects such as instruction following, question answering, trivia knowledge, and reasoning. Finally, we evaluate our speech generation module using zero-shot TTS evaluation the English subset of the SEED benchmark [4]. To calculate accuracy. we use either exact string matching with the ground truth or CLAP-based retrieval following [27], implemented with open-source AF-CLAP [40]. For MCQ, AF3 typically outputs only the selected option. In cases where the model provides more verbose or open-ended responses (e.g., with thinking mode), we apply multiple regex patterns extract the chosen option.
which we
6.1 Audio Understanding and Reasoning Evaluation
AF3 is the strongest and fully open-source LALM. Table 2 shows AF3 outperforming previous SOTA open-weight and closed-source models across wide range of audio understanding and reasoning benchmarks. AF3 sets new highs on MMAU (72.42) (note for Qwen2.5-0mni on MMAU we report the "parsed score" for fair evaluation), ClothoAQA (91.1). Clotho Entailment (92.9), and CMM Hallucination (86.7). On tasks like NSynth and MusicInstruct, it shows significant gains, highlighting strong sound and music understanding. For Aunh (sound and speech), AF3 outperforms Gemini 2.5 Pro by wide margin, demonstrating its strength in long-context reasoning. We also evaluate AF3 with thinking prompts (+Think) on reasoning-heay benchmarks like MMAU and MuchoMusie, observing a performance boost. Although the thinking mode is activated after Stage 3.5 only when using our specific thinking prompt, the checkpoint remains usable without it. We report average scores of 73.16 and 74.26 on MMAU-test and MMAUUttst-mini, respectively. Additionally, AF3 achieves state-of-the the--at ASR results on LibriSpeech, SPGISpeech, and VoxPopuli- even compared to dedicated ASR models-despite not being trained on large-scale ASR datasets like many open- weight models. We illustrate demo of AF3's capabilities in Fig. 14.
6.2 Chat and TTS Evaluation
Multi-turn multi-audio chat evaluation. On AF-Chat-test AF3-Chat shows a relative improvement of 30% over Qwen2.5-Omm) thereby showing the capability of effectively handling extended dialog turns, allowing for deeper contextual reasoning and more accurate references multiple audio inputs.
Voice-Texx and Speech Generation Evaluation. Table 3 evaluates AF3-Chat on two key tasks: voice-to-text and text-to-speech generation. In the voice-to-text setting (spoken QA), AF3-Chat achieves strong gains across all Open surpassing Qwen2.5-Omni. On VoiceBench, which tests spoken QA robustness across AdvBench, CommonEval, and OpenBookQA, AF3-Chat

---

performs comparably to Owen2.5-Omin and Qwen2-Audio Chat. For TTS (evaluated on SEED test- en), AF3- Chat shows improved performance with lower WER of 2.02 (vs. 2.72 for Qwen2.5-Omni) and speaker similarity score of 0.61, closely matching Qwen2.5's 0.63.
Furthermore, AF3-Chat exhibits significant advantages in generation speed. For 10-second audio generation on an A 110 GPU, AF3-Chat's text-to-audio token generation is 5.94 seconds with an additional 0.02 seconds for waveform synthesis. In comparison, the Talker model of Qwen2.5- Omni requires 14.62 seconds for token generation and an additional 1.26 seconds for waveform synthesis. This efficiency allows our streaming text-to-speech to achieve time-to- tor-t----otttt of 0.15 seconds and an inter-token latency of 0.06 seconds (both including waveform synthesis), producing 10-second audio clip in 6.68 seconds.
6.3 Ablation Studies
In this section, we ablate our key components (using just 10% of the training data) to support the paper's main claims.
Evaluating AF- AF-Whisper as a Unified Encoder. Table compares AF3 trained with our unified AF-Whisper encoder against dual-encoder setup using CLAP for sounds/music and Whisper-v3 for speech [33, 96]. AF- Whisper outperforms the dual-encoder model under the same data budget, demonstrating its effectiveness
single encoder for sound, music, and speech.
AudioSkills-XI: A Key Dataset for Performance Gains.: To measure the impact of AudioSkills-XX we ablate it from Stage 3 of training and compare results to the full setup. As shown in Table 4, remov- ing AudioSkills-XO causes a significant performance caap--patcuularly on MMAU underscronn its role in improving generalization and robustness. These findings highlight the value of large-scale, skill-targeted audio QA data for fine-tuning multi-modal models.
Table 4: Comparison of AF3 w/ 10% data, w/o AF- AF-Whisper and wo AudioSkills XL.
Model
MMAU-Sound ACC
MMAU-Music ACC
MMAU-Speech ACC
Librispeech-clean WER J
Librispecc--thher
WER 1
w/ 10% data
66.7
65.9
57.4
2.0
4.1
AF- Whisper
63.7
68.3
45.2
3.7
7.2
AudioSkills-XL
56.1
42.1
14.3
3.6
Audio Flamingo
75.8
74.4
66.9
3.1
Conclusion, Limitations and Future Work
In this paper, we introduce Audio Flamingo 3, the most capable and open LALM. Our model leverages a custom Whisper, novel data curation techniques, and 5-stage curriculum learning strategy. Audio Flamingo 3 not only achieves SOTA performance in audio understanding and reasoning but also introduces capabilities, including multi-turn multi-audio chat, on-demand thinking, and voice chat, We detail our practices, including architecture, training, inference, and the evaluation pipeline, and open-soures two large datasets. For future work, we aim to address current limitations, including: (1) mitigating the need for a cascaded system for voice chat, (2) making AF3 multi-linguss and (3) reducing dependency on fouuee- models for synthetic data.
10

---

References
[1] A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai,
V. Chaudhary, C. Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.0174. 2025.
[2] S. Abu-EI-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. Vijaya- narasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.0800 2016.
[3] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, Roberts, M. Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint 1rXiv:2301.11333 2023.
[4] P. Anastassiou, J. Chen, J. Chen, Y. Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding. L. Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint 0tti:244060243300 2024.
[5] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and Weber. Common voice: A massively-multilinge speech corpus. In Proceedings of the 12th Conference Language Resources and Evaluation (LREC 2020), pages 4211 1-155,, 2020.
[6] J. Bai, H. Liu, M. Wang, D. Shi, W. Wang, M. D. Plumbley, W.S. Gan, and J. Chen. Audioset- caps: Enriched audio captioning dataset generation using large audio language models. In Audio Imagination: NeurlPS 2024 Workshop Al-Driven Speech, Music, and Sound Generation, 2024.
[7] P. Barros, N. Churamani, E. Lakomkin, H. Siqueira, A. Sutherland, and S. Wermter. The omg-emotion behavior dataset. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE, 2018.
[8] T. Bertin-Mahieux, D. P. Ellis, B. Whitman, and P. Lamere. The million song dataset. In Proceedings of the 12th International Conference Music Information Retrieval (ISMIR 2011), 2011.
[9] T. volume 2, page 10, 2011.
D. P. Ellis, B. Whitman, and P. Lamere. The million song dataset. In Ismir,
[10] R. M. Bittner, J. Salamon, M. Tiemey, M. Mauch, C. Cannam, and J. P. Bello. Medleydb: A multitrack dataset for annotation-intensive mir research. In Ismir, volume 14, pages 155-160, 2014.
[11] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359, 2008
[12] M. Cartwright, Cramer, A. E. M. Mendez, Y. Wang, H.-H. Wu, Lostanlen, M. Fuentes, G. Dove, C. Mydlarz, J. Salamon, et al. Sonyc-ust-v2: An urban sound tagging dataset with spatiotemporal context. arXiv preprint arXiv:2009.05788, 2020.
[13] C. Chen, P. Peng, A. Baid, Z. Xue, W.-N. Hsu, D. Harwath, and K. Grauman. Action2sound: Ambient-aware generation of action sounds from egocentric videos. In European Conference on Computer Vision, pages 277-295. Springer, 2024.
[14] G. Chen, Chai, G. Wang, Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2 2021.
[15] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEFE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.

---

[16] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei. Beats: Audio pre- -re-training with acoustic tokenizers, 2022.
[17] Y. Chen, F. Xue, D. Li, Q. Hu, L. Zhu, X. Li, Y. Fang, H. Tang. S. Yang, Z. Liu, al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
[18] Y. Chen, Yue, C. Zhang, X. Gao, Tan, and H. Li. Voicebench: Benchmarking Ilm-based voice assistants. arXiv preprint arXiv:2410.1711 2024.
[19] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Lv, He, J. Lin, C. Zhou, and
J. Zhou. Qwen2-audio technical report, 2024.
[20] Y. Chu, J. Xu, X. Zhou, Q. Yang, Zhang, Z. Yan, C. Zhou, and J. Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio audio-language models, 2023.
[21] J. S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. arXiv preprint arXiv: /006. 05622, 2018.
[22] C. Cieri, D. Miller, and Walker. The fisher corpus: resource for the next generations of speech-to-text. In IREC, volume 4, pages 69-71, 2004.
[23] A. Clifton, S. Reddy, Y. Yu, A. Pappu, R. Rezapour, H. Bonab, M. Eskevich, G. Jones, J. Karlgren, B. Carterette, and R. Jones. 100,000 podcasts: A spoken English document corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5903-5917, Barcelona, Spain (Online), Dec. 2020. International Committee on Computational Linguistics.
[24] F. Daniel, M. Matera, V. Zaccaria, and A. Orto. Toward truly personal chatbots: on the development custom conversational assistants. In Proceedings of the Ist international workshop on software engineering for cognitive services, pages 31 31-36, 2018.
[25] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson. Fma: A dataset for music analysis. arXiv preprint arXiv: 1602.018444 2016.
[26] Z.. Deng, Y. Ma, Y. Liu, R. Guo, G. Zhang, W. Chen, W. Huang, and E. Benetos. Musilingo: Bridging music and text with pre-trained language models for music captioning and query response. arXiv preprint arXiv:2309.08730 2023.
[27] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang. Pengi: audio language model for audio tasks, 2023.
[28] S. Deshmukh, B. Elizalde, and H. Wang. Audio retrieval with wavtext5k and clap training. arXiv preprint arXiv:2209. 14275, 2022.
[29] S. Deshmukh, S. Han, H. Bukhari, B. Elizalde, H. Gamper, R. Singh, and Raj. Audio entailment: Assessing deductive reasoning for audio understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23769-23777, 2025.
[30] S. Doh, K. Choi, Lee, and J. Nam. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307. 16372, 2023.
[31] M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou. The faiss library. arXiv preprint arXiv:2401.08288 2024.
[32] Drossos, S. Lipping, and T. Virtanen. Clotho: An audio captioning dataset. ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736-740. IEEE, 2020.
[33] Flizalde, Deshmukh, M. Al Ismail, and Wang. Clap: Learning audio concepts from natural language supervision, 2022.
[34] J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan. Neural audio synthesis of musical notes with wavenet autoencoders. In International conference on machine learning, pages 1068-1077. PMLR, 2017.
12

---

[35] J. Feng, Q. Sun, C. Xu, P. Zhao, Y. Yang, C. Tao, D. Zhao, and Q. Lin. Mmdialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:221 1,05719, 2022.
[36] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra. Fsd50k: an open dataset of human- labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 50:829-8522 2021.
[37] E. Fonseca, J. Pons, X. Favory, F. Font, D. Bogdanov, A. Ferraro, S. Oramas, A. Porter, and X. Serra. Freesound datasets: A platform for the creation of open audio datasets. In ISMIR, pages 486 493,-9932200 2017.
[38] P. Foster, S. Sigtia, S. Krstulovic, J. Barker, and M. D. Plumbley. Chime-home: A dataset for sound source recognition in domestic environment. In 2015 IEEE Workshop on Applications of Signal Processing Audio and Acoustics (WASPAA), pages 1-5, IEEE, 2015.
[39] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
Moore, M. Plakal,
[40] S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and Catanzaro. Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities, 2025.
[41] S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, Sakshi, O. Nieto, R. Duraiswami, and D. Manocha. Gama: A large oudio-language model with advanced audio understanding and complex reasoning abilities, 2024.
[42] S. Ghosh, A. Seth, S. Kumar, U. Tyagi, C. K. R. Evuru, S. Ramaneswaran, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha. Compa: Addressing the gap in compositional reasoning in audio-language models. In The Twelfth International Conference Learning Representations.
[43] J. Godfrey, E. C. Holliman, and J. McDaniel. Switchboard: Telephone speech corpus for research and development. In Acoustics, speech, and signal processing, ieee international conference on, volume pages 517-520. IEEE Computer Society, 1992.
[44] A. Goel, Z. Kong, R. Valle, and B. Catanzaro. Audio dialogues: Dialogues dataset for audio and music understanding, arXiv preprint 2404::2004.061111 2024.
[45] Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass. Joint audio and speech understanding, 2023.
[46] Gong, H. Luo, A. Liu, L. Karlinsky, and J. Glass. Listen, think, and understand, 2023.
[47] D. Guo, D. Yang, H. Zhang, Song, R. Zhang, R. Xu, Q. Zhu, S. Ma. P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
[48] A. Guzhov, F. Raue, J. Hees, and A. Dengel. Audioclip: Extending clip to image, text and audio, 2021.
[49] F. Hernandez, V. Nguyen, S. Ghannay, N. Tomashenko, and Y. Esteve. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings 20, pages 198-208. Springer, 2018.
[50] S. Hershey, D. P. Ellis. E. Fonseca. A. Jansen, C. Liu, R. C. Moore, and M. Plakal. The benefit of remporally-strong labels audio event classification. In ICASSP 202/-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 366- 370. IEEE, 2021.
[51] J. Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Li, S. Wang, I. Wang, W. Chen, et Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
13

---

[52] A. Huang, B. Wu, B. Wang, C. Yan, C. Hu. C. Feng, F. Tian, F. Shen, J. Li, M. Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502. 1946, 2025.
[53] Huang, M. Li, Yang. J. Shi, X. Chang, Z. Ye, Y. Wu, Z. Hong, J. Huang, J. Liu, Ren, 7. Zhao, and S. Watanabe. Audiogpt: Understanding and generating speech, music, sound, and talking head, 2023.
[54] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[55] M. M. Islam, N. Ho, X. Yang, T. Nagarajan, I.. Torresani, and G. Bertasius. Video recap: Recursive captioning of hour-long videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18 18198-18208 2024.
[56] J. James, L. Tian, and C. I. Watson. An open source emotional speech corpus for human robot interaction applications. In Interspeech, pages 2768-2772, 2018.
[57] I.-Y. Jeong and J. Park. Cochlscene; Acquisition of acoustic scene data using crowdsourcing. Signal and Information Processing Association Annual Summit and
In 2022 Conference (APSIPA ASC), pages 17-21. IEEE, 2022.
[58] X. Ju, Y. Gao, Z. Zhang, Z. Yuan, X. Wang, A. Zeng, Y. Xiong, Q. Xu, and Y. Shan. Miradata: A large-scale video dataset with long durations and structured captions. Advances in Neural Information Processing Systems, 04448955-49777 2024.
[59] W. Kang, X. Yang, Z. Yao, F. Kuang, Y. Yang, L. Guo, L. Lin, and D. Povey. Libriheavy: A 50,000 hours asr corpus with punctuation casing and context. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10991-10995. IEEE, 2024.
[60] C. D. Kim, B. Kim, H. Lee, and G. Kim. Audiocaps: Generating captions for audios in the wild. In NAACL-HLT, 2019.
[61] J. Kim, T. Moon, K. Lee, and J. Cho. Efficient generative modeling with residual vector quantization-basel tokens. arXiv preprint arXiv:2412. 10208, 2024.
[62] P. Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79-86, 2005.
[63] A. S. Koepke, A.-M. Oncescu, J. F. Henriques, Z. Akata, and S. Albanie. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Multimedia, 25:2675- 2685, 2022.
[64] Y. Koizumi, H. Zen, S. Karita, Ding, K. Yatabe, N. Morioka, M. Bacchiani, Y. Zhang, W. Han, and A. Bapna. Libritts-r: A restored multi-speaker text-to-speech corpus. INTER SPEECH 2023, 2023.
[65] Z. Kong, A. Goel, R. Badlani, W. Ping. R. Valle, and B. Catanzaro. Audio flamingo: A novel audio language model with fhw-shot learning and dialogue abilities, 2024.
[66] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar. High-fidelity audio compression with improved rvggan. In Thirty-seventl Conference on Neural Information Processing Systems.
[67] E. Law, K. West, M. Mandel, M. Bay, and J. Downie. Evaluation of algorithms using games: the case of music annotation. In Proceedings of the 11th International Society for Music Information Retrieval Conference (ISMIR). Utrecht, the Netherlands, 2010.
[68] C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping. Ny-embed: Improved techniques for training llss as generalist embedding models. arXiv preprint arXiv:2405.174288 2024.
14

---

[69] D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1152--11532, 2022.
[70] K. Lee, D. W. Kim, J. Kim, S. Chung, and J. Cho. DiTTo-TTS: Diffusion transformers for scalable text-to-speech without domain-specific factors. In The Thirteenth International Conference on Learning Representations, 2025.
[71] K. Lee, K. Park, and D. Kim. Dailytalk: Spoken dialogue dataset for conversational text- to-speech. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[72] S. Lengg Y. Xing, Z. Cheng, Y. Zhou, H. Zhang, X. Li, D. Zhao, S. Lu, C. Miao, and L., Bing. The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio. arXiv preprint arXiv:2410. 122.22777..77777 2024.
[73] G. Li, J. Liu, H. Dinkel, Y. Niu, J. Zhang, and J. Luan. Reinforcement learning outper- forms supervised sae-ttuing: A case study on audio question answering. arXiv preprint arXiv:2503. 11197, 2025.
[74] Li Y. Wei, Y. Tian, Xu, J.-R. Wen, and D. Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108-19111 18,0111110011 2022.
[75] T. Li. J. Liu, T. Zhang, Y. Fang, D. Pan, M. Wang, Z. Liang, Z. Li, M. Lin, G. Dong, al. Maichuan-audics A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502. 17239, 2025.
[76] S. Lipping, P. Sudarsanam, K. Drossos, and T. Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1 1140-1144. IEEE, 2022.
[77] S. Liu, J. Cho, M. Freedman, X. Ma, and J. May. Recap: retrieval-enhanced context-aware prefix encoder for personalized dialogue response generation. arXiv preprint arXiv:2306.07206, 2023.
[78] S. Liu, A. S. Hussain, C. Sun, and Y. Shan. Music understanding Ilama: Advancing text- to-music generation with question answering and captioning. In ICASSP 2024-2024 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 286- 290. IEEE, 2024.
[79] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-11986, 2022.
[80] Z. Ma, Z. Chen, Y. Wang, E. S. Chng, and X. Chen. Audio-cot: Exploring chain of of-thought reasoning large audio language model. arXiv preprint 2tt.::2001.00244.. 2025.
[81] Z. Ma, Y. Ma, Y. Zhu, C. Yang, Y.-W. Chao, R. Xu, W. Chen, Y. Chen, Z. Chen, J. Cong, K. Li, K. Li, S. Li, X. Li, X. Li, Z. Lian, Y. Liang, M. Liu, Z. Niu, T. Wang. Y. Wang. Y. Wang. Y. Wu, G. Yang. Yu, R. Yuan, Z. Zheng, Z. Zhou, H. Zhu, W. Xue, E. Benetos, K. Yu, E.-S. Chng, and X. Chen. Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix, 2025.
[82] L. Martinez-LLuas, M. Abdelwahab, and C. Busso. The mss-conversation corpus. Interspeech 2020, 2020.
[83] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang. Waveaps: A chatgpt-assisted weakly-lahelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2024.
[84] J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and Poria. Mustango: Toward controllable text-to-music generation. arXiv preprint arXiv:2311.08355. 2023.
15

---

[85] A. Mesaros, T. Heittola, and T. Virtanen. A multi-device dataset for urban acoustic scene classification. arXiv preprint arXiv:1807.0980 00w::08077.9940... 2018.
[86] J. Moon, Y. Kong, and K. H. Chon. Language-independant sleepy speech detection. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Bioology Society (EMBC), pages 1981-1984. IEEE, 2022.
[87] I. M. Morato and A. Mesaros. Diversity and bias in audio captioning datasets. In Detection and Classication of Acoustic Scenes and Events, pages 90-94, 2021.
[88] M. R. Morris, J. Sohl-Dickstein, N. Fiedel, T. Warkentin, A. Dafoe, A. Faust, C. Farabet, and S. Legg. Position: Levels of agi for operationalizing progress on the path to agi. Forty-first International Conference on Machine Leurning, 2024.
[89] A.M. Oncescu, A. Koepke, Henriques, Z. Akata, and Albanie. Audio retrieval with natural language queries. arXiv preprint ar 1rXv:2200.00119,, 2021.
[90] P. K. O'Neill, V. Lavrukhin, S. Majumdar,
V. Noroozi, Y. Zhang, O. Kuchaiev, J. Balam, Y. Dovzhenko, K. Freyberg, M. D. Shulman, et al. Spgispeech: 5,000 hours of transcribed finan- cial audio for fully formatted end-to-end speech recognition. arXiv preprint arXiv:2104.00 104004404..0014, 2021.
[91] Z. Ouyang, J.-C. Wang, D. Zhang, B. Chen, S. Li, and Q. Lin. Mqad: A large-scale question answering dataset for training music large language models. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[92] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[93] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205 2023.
[94] S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprins 2018.
[95] V. Pratap, Q Xu, A. Sriram, G. Synnaeve, and R. Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411 2020.
[96] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and Sutskever. Robust speech recognition via large-scale weak supervision, 2022.
[97] Z. Rafii, A. Liutkus, F.R. Stöter, S. I. Mimilakis, and R. Bittner. The musdb18 corpus for music separation. 2017.
[98] M. A. Rahman, Z. I. A. Hakim, N. H. Sarker, B. Paul, and S. A. Fattah. Sonies: Synthetic or not-identifying counterfeit songs, arXiv preprint arXiv:2408..4400 2024.
[99] M. M. Rashid, G. Li, and C. Du. Nonspeech7k dataset: Classification and analysis of human non-speech sound. IET Signal Processing. 17(6)ee12233 2023.
[100] A. Rousseau, P. Deléglise, and Y. Esteve. Ted-lium: an automatic speech recognition dedicated corpus. In LREC. pages 125-129, 2012.
[101] S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410. 19168, 2024.
[102] I. A. P. Santana, F. Pinhelli, J. Donini, L. Catharin, R. B. Mangolin, V. D. Feltrim, M. A. Domingues, et Music4all: new music database and its applications. 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), pages 399-404. IEEE, 2020.
16

---

[103] H. Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. In The Twelfth International Conference on Learning Repre- sentations.
[104] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang. Salmonn: Towards generic hearing abilities for large language models, 2023.
[105] G. Team, R. Anil. S. Borgeaud, J.-B, Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 18005, 2023.
[106] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13997
2023.
[107] C. Wang, M. Riviere, A. Lee. A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390, 2021.
[108] D. Wang, J. Wu, J. Li, D. Yang, X. Chen, T. Zhang, and H. Meng. Mmsu: A massive multi-task spoken language understanding and reasoning benchmark. arXiv preprint arXiv:2506.04779, 2025.
[109] Y. Wang, S. Wu, Y. Zhang, S. Yan, Z. Liu, J. Luo, and H. Fei. Multimodal chain-of-thoughe reasoning: A comprehensive survey. arXiv preprint 1xXiv:2503.12600 2025.
[110] Weck, Manco, Benetos, E. Quinton, G. Fazekas, and D. Bogdanov. Muchomu- sic: Evaluating music understanding in multimodal audio-language models. arXiv preprint arXiv:2408.04 0at.:24488.1133,, 2024.
[111] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
[112] H.-H. Wu, Seetharaman, K. Kumar, and P. Bello. Wav2clip: Learning robust audio representations from clip, 2021.
[113] Y. Wu, K. Chen, T. Zhang. Y. Hui, T. Berg-Kirkpatricke and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1-5. IEEE, 2023.
[114] Z. Xie, M. Lin, Z. Liu, P. Wu, S. Yan, and C. Miao. Audio-reasoner, Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02311, 2025.
[115] Z. Xie and C. Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408. 16725, 2024.
[116) G. Xu, P. Jin, L. Hao, Y. Song, L. Sun, and L. Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint 1.XXx:2411.10444 2024.
[117] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20225, 2025.
[118] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng. B. Yu, Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint 2XXw:2412.11111. 2024.
[119] Y. Yuan, D. Jia, X. Zhuang, Y. Chen, Z. Chen, Y. Wang, Y. Wang, X. Liu, X. Kang, M. D. Plumbley, al. Sound-vecaps: Improving audio generation with visually enhanced cap- tions. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[120] Y. Zang, S. O'Brinn, T. Berg- Kirkpatrick, J. McAuley, and Z. Novack. Are you really listening? boosting perceptual awareness in music-qa benchmarks. arXiv preprint arXiv:2504.00:: 2025.
17

---

[121] Z. Zhao, Y. Jiang, H. Liu, Y. Wang, and Y. Wang. Librisqa: Advancing free-form and open-ended spoken question answering with novel dataset and framework. arXiv preprint arXivv2308. 10390, 2023.
[122] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006 55021, 2023.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Detailed experiments and evaluation in Section 6 support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations our approach are discussed Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
There is no theoretical result in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give details about datasets, experimental setup and training hyperparame- ters in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: All the code, data and models are open sourced at https://github. com/ NVIDIA/ BANDIA/aadio-aa www.iagg/ tareg///reeeaeeeee aundoo_lamingo..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. ec/ Sttii///uiides//eeeubmniin
icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips http:pppllcgdcds/eeeeepee
http///ooeSeiissseeeeee hwt.SSbbbiiiiooooiiiiioooooo onPolicy)) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the settings in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
is expensive to run multiple experiments and report such metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all the details in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the impact of our work in Appendix K.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provide all the details in Appendix G.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details about all datasets, including the method for curation, are provided in the main paper. More detailed statistics and examples provided in the Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We provide details about this in Appendix E.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
Justification: Most of the human manual analysis was done by the authors of the paper, except chat evaluation, for which we obtained IRB approval from our institution. More details will be provided in the camera-ready.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
24

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: LLMs have been used only to help in writing and writing code for parts of experimentation.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to LLM policy ptctps://neurigee /00/ /LLM) for what should or should not be described.
25

---

Appendix
A AF-Whisper
A.1 Training Details
We train AF-Whisper on 512 NVIDIA A100 80GB GPÜs. During training, effective batch size of 1024, the AdamW optimizer (learning rate = 10-4, weight decay = I),, and train using fpl6 precision. We train for epochs on the complete dataset and sample instances randomly from the entire pool for each batch.
A.2 Training Datasets
Table 5 lists the datasets used to train AF-Whisper. For each dataset, we follow the same process outlined in Section 3 of the main paper: generating transcripts, spoken language characteristics, and audio captions. When available, we incorporate gold-standard metadata for these elements (for e.g., transcripts for LibriSpeech or captions for AudioCaps). GPT-4. is prompted to produce the final caption using a format similar to Fig. 50, with a modified exemplar. For extracting spoken language characteristics using AF2, use the following prompt: "There is human speaking in the audio. Describe in detail the characteristics of the spoken utterance, including pitch, emotion, mood, speed, and other speech dynamics."
Table 5: Statistics of audio-caption datasets used for AF-Whisper training.
Dataset
#Audio- Text Pairs
GigaSpeech (L) [14]
2,266.371
Sraech-in--oundee Captions" [2]
1,999,959
SPGISpeech [90]
1,966,109
Sound -BCdpp 9
1,657,029
Million Songs Dataset
1,169.997
Common Voice [5]
1,109,689
MiraData [58]
748,320
Action2sound [13]
306,602
NSynth [34]]
289,205
LibriSpeech [92]
281.241
Freesound [37]
256,695
AudioSet Strong [50]
216,622
VGGSound [15]
185,161
VoxPopuli (en) [107
177.019
FMA [25]
106,412
Video Recap [55]
64,627
CochlScene [7
60,855
Music4All [02]
109269
Switchboard [43
76,652
FSD50k [36]
40,966
MACS [87]
31,675
BBC²
31,201
MagnaTaggATTine [67
25,863
SoundDescs [63
23,085
Clotho [32]
19,195
TAU-Urban [85]
14,400
MusicCaps [
5,479
WavText5K [28]
4,347
SONICS [98]
1,602
SoundBible
935
MUSDBIS 18 [97]
276
Medleydb-Pitch [10]
103
Total
13,246,961
26

---

B AudioSkills-XL
Table 6 provides all details, including statistics and references prompts used for generating AudioSkills-XI.
Table 6: Detailed statistics of AudioSkills XL, categorized into individual reasoning types, together with details on sourco datasets, additional meta-data, and prompts used for QA generation. indicates that these types are further categorized into skills, and we elaborate on this in Section B.1. Rows not grayed out are the contributions of this paper. Speech QA types are the same Long XI. and explained in Section 4.2, with examples in Figure 3 and examples in Appendix C.
Question Type
Size
Datasets Used
Meto-Data Used
Prompt Reference
Tempural
188K
Table 14 in [00)
Table 14 [0
Table 14 in (10)
pythomic
Attribte Meentification
201K
Tiable
[4]
Table
Counting
50K
Table 14 in 40
Table 14
40
Table in 420
Contextual Sound Event Reasoning
982K
Table 14 in 100
Table 14 im
10
Table 14 in 000
Contextual Speech Event Ressoming
1,272K
Table 14 in M0
Table 14
40
Table 14 in 440
Information Extraction
858K
Table in
Table
Tablee in
Reasoning
Table
Table
ours (only sound)
300K
You RouTaheMM
caption
Fig.
Soand Rensoning" (ours)
300K
YorThWWM
caption
Fig. 20: 48, 47
Music Knowledge* (ours)
LO00K
MusccBench, MusiclAlL. MSD
captions, datasct-specilic data
Fig. 31,26
Music Reasoning" (ours)
LO00K
MusicBends, MusicAAll. MSD
captions, datasct-specilic metu-data
Fig.
Speach-in-Saund (ouns)
Speach-in-Sounl Caps (YouTabbetM))
Caption. Transeripes, Speech Chameteristies
Fig.
Spench QA" (aurs)
LibeiSpeach, GiguSpocch, VoxCeleh?
Transoripes
Fig. T5, T8,
B.1
Skill-Wise Breakdown
B.1.1
Music Reasoning
Focuses on the model's ability to infer musical genre or stylistic influences by analyzing instrumentation, arrangement, and production characteristics.
Genre and Style:
Mood and Expression: Focuses on how well the model interprets the emotional tone or affective content conveyed by the music, such melancholy, uplifting, aggressive moods.
Temporal Relations Between Elements: Focuses on the model's understanding of structural evolution within the music over time, including transitions in energy, tempo, instrumentation across different sections.
Focuses on the model to link the music with real-world settings or usage contexts (e.g., movie scenes, events), requiring understanding of appropriateness and intent.
Functional Context:
Lyrics: Focuses on interpretation of lyrical themes and content where applicable, often demanding a blend of semantic understanding and musical context awareness.
Historical and Cultural Context: Focuses on whether the model can connect musical elements to their broader cultural historical origins (e.g.. jazz fusion, protest music), relying external world knowledge.
Music Texture: Focuses on knowledge of the audio's timbral and sonic character by evaluating aspects such the layering of instruments, vocal texture, and overall audio quality. This skill captures how dense, sparse, smooth, or gritty piece sounds, requiring models to interpret descriptive attributes and production characteristics.
Melody: Focuses on understanding the primary musical contour or thematic tune in the audio. Melody-based QAs evaluate recognition of pitch movement, vocal/instrumental phrasing, and stylistic traits such as ornamentation or melodic structure, encouraging indirect inference over simple labeling.
Rhythm and Tempo: Focuses on the temporal structure of the music, including pulse, beat, speed, and time signature. These questions test whether the model can identify rhythmic complexity, tempo changes, and groove characteristics that define track's pacing drive.
Harmony and Chords:
Focuses on the models' ability to reason about harmonic progressions and chordal structures that shape the emotional and tonal qualities of the audio. This includes interpreting transitions, key relationships, and compositional patterns in harmony using indirect reasoning from musical cues.
27

---

GABrE and Sigle Rassslat
Sound Iestare
Questier en sort desere led deciiie some hastis audt Choose and Milwinng
Simuthwazi Juaii
(4 Aonudi palti teigs ans skrr flecie cytil cune, Pilns addiica
Araver: ( Aoxulic palai. banty wotn
Meed and Experodos
Instranuntaaioon
and sptes Tiieeng and dream, Aggna maree andrc
be tabweng phooue
flaluing
Ansuer Trangel
Bly and Vongo
Caplasniag
5994, Answar 1iMi, 6'
Deeiiie: (Weul semo time acniltrr daa the baok walot Choese hor the kilceing
A-te Paat"
Abus ham LB
part
Kep a arrine"
Doeeiien iieu um sou divor to rededy the song Choose and von falloiing chenwe (h Answar scatting
Tampo (TPM
Captery Aigheenngg elictosp sacd corout meled es castarty alamio ae winr with 1h0 kinges foat
cartfyy
Quastionn whanic the the dpeopincii cddl acadti ourgi chocse ton talkeing choisec
Clessic
ASSF Colur Hape
knower: Clesse pvgrassoo
Alaum hure Cokor Haur"
Comples Reoesing Q4
Tarpe 6TPM 19.31
Ousetion Vitu coundrasck pame. Criwe Documatton marbory
saladle Chooss
For Lstinn dl ereecialed lin
Genereter Captaon: Noo iemoniee paydhade mock back sroee rach cosrattrbbe parcipaccio pl hanaga les mediadio
Figure 4: Examples of Music Reasoning and Knowledge Questions from AudioSkills-XL. Additionally, we also illustrate examples of music captions generated audios Music4All by prompting GPT-4.1 with metadata obtained from the dataset.
General Complex Reasoning QA: Evaluates the model's ability to perform multi-dimensional inference on short music segments by combining musical knowledge, perceptual cues, and contextual understanding. These questions grounded in rich musical attributes, such dynamics, structure, genre fusion, narrative cues, emotional evolution, and historical style, and require the model to synthesize diverse information to arrive at the correct answer. This category tests higher-order music comprehension across expressive, structural, technical, and cultural dimensions, aiming to emulate how humans make sense of music beyond surface-level tagging.
B.1.2 Music Knowledge
Focuses on the model's s ability to recognize the instruments used in the music and how their timbre, arrangement, or presence contributes the overall sound and suitability for various contexts.
Instrumentation:
Performance:
Focuses understanding of the vocal instrumental delivery, including vocal tone, articulation, expression, or the presence of unique performance techniques.
Focuses on the density and layering of sound, such sparse vs. rich textures, acoustic vs. electronic timbres, and how these contribute to the sonic identity of the piece.
Sound Texture:
Metre and Rhythm: Focuses on the temporal structure of the piece, including rhythmic patterns, tempo consistency or variation, and the use of syncopation or groove, which are essential for identifying genre or compositional style.
Melody: Focuses how the model interprets the musical contour and phrasing of the primary tune, including vocal stylings, tonal range, and melodic progression.
Dynamics and Expression: Focuses on the model's sensitivity dynamic shifts (e.g., soft to loud passages), expressive techniques, and emotional delivery throughout the performance.
Harmony: Focuses on the model's ability to recognize chord progressions, harmonic structure, and tonal relationships, which contribute the music's 's emotional or stylistic impact.
B.1.3 Sound Reasoning
Speech-in-Soune QA: Focuses on reasoning over spoken content in addition to ambient sounds
or music to answer complex questions about the input audio, including scene interpretation, action reasoning, etc.
28

---

Eco-Acoustic Sounds QA: Focuses on the model's ability to interpret natural environmental con- ditions based on ambient audio cues. This includes reasoning over weather phenomena such as thunderstorms, snowfall, or rain using non-speech acoustic indicators like wind, water, or animal sounds.
Acoustic Scene Reasoning: Evaluates the model's capability to infer real-world environments from ambient and structural sound patterns. These include background music, reverberation, crowd noise, and electronic elements, enabling scene classification (c.g., arcade, mall, theater) from complex audio mixes.
Sound-Based Event Reasoning: Focuses on identifying and reasoning over specific audio features or events, such musical motifs, instrument timbres, recurring sonic patterns, to infer event types or characteristic actions.
Eco-Acoustlc Sounds QA
Question: Based on the audio, what weather condition likely occurrig? Choose one option from the choices: (A) Thunderstorm, (B) skies, (C) Light drizzle, (D) Heavy snow
Answer: (A) Thunderstorm
Acoustic Seene Reasoning
Question:
environment is suggested by electronic synthesized sounds?
Choose one option from the following choices: (A) A live concert, (B) A video game arcadie, (C) A movie theater, (D) shopping
Answer: (B) video game arcade
Sound Event Reasoning
Question: Based on the given audio, what defines the musical genre? Chosase one option from the following choices: Rhythmic bass and guitar melody, (B) High-pitched vocals and synthesizers, Fast drumming and heavy distortion, (D) Orchestral strings and wind instruments
Answer: Rhythmic bass gulter melody
Figure 5: Examples of Sound Reasoning QA, together with the metadata used for generating them.
Speech-nn-Sonnd QA
What can inferred about the speaker's foolings towards his location based on his tono and the surrounding ambient sounds?
Question:
Answer: speaker likely nogative lowards living Oktahoma, as indicated his disgusted tone and the flat deliverry his singing, accoppanee by monotooouu hum utillyy poles and rustlng leaves.
Speech-in-Soune Sound Caps
Metadats:
Sound Caption: A woman is having conversation, and then she continues talking while child joins in, pig
followed the
objocts being seen before. emotional. People really go through things. makes me feel so bad. told him worse.
the background.
Speech Transcript:
him
Spoken Language Characteristic: The speech English by a male speakkr with a disgusted mood. The
speech charactonsties include flat male singing voice at tempo of around 99.889 bpm.
distreesing while embient sounds include a woman corversingg a child joining in, objects being moved. and pig oirkkig.
Speech Aware
Caption: speaker, sounding diegusted,
Metadata:
Sound Caption: Sound of restaurant.
Spesch Transcript: I'm to oat lot of trookbokki
Spoken Language Characteristic: The speech lemale who speaks Mandarin, aged between 26-40 years a mood. speech content 951 Will ('That was eat").
Speech Aware Generated Caption: woman restaurant planning to oat lot of moakbolde).
Figure 6: Examples of Speech-in-Sound Caps and QA, together with the metadata used for generating them.
29

---

Arewon
Figure 7: Examples of general audio QA generated as part of AudioSkills, We generate this as we find models struggle say "No" while responding to questions.
C LongAudio-XL
Tables 9 and 10 present detailed skill-wise statistics for LongAudio-XLL including the source datasets and the minimum, maximum, and average durations of the audio samples.
Below, we also show some examples form LongAudio- XI in Fig 8
Surcaen Iduniification
Question: the ststement Friorides honey, prirlles mode corcasticall in the contest conversation?
Arrawerr You, the statement can intepritedd sled surcast comes afer neated discussion acout cheosing between family and canser, that
priorisdig
Qusetion: the sen socken comvenotion, the comment about better ther way" made by spesser conouste?
spasker sylnn
person by suggening
Tople Relationsip Reesoning
Question: what did act
demonstrate strateglo
Romons counter
Inxacing lorcal
Answer:: actions three men, par mauurattyyy tramtaus who staod thei ground apainsl the enamy at the bratge, alloeed Homman
eneryy
rg
and tad dkiterse the itmmn st atritage
Question: dieas the apsakar's dleserge reseripton of the acenery contrat with sundhine faced during the escent mountaant
desorioze the conmusia with the challengeaa asconccing mountain, such the koose frish snow and warninga about pertous nature the ceason dlimt.
fiss
Emnttio
Brasening
Question:
the foel
Answer: The spoaker fools loccuase despit having prest glous oducal from Harvard, tley are porooked us overquallited for oonain Jobo and sill laco rejection adding to thiir frustaation over thek inatiily to secure a sultable position
speaker tool trustrated when Anawver: The spenkerts meadrationn stams from the astrree the hour and poieiily a lack or intereet or erthhue sam the acticly they are engaged Thes atatement exdicates a destse leave, which conineda with the other parsori socilamend, creating tereaon and impatience.
Question: Why
ti's ofter
Information Extraction Response QA
Question: How one spoaker responc
they regres the
Question: What is the rosponse the request for transforingg the forn informtion?
everyone
trin.
Summarization
Qusstion: Summantee actioee taker the Homanns dekens against tneadingg torcs
letially enemy tridge
destrped One Roman sayed
enemy
Queetion: Summortie the key cha recteietiio and Mledtye of the peocle desorthed in the provided conversaion.
Araaver:: speaker describcu lambermen know indgeadee
and boe roamera stiloo, alart lkvin harmoriouslly with
Figure 8: Examples of LongAudio-XI.
D AF-Think
Table 7 provides all details, including statistics and prompts for generating AF-Think.
Table 7: Detailed statistics of AF AF-Think. Most speech QA examples in this benchmark involve reasoning about ambient sounds in addition spoken content. As our analysis shows, this added requirement increases task complexity, necessitating deeper inference answer questions accurately.
Modality Type
Size
Datasets Used
Meta-Data Used
Prompt Reference
Speech
100K
Speech in Sound QA, Long/uuioo XL.
transcripis, generated QAs captions, QAs, dataset--pecific meta-date data captions, QAs, datasel specific meta data
Fig. 38, 42
Sound only
50K
AudinSkills-Xi. (AudioSer-SI. Youthhe8M
Fig. ,1,
Music
100K
AudioSkills XL. (Music4All, MSD)
Fig. 39, =0"
Below, we also provide several examples from AF-Think in Fig. 9:
30

---

Went fncroroaeed ro wit soteoaaee Claasi cannininn lalloing
[R] Pip
yone
Duacc dicahic palar ri,, oui las duve (icluding what sound Bhe bdaat
Dsaed the emciii reiicty Spenia langage corna and the comail anr snmoupheee mek,
would
edgenc ne Company this
poonded "Mhh Cheak wooe mecd
Deard Coll one
content, what lin meal
age
St vviid tormanc tas and aaavarmnnt
201 desas diccatioo
Rosocd sntton sadlo priu ded, vntee connto ou land seioary dridig uniodyy.on leaia muttammentt lac TMA Gutar"
Th
yhout cur best plumiing
pugget mulh ai and petertee cievoumtie diyy of reficle or mecrunce zyseess which talt papporad apeciic aounds speech praant
Weate moal procbble contector infenction hwar?
calora
Calore
Alriaal stonie
dester Bunoo aove dew
SoJ schac leporinee pocupp
The weth conarre comernaiinal upavih with lough in eugoniiing informalty aoundi pat of tai dures This maindlee puéra ioomall auchang sccurting while as infernal end target da tamat.
Noar te roceddo uatay tho penvirunt and padke parae vouu style ond evart mnteenhheeedd move
The
atince letings mekin g Hniplyy
Figure 9; Examples of AF-Think, for music, speech and sounds.
E
AF-Chat
Table 8 provides all details, including statistics and prompts for generating AF-Chat.
Table 8: Detailed statistics of AF-Chat.
Modality Type
Size
Datasets Used
Meta-Data Used
Prompt Reference
Sound & Speech
35K YouTubeB8
captions, transcripts, speech characteristics
Fig. 35
Music
40K
Music4AlI, MSD
captions, datasct-specifiic meta-data
Fig. 36
Below, in Fig, 10 and 11 we also provide examples from AF-Chat, one for each sound and music.
E.1 Human Study for AF-Chat-test
The human verification process has been approved by our institution's Institutional Review Board (IRB). For the human study, we hire 4 Ph.D. students proficient in audio research, including music. For each instance in each test-set dialogue, the students were asked to rate the output of the model on a scale of 1-5 across Factuality (how correct the response is), Usefulness (how useful the response with respect to the context of the conversation), and Depth (how detailed the response is). For reference, we, the authors of the paper, provide responses scored 1-5 across the 3 aspects. The final score provided Table is an average of scores across all instances.
E.2 Clustering for constructing AF-Chat
To construct high-quality multi-turn, multi-audio dialogues for AF-Chat, we implement targeted clustering strategy that ensures each dialogue is grounded in a semantically diverse but coherent audio context. Rather than sampling audio clips random, which often leads to incoherent or loosely connected conversations, we curate each dialogue from a controlled pool of semantically related audio samples.
Specifically, for each seed audio, we retrieve its top most semantically similar and top 8 most dissimilar clips from the dataset. Similarity is computed using captions, NV-Embed-v2 embeddings of the captions, and FAISS-bass similarity search [31] of the embeddings.
For speech and environmental sounds, we use clips from Speech-in-Sound Caps. For music, we source from Music4All and the Million Song Dataset (MSD). Once the 16-candidate pool is formed (8 similar + 8 dissimilar), we restrict the dialogue construction process to this subset. GPT-4.1 is then
31

---

<sound-1> The man here talks calmly about being in other countries, with what sounds like a vehicle running nearby. Is his tone affected by the setting, or does he stay neutral throughout?
His tone stays quite neutral, even as the background engine hums along. The steady, even delivery suggests he's focused on sharing information rather than reacting to setting. The presence vehicle sound doesn't
distract heighten his emotion it adds context but lension.
So even with the background noise, would you say the message comes across as confident more reserved?
It's reserved. The neutral delivery. steady pacing, and lack of strong emotion his voice point toward composed, thoughtful sharing of experience rather than assertiveness enthuslasm.
<sound-2> In this restaurant clip, woman sounds apologetic and afraid. How does that emotional shift affect the almosphere compared to the earlier, calmer speech?
creates uneasy atrnosphere. The fearfulness tone paliabble and though the words are few, the emotion comes through strongly. Unlike the calm, neutral narration before. this moment feels much more personal and vulnerable, shifting focus from information to emation.
<sound-3>> Here, there's laughter and playful breathing as a woman and a child play with balloons. Does this kind of playful sound immediataly change the mood, if speech isn't clear?
Absolutely. Laughter and playful breaths are universally uplifting. Even without understanding the exact words, the sounds of enjoyment and lighthearted activity signal a positive, carefree environment very different from the reserved tone of the first audio or the tension in the restaurant
<sound-4> Now, this features female singer performing with musicians, and the crowd engaged. How does the emotional delivery of singing compare to spoken words in the other
samples?
Singing often amplifies emotion through melody, myyhhm, and dynamics. Here, the singer's intensity and the musical backing foster collective emotional response. making more immersive and expressive than plain speech. audience's 's enjoyment adds another layer, creating shared experience that's distinct from the solitary or conversational tones in the spoken clips.
you were to group these audios by kind feelings they evoke, would the calm travel discussion and the playful balloon scene belong together?
They both evoke positive or neutral feelings, but in different ways. The travel discussion feels composed informative, while balloon scene joylul and energetic. They contrast fear and apology the restaurant and the heightened emotion the musical performance, but each brings its own kind of calm or happiness.
Figure 10: Example of AF-Chat for sound and specch.
32

---

<music-0> This track has such rich, emotional feel. What production choices give that signature early 2000s indie vibe?
The emotional quality comes from blend factors: first, the guitar tones are slightly overdriven but not harsh, creating warmth and clarity. The drum patterns are tight, olten with dry snare, lending intimacy: Vocals are uptront dynamic, building and releasing lension match the lyrical introspection haalimarksns of early 2000s indie rock.
subtle reverb keep personal. The arrangement
<music-1> The mood here is also quite reflective, but the sonic palette feels different. How do the electronic elements affect the overall emotion of this piece compared to the first one?
The electronic textures in this track-llke gentle synth pads and programmed percussion contribute a smooth, polished atmosphere. This contrasts with the organic edge of the indie rock track. The electronics soften the emotion, making feel nostalgic and dreamy, whereas the more traditional rock instrumentation in the previous track delivers the emotion in rawer, more immediate way.
wanted
rawness first track the electronic atmosphere the second, what should focus terms arrangement?
Focus layering: keep the live drums and guitar foundation from the first track to maintain rawness, but introduce subtle synth pads or electronic percussion from the second for atmosphere. arrangement
organic band sound, then add electronic textures in the chorus or bridge to build emotional depth and contrast.
element shine perhaps verse
<music-2> This one feels huge and immersive, almost overwhelming. What's making the sound so dense stmospheric?
The density comes from the layered, heavily processed guitars- lots of reverb, delay, and modulation effects create a wall sound. The vocals are often blended into another instrument, and drums techniques envelop the listener, making the sound immersive and emotionally intense.
almost driving yet washed-out quality. These shoegaze
Could take those lush guitar textures and them backdrop for the intimate vocals from the first track?
Absolutely. Pairing lush, reverberant guitars with upfront, intimate vocals can create a powerful contrest. careful with mix space in the midrange so the above the dense textures. This approach amplify the emotional impact while maintaining clarity.
clear
Figure 11: Example AF-Chat for music.
prompted to construct multi-turn conversations (up to 10 turns) using any combination of these audio clips. This ensures:
1. Topical consistency across turns using similar clips.
2. Diversity and contrast through the inclusion of dissimilar audio.
3. Clear referential structure, as questions may depend on or refer back to earlier audio.
Our clustering strategy was informed by a preliminary human study (participant details similar to Section E.1), where participants engaged in multi-audio, multi-turn conversations with an LALM, focused on tasks such as sound design and music information retrieval. We observed that participants naturally gravitated toward using either highly similar or strongly contrasting audio clips within a dialogue. This behavioral insight motivated our of similar and dissimilar audio clustering.
Empirically, this approach produced dialogues that were more natural, coherent, and diverse compared to those built from randomly selected audio pools. Moreover, AF3-Chat, when trained on this clustered dataset, outperformed the variant trained on randomly selected audio clips, both in terms of response relevance and conversational depth.
33

---

AF-Chat Clustering
Original Audio (Caption): A young male speaker, sounding fearful, repeatedly accuses someone of being a monster destroylng the world,
conversation between couple people.
young male voice filled with fear dominates an energetic and intense conversation among a large group, as he questions and demands action regarding 'monster' and intervention.
Positive Audio (Caption):
Positive Audio (Caption): A male speaker with serious and disgusted tone accuses someone of being a monster, amidst tense almosshere..
Negative Audio (Caption): woman instructs to use hands to dip something in water, while mixing ingredients bowl silently.
Negative (Caption): parade features a celebration with crowd marching performing music, while vehicle moves through the formation.
Figure 12: Examples of audio clusters obtained after clustering (Section E.1), used constructing AF-Chat.
F
Prompts
We provide all prompting templates used across our datasets and QA types in Figures 15, 16, 17, 18. 22,
19, 20, 21,
23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45.
46, 47, 48, 49, and 50.
G AF3 Training Datasets
Table 11 summarizes all datasets used train AF3, including total hours, number of audio-QA pairs, and the number of epochs (passes over the dataset) used at each training stage. Similar to [40], we convert all foundational datasets (captioning. classification, etc.) into QA formats, using the same of prompts for each task mentioned in [40].
H AF3 Training Details
In this section, we present the training settings of our models across all 5 stages, each with specific configurations. Details are in Table 12.
I Streaming TTS System Architecture and Training Details
To enable voice output capabilities within our system, we incorporate a text-to-speech (TTS) module that operates on subword text tokens. For efficient and simplified streaming speech synthesis, our TTS module employs decoder-only architecture.
As illustrated in Fig. 13, the TTS module predicts the subsequent audio token conditioned on incoming subword text tokens from the main AF3 model and the history of previously generated audio tokens. These audio tokens then decoded into voice output by the neural audio codec. This design simplifies the speech generation pipeline and minimizes latency, which are critical for real-time -imee speech streaming.
L1 Neural Audio Codec
We utilize a fully causal convolutional neural audio codec for efficient streaming audio decoding, following [61, 103].
Encoder. Input audio is first resampled to 44.1 kHz. It is then converted into Short-Time Fourier Transform (STFT) parameters using a hop size of 8 and window size of 32. This STFT representation processed by an initial 1x1 convolutional layer to produce 384-dimensional hidden embeddings. Following this, the signal undergoes three downsampling stages. Each stage consists of three causal
34

---

Voice Output
Coder
ThAt
good!
Audio Takens
S
Output
End Prid.
KIn Vuentier
Seetch-Sporch Medulk
SIP
MLPP
MLP
MLP
SEP
Trchedling
Takon Exenwting
Transformer Docoder
MI.P
Block
That
Audio Tokens
Kaaked KVQ Tolona
Input
Tokens
Tranforta Oulpal
(a) Streaming TTS system architecture.
(b) Iterative unmasking of RVQ audio tokens.
Figure 13: Streaming TTS is enabled by autoregressive audio token generation coupled with neural audio codec decoder. (a) The streaming TTS system predicts audio tokens conditioned on incoming subword text tokens (e.g.. from the main AF3 model) and the history of previously generated audio tokens; these audio tokens are then decoded into voice output the neural audio codec. (b) The iterative audio token unmasking process relies on MLP block. This block takes partially masked RVQ tokens and transformer decoder output as input, predicts a cumulative embedding vector, which is subsequently quantized into progressively more unmasked RVQ tokens.
ID-ConVNeXt blocks [79, 103] followed by a strided convolutional layer for downsampling. These strided convolutional layers use stride and kemnel size of 8. Each such layer doubles the hidden dimension, except for the final one, which produces a 111-dimmnsiooal output. The encoded output sequence is 4096 times shorter than the raw waveform, corresponding to approximately 10.8 frames per second.
Quantization. The encoded output is quantized into audio tokens using Residual Vector Quantization (RVQ) [69, 66). The number of RVQ levels is set to 72.
Decoder. The decoder mirrors the encoder's 's architecture symmetrically, employing ID transposed convolutional layers for upsampling and causal ID-ConvNeXt blocks. The final convolutional layer reconstructs the STFT parameters, which are then transformed back into raw audio waveform via an inverse STFT (iSTFT) similar to Vocos [103].
Training. The codec is trained using a combination of adversarial training and a mel-spectrogram reconstruction loss, following methodologies from DAC [66].
1.2 Text-to-Speech (TTS) Module
Architecture and Operation. The TTS module's transformer decoder processes a sequence formed by concatenating subword text tokens (from the main AF3 model) and previously generated audio tokens. The resulting hidden states from the transformer serve as conditional input to multi-layer perceptron (MLP) block. This MLP block then iteratively predicts progressively higher levels of the RVQ tokens, technique inspired by [61]. In practice, we employ iteration steps during inference. A key aspect contributing to the system's 's simplicity and low latency is that the model is designed to generate an audio token whenever text token is emitted by the AF3 model, without requiring explicit alignments between text and speech.
Training and Configuration. During training, the transformer decoder utilizes teacher-forcing with ground-truth audio tokens. The MIP block is trained to estimate the parameters of a mixture-of- gaussians distribution where the number of mixtures is 1024. The objective to maximize the ligeliill of predicting the cumulative RVQ token embedding, following [61]. The decoder-only
35

---

transformer has configuration similar to DiT-XL 193, 70]. The MLP block consists of layers, totaling 644 M parameters.
1.3 Training Data and Processing
The models are trained on a comprehensive dataset comprising LibriTTS-R [64], LibriHeavy [59], the English portion of Multilingual LibriSpeech [95], and proprietary datasets, totaling approximately 100,000 hours of audio. To support long-form speech generation, audio segments from the same speaker are randomly concatenated to create training samples with durations ranging from 120 seconds.
J Qualitative Examples
Prompt: Generate a detailed caption for the input audio, describing all notable speech, sound, and musical events comprehensively. In the caption, transcribe all spoken content by all speakers in the audio precisely.
AF3 Response: A tense, cinematic soundscape unfolds over roughly three minutes. opens with woman's surprised question-"Arr you being serious right now?" followed by man's 's quick reply, "Ycah." A door creaks, camera shutter clicks, and soft whoosh ushers in female interviewer who states, "You'd d let me interview you as Superman. Sure. Ready? Let's do it, Cronkite. Superman. Miss Lane. Recently, you's ve electronic beep and swelling orchestral bed that will underscore the rest of the piece. Over the music she continues, "It's lot. Today the Secretary of Defense said he was going look into your actions. That's funny. My actions? stopped war. Maybe. Not maybe. did. effect, you illegally entered a country. This how you't re going to be? I'm not the one being interviewed, Superman. Did you consult with the president? No. You seemingly acting as representative of the United States, doing good. I would question myself in the same situation and consider word is drowned by sudden explosion and a deep rumble that echoes like a sonic boom. A male voice, urgent and breathless, shouts, "Hey, buddy. Eyes here." A brief pause, then softer male voice reflects, "Your choices. Your actions. That's what makes who you Superman. Hc's not man. Hc's it. Hc's somehow become the focal point of the entire world's conversation. I will not accept that. cleaned your boots. I'll go get them for you." A door creaks open, slams shut. The music swells again, single word pordd""ou""-ai spoken, and triumphant chord that fades silence.
under a lot of fire for what some might-" Her sentence is swallowed by low
consequences. People are going to die! Superman"" Her last
bruee
asks, "You have dog?" and the door
resolves into final,
Demonstration of AF3's capabilities an unseen audio elip-extracted 2woCZg50dvE)- capppuue in the wild. The model accurately identifies and describes background sounds, spoken content, speaker turns. and transcriptions. demonstrating strong audio understanding. Beyond this example, AF3 supports significantly more complex reasoning tasks. We invite readers to explore these capabilities via public demo: https:/ https://huggingaaee ace. cttcco/.paa.o coo/ppace//vviiii ieciess//viiiiaa a/ wwwaaaddolllee -fl.oofolmmm..3....
Figure 14:
audio captioning task. We prompt AF3 with youtube. com/vatchPv
extracted from the Superman 2025 trailer
36

---

You are helpful Al sseistant. You need act questior-ansser generator for long speech. will only provide you with the transcript of single-spesker speach lacture and you need to generate complex reasoning based quastioa-nnswe pairs iron the conversation. The speech might diverse nature, which might be story from an audio book, a parlianentary speech, or some etaer diverse kind. spacitically, the question should require ideas coaveyed
listener or the model speech relate to each other in context the entire content, My finel objective is to train an audic agent with these question-anaver pairs with leng-speech understanding and abilities. Senerate the Question-Mnexes pairs with the following conditions:
underetand how
1. The quastion should require reasoning
answered correctly.
2. Kention everything abstractly.
3. Bo not rame the speakers even if nane le provided.
4. Noat inportantly, underatand the aature of the speech and gererate the QA pair sccordingly.
Below is ar examp_e speken speech. and types of wasziions/aasver below:
me preaidart the comnission thanks the rapporteux mea regner and the parliament issue at juriadiction rules in employment matters. we share the ain of ensuring strone protection
raising the
the rights of employees general and in particular when they are involved amployment dispetes, let me recall that the rights at emplcyees have alrescy been specitical-y addressed and streagthered thousand and twaive for instance the new brussels regulation provides for the rigat for employees
the
brussels requlation adopted
december. two
several employers togethes and the courts in europe even if the empleyer is domiciled outside europe, the commission will careft' ly nonitor the application of the new rules over the coming years in the process paying close attentior. to the employment mattera raised by pariiamant. this can then be considered in the contex: of the zeview provised for in article seventy nice of the recast regulation. in this perspective the commission sould employment centracts shich has already been cantiored.thia could be done by looking more specifieally into amployee against the employer detining as relevant the place of business from which the employee receives which engaged the employee is situated.
for smployees
have access
tuture consider looxing into the issus
auggested fallback clause article twenty
ir
brought
received day day instructions rasher than the place where the business
Example Question: According to the input speech, how does the Bruseels I Regulation adopzed in 2012 relate to the protection of enployees' rights in engloyment diaputes?
Example Answers The Brussels Regulation atrengthens employeas' rights by allowing them to aue multip_e employers togsthar
access courts in Burope ever. it their employer is comiciled outaide Europe, thos enhancing their protection in employment diaputes. Generate two such questions,
you think good quality cannot be made do not generate a question and only recurn "None" for both question and answer, Return a J8oN the following format: ("Quastion queation, "Anawer 1"; "Answer 2": Answer)
the conversation,
"Question Quastion,
Only return the JSON and nothing else.
Heree the input transccipt of the converestion:
Figure 15: Prompt used for generating Topic Relationship QA for LongAudioXL.
37

---

Table 9: Detailed -kie and dataset-wise statistics of LongAudio-Xe
QA Type
Dataset
#lnstances
Min Dur((s)
Max Dur(())
Avg. Dur.(s)
Order
VoxPopuli [107]
16,926
1.87
294.80
89.55
LibriSpeech [92]
2,340
16.02
147.59
82.01
MELD [94]
4.135
1.06
108.01
30.42
IEMOCAP [11]
599
82.00
542.00
272.45
EuroParl [62]
11,885
2.59
176.14
69.34
Fisher [22]
25,962
33.34
240.00
136.84
Switchboard [43]
2,702
22.81
148.96
87.38
MultiDialog [85]
27,927
1.31
499.33
135.10
VoxCeleb2 [21]
12,855
8.00
1273.60
71.12
Emotion Ident.
IEMOCAP [11]
300
82.00
542.00
272.22
MELD [94]
1,847
1.78
108.01
33.20
Emotion Causal Reason.
IEMOCAP [11]
300
82.00
542.00
272.22
MELD [94]
1,850
1.57
108.01
33.13
Emotion Flip Reason.
IEMOCAP [
299
82.00
542.00
272.62
MELD [94]
1,807
1.64
108.01
33.57
Topic Relation. Reason.
VoxPopuli [107]
13,651
3.58
240.44
97.14
LibriSpeech [92]
1,165
16.02
147.59
82.11
MELD [94]
1,518
1.89
108.01
34.45
IEMOCAP [11]
188
82.00
542.00
270.99
EuroParl [62]
9,381
7.97
176.14
70.14
Fisher [22]
20,453
33.34
240.00
136.10
Switchboard [43]
998
24.58
148.96
90.05
MultiDialog [85]
14,906
5.11
499.33
135.35
DailyTTakk [71]
3,141
8.05
103.66
35.32
VoxCeleb2 [21
5,414
8.51
1193.60
78.96
Sarcasm Ident.
IEMOCAP [11
299
82.00
542.00
271.58
MELD [94]
1,958
1.10
108.01
31.82
Summarization
VoxPopuli [107]
13,913
2.12
294.80
91.38
LibriSpeech [92]
1,057
16.02
147.59
83.15
MELD [94]
2,803
1.84
108.01
32.92
IEMOCAP [11]
300
82.00
542.00
272.22
EuroParl [62]
8,905
6.62
176.14
70.03
Fisher [22]
15,500
0.33
240.00
135.60
Switchboard [43]
1,346
24.58
148.96
87.60
MultiDialog [85]
20,838
1.93
499.33
135.73
DailyTalk [71]
7,218
8.05
103.66
31.42
VoxCeleb2 [21]
5,894
7.94
1193.60
70.87
Spotify Podcasts [23]
103920
0.06
18206.44
2002.99
Needle QA (IE)
DailyTalk [71]
13,563
5.72
103.66
31.12
EuroParl [62]
18,426
6.57
176.14
70.10
Fisher [22]
37,779
18.59
240.00
135.99
IEMOCAP [11]
542
82.00
542.00
272.03
LibriSpeech [92]
2,248
16.02
147.59
82.82
Spotify Podcasts [23]
103920
0.06
18206.44
2002.99
Response QA (IE)
VoxPopuli [107]
13,913
2.12
294.80
91.38
MELD [94]
1,660
1.57
108.01
31.83
IEMOCAP [11]
177
82.00
542.00
272.52
MultiDialog [85]
13,505
1.95
499.33
135.40
DallyTalk [71]
4.516
5.72
103.66
30.91
Switchboard [43]
862
22.81
148.96
88.75
38

---

Table 10: Detailed skill-wise and dataset wise statisties LongAudio-
QA Type
Dataset
#Instances
Min Dur. (s)
Max Dur:(s)
Avg. Dur((s)
Causal QA (IE)
VoxPopuli [107]
12,264
4.10
240.44
92.88
LibriSpeech [92]
1,166
16.02
147.59
82.04
MELD [94]
2,957
1.27
108.01
31.74
IEMOCAP [11]
298
82.00
542.00
273.10
EuroParl [62]
7,457
7.97
176.14
70.24
Fisher [22]
19,335
37.17
240.00
135.87
Switchboard [43]
1,352
22.81
148.96
87.40
MultiDialog [85]
20,811
3.17
499.33
135.62
DailyTalk [71]
7,368
8.05
103.66
31.15
VoxCeleb2 [1
6,171
8.06
1193.60
71.08
You are helpful AI assistant. You need to act se sesttion-snwee generator for long speech. will only previde you the transcript ot single-speaker speech to generate complex reasoning kased question-answer pairs iron the converration. the speech might be at diverse nature, which might se story tron. an audio book, paritamentary speech, some other diverse kind. specifically, to understand the contents of the speech and snewer about specific detail in it, By finel audio agent speech understarding and Q8 sbilities. Generate the question-Answer paics with the following condit'
and you
question should require listener or
model
objective
question-ansxer pairs to endow
long-
1. The quastion ahould zequire reasoning to be anavered correctly.
2. Keation everything abstractly.
3. Do not rame the speakers even if a name it provided.
4, The question should ask about particular detail in the conversation.
5. Noat iaportantly, underatand
nature of speech and gererate the @A pair sccordingly. Balow is examp_e spoken speech and types of questions/aaawers below:
mi president the comnission thenks the capporteus issue of jurisdiction rules in employment natters. we share the ain of ensuring strong protection
regner and the parliament
rights employees general employment dieputes, let me recall that the rights of employees have alresdy been specificalls addreased strengthered thousand and twaive for instance tae new brussels regulation provides for the right for employees
in particular when they are
regulation adopted
decenber,
several employers togethes courts in europe even if the employer is domiciled ousside europe, the cemmission will careft' ly noni tor the appl close attention. to application fcation of the new rules over the coming years in the process paying employment mattera raised by pariismant. this the contex: of the review provided for in article seventy nice of the recast zequlation. in perepective
the right for smployees
have access
be considered
commission could employnent contracts which has already been contiored,this could be done by looking more specifically
censider looking into
issue
suggested fallback employen oyee agsinst the employer defining as relevant the place of business from which the employee receives which engaged the employee situsted.
artiels twenty
brought
received day day instructions
than the place
Exarple Question: Tn the input apeech, on what date did the apesker that the Brusaels Regulation was adopted?
Exarglle Answer: The speaker mentioned that the Rrussela Regulation adopted on December 2012.
questions, you think good quality cannot do noz generaze question and only resern "Hone" for beth question and answer. Betura a decN the follewing fomat: ("Question 1"; Question, "Answer Anewes, "Question Question, "Ansuer 2"; Anguer] only return the JSON and nothing else.
conversation,
Hare sa the ingut transorigt of the conversation:
Figure 16: Prompt used for generating Needle QA (Information Extraction type) for LongAudioXL.
39

---

Table 11: List of fine pre- treiirinn and fine- tin-ttning datasets together with their training composition.
Dataset
Hours
Num. Pairs
St. 1
St. 2
St. 3
St. 3.5
St. 4
AudioSkills (Uurs)
9700K
2.0
2.0
LongAudioXL (Ours)
1000K
1.0
1.0
1.0
1.0
AF-Think (Ours)
250K
1.0
1.0
1.0
2.0
AF-Chat (Ours)
75K
1.0
CompA-R [42]
159
350k
2.0
2.0
MusicBench [84]
115.5 hrs
686k
1.0
1.0
Mu-LLAAMA [78]
62.9 hrs
70k
1.0
2.0
2.0
Salmonn AQA [104]
800 hrs
270k
1.0
1.0
ClothoAQA [76]
7.4 hrs
9.7K
8.0
8.0
OpenAQA [46]
693.2 hrs
1959.8K
1.0
1.0
Clotho-v2 [32]
24.0 hrs
19.2K
1.0
2.0
2.0
MACS [87]
10.9 hrs
17.3K
1.0
1.0
FSD50k [36]
80.8 hrs
41.0K
1.0
1.0
1.0
CochlScene [57]
169.0 hrs
60.9K
1.0
1.0
NonSpeech 7k [99
6.2 hrs
6.3K
4.0
4.0
Chime-home [38
5.0 hrs
4.5K
1.0
1.0
Sonyc-UST [12]
34.9 hrs
27.9K
1.0
1.0
Emov-DB [86]
7.8 hrs
6.8K
1.0
1.0
JL-Corpus [.6]]
1.4 hrs
2.4K
6.0
6.0
Tess
1.6 hrs
2.8K
2.0
2.0
OMGEmotion 77]
3.0 hrs
1.7K
3.0
3.0
MusicAVQA auudio-only
77
77.1 hrs
5.7K
6.0
6.0
MusicQA [91]
62.9 hrs
70K
1.0
1.0
LP-MusicCapsMan
30
5805.7 hrs
1331.8K
1.0
1.0
1.0
LP-Music ..-MusscCaapsMT
30
126.4 hrs
46.9K
1.0
1.0
1.0
CPMMusicCapsMc [30]
7.4 hrs
7.9K
1.0
2.0
2.0
MusicCaps [3]
7.4 hrs
2.6K
1.0
6.0
6.0
NSynth [34]
321.3 hrs
289.2K
8.0
8.0
MusDB-HQ [97
29.1 hrs
10.2K
2.0
2.0
FMA [25]
860.7 hrs
104.2K
1.0
1.0
Laion630kgscSommderr [113]
456.9 hrs
15.1K
1.0
1.0
Laion630k [113]
2494.8 hrs
306.5K
1.0
1.0
SoundDescs [63]
749.7 hrs
23.1K
1.0
1.0
WavCaps [83]
3793.3 hrs
402.6
1.0
1.0
AudioSet [39]
2617.8 8 hrs
950.8K
1.0
1.0
Wav [28]
23.8 hrs
4.3K
1.0
1.0
MSP-Podcast [82]
73.9 hrs
45.1K
1.0
1.0
1.0
MELD [94]
8.7 hrs
32.9K
1.0
1.0
1.0
vuudiovviiual [74]
142.4 hrs
17.9K
1.0
6.0
6.0
Music4All Captions (ours)
910.5 hrs
55.6K
1.0
1.0
MSD Captions (ours)
15449.9 hrs
55.6K
1.0
1.0
Speech-in-Sound Captions (ours)
6227.6 hrs
1999959
1.0
1.0
LibriSpeech [92]
960 hrs
281.2K
1.0
1.0
1.0
Switchboard [43]
109.9 hrs
76.6K
1.0
1.0
1.0
GigaSpeech (L) [1]]
2499.8 8 hrs
2266.3K
1.0
1.0
1.0
Common Voice 15 [5]
1752.1 hrs
1109.6K
1.0
1.0
1.0
VoxPopuli (en) [107]
501.8 8 hrs
177K
1.0
1.0
1.0
TEDLIUM (en) [49]
472.3 hrs
68K
1.0
1.0
1.0
SPGISpeech [90]
4999.8 8 hrs
1966. IK
1.0
1.0
1.0
VoiceAssistant400K [115]
684 hrs
470K
1.0
40

---

Settings
Stagel
Stage2
Stage3
Stage3.5
Stage4
per device batch size
64
16
4
2
learning rate
le-3
2e-5
2e-5
5e-5
5e-5
learning schedule
Cosine decay
warm up ratio
0.03
weight decay
0.0
epoch
2
2
bf16
V
V
grad accumulate
2
4
4
8
DeepSpeed stage
Zero3
GPUs
128xA100
Table 12: Training settings across stages.
You are helpful sssistant. need act as queatlor-answer generator for long speech. I will only provide with the transcript ot single-speaser speech lacture and you need generate complex reasoning baaed queation-snswer pairs from the conversation. the speech night be of diverse nature, which night be a story from an audio book, a parliamentas speech, diverse kind. Spellfically, to understand the contents of the speech and understars the main topic of the speech, My tinal obfective is train audio
question should require 1fatener
rodel
these question-anrzer pairs endov with speech understanding and QA sbilities. Generate the Question-Anewee paics with the following conditions:
1. the question should require reasoning to be answered correctly.
2. Nention everything abstractly,
3. Do not rame the speakers even nane is provided.
4, Nost isportant"y, understand the nature of the speech and gererate the 08 pair accordingly.
Below ar example speken speech
types questions/snswers below:
me president the comnission thanks the capporteur mes regner and the parliament for rsising the juriadietion rules eypentment protection for the rights of amployees in genaral and in particular wsen they are involved in employment disputes. let retall that rights of employees have alresdy besn specifically adcressed and streagthared in the recent brussels : regulation adopted or six dacenber. two thousand twelve for employees to sue reveral employers together and the right for employees to have access to courta in europe even if the empleyer is coniciled outside scrope. the commisatin wi"l earefully noniter close attention for to the employment matters raised by parliament. this can then be considered in the contex:
ain ensuring strong
the brussels regulation provides for
right for
application
rules over coming years
process paying
provided for this pe perspective the comiission could in the future consider looking into the issue of employment
seventy nine
recast requlation.
by looking speciflcally into the suggested tallback clause in articis twenty one in cases brought by the employee agsinst the employer defining employee receives or recsived eay to day instructions racter than the place where the business which engaçed the employee situated.
already
nentiored,this
relevant the place of business from which the
Example Question: what the nain topic the input speech? Example Answer: The speaker talks about enhancing employes rights in employment disputes, focusing jurisdiction rules
the Brussele regulation
potential Suture
improvenents.
you think good quality cannot made do not generate a quest and only recurn "None" for both question and answer, Return a J80N follewing fornat: ["Question 1": Only return. the JSON and nothing else.
one question.
conversation,
1": Arswer]
Here is the input transcsipt of the conversation:
Figure 17: Prompt used generating Topic QA (Information Extraction type) for LongAudio
41

---

You are helpful AI assistant. You need to act as questior-snswer generator for long speech. will only provide you with the transcript st single-speakex speach lacture and you need to generate complex reasoning based question-anzwet pairs iron the conversation. The speech diverse nature, shich might be story feem. an or some other diverse kind, specitically, the question shoulc require a listener or the model understand the contents of the spech//ecture/steee of topics being talked about in the epeech, My finel objective is to trsin an audio agent with these question-answer pairs to endow with Long-speech underetanding and sbilities. Generate the Qpestion-Answer paice with the following conditione:
night
book, a parliementary speech,
and answer questions regerding the order
1. The question ahould require reasoning be anavered correctly.
2. Neation everything abstractly.
3. Do not name the speakers even if name is provided.
Balow ar examp_e conversation and types of questions/answers balcw:
preaident the comnission thanks the rapporteux issue of jurisdiction rules in employment matters. we share the ain of ensuring strong protection for the rights of employees in general and in particular when they involved employment disputes, let me recall that the rights of emplayees have airezdy been specifically addressed and strengthered in the thousand and twelve for instance the new brussels regulation provides for the right for employees to sue several enployara together and the right for employees to have acceaa to courts in europe even it the empicyer is comiciled ouzside europe. the ccmmission wi-1 carefully nonitor the spplication of the nev rules over the coming years in the process paying close attentior to empleyment matters raised by parlisment, this can
regner and the parliament raising the
brussels regulation adopted
decenber. two
be considered the context of the review pravided for in article seventy nine of the recast regulation, in perspectiva
commission sould in employment centracts which has alzeady been mentiored, could be dons by looking more specifleslly into the sugpested fsllback clause in article twenty employee against the employer defining as relevent the place of business from which the employee receives which engaged the employee situated.
tuture consider looking into the issue
it
brought the
received day day instructions rasher than the place where the business
Queation (Ordar typel: What is the correct order of topics discussed in the input apeech?
optioas 10raer type! (1) The Comnission thanks the rapporteur and Parliament., (B) The nev Brusseis I requlation's provisions for employee rights are introduced, (C) The possibility of future considerations regarding employment contracts nentioned.. (0), Connission plans monitor the application of new rules.)
Answer (Order type) the iaput speech according to their sequence of occurrence" and then you provide the options and the
(DJ This questicn be also phrased "Arrange topics
Question (Tengore". Refercing type): what topic is introduced early in the epeech? Optiona (Temporal Referring typel: [(A]hhe new Bruasels regulatton's proviatona for employee rights, 1B) ruture considecations on engloynent contracts, (C) The impoctance of protecting employee rights, Answer [Temporal Referring type): (C) The importance of protecsing employee rights
Menitoring the "egulatiora]
Queation (Temporal Grounding type): When in the speech does the speaker discusa the possibility of revieving employment contracts?
Options (Tenporal Grounding sype): (A) the begin end)
(B) the midsle, Answer [Pemporal Grounding type): (C) The end
Question latteibure sype): How does the tone the speech shift over time? Answer (Atcribute sypel:lion (() It becones nore speculative as potential future actions are diacuased., (9) remainn fornsl informative shroughout.. (C) shifta to an opt inist'' tone as the new rights are discussed.].
Answer [Attribute type): (A) discussed.
speculative potential future sations
Return a JSON in the following format: ("Cemporal Referring Question"; Quession, "Temporal Reierring Op:icas":
Options, "Tengoral Referring Answer": Answes, "Temporal Grounding Question"; Queetien, "Temporal Grounding Options"; Options, "fenporel Grounding Answer": Answer, "Crder Quession": questica, "Order Options": opsions, "Order Answer": Answer, "Attribute question": Question, "Astribute Options": Options, "Attribute Answer"; Answer). only return the USON and rothing elae. If a question type is not possible, don't output and only output "Sone". Bere is the input information:
Figure 18: Prompt used for generating Order QA for LongAudioXL.
42

---

You are helptui assistant. read act queatior-answer generator tor long speech. I will only provide you with the transcript of single-speaker speech lecture and you need generata complex reasoning based question-answer pairs from the conversation. The speech might be of diverse nature, which might be a story from. an audio book, parliementary speech, er some other diverse kind, Specifically,
question should require listener or model to urderstand the contents of the speech and infer the actual reason behind a particular statement utterancs made by the speekez, model to look further than the surface and literal maaning of the speech, ideas, or sentences find the correct answer pairs to endow it with leng-speech understanding and OA abilities. Generate the Queation-Arswer pairs with the following conditiona:
questien should involve the listenez
the
My final objective is train audio agent with these question-
1. quastica should zequire zeasoning be anavered correctly.
2. Reation everything abstractly.
2. Do not rame tae speakers even it nane is provided.
4. Rost importantiy, understand the nature of the speech and generate the @A pair accordingly.
example spoken speech mu president the commissicn thanks the zapportsuz mas regner and the parliament for raising the issue of jurisdietion rules in employment natters. protection for the rights of employees in genaral and in particular when they are involved in employment dieputes, let recall that the rights of employees have alresdy been specifically addressed
types questions/nssers below:
the ain of ensuring strong
the recent brussels regulation adopted or aix decenber, two thousand and txalve for instance the new brussela regulation provides for the right for employees to sus several employers togethar courta in europe even it the empleyer is comiciled outaide acrope. the commistion will carefally nonitor the application the close attention. to the employment matters raised by parlisment. this can then be considered in provised for articie seventy nice the this perspective the commitrion could in the future censider looking into the issue of employment contracts which has already been mentioned, specifieally into the suggested fallback clause in articls twenty one in cases brought by the employes against the employer defining employee recetves received day to day tnstruct! ons rasher than the place where the busineas which engaged the employee is situated.
strengthered
right for amployees
have access
rules over coming years in process paying
requiation.
could be done by looking
relevant the place of business from which the
Example Quastion: According to the input speech, what might the Connission consider the tuture regarding employment contracts based on the talk? Example Answer: According to
given input spesca, the commission might consider addzessing the issue of employmen: contracts by examining the fallback clause in Article 21, which could inveive defining the relevant pLace of business for day-to-day instructions, sataer than the employer's prinary business location.
Generate two such questions. do not generate a question and only resuen "None" for both question and answer, Return a JooN in the following format: ("Question I": Question, "Anawer 1"; Answet, "Question 2*: Question, "Ansver 2"; Anaver]
you think
quality cannot be made the conversation,
Only return. tae JS0N and nothing eise.
Hare is the ingut transcript ot the conversation:
Figure 19: Prompt used for generating Causal QA for LongAudioXL.
43

---

You are helpful AI sssistant. You need act queatior-answer generator for long speech. will only provide you the transcript single-speaker speach lacture you need to generate complex reasoning bazed question-antwet pairs iron the convereation. The speech diverse nature,
night
might etory an audio or some other diverse xind, Speclficalll: the question should ask for a summary of the entire speech/lettur/esoorf
perliementary speech,
pert the conversation ion and srawer the queation by providing a with leng-apeach underatanding and QA abi_ities. Generate the questicn-Inswer paizs with the following conditions:
model question-answer paira to summry, My final objective is to train
conterts
ausic
1. The quastion ahould require reasoning be anavered correctly.
2. Nontion everything abstractly.
3. To not rame the speakers even it nane is provided. sumaary should max 25-30 words.
5. The cummary can be of the entice conversation or part of the speech. If for part of the important part.
speech, qnestion
1. The sumnary should contain the key points, decisions made, main action itens, main tapics discussed. additionally, you ney generate question Exarple Cuestions: Summarize the input speken conversation provided OR Prom the inpat apoken converaation,
any
these.
participants spoke
for the key pointa Por given trput spoken
atter-eftects
given input spoken conversation, sumariie converaation, sumariiz the action i:ems it any. For the part conversation, mention the particular topte an abatract fashion.
It you think good quality cannot be nade tron the conversation, do not generate a question and only zeturn "Xone" koth question psirs, although you can generate one of two caanot be made, Return JSON in the following (*Quest
answer. Generate maximun two such question-anweer
Question,
"Question Queation, "Ansuer
Answer; Only
JSON nothing
Heree the input transcript of the converestion:
Figure 20: Prompt used for generating Summarization QA (Summary QA) for LongAudioXI.
44

---

You are helpful AI assistant. You need to act questior-snswer generator for long speech. only provide you
transcript you to generate complex questions that require nodel to understand the enotiona" state of each speaker in the conversat ion from an input speech to srewer
question should require Histner or the model to understand the contents of the conversation and identify the cause for objective
question correctly. specifically,
=lip in speaker'a enotional question-antmet pairs to erdox it with lone-speach understansing and QA abilitiss. will provide you with conversation between multiple speakers in which each speaker is denoted with separate spesker ID, such as 'Speaker 1', 'Speaker provide you information about the emational atate, suct aa happy, sad, fruatrated, esc., is separately sencted inside square bracket I. (''aapy"],, I'sad"l, etc.) tor the utterances spoken by each speaker. Generate the Questicn-Anewer with the fellowing conditions: question
ausio
with
or by name. additionslly,
will
require reasoning 2. Nention everything abstractly.
correctly.
if Belou ta ar examp'e e conversation and types of questions oos//navs/s below. Example Corversation:
speakers
is provided.
"Rache"s guesa just done the whole dating thing. (sadness)
Rachel: Its one more thing in ny 1ite thate suddanly completely different. (sadneas) Rachels This hard. (sadneas)
Rosa: Seah I know. [neutral]
other hand un, in about seven months yeu're gonna semething that you're gouna Love more then any you've ever gone out with. (neutral)
Ross:
Ineutral;
Ross: Heit until ub, wait un:il the first time your baby grabs your finger. (jay)
Ross: You
idea. (Joy)
Rachel; Thanks sweetie. (neutrel) grab Rachels oh no, think gonna go home and eat ten candy bars. (sadneas]
Ross: You
coffee? (neutral)
Rosa: Bey, I thought cheered you (neutral)
Rachel: oh you did, shere are twenty here. (neutral)
Rosa: Right. Good night. (neutral)
Rachel: night. incutrali"
input conversation, her dating life, feel okay as the conversation progresses?
Exarple Question:
speaker, initia'ly due
Example Answer: future baby she is gonna have which cheers her up. You shou'd strongly take into consideration the prise conversation when generating the queation. you think the reason cannet inferred from the conversation and there is auch fon that and answer. Generate two such queations. you
oshes speaker
talking consoles speaker
about her
nade, do not generate question and only return "None" for both question
there no auch question that can made, do not generate question and just retura None for bosh question ane anaver. Return Question, "Anewer Answer, "Questicn 2":
folloving format: ("Question
Question, "Answer 2"; Answer:
JSON and nothing else.
the input transcsipt of the conversation:
Figure 21: Prompt used for generating Emotion Flip QA (Emotional State Reasoning type) for Long
45

---

You are helpful AI assistant. You need to act questior-antwer generator for Long speech. you
will only provide you require nodel to understand the enotiona" state of each speaker in the conversation from en question should require listener or the model to understand why the speaker teels a particuiar smotion at particular with these questicn-snewer pairs
transcript
complex questions that
input speech
question correctly. Specitically,
My final objective endow it with long-epeech understanding and GA abilities. 1 will provide you with a converaation between meltiple speakera in which each speaker is denoted with separate speaker 10, such as 'Speaker or 'Speaker 2', Additienally, 1 will provide you infomatioo about the emot fonal state, such as happy, sad, truatrated, etc., la ssparately denoted trside aquare bracket I["tapoy'l, ["aas"], etc.) for utterances speken by each speaker. Senerate the Quest fon-Arswer w'th the fol' owing conditions:
train audio
1. question ahould
reasoning 2. fon everything abstractly.
correctly.
Do
speakera
is provided.
Below is ar examp_e conversation and types of questions and answers below. Example Corversation:
"Speaker 1: don't shink can this anymore [frustrated]
Speaker 2: Well guese you aren't trying hard enough [neutral]
Speaker 1: It's been three year. have tried everyshing [frussrated]
Speaker 2: Maybe
are not smart enough to [aeutral] really good
Speaker
snert enough.
[anges]"
Exarpie Question:
ingut convexsation, why
angry saying,
smart enough"?
conversing with tols the speaker that they aren't smart enough. The speaker was already frustrated with their situation sbout nultiple failures.
Anaver: The speaker
angry
You shou'd strongly take into consideration the prige conversation when generating the generste question
such question that be made, both question and answer. Generate suza guesticns. Retura a JacN in the follewing format: ("Question 1": Question, "Anzwer 1": Answez, "Question 2*: Question, Only reture the J80N and nothing else.
If think there and just return
Hare ia the input transorigt of the conversation:
Figure 22: Prompt used for generating Causal Reasoning (Emotional State Reasoning type) for LongAudioXL.
46

---

You are helptul AI assistant. You need to act questior-answer generator for Long speech. will only provide you the tranacript and you need to generate complex queation-answer psirs from the conversation, Specifically, the question should be auch that it should require identify
model underatand the conterts of conversatior converaction ia sarcastic. Additionally, through world knowledge, the modal saould alse se ab_e final cbjective
made in the
why the with these quertion-antwer pairs to endow it wish leng-speect: understanding and 0A sbilities. multiple speakers which each speaker is denoted
made is sarcastric.
train sudio
will provids you with conversazion with a separate spesker ID, such ss 'Speaker 1' or 'Spesker 2', Additionslly, I will provide you information about the enotional state, such as hapgy, sad, frustrated, etc., is separate'y cancted insida square bracket [['happy"I, ['sa6"), etc.) for the utterances spoken by each speaker. Senerate the Quest Question-Answer with the following conditions: questicn
require reasoning 2, Fention everything abstractly,
answered correctly.
if Balcu is ar. examp_e conversation and types of questions/answer. belcw. Example Conversation:
Do
speakers
is provided.
"Speaker 1: don't think can do this anymore [trustrated]
Speaker 2: Well quess you aren't trying hard enough [neutral]
Speaker 1: It's been three years. have tried everything [frustrsted]
Speaker 2: Maybe
are sna-t enough to [neutral]
[anger]* enozions should serve as extra ques to 'dentiff ffy sarceen based on the context of the conversation
Speaker
snart enough. I reslly good
Exarple Question: In the giver spoken conversation, is the comment 'Naybe you are not seart enough to' made by speaker sarcaatic?
Example Answer: Yes, the connent is sarcastic from the tone of the speaker.
"why?" ansvered from any you may ask "why" the question and frane ascordingly. You should strongly take inte consideration the pricr conversation when generating longer conversations you many
the question. the conversation around particular idea being conveyed. If you think the reason cennos inferred from the conversation and there is 20 such question that can be made, do not generate question.Generst Generate two such questions. conversation and there is no such question that can be made, do not generate question and question
point the exact utterance and just frane
you
resson cennot
the
only recurn "None" ("Oueatton 1"; Cuestion, "Anaver 1"; Anawer, "Queation on 2"; Cueat on, "Anawer 2"; Answer] Only reture J808 nothing
snexes, Retorn
following
the input transccipt of the converestion:
Figure 23: Prompt used for generating Sarcasm Identification QA for LongAudioXL.
47

---

You are helptul AI assistant. You need to act questior-answer generator for Long speech. transcript
only previde you require model to understand the encticna- state of each speaker in the conversaticn from input speech
you need to generate complex questiona that
queation correctly. Specifically, the question should requi listener to idertity the amstioral ataze of a speaker during an utterance. My tinal objective train audio understending and OA
endow with long-speech which sbilities. will provide you with a conversation between multiple 'Speakes
with these guestion-ansver
speakes 'Speaker 2', ndditienally, will provide you information about the enotional state, such as bracket (("hapey"),
denoted
happy, sad, frustrated, etc., is separately denoted inaide utterancea spokan by each speaker. Generate the Quastion-Anawe: the tollowing conditions:
['sad",, stc.)
quastion ahould require reasoning 2, Nention everything abstractly,
anavored correctly.
speakera
is provided.
Balcu is ar. examp_e conversation and types of questions/answer. belcw. Example Conversation:
"Speaker 1: don't think can do this anymore [trustrated]
Speaker 2: Well quess you aren't trying hard enough [neutral]
Speaker 1: It's been three year. have tried everything [frus:rated]
Speaker 2: Maybe
are sna-t enough to [neutral] anart enougn.
Speaker 1:
really good at what [anger]"
Example question expreasing that they had tried everything over nultiple years but failed?
input conversation,
speaker's emotional
Exarple
the speaker seems be frustrated.
Example question 2: In the given spoken conversation, what was the speaker's enotional state during saying
really good what do"?
Example ie Answer 1: The speaker seems to be angry while saying this.
Generate two such questions. 15 you think there is no such question that can be made, not generate queatior and just reture Nore for both question and ansuer. For long conversationa, transcripte and brozder and abscract versien of what is being talked about by the speaker. Return a J508 in ("Question question,
aveid having questions
exect
question
following "Ansuer 2"; Anguer]
Answer, "Question 2"; Question,
380N nothing
the input transccipt of the converestion:
Figure 24; Prompt used for generating Identification QA (Emotional State Reasoning type) LongAudioXL.
48

---

You are helpful assistant. I will only provide you with the transcript of nulti-spearer conversation and you need generate complex reasoring based questica-anawez pairs from the coaversation. Specifisally, the question should require listerer or the model to underetard what or how speaker responded another speaker train an audio agent with these question-anaver paixs to endow it with long-speech underatending and OA sbilities.
act question-answer generator for long speech.
quest'
My
objective
proviée you with speakers in which each specker is denoted with a separate spesker ID, such as "Speaker 1', 'Speaker 2 or
multip'e
Generste the Duestion-knawer pairs with the following conditions: 1. The question ahould require reasoning to be anavored correctly.
2. Keation everything acstractiy.
3. Do not rame the speakers even nane i8 provided.
4. The quastion should be about particciar and important (or signiticant) utterance in conversacion.
5, Neither the question, not the response should be more than 25-30 words. examp'
Relow
types of quest ons/anawera below: Speaker 1: (f [noise] hello this is thonda })
Speaker 2: hi i'm jerzy
Speaker 1: [noise". hi what changes have you made since reptember eleventh (noise) [noize]
Speaker 2: um
not aure i've made any personal changes my 1ife un go to college new york and actually
Speaker 2:
Speaker 1: {( okay ))
Speaker 2: 1 saw it all aappen so i nean it attected me lot un just as to
Speaker 2:
Speaker 2: (( like 11
Speaker 2: daily things like shat subway line could couldn't and it had very like personal impact on me but as far as like daily zoutines and Like things in lite don't think L've artually changed anything because it
Speaker 1: (( [noise] [noise" oh okay 11
Speaker 2: what about you
Speaker 1: [noise;
live in new jersey you know so kinda close um [1ipsnack]
Speaker 2: {( ch yeah ))
Speaker 1: you know
york well new york is about two half hours away trom where Speaker 1: (( you know [acise] but un
Speaker
haven't really
any peraonal changes either [noise]
Speaker 1: since then
Speaker 1: when i'n at the airport and stuff have noticed that you know they have more security [noise] you know so make sure that un follow the rules and stuÉÊ regarding
Speaker 1:
know
Speaker 1: um flying
Speaker 2: [( yeah yeah
Speaker 1: [( you know ))
Speaker 2: yeah and also when i'n un home from col college 1. 1ive in waahington state and go more secure taan used to be
canada pratty otter. and the borsers
Speaker 1: (( un (nof se] })
atrects like have always bring my pasaport 1st of the tine they give me a big hassle just because they give evezycne a hassle that goes through 'cause they don't
Speaker that's
changed
any terroriste Example Questicn: in the given conversation, what is the response of the speaker when the other speaker aske if they travel lot by train?
Exarple Answers The speaker reeponds saying they take the train a lot and specifloally the subway.
conversacion, not generate a question and only return "None" for both questien and ansuer, Return 390M in the rollowing format: ("Question Question,
you think good quality cannot made
1"; Anawer, "Question Question, "Anawer Anewer) Only reture the J80M and nothing else.
Here is the input transcript of the conversation:
Figure 25: Prompt used for generating ResponseQA (Information Extraction type) for LongAudioXI.
49

---

You are a helpful AL assistant. You need to act a question-answer generator for 10-second auáío sarples and generate complex reesoning- answering, Complex reasening-based gas refer to types of questions that would ask the model to understand
paira audio question-
input audio, contextual sound understending, which refers to underetanding all sounds and reason abouz sone sounda to
question.
example
particular question. agent with these question-arawer pairs to endow with audio-understande To generate tais guestion-snaver pair, will provide you with three pieces of iatornation: 1) An audiocapa style audio ospcion.
final objective
train audio-understanding CA abilitiea.
2) A transcripted generated using Whisper.
3) transcription generated using Whisper
for the questien. Bere 1) Do not previde too much information in the question. 21 Keep both the question and ansver complex, multi-hop thinking the question audie/sount sound specific without requiring too much understanding of the speech content. You
Generaze
crisp. 3) questions should
4)
the tranacription to understand the audio better but rot to make queations completely based it. Transcript: each other secause I'm not going be around to keep out of trouble. Audio-anly Caption: adult female speaking while driving, accorpanted by thene muate and music in the background, as she navigates tarougt ariving and walking, with occasional glitches and distortious.
are some examples:
Speecn Charscseristica: The speech characteriatics include a tlat pitch and a speaking rate of 90.36
Question: might speaker's Options: A) She is focused on navigating hez surroundings
rensin flat despite the dynamic background?
8) She is trying not to wake soneone nearby
() music makes
hear her enocions
D) She is zeading from prepared script
Answer:
rocused on navigating her surroundings
that's what's going with the movies. I'm. Dave Karger. I'll see you next week. Be sure to comioad the Fandango app for the lstest novies, showtimes, and
Transoript:
in film.
really
Audio-only Caption: Two man converse in foreign language over clectronic music with distinct eymžal bests
Speech characteristies, The speaker English-epeuking, male with an age between 16-25 years old, conveying a neutral mood in his speach.
Queation: Why there music and people talking in another language behind the speaker? options: A) To maxe sound he's at an event
R) no cover up recording mtatakes
с)
sound atronger
p) To show he is feeling emozional AnswersA) nake sound like he's event.
Output JSON the following format: ("Cuestion"; oot The answer).
The question, "Options"; the
Output
and nothing
input information:
Figure 26: Prompt used for generating Speech-in-Sound QA for AudioSkills-Xi
50

---

You helpful aI assistant. You need esptions in HusieCaps style. My final objective to train an audio understanding agent with these captions to endow it with music unserstanding abi-ities. To generate this opation, I wi.l provide you
caption
for and
provide you as follows:
neta-data
inpu: music. meta
title of the song, relesse of the song, arcist name for the song, year the song was launched, optionally
tags deseribing provide you with a list of captions generated by a misic captioning model. The list has maste captions generated noisy, Your caption should just refer to this but should be of better quality and aleo use world knovledge from the meta-datz. *Flease provide understand the masie, inclusing world knowledge about the music from the ari:st information other mata-data, pricrtize it capticn verufy integrity',
song |may may provide). Adsitionally,
will
music captiones for every 30 secends segments of the masic and may
world knowledgs with all the song, tempe, stc. It you
Generate caption using following
1) Keep it within 1-3 sentences.
2) Keep it within 20-50 worda.
3) Similar to musiccaps, mention the details about all instruments, fsel, nood, anbience, and all other festures and characteristive of the music, including
best, applicatien, etc. Only provide desalls you contident about. It not compuisory to provide
dataiis,
but do no: hallucinate. An example caption is "This ia an Rab/soul muate plece. There is a male vocalist singing noiodically string sample. There is strong bass sound in the piece. The zbythm is beat,
sessual nanner over maitipie tracka. The maiody the
beat is provided groovy electronic used in the soundtracks of a TV series where two charactera are sharing intimste nonents". this caption, Uo
provided
to piece. it
shows style case.
song
esption any
Only return the capt fon and nothing else. Return the capt fon J80N with single key "cagtion", Bare are the ingut detaila:
Prompt used for generating captions for Million Songs Dataset. Noisy captions for the prompt generated using AF2.
Figure 27:
51

---

You aze a halptul AI aasiatant. You need act as question-anave: generatoz for songs and generate know knouledge-bae rdgeebesed ion-tionn pairs. for knowledge
it requires the model to understend the and its internal knoxledge anever the question. By tinal objective to trein audio understanding agent with thase questiaa-anavee paics to endow abilitics. To generate this questisn-onswer pats, will provide you with scveral neta-data about the inpu: muilc. The seta daia will provide you is follows:
nusic undarstanding and gA
title of the cong, release of the seng, actist name for the song, year the cong was launched, and optionally some tags describing the seng (may may not providel, fdditionally, I will provide you Hist has
list captions grnessted by by music captioner for evesy aeconda segnents world knovledge with all the reta-data I provide to understand the music, including name of the song, tetpo,
saptiening nodel. the muii and may be noiay *Flease use your
captions
you have verld inexcedee about the maric the artist information other meta- dats, priorstize over coption or verity integrity.
Generale Lsing the following ruless
11 Generate poics with criep question anewer, not have too much information the question. 2) Generate NDD based quest The choices should be contrastive that nay rise confusionn to the model.
questions should have level of dirficulty.
For both reascaing and knowledge, you are supposed to generate Dhs using a pre-detined set or skills. will now provide you with the nere of skil's and exaup OA pair for each seill; Krowledge skills; Instrunentation:
Example quaation: What type of scardtrack would this place be most suitable foz?
Example Options: (A) Puturistic novie car racing video game 18) Crine theiller movie (C) (D) RonanLic comedy
Docunenlasy
Example Answers [B] Crime thriller novie
Knowledge skillas pezfornance
Pxample Questionn fee would you describe the voice used in the song?
Example Options: [A] Distoried fetale singing voice (B) Cleas nale singing valce (C) Peatures electronic music elenente (D) The lyries are about the power of tesendentp
Example AnSMEE: (M] Faturistic novic or car racing video gane
Knowledge sxilla: sound texcura
Example Questionn Which sype soundbrack would chis piece
suitable For?
Example options: (K) Faturistic novic or car racing video gene (B) Crime thriller movie (CI
tocunentary narration (D) Romantic conady novie
Example
Knowledge Skills: netre and rhythe
Example Quastion: What notable eleneat is presect the chorue?
Example Options:
Backezound hamonies suac in Enclish (B) Use quitars and bess (CI Children's voices doubling the main melody (D) spoken word poctry interludes
Example Ansvec: (D) Spoken word postzy interludes
Knewiedge Skills: melody
Example quaation: What is the vocal atyle used in this piece?
Example options:
Whispered noiodi: (B) Bolting and singing Gregorian chant Portorned by a Sunale vocaliat
Exemple Arsvers [R] Belting and eest singing Reasoning Skills: dynamics and exprescion
Example quastion: Which zyps of soundtrack would this piece be most puitable for?
Example Optionas (A) Futuriatic novie or par racing video gaze 1B) Colow (heillor novie (C)
bocunentary nerration (DI Ronantic conedy movie
Example Answer: A
Knowledge Skills: harneny
Examlle Queations Question: Which type of soundcrack sould this piece be noss suitable for? Optiones (A) Futuristic movie or car racing video game 1BI ceime thriller nevie (C) Documentary nazration (DI Romantic conedy novic The correct Example Ansver: (B) Crime thrilier novic.
*It is not necessary generate all krouledge and ressoning based questions, generate 1-3 GAs for unique skills
unique conbination skills) tron esct. input for the skills you teel is the fit and can generste the best quality of questions, If nothing is return Nore*, Please generate Cutput a JECN of the folloving fornst: ("quastion Type": Rossonaeg Type of Questicn 1, "Question
queation mininun and opnions from the original, *Aneve Type": Answer for the question 1, "Question "ype"; Reasoning ng Type of question 2, "Qaestion 2": gusstion 2, "Options 2"; List at optiors ter question 2, minimun 4 and contrestive options iron the original, "Answer Type": Answer for the question 21.It queetion is possible, just return Nono, don't retera wreng QA.
Queation
"Cptions 21"; List optiona
Cotgut
JS0N of the sini lar fornat and nothing else
Heze is the input intormation:
Figure 28: Prompt used for generating Music Knowledge QA from Million Songs Dataset for AudioSkil Noisy captions for the prompt are generated using AF2.
52

---

You are helptut AI assistant. You need to act a question-ansxer genezator for songa and generate seascning-basee Question-Anewer paize. Reasoning-based that would ask the model to understand an input song (music and vocals), reason about it, and answer question, My rinal objective question-snawes psirs to endow it with maiic understanding and DA sbilities. To generate this quest fon-srewer pair, wi1l provide you with Esta I will provide you is as follexs:
refer types questions
train audio understanding agent
these
the input musfc, The
title of the song, relesse of the song, ercist nane the song, year the song waa launched, not provida) Aacitionally, I wi.l a list captions generated by music captioning model. The 11st
and opticnally aone tags descriping the song imay provide you captions generated by a music captionar for every 20 secenda segments of the masic and may be naisy "Flease use your world knowledge music, including name of the song, cengo, etc. if you have world knowledge about the music fron the infortation other meta-dats, prfortine
muate
xll meta-data provide to understand the
the caption verufy integrity.
Generate using the folloxing rules: 1) Generate QA psirs with criep question and answer, Do not have teo much informezion in the quest fon, 2) Generate MCD based quest! one. The ceo should be contrastive that rise contuaion the nodel.
3) The questions shoulé have Tevel of di difficulty for the model..
You are supposed to gererate gās using pre-detined set of skills. will now provide you with the name skills
example QA pair each skill: Bessoning Skills: gence style Exarple Questions Which of the following would be an approprieze setting for this music track? It festures diverse syntherizers znding credita of a video game or sei-fi movie ID) & motivational ritness workou: features diverse synthesizers
Example Options: (A) romantic conedy montage
Example Answer:
Reason' ng Skills: nood and expression
Exampie Question: Which the tollowing would be appropriate setting this muaic track? Examrle Options: (A) A ronentic conedy montage (3) It features diverse syntheaizers (C) Rading credita of video game or sci-fi movie (D) motivational ritness workous
Example Answer: (B) It features diverse synthasizers
Reasoning Skills: temporal reiatiens between elements
Exarple Question: How kould you desoribe energy level
Example Options: (M) I: features sudden tango change (B) Gradually evolving from creany to driving (0) liihh energy throughout (D) Instrunental electronic piece
Exarple Anewers (B) Instrumental electronic piece
Reasoning Skilis: funstional context
Exarrle Queation: Which of the following would be appropriace setting for this muatc track?
Exampie Options: (A) ronantic comedy montage (3) teatures diverse synthesizers (C) Eading credita of a video game or act-fi movie (n) A mot fostistion onall fitnees workout romantic comady montage
Exarple Answer:
Reasoning
Lyrics
Example Question: What is the main thene of the song's ss lyrics?
Example Options: (8) Pop-rock (B) Politicsl commentary (C) Heppiness Love
Exarple Anewers (A) Pop-rock
Reasoning Skills: historical and cultural context
Exarple Question: What unique combination of muaical stylea is evident in this song? country music (2) The song played by a farosa bard (D) Classical blues chorda with big band style Exampie Answer:
Exarple Options:
The song has fast tenpo
and country music
You may also generate QA pairs with multiple skills requized.
It is not necessacy to generate all knoweldge and reasoning based questicns, generate 1-3 gis for unique skills
unique combination of skills) from each ingut for the skills you feel is the beat fit and can generate the best quality of questions. ganerate Output JSON of the tollowing rormat: ('Cueatior. 1 Type": Reasoning Type Queation 1, "Queat 1*s Question 1, "Answer 1": Ansuer for the question 1, "Options 1*; 14at options for question i, minimum 4 and contrastive options frem the criginal, "Question 2 Type": Reasoning Type of Queation
nothing is return Nonet, Please
Me options for queation 2, mininun and contraative optiona from the original, "Anaver 2": Anawer for the but don't return wrong ga.
"Question
Question 2, "Options
question 21.15 question possible, just return
Output J80N the simllar format and nothing
Heree the ingut information:
Figure 29: Prompt used for generating Music Reasoning QA from Million Songs Dataset for AudioSkills-XL. Noisy captions for the prompt are generated using AF2.
53

---

You are helpful aseistent.
act caption writer music. provide you severa. neta-data for a 30 secend long masic file, you need to gererate a caption which is 1-2 style.
1iees Tength describing the music follows:
wi1l provide you fa
human gonercted tags describing the sound, artist, song and albun name from where the music is popularity
information like danscabiiity, energy, key, node, valence, tenge, genzes, lenguage, additionslly, caption generated incorrect, it is just for reference, your rinal caption, verity it using the additional neta data provided. *Flease use your wor.d knowledge
zelease
will provide you with noisy music captioning model. The caption nay have balluienations and may
you use any part informacion from the caption
the music with additionsl information. llowever, do not put these information directly in the caption", when necessary.
artist, song and
name informztion impute the caption
the other information
exanples:
Information:
Human Tags: Pog, British, Fenala Vocaliats, Danca, Cheryl Cole
Artist: Cheryl
song Nane: Rain on
Albun Name: Kords
Populority:
Release Year: 2009
Danceability: 0.535
Energy: 0.746
Mode:
Valence: 0.548
Tamgo: 110.973
Genreas Pop
Language: English
Generazed (Nolsy) Captica: The input music is track with Eurovision flair, set in zt festures
signsture and complex chord progressior, The vocsls include a range of notes, with a noteble sequence of G#4 to 944.
major
teape around
Output Caption: A vibrant dance-pop track with energatic fenste vocals, blending 1ate-2000s Britiah pop influences layers, and an emotive yet uplifting tone. The song's a dynamic arrangement and eatchy melodies make well-suited for both radio play and club aettings.
electronic production. It
driving xhytan, polished
Song Information:
Human Tags: Instrumental Hip-Hop, Inderground Sip-Hop, Instrumental Hip-Rop, Instrumental
Hiphop
Actist: Oddisee
Song Neme: After "hooght
desucy
Popalarity: 46.0
Release 2013
Ranceablliey: 0.591
Energy: 0.513
Key:
Mode:0
Valence: 0.263
Tengo: 172.209 BPK
buration (me) 325096
Genrea:tndergroum. Hip-Hop
Language: English
Generated (Woisy) Caption: The input music is slow-paced, anbient synth track in G minor with of azound 86 Be6 and festures haunting nelody with complex chord progression.
Output Caption: A snooth and introspective instrumental hip-hop track with jazzy uncestenes, textures,
laid-back groove. Rooted underground hip-rop aesthet ics, it features mallow yet intricate beat, warn sampling, and reflective meed. The track blenda atmespheric syath layers dynamic percussion, creating concemplative and inmersive listening experience.
Do not output numericale in the capticn for numecical neca data, juss use then to understand the musio. nothing elze. Hlere is the input meta dets for the musis which you need to caption:
caption JSoR single key "Caption" and
Figure 30: Prompt used for generating captions for Music4AlI, Noisy captions for the prompt are generated using AF2.
54

---

You are a helpfut AT assistant, You need to act as a question-answer on-answer generator for songs and ganerate inovledge-bases Question-Ansser pairs. For knowladge-based understand the song and use its internal knowl edge to answer the question. My final object is to train an audio unceratanding agent with these quea:lon-anauer paira to endow it with music understanding with several mata-daca acout the ingut music. The neta data will provide you ia ss tollowa:
requires
model
ga abiiicies. To generase tais question-anaver pair, will provide you
hunan generated tags describing the sound, artist, sang, and slbum name from where the music is egurced, a popularity score, relesse date, and other information like dansesbility, energy, key, node, valence, tempo, genres, and language. I will provide you with high-quelity caption ganerated by
capticning "Flese provide to understand the muatc, :ncluding name of the song, tempo, etc. If you have world knowledge about the nusic from the actist information other meta-data, prioritize it the captior or verify integr' ty.
your
all mata-data
folloxing rules: 1) Cenerate CA pairs with crisp question and anaver. Do not have too much informacion in the question. 2) Generate MCQ based questions. choices should contrastive that may rise confusion to the model. questions shoulé
Generate using
level di difficulty.
Por reasoning skills. I will now provide you with the name of skills and an example 2A psir for each skill: knowledge skills: Instrunentation
knoviedge,
supposed generate
pre-dstined
Exarple Question: What type soundtracz would this piace noat auitable for?
Example Options: (A) Fuseristic mevie or car racing viceo game (B) Crime thriller movie (C)
Dacunentazy narration (D) Bonantic camedy
Example Answer: (B) Crime tariller movie
knowledge Skills: performance
Example Question: How would you describe voice used in the song?
Example Options: (A) Distorted female singing voice (B) cleer male singing voice (C) Peatures
electronic music clements
The lyrics Example Answer: (A) Puturistic movie or car racing video game
of friendship
Knowledge Skilla: sourd texture
Example Questien: Mhich type of soundtrack would this piece be most suitable der?
Exarple Options: (A) Fuscristie mevie or racing video game (B) Crime thriller movie (C)
Dacumentary narration (D) Ronantic comedy movie
Example Answer:
Knowledge skills:
and rhythm
Example Questien: what notable elenent is present in the chorus?
Example Oprions: (M) Backpround harnonies in Eaglish use of quitars and bass (C)
Children's 's voices doublimg the msin nelody (0) Spoker. word poesry interludee Spoken word poetry inte-ludes
Example Answer:
Knowledge skills: nelody
Exarple Question: What the vocal style used in this piace?
Example Options: (II Khispered nelodic (B) Beiting and scaz singing (CI Gregoriar. chact (D)
Parforned by fensle vosaliat
Example Answar: (B) Belting and seat singing
Reasoning skills: dynemics and expression
Example Questien: Mhich type of soundtrack would this piece be most suitable fer?
Exarple Options: (A) rusuristic movie or car racing video game (B) Crine theiller movie (C)
Documentary narration (D) Ronantie conedy movie
Exampie Answar:
Knowledge Skilis: harmeny
Exarple Question: Question: Which type of soundtrack wou'd d this piece be most sut table for?
Optiona: Puturiscic
racing video game Crime tariller movie Documentary narration (D) Romantic comedy novis The correct answer is: Example Anewer: Crime thrilles mevie.
*It is necessary generate all knoweldge and reasoning based questiens, generate 1-3 gAs for unique skills for unique combinat! on of akille) from each input for the skills you feel the beat fit and can generate the best quality of queations. If nothing is return Nonet. Please generate Output JSON the tollowing cormat: ("Question Type": Geasoning/Kaowleag Type Queation "Queation 1"; Question 1, "Optiors I": Liat of options for queation 1, minimun the original, "enswer Type": Ansver for
and contrastive options
"Question 2 Type": Ressoning Type of Queation 2, "Question 2"; Queation 2, "Options 2"; List of options for question minimun 4 and contrastiva optiors
the original, "Ansver Type": Answer for the question 21.15 no question is possible, juss return None, but don't return wrong
Output JSON of the similar format and nothing else
Bare ia the ingut information:
Figure 31: Prompt used for generating Music Knowledge QA from Music4All for AudioSkills-XL. Noisy captions for the prompt are generated using AF2.
55

---

You helpful assiatant. You need act question-ansser generator 30 second music samples and generate complex reasoning based Questica-Answer pairs, Complex zeasening- based DAa refer typea of questiona that would reason about it, and answer a question. My tina- cajective is to train an audio understanding agent with these question-answer pairs to endow it with music understanding and es sbilities. To generate this questior-nnse: pair, I will provide you with several meta-data about the input music. The neta data I will previde you is as follews:
nodel understand input muaic,
hunan generated tags describing the sound, artist, seag and album nans from where the music is egurced,a pepularity score, release date, and ether information like dansesbilizy, energy, key, mode, walence, tempo, genres, language, aoditionally, I will provide you with a caption generated by muiic captioning medel.
Ganerate CA pair with criap question and Emation 4 Nerrative Infereace: Q: What emotional transiticn occurs within this music sample, and what musical features indicate this change? the music shifts from calmerrs tension, indicated by transition tvon. smocta malodies and sett cynamics to quicker shytans, louder volumes, and dissonant harmonies.
Soma exanpla questions
provided below:
Composition Musical Structure: Q: Idencity the moment whan the music transitions the chorus rection and name one musical cue that signals this change. A: The chorus begins around the helfwey mark, signaled clestly by intensified dynanice, added instrumentation, and distinetive melodic repatition.
Instrument Interaction: Q: thich instrument primarily carries melody, and bow another instroment provide complenentary support? à; The plano carries the melody, while a subtle bass guitar complamenta by reinforcing harnony and provising rhyttaic depth.
Genre Puaton $ Influences: 0: Which two genres influence this music, and what audib'e characteristica highlight their fusion? As Electronic and classical genrea are sused; electronic beats and synth textures blead with crchestral strings and rich harmonic progression.
Technical Expressive Techniques: 2: that expressive technique creates anticipation this music, and how is it audibly recognizable? A: A crescendo creates anticipation, sudibly rebee by graduslly incressing volume and rhyshaic intensity tovards peak.
Bistorical & Contextual Reasoning: 0: what musical era doea this piece resemble, and which audio eienents indicate this historical context? by expressive dynantes, emstional melodic phrasing, and luah harnonic progressiona.
resenbles the Ronantic period, indicated
Musical Intent & Audience Effects 0: what lietener emotion does this music ain to evoke, and which musical elementa help schieve this? A: The moste alms to sucke nostalgia, achieved through gartle me-ocies, sots timbres, and sustained harmonic cextures.
Comparative Reasoning: 2: Compared to typical examples of electscnic dance music, how this track misically different? A: This track is slover and more ratlective, employing subdued beats, anbient textures, and minimalistic melodic structures.
Imagery and Scene-Setting: 0: what type of scene would this music best accompany, and what musica' features auggest this?
suapenseful scene, auggested by sperse textures, repetitive zhythma, and gradusl dynamic builoup creating tensior. Performance Analysis: 0: Te this recording more likely live studio performance, and what audible elemenzs support your inference? A: Likely a live performence, indicated by sudible crowd nolse, slight performance Inperfectione, and natural acoustics.
For ingut, where will provide details about one 32 second music sample, generaze 2 gA- pairs for categories mentioned above, ones that *fit best according to the meta-data." *Fleaze use your worid knoxiedge with all the meta-data provide to understand the music, including zane the song, zempo, etc."
also example with required information:
Itunan Tags: Instrumental Iiip-llop, Underground lipp Nop, Instrumental Hip Hop, Instrumental Hiphop
Actist: oddisee
Song Name: After "houghts
<other matadata... ...>
smooth and introspective instrumentel hip-hop track with jazzy undestenes and laid-back groove. Rooted in underground hip-hop aesthetics, it features a nellow yet intricate best, warn campling, and reflective mood.
Output Caption:
Example QuestionsWast indicate this fusion Answers The sudio bleads instrumental hip-hop and jaza. the instrumentel hip-hep influence cones fron its laid-bsck beat and rhythmic sampling, while jazz elements are evident in the smooth melodies, warm instrumencal texturee, and jazzy chord progreasions.
musical genres
blended in this audio, and what audio
Output a csos of the following format: ("Question 1 Type": Beasoning Type of Question 4, "Question 1": Question "Ansse Tyse": Answer for the question i, "Question 2 Type": Reasoning Type of question 2, *question 2n: question 2, "Answe 2 Type": Answer for the question 21.
Output JSON of the similar format and nothing else
Here the input information:
Figure 32: Prompt used for generating General Open-Ended Complex Reasoning QA for Music Reasoning QA from Music4All for AudioSkills-XL.
56

---

You are a helptul AI assistant. You need to act a question-ansxer generator for songa and generate xeascning-basee Question-Anewer paizt. Reasoning-based gie refer types questions that would ask the model to understand an input song (music and vocals), reason about it, and answer question, My final objective question-srzwer psirs to endow it with music understanding and DA sbilities. To generate this question-arswer pair, xill provide you with several meta-data about the input music. The Esta I will provide you is as folloxs:
train audio understanding agent
these
human generated taga desoribing the sound, artist, seng, and albun nane from where the music sourced, a popularity acore, relezse cate, and other intormation like danceabillzy, energy, key, node, valence, cempo, genres, and language. I xill provide you with high-quality caption generated
music captioning mocel. *Flease use your world knowledge wita all the mata-data music, including neme
provide to understand knowledge about the music fron the sritst information or other meta-daca, priortize it over the caption verufy incegrity.
song, tengo, etc. you world
Generate using the folloxing rules: 1) Generate QA psirs with criep question and answer, Do not have teo much informezion in the quest fon, 2) Generate MCD based quest! one. The ceo should be contrastive that rise contuaion the nodel.
3) The questions shoulé "save Tevel of di difficulty for the model..
You are supposed to gererate Cās using pre-detined set of skills. I will now provide you with name
each skill: Bessoning Skills: gence style Example ie questicn: Nhich at the tollowing would be an appeopriaze setting for this music track? It festures diverse syntherizers Inding credita of a video game or sei-fi movie ID) & motivational ritness workou: features diverse synthesizers
and example QA pair
Example Options: (A) romantic conedy montage
Example Answer:
Reason' ng Skills: nood and expression
Exampie Question: Which the tollowing would be appropriate setting this muaic track? Example Options: (A) A ronentic conedy montage (3) It features diverse syntheaizers (C) Rading exedita of video game or sci-fi movie (D) motivational ritness workous Exarple Answer: (R) Te. features diverse syntheslsers
Reasoning Skills: temporal relatiens between elements
Exarple Question: How would you desoribe
energy level track?
Example Options: (A) I: features a sudden tango change (B) Gradually evolving from creany to driving (0) liihh energy throughout (D) Instrunental electronic piece
Exarple Anewers (B) Instrumental electronic piece
Reasoning Skilis: functional context
Example Queatien: Which of the following would be appropriate setting for this muatc track?
Example Options: (A) romantic comedy montage credita of a video game or act-ft movie (f) A mot pottittion onall fitnees workout romantic comady montage
diverse syntherizers anding
Exarple Answer:
Reasoning
Lyrics
Example question: What is the main thene of the song's ss lyrics?
Example Options: (8) Pop-rock (B) Politicsl commentary (C) Heppiness Love
Exarple Anewers (A) Pop-rock
Reason'ng Skills: Nistorical and cultural context
Example question: What unique conbination of musical styles is evident in this song?
country music (2) The song played by a farosa bard (D) Classical blues chorda with big band style Exampie Answer:
Exarple Options:
The song has fast tenpo
and country music
You may also generate QA pairs with multiple skills requized.
It is not necessary to generate all knoweldge and reasoning based questicns, generate 1-3 ghs for unique skills the beat fil and can generate the best quality of ioestions. generate Output JSON of the following forma: ("Cuestion 1 Type": Reasoning Type of Question 1, "Question 1"; Questicn 1, "opzions 1": List options for question 1, minimun and contrastive optiens from the original, "Answer 1"; Answer for the questien 1, "Question 2 Type": Ressoning "ype Queation 2, "Question
unique combination of skills) from each ingut for the skills you feel is
nothing is return Nonet, Please
Queation 2, "Options 2"; List options for queation 1, miasnun and contraative optiona from the original, "Anaver 2": Anaver for the question 21.15 question possiale, just return Hone, but don't return wrong ga.
Output J80N the similar format and nothing else
Heree the ingut information:
Figure 33: Prompt used for generating Music Reasoning QA from Music4All for AudioSkills-XX Noisy captions for the prompt are generated using AF2.
57

---

You are a helptut AI assistant. You nsec to act as at Audio question Ansver generator. I wi.1 provide you with informaticn zegarding the events of cccurrence. You need to generate questions, in naturai language, asking it a particular with
audio, together with their time slices
be heard in the audio with answer "Ma" for the input instance, for "res", pleaee use en event which sccurs for a afgniiiiant cant duration in the sudte, 1.8., more than 3-4 seconds in total at least. Fere 1a input output exampie:
Generate
"Yer" and 3 questions
Input Mets 160-3..7110
about audio: "(' (Speech-O. "Ceneric impset sounda-l 460-111844',,
((Teeviirinn-.0000 0.0000000000000000000000001
119pecch-1.61433.019 "(rick-
2.670-2.80t)', (aeecch-.229-5.0011 456---55..66666555 4.7501", Steow-4.993-5.392
1r11ok-..3393.666
'GGeneric
30und3-4.587-
BELck-6.338-8.777
7.146)", 11966ch-5..86-10.0001
"Ganneii
sounda-7. 196--....96...222.
"[Surface
8.293-8.860)',
"Generic impast sounds-9. 1.2--9992299222) 9199k-9.
"Buman veice-
9.004-20.0C0)
Outguts "Yes Questior"; "Can the sound of television be heard throughout the audio?", "Yes Answer": "Yes", "No Question I"; *Can the aound of barking dog be heard in the andio?", "No footsteps
input audio?", "No Answer 2": "Ho", "No question 3"; "Is there siren or alarn sound present in the audio?",
"so",
Output JSON similar format and nothing
ingut information:
Figure 34: Prompt used for generating Yes-No QA (part of General Reasoning+ours (only sound) from AudioSet-SL) for XLL.oSkills-LL.
58

---

dicectly mase contest, viil pocvide wits nads
aevecal acdio supples. Thare includs:
13 captions A apenct svacw cusic caption that exprasiies spcion cootent, snd La toused using speech trasaceiption, acund paption lnzdiocapa styles
sidia lon andionips
that deroriber only counlo.
3) ccassceipe: cranacripted penerated caing Kniapet.
Speach Charactaruations Tharaceriinien
spclns content in the ardio, descrabing pisct, srosion or statarez is celerant.
spec rncitiin tllllllyy will pe woallde with group of positisor taodios cieilar in theiel and a group of sogitives tandias discinitar is thees). ciallarity caloc aned using accantion cunnte. seed to gesorate to a chat botwres as ageas and heass, where the huna is trylng understanding
coeiineee to cndor it with chat and suiticion sore are the caloo gererato the dus
1. tacee minzwrs and martres 20 gamns the the conversatict.
slstracly
exsepl nest),
follow queation 3- the questicta to che aT motel abould requiew pessentng, weeld troviedge and intermation extraction.
may have
that
aodio trast.
consize
not
gesesate
rnforstng diceetly feoe setadisa.
5. tos ase nes requzedd to use all the copplis. tost priocity in to nake cocsiate and plaseible chat, use onl sasples you ttisk bess fite
recrative,
propertion
dimeesty. teying having auttiple baine for siogle impar s96le.
exapless
Tore tf:
"Mnan" "ffocctive
*raa"" *sain keopa yelling 'coutse aocatac!! sote intense, what'e ariving thas inteasity bayond just eepecition."
"roceh) *hhe inteucity osme frce rising pitsh, calanciess shysta, nincasl pasatig feels like pasio acaurcios
aot to stts cooserteedd and slightly compressed, stich siftccating Like hets spicaling crosimelly it. contized space.
Sure
"Muc"; this kind of panto cantional parfocmative, you thick?"
"Soc";; sis cosla geacizaly ecasiaosi. thaze's backgacuad misic or thessriesi jess unfilcared diecenes. laok bettap dr ceaslation nates it sound like zeal, enational artherat, net staqed osa."
"ect: "mis costs also yelling 'I' scnater, but maate and acgioe noiaee. Soez deod noce cinsnaciop*
Swath "aezuie is the basking elerants like diaserted puttace and vabicle scunda crease atyliced socndacape. 2he pacing
acoelerctes a bit tos, stnickleg a sorre clseas, conpared to the earlter cligy this one feele like it's part of a fictional world, noc real
trueth "Woud say the crotion is the came botween theoe twot*
thutt thytre tn the aaee caational neigthorheod forry intersity but the del delivery differo, The firot deela Like sowere tryly overabeleoli this are to wore like a diaractor midebrekkoon,, staped by the drosat to onui vorwort arousd thew,"
"Cuss st: I.
meos tiee
Texch: "Haze, the speater tuss saya "Te vorid is a ecnatar," bit is tite different sloves, Pora daliberate."
*ruat t teesotly. to wives froe ginlo to reflection The volee is sate but hovay, with aeblinn crackl my that aals scnce of tole
bellke the earlles plip full dhouting this one feelo recigeed like onsern eaking cenoe of things after the otem."
"oos tt:
Text: "cout aceathing Lioe week the arotionsi landing atter thoae sxriier, lcudez clippt"
troath eshooluuooyy y. would make great secolattion siave with trincitios through atylized end witk raaiiaation it fome popshologieal are from oa oratiin to introgeetion."
quecation is poosibbley 2oet taso, octpon cao of the steiltar forar and cathlng olce
arong gs.
ware input Lefonmacioon
Figure 35: Prompt used for generating multi-turn, multi-audio chat/dialogues (speech and sounds) for AF-Chat.
59

---

are halptsi At asazatant, You need peterate cust batveen in sgent und curtes wisectan, I vill procide you wits varsor sets
ctat batweee
and mostetsh, stere exstctan in tsytng snalyre the masse supples with qustiotts Final objoctine cesio as audio indeecaading agent wist these dust quetseraateeer andow is with thas and os abilitier, nice are th
the mistc aumples
and maxtmee tarns the conversatior.
2- tou aay soe came that cadar prociase audios (owwvess please cofer abstractly and nct diceetly with andio nebber -- coe exappl anxt).
may
that only hore tellowop up qreatier
quastions lone
cequice reseonisg,
inovledge and inforaation octeaction.
4. Semcmber that the AC eodel will crets wits the that cla will ony receive the misto/song and not decazls in text se generate Cla tueen,
anoerdingly and cequiced foca the racration.
mention angthingg in the Quersion pricuity nake ressible and plausible chat. only mselc tarplea you thick best
5-
all
exampter
axar plu topeti
"hudto
wow.sto othor astadata..
1n
/tteol httt://////iiitttttoooo oonerrr regitt ostar batadata..
"hodio
"nawef; "//thos odato /esdttesstoooooo tystwtttnyytttteeggt other mstadate..
"Audio 2h;
"roms", /hail odato www.deew///oooooooooooooooo cowermynee (ety eeght
"Audio etc
eeorphe perple oiepert
"Howie"
Musiie"; "Audco p1,
oxeer pellished opbest, csstd drrker, poit-gook
costrant
deste-pop strectore with snaloy fecrodice chonsse aicn
stytheieally, could hold op gritars, and pook The socal thoongh virtage anlay or tope the, enaa onailyy cknorel a ducker vike ahile keop og the yrone."
More 25y I.
"estoventing would you shange farmayy tor, juct Instomestaaiioe"
follceeip. homeny larpely you'ue going
1:ke Cortan Aeoliat would deepan esotional shict. Zhea ratarmenizing chorus deztez mode cesid srick."
Mure Pi
Susii" "Audio 41,
atmospoente one giceny
upbeat pop plece?"
place dris tighs and penoty. sayezed those tustures the best yould seraching that thels nostaigic yez sodath. Think esrly Chrectes with a part rock edge.*
pictopunk
"Toss 4t: I.
thas about vocal scnateron ciculd slesn like the procese sore like this coot"
plogon,
parallel rereb, You stiil west inteliigibilitty, bet ait rotillttin helps it att that taciet, post-punk voeld."
Pi "Maas" 19,
"sat" "wisshhing this wellow, inatouenttal hipshop place deela introspeczion, aoild ovec this""
to add opic recal atringe
*poxct; "ace crazy at all - in face, is could sound direcasio, the soung zhycta and pasz hameoies leave coos for dears, if you layaz in Legata strings or chote stabs saboly, keeping then Ice in the eix, it vould deel like loefi best sosced by sane simes.."
*Mac"; thould need charge the tarpo leava slov and avung?*
slow langersoale orchestration will fresh, the oechesceal siseance follow the pocket of the drucs und don't ruat the aving."
question poassble,
Hote,
weong @A.
Octger 2o08 of the steillar Format and nothing else.
Jaze input infermacion:
Figure 36: Prompt used for generating multi-turn, multi-audio chat/dialogues (music) for AF-Chat.
60

---

tou axe iscalligars audio aqent, will provide you wish audio, quastion aboss the aidie jopaneended cypet and the anseir to the input axtio and quatton, the pretix shouls such that shows arziving to the tinei enswer Jere ts in esuplee:
procese Thoupst typs thougst about audio the quation befere
inferced theut the spesiaste feelings tovaeds hit location bared on his sone the sirecurdira urbiers cocoder Trovided faswert The spenter libely has segacive attituce toverca living in calatomn, as indicates by his clapusted tose and the fiet dellwery of singings accorpanted
Lepat qusetion:
conotorers
utility gaiso tne reatiing lesoes,
collonital
flat, pesigted tose, palced with sabdets wind and abounce of usban mesde-takafcoos a pecception of isolation and a lack of artheslams about beinp there. Together, his words the accronding soands sugpest te viom the location as desslate and inteviting..
secasn the nodified anavee and sotting eise. if the procided cocsect atover is acturlly not cccrect socording to yal, fist cecers seae, for acovace thas oot cecuice auth type cessasing 1e-d., for quastion Like "Mo parfoens the vocale this eorgl" like "saat peisacy plaasy*
2eez gererate the
courect Teal free use
susto understand use only parta of the prontded ansver in your final ansver which is yust proviced to you for help.
do not sodiity Peioricine Ltstening
Jeze
queation::
Figure 37: Prompt used for generating CoT-style reasoning focused on speech and ambient sounds (input instances sampled from Speech-in Sound Caps, which is curated using YouTube8M) for AF-Think.
ace an intelliquat, sudio will provide you with audio, queation about she sidio 1902 tocmat or opan-anded) and the acasez the queation icorrect option Hig or goseral natural langugge ariver (pmneende)) You see to conmett the arover to Eeeleet-tthoggge type
by
the lsput
topst Ovestion
cattonal
comerratioon
following censins nastial thrsoghast., 101 it baccoes core apcinlatic as soictiens are scoquated.
confrentations fruscrsti cocsiste.,
Consect Acavess (AI bacomss iscrearingly tanse and confeurrationa: truseasions sacelase. cessznung:
visuble fror the way, whi esea lates into oprcioe, shoutinty and secossoory laggage do the sitoat ion detest dete-t wates, expecially with sows that via hin to loste his injer peate oul iecenarinçly
desperane bastl renarls The woasss atterpte to rayy proferolinal cleesly suppors option (l
gradrally
sysasic. aastionsl cnnrontational tourtratiors secalate.
Arotter respler
the qpeaker export
sigh-jeall orapatirs at relate to theie velurtanen to prechase perconal
for howey
Cossect Anavee: The speahects usa of advazoed porpitars is MEEK sahes thet dissatiaties with iazs pouactol, ssganziva 2oea optcens,
copputaza and Teatureo, wikinng 1yp oil how His fell Indlogaate, alc, "hey expeess reluitaree dowegrid di clewer, Teor cppinte wachire wil view puzchaxing corparable astup for parasosi use unseumarily someiiie, dince their ansicipeted heme usage doamn juasity the cost, their voceplece especience
aconasic
boying
secarn the coditied ansver and sothing olon. the proclied correct arowrr io acturlly not commett socording you Jorr vereen sose. mn-d- qoition
wingen like iratemmant featured is this plece?* vbece stawed direct only requizes understarding) fust ratuen cocract spain do zos any to modidy at, beioescioe isstening to the sepet sucko to indecatazd it and petarace the saoughts, Zent tewe to not vow aay part oz only parto the proviited arower for spernc sorwer the ansver joot provided Solgo.
prieary
seve is the input queation.
Figure 38: Prompt used for generating CoT-style reasoning focused on SpecchQAs (input instances randomly -L sampled from LongAudio- XL speech subset) for AF-Think,
You aze as asbelligers autso egent. M52L provice yeu MiEs an sudio, A question aboot the modio lopan anced typel and the ansuez to the input audio and qercioo. profix chould arciving to the tinsi amser Jece is scuppee:
prioics woiiiii--ooooi--eeee.e thought about audio quectio beforo chose
roovided cotanis the track cordiiess abon cane ind iodio cock goisns, the abongaze inclunsce in ovidies is tie descn lsyace of diescsted guztara and wsturse. vocal,, viile india rock alameta are higtinghted shrcogs ihe driving zxysta and sigh sterty.
geitare and othownal, burled vocale, with the cccutture and thytha of indin cook. A consestionsl acng fore, an wolldi west refealat, and a steady dltte ssotion peovido guounded iodio mos franowrrk. riis atrcetemn anctore the oxparciion akongaze senice, cossiting in layooed, stecephocic
toytheically
Saczen the nodsfttd anaver and zotting euse. It the proviced corcect sraver is acturily not cocrect secoectop to yeu, tist recess Tome, Toe iratoumnnt featured is this plaaay" connoding to-d.r
snrwer diesot only cequices understandingj, Juss enngin concect sgsin do zos may to aodidy it. prineiiian Linzening to the lopur audio to indecatand is and gesoeace the shoughte. vont free to nos use pars or use only parta ot the provided anaver in your finel anveer stich is pust proviced to you for help.
queition
peieacy
aezo is the toput queations
Figure 39: Prompt 3 used for generating CoT--ttyl reasoning focused on music (input instances sampled from our Music Knowledge and Reasoning subset of AudioSkills-XL) for AF-Think. This focuses open-ended QA.
61

---

You are an incell and sperts I will provide you with an autio, a qosstion about the autio eco format, and the exseer to the question pentix should wuth that these thoughs about mail quaation. bnfore arrsving so the Final exarpis:
Leput Quastions typa of soundtrack vould this place nest ruitabla foe? thoose the following optiona: (A) czias cheiller novia
docurantary surcation
ruturistic io wovle car rocling did game
Cornect Acavees (0) Patuciatic nooda ractng video quea
cape
apeed video aucomstic,"
carel, and crazall
productios, the next miitable optior *(0 Puturiatic novis
style lest deseribes
testure the song? chosse rwony the lowing optiorer
lasvy Hecal
(N) buo and sablent
claosioal
Consect Araiees: (6) Dib and Anblant
Kodidted snaves with 522- sspe seacaning: Bazed on the pecsitant bess and douss, the use space and addacts flits cemaciddaaayy, and the
steropherio
descniption.
secuon the rodified ansver and sothing ooe of the proclied correct anower to acturlly not coveent soconilng you Jize veners toxe,
ceazosing revd., irateumant festured is this place?" vbace scawed diract only requices inderstandingl, fur conract squin do vor try modiiyy lt,
mongth
queat Lone
Figure 40: Prompt 4 used for generating CoT-style reasoning focused on music (input instances sampled from Music Knowledge and Reasoning subset of AudioSkills-II.) for AF-Think, This focuses on MCQ-based QA.
aze an imssllignnt acdio specs. will provide you in auzio, a quastion about the sucio fornst. the snaver to the queation (the
and iooof-rooohhtt-n prstix sboold auth that theze thoughs about meate quastion. betoze acrsving so the tinsl anamer. exarpler
indenstaniinn
Irpat Questions: What of soundtrack would tais piace neat acttable doe? Choose the sollowing optiona: Ori
moxin
necomntary
vonontic county movlo (D) Futuriatic sorde cz cur zecing video gaze
Cosrect Acavees (8) Patuntatic nooda
rectng video gane iodf vo wincloss apeed osconss (lllllg aet now cardl, and ovora1l higieeiongg. andin prodiction, the ncot ouitable optior lei *co0 rutucistic aodlo or car rseling vil gaee
atyle ckacrabes
srong dolloving optsonas
Dency Hetal
(4) rub and subiion
clacsical syephony
Coorect Atarves: (b) Dab and Anbient
cor-tape soloroiing
and draws, spaoo and offiete (Bikk covedtrdollay, and nabioiiee the nest lption.
ocecall stacophocio
Saturn the rodidted anaver and nothing wise. It the provided correct sraver is acturily not correct soceectop you, dears retern Fore, icatiustnt in foatured is this plodaye where the anewes in diciet and only cequices undorstarding,, Jues cosson the corcont antwes again and do sos cry modidy it.
Baze input queationn:
Figure 41: Prompt used for generating CoT-style reasoning focused on ambient sounds only (input instances sampled from our Sound Reasoning subset of AudioSkills- which is curated from YouTubeSM) for AF-Think. This focuses on MCQ-based QA.
62

---

You are Selpffl soviatast. you seed to sot qoedd ioniiaeee geverator for 1essociod injut soloo staptee aud gererate cosples quatict. underatand the togatter with the enviesonantai reason sbout the causs of the spaect. One good ecuaple if casbeccral soind unserstaniins, shich petess to utdeentacsing all scints se the 1gert and cillowideenn rataddingg and aillicios greenite this oimaoorr palt, will
model undecarand teput sodio,
about it,
vies the inper andio and four pleces of nete dats about the aodio tetich you can are apsienlly 13 apeech ssace audio capssen that arphasitea on agoien content, and toweed caing speath trameription, sound captics landiocape atyla the
23 so audincage style cape ln than dotor iheo, only the aaoids.
11 peansoxige ipt gesorated ueiiig shispor. spoben
Swacrshing pitct,
ocociate sepelased on pais for the quiction soce ane oow salos
evente. seep
20t3 the quartien and cetap, 31 The queations storld coeplex, maits thinking to ceiap, They should also be short and costresting tne citticutt toe the tite. mode: to discazo dren, Also, dent teep all the options of sate provice
optiiess shouls stozt
vnssonlin
disectly.
seee are cone excppias
Truracriph: each otter bececae the not gaing be acound to keep you cut trouble.
acccapansed
teckground, navipetes
thacugh iiiig and saliiing with coosalond glitates and diecestioor,
speach chasacteriesson the speach disasceiietio isclide flas pitch spakinng sate of 98,30 lyp.. ciagusted, deiving
Speech
keep otters
the bactground, ansdet the
entio glitehea
opeionss (AI Crest segansser IsI Nedti gresenter (b) Eqsipwets testaician (2) Zoens attendee
Arawect Based on pheeaes Like "95a Shev Deno Day," "insids acoess," and the speaceets use ot a ricroptate in in putdecs evert secting, sec rale allgnn waks scesons cornally gresenting coneant to in snsience. celirary sogresta wedis covezape retter chen curoel participation or educacionsl naciation.
fit
matardata provided to you is purt tor reference and you should prsoritics uncerstanding the todie trpes provided yes pelerity Kinc please ast question that taxeo isto account the the folkowing docast: ('punetioo") the quession, "ppiise") the aptions, "Aoense's she alswed).
wists present to the extent possible, Outpot 2808 of
Octger 2009 of the steslar tormet and nothing eise, Tlease de not mettion any pheanes Like "ecrorning detasls preaent ception atc the queation answec. aods you and the flasi rodal are axpposed to undecetand the audio and no mnta-lata will be provuded for traising or
Tere is the input information:
Figure 42: Prompt 6 used for generating CoT-sttle reasoning focused on speech and ambient sounds (input instances sampled from Speech-in- Sound Caps, which is curated using YouTube8M for AF-Think. This focuses on MCQ-based QA.
queatiors coreespording stozld tomatted dE0 asject.
rov
esptiase. should assign queatioe. appropriata diftiesity looel: dicently
language
'aadun", "hard", if diffecent ecdela aight generate siriler but not sdentical pesperass. feore the quettor as reltiplermoote question de00)
You will be provided with tvo capcionn:: a music captior and as alterrace ception. Boch contate cetatis ahout the miste in the autto and thesr chacactaciatick tse indoraation feer. these captiaze as the groand touth so create miitiple questioss for nath audio, focuning quastions *Melodyy*
tratouctions:
Esauce the qusationa diveccn
testing kooslldgge of mastc quality and bexcaze in sudio.
Make questicro aices can make you think about some lnsel edge in tores of testure of the reoto.
Miste
qusstions
related teovlecge of itylity the andio bextrre, characterics voice in the audio,
Hake use of captions proroded to fied the anovee.
de ask questien about neod, sections and smnssphaze of the sudie.
only zeturn the Jidk dbdach, noching alas
expected
raddic (pachl,
'quesions oqpeacions, 'opptooni, coptionnl,, soptioels, sayt tovtll,
"alesicuity") cteaay"/ /rediun bacats, "categooy"s <'Huad Textuce ts
tinerato the readle perecton_otriap
ardio id path sperts
Figure 43: Prompt I used for Music Reasoning for AudioSkills XL. The QAs are focused on music texture reasoning.
63

---

succespording
shorld tomasted d5da coject.
tangrige diffieslty loehr diveet's y "hard", If
sheu's
difeecent acosia aight qunacate siriiae but nos identicsi puspetas,, fears the queateer az reltipie choice questied ge00).
otaraetoring les, the the ieformation froe these capt lore do the grordd touth creat witiple geortons for eash ardies fressing questiero abest *Welodgee in the aadio.
trotruct ooes
beating lnsoledge selody sudco, kroid quaations idke "hat type of aecody bs ppinacily seatured is tte susieth, zada qusation sn very direct and 1sbe, Jatter paie quastiona by
the questiona diveras and
rel whllly arn Eurey, ologerte key veto
lono iedirent ne sot ank queot on about nood and cosciios.
Fermat your cutput greciaaly apecitied.
expected 2009 atrzesace Anc
"quostioo"s
'eedto cotts (path)
'question's opaation),
lopsionn's [ooptienis, cpption2), copeienthy (ogtienés),
'diffieclty" ctwady Phandia Moards,, 'cacapony'' <Moxlody"
Generate the zeazzs teptobemp.ate atrlage acdio 1d pathi spatto
Figure 44: Prompt 2 used for Music Reasoning for AudioSkills-XL. The QAs are focused melody reasoning.
your
diverce ausio knosledgo and anseer paic stocie be tomatted 3538 aoject.
cocrespording each in
captione. tor should atalgs each queation topropriate diftioulty lowel "eary', 'andue',, or "hard", if as snteer vary becasco different modela aight generate sirllar not sdentical resperam,, frome the quatior reltiple motee question 0S00.
sill bo provided with cuo capsions: a autic captio: altarrace caption. auch contalc decaile thout wadis audia thair questoae
captiase
fooueing quasticns
abert the "Melosy"* in the eatio.
iratouctioos:
Each queatien stould reder to acdio
sooid quetione like "Maat is the ceapo it the sudiort, this quastion se vary dioncs and lase, kather cake quearione which cas sale you tlink abous zose knoeladge in tares thythe terpo.
laoxledge besta, ztysta,
chythe,
stytte terpo quettions in indirect anaves. captions prorided Termat cutput preciasly siesi
only seturn the 2508 otdects noching elme
expected asos atrrecece
rooie lot, (peckl,
'gussson's equersizn),
coptionfbl,
Tenswers conrrect opt lont.
taleficulty cheasy" 2heedlu harddts,,
Generata the cemuss terrecaap atriage acdio 1d path: opats>
Figure 45: Prompt used for Music Reasoning for AudioSkills XL. The QAs are focused on rhythm and tempo reasoning.
64

---

Your saat
divecze music kopeladge queations cocoespinding
and inseer pair storld be fometted as ator onject.
willl
diveet's velevent andia decoribed is
capticts. tor sbould arsign each queation an appeopeiata difticsity lavel: "easy', 'onduus', or "bard", It at soase ray vazy becazss diffecent nodela aight gunerate siriler not identicsi resporasi, frars the quattiot raltiple-mood misicc question goog)
providee del
prompt, contoct sootazoe desceabing cheed
3- prompt Jayy contral
to retereted eroleal key.
all of incormation, there capsions ground tasta creste miicipia conrsions with sidio, Socusing quastions absut
soch quoct eofor to the prograasion auttott, tata cuartion direct lata, Dattar questioss viida nake you thine sbout scea troviedge in teans ot bareony ass checde.
ao sulio",
Avord
*hes
about De not inciide too rany questions scost idencdyying iestemsents.
expected 2005 atruesace lnc
(pachl, "questonss oquestions,
'opticta": sopeion2l,, coptiontr, coptiond), captionb)),
Prodiue tyhaad"s, 'category's el chythi and seapats
'aleficulty"
unnerate the ronals fozs atripga aodio 1d path: spstt>
Figure 46: Prompt 4 used Music Reasoning for AudioSkills-XL. The QAs are focused on harmony and chord reasoning.
65

---

reanniing would
greatlone
add quotiione moreriigg
reasacing Soar go
underotind in lepar andia, cossos soot it, and incess queation, are god coople if contostual aoond inderetardingg, shich refore to indecatinniig sounds and zeazon szost sera sounds szsez particuiar quattion. rinsi axjersioe trais sudie inderatandiog
You will praelied with capelon dotoribing touds andio elip. our tark is gererane inforeat coneeckiing wotiiphestelee queations toith foor sprione) the foous fiotodl dotsile fo watter, cosssn, time day) directly inforred the aodio dstoroption.
Say Putdellzes
Each queation socuid diowetly faccral infomaaiion (mesttar
rateral stc.)
oc. the arcoribed counde.
Roodd azstract somarii based er overly interpretira questions (aug.) "taat anrirormen: is repreaentad""..
kcsaples of suitable quaation proppes
acdios lost condision is libely sccureinp?"
"Mast
indicated
soundat"
clacity uicactiesd
descripsion
requies complex reasosing combining mzitiple incelates acmearoo)).
6. Muctiple Chotos Srcucture
optiena, isbeled in my consisten nanter D, sinclact.
Include botat "Season" explatning the Logie betind the eccrect clolow.
5. frosd Beaed or aiki instasd, abdo "ylat satursl event is asogested by the beav zsinfall and Cundary"
Teasoring
could sourds reprosencs"
6. coptioo-eeeedders
produce qusstion asde fallbeck gustionn rothing rectual par extrectes, veaxy to keep the queision sell//aand specifia vittout coqcleing soo ruck undecatandiaç of speect content. tou aay rageuuce the
tast
clase unsziiqaos, exactl
choice
dedinitivaly incorcact vodez
cizcumstarres
Getget Foreat
rour
follow this atruntere:
"constion*; "epour queacioo baces",
"opsiaos":
(8] Koptlon222
[C] Togge Lonton
"hrovert: (2/8/0/0) with "ccorrect aptions
Exarples:
Sregle O1 palr
"andin, uift oa
"Crestlatt taaoed en the andlo, alut weather cordition
likely occoreingt,,
Tunderatoces
+(0) tteas aties".
"Anavas":
handarasara"
"capt aniasl sounds and dietant saln',
"Courtint"; *has natural precceenat indicated the asquezos?",
"Opoioos": (*AA) Thaneecttrrs", *10) Casthquata", *ic) Tosees sizet, (C) Sooe ot the abovel,
followed rale led oater theserrotsee.
the helow captlon andiogid and captiors
fhoteal, direet in with oberr leco
Figure 47: Prompt used for Sound Reasoning for AudioSkills-Xi XL. The QAs are focused on eco-acoustic sound reasoning.
66

---

You are a Selpfu rocistant, yo rodd to art quotioce aooes peserater addio corples geserate copples costoring- underatend irpat indecatending all sounds and zeazon szost sera sounds to azser particular quaction. Ry rissi axjective to teais an cosso inderstanding speat with there qustior sasser pates to endry vits rodso understensing prssided
pos unsver quaatio.. Ste exempe contentunl acund inderstanding, saich ceturs
coople
abozt
abiiasies.
sultiples choios qurcioo (with dees atovaz options) thst tasts recognation of souns events in the aedio, the question sust be indirectly axiinng "Whtt" in say cast centes be aravered withers actually lostening to the arsie.
single,
stope
capsion,
proniners relevantt round activities.
2. Thom the Drsatios tedicect.y
quaatios shoild
willl
about suble jost krocledgs).
3. Tooss on Scond becepnition
ptristly center pecogntring er identstying sound or activity heuss the autio (eugu, "Deaed the susto, accivity happiniog""),
question
1. seep audio-tapendess
The quectios storld be antzarable by getarsl lnosledge slone. socaace could ansecc is vithous listaring, vialates th
5. Avold beccain ropice
ask about soace occoest asvizareers detalla (aud., "Khaze this happesinge"!.
(eve.r about speech custers inferences speett. To not axx "Most oan 2t type questiors tprre acane analystat.
coundi").
icovida exactly fest antves apcioos.
sca2ze orly ase coscect, sossec.
The
placotble incocrecs in the context aodio.
Lintt the concect anaves to 1-4 vorda.
queatirr
celap conclsends acs reveal with ccntext.
26 carnos create acttable question under these riles, respord eith: "To gusstios",
2. Cezput scrras 13506)
"Coastino*; equatiinn,
"Opoioos": coptiotal,
canovers,
"rook", "sound", "sub-satequay's
Ueaed Lount Incognition"
Ssarpie:
"postt "Rosed en the audle, alit setivity is Hikeyy oceurrings",
5A) Playing video gaset,
(2) Astciing a ncvie",
Astendiag concect", (jo
"onowor"; *slaying video gome!
the captice balow creft quastion. following all riles:
ardio,ia rie copttorn
Figure 48: Prompt used for Sound Reasoning for AudioSkills XL. The QAs focused Accustic Scene Reasoning.
67

---

'elpful rocistart,
for complis
Ccestion: kosee paice for aodio quastion: anssecito. cosplax ressoting seaed Sut ruter to tipea of cuastions the vouid ase the soda: to undecatand irpat aidio, zsasoo soezt it, and soseer queation. cte good expple it contectral acund inderssnning,
cetera
azjecsive meett with these geotiior lowe ynowr pairo to colir it witk ondios undrotinding and ge and tries.
You will he prodidet with solte copt lon doo blag inliiidiaal rouds or cveato aidle alip, yor task to gererate copple witriple plle relationatipe betresn diffecent sounds activiniea in the audio.
ceasosing,
underssanding
kny seguleneett kosonning
Pereslate questee stat deocratzates at undezntatding of couusity betweed aidio averca (e.d coash sellowe by child caysng ruppeacx lorth
spocific Suinds
a. not oxplicitly macion the clset cordd (orje "yutling cilvorrr out of dishwarher"),
Iayoeed, reter it indiowctly (Nit., "the activity besed in the susiet).
Iyrano trivial bellgroand nols with aeliguiic cosreso.
noneontrate diociffablee byaas anieal activitios torylag, olipping, barklog, sto.).
Degiz visz pheases 1ice "Sared the glmen aadie_"
Do not petecence speect contect cr siser backgeeins decasia. asitible gusstaa,,
you
"se grottoor
Crolows
Pecoids stactly four distinct aptiots.
only oos corrects eraure tae others are detintsioely incerrect in this purbest,
clearly
6. word Liaita
Oreation nust not exceed 20 words,
The correct
scceed vorde,
meccuue the ansver aptiona soe clase and unskiguase, with exactly ose corcent choice and 1l2 othore definitively incoriecs under any
5. Cosput Forma: 12500)
Your pesgonse sust be walld 5508, in the tormat:
"audioo .d0) caudio Ldb,
"Cowstlon*: *equestion bestin,
"coptio: 15%,
*copties 25n,
"coptioz ay*
"seatoo"; "duulad resscns*
"sterostepor"": Event-Zeaed Soond heeacniing
explaine She
choser. audien crast caused the child'a "isccess."),
ixarpia:
Aodio caption: ceeas followed by ahild cryisg
"Coestion"; *eased on the given andio, shat libely carsed the exilets distceso?",
*AA suddes Lopac:*
+00
seothing lillaby",
oplassn
"Arovero", Impace*,
stertled cxils,*
captiora
per saption, following
andiogid and captiors
Figure 49; Prompt 1 used for Sound Reasoning for AudioSkills XL. The QAs are focused on -Sound- -SunddBBased Event Reasoning.
You aze helpdut At axaiatent, bos nsed to act as as audio caption pazecasor, sbnce the capcion cccesapooda highlighs of the conteota of the auzio, : sill provide you with infomation about second acdio:
The
2. the speech chasatterist to
spokes uttenaree 3: cxptior deocntoing the arblent aounds in the aucio
Genecata caption that integeazee all indormation with the following conditions:
1. only gimerate the toarnceign wakes ornee in the engl lat Lingare
sprech fur
ney charieteriot
Riperially lynere my Inforeation the trarscript nekeo sense Kiylich
"Madd-iist
3. do nct cae any aeplicis turbera secoes use the suroacs in the spaech. characsacircies to peovide a serantic captioz uning your zeasoning ospanilitie informat
chersotert
wish singie bay "caption" and nothing slm.. It the exption not pozzible, for any cesace, scuepl Gcopes caption in the torn capsion. Inglish, juat
Figure 50: Prompt used for generating Speech-in-Sound captions used in pre-training and further used in generating other QAs.
68


Reviews:
[{"review_id": "LyYSBUH65I", "reviewer": "Reviewer_mZSh", "summary": "This paper proposes a third version of Audio Flamingo, abbreviated as AF3, a large audio language model with new capabilities and datasets to enable it. In particular, capabilities added over AF2 include tackling speech through a unified encoder (AF-Whisper) for speech, music and sound, multi-turn multi audio chat and CoT style thinking. The paper proposes better, larger versions of AudioSkills(-XL) and LongAudio(-XL), and new datasets AF-Chat and AF-think for the respective capabilities. Evaluation is carried out over several benchmarks demonstrating superior performance. AF-Whisper encoder and AudioSkills data contributions are also analyzed to show improved performance.", "questions": "Please also see weaknesses\n\n- Could the authors clarify how the +Think version is different from AF3. In particular,  they mention it is AF3 with thinking prompts; is it different from those incorporated in the AF-think dataset and training curriculum for CoT style thinking? I also ask because, in Table 2, +Think is marked in gray (closed source), but the authors mention in the abstract that they will open source “..all our code, data, and checkpoints upon paper acceptance”. If there is indeed a discrepancy, it will be good to clarify up front what will and won't be released.\n \n- Something I am curious about: on the demo webpage the authors show \"Emergent Audio Understanding\" results. What would happen if the instructions are more general and do not specify one or both of the events in the audio - it one just asked the model to say what is surprising, atypical or unlikely (respectively) about the events in this audio?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- In my view, datasets are the main contribution of this paper, as they are also the main drivers of the capabilities of AF3. In particular, speech-in-sound QA for audio skills and LongSpeech QA for long audio seem to be very timely contributions for audio reasoning benchmarks.\n- AF-whisper is proposed as a unified encoder that does better than the dual encoder approach with one for sounds/music and another for speech.\n- Extensive evaluation over multiple benchmarks\n- Fully open source model and datasets to help advance research on the topic.\n- The paper is well written and easy to follow.\n\nWeaknesses\n- The paper has incremental novelty in technical approach, limited to an additional step in curriculum learning over AF2 and the unified encoder.\n- It would have been interesting to analyze when and how the model fails after training with proposed well-curated datasets.", "was_revised": true, "final_justification": "After the rebuttal, I have a more nuanced understanding of the contributions. It also addressed some of my questions around limitations and model release plan. Increasing my score to 5.", "rebuttal": "Dear Reviewer mZSh,\n\nWe thank you for your thorough review and constructive feedback. We're grateful for your recommendation to accept our paper. We have tried to address each of your concerns point by point.\n\n------\n### Questions:\n\n> Could the authors clarify how the +Think version is different from AF3. In particular, they mention it is AF3 with thinking prompts ....\n\n**Ans.** The *+Think* model refers to the version trained in Stage 3.5 using the AF-Think (and LongAudio XL) dataset, where a) we append a thinking prompt—“Please think and reason before you respond”—to each question in the AQA instances and b) each response has a thought process before arriving at the actual answer to encourage chain-of-thought (CoT) -type reasoning. This differs from the Stage 3 AF3 model, which is trained without such prompting and does not reason by default (the version trained in Stage 3.5 is also able to process longer audios, up to 10 minutes).\n\nThis design is intentional. Unlike domains such as math or code, most audio understanding tasks do not inherently require structured reasoning. We therefore propose a novel *on-demand thinking* mechanism, where the model engages in reasoning only when explicitly prompted by the user. This setup provides greater flexibility and better aligns with real-world usage, where reasoning is not always needed.\n\n**Regarding model release**: *We (will) open-source all models*, including +Think, as stated in the abstract. The use of gray shading for +Think in Table 2 was unintentional. They are actually different shades of grey. Thank you for pointing out that there is confusion, and we will change this in the final version to avoid confusion with closed-source models.\n\n> Something I am curious about: on the demo webpage the authors show \"Emergent Audio Understanding\" results. .....\n\n**Ans.** Thank you for the thoughtful question. In the first demo example, we tested the model with a more general prompt: “What is atypical about the events in the audio?”—without explicitly naming any events. The model responded: “The dog’s barking is atypical as it is accompanied by music.” This demonstrates that the model can identify salient events and reason about their relationships, even when the prompt is underspecified. It suggests an emergent ability to infer context and highlight surprising or unusual aspects in complex audio scenes.\n\n------\n\n### Weaknesses\n\n> The paper has incremental novelty in technical approach, limited to an additional step in curriculum learning over AF2 and the unified encoder.\n\n**Ans.** Thank you for the comment! The paper proposes much more novel findings beyond just a minor curriculum update. We would like to take this chance to highlight the same: The core novelty of our work lies in unlocking and demonstrating advanced capabilities in audio-language models—including long audio comprehension, multi-audio and multi-turn dialogues, on-demand reasoning, and emergent thinking—*capabilities that were largely absent from the literature, and how to enable them remained unknown in open-access models, which did not expose or document such behaviors, just open-sourced weights. As a result, how to effectively elicit and train these capabilities remained an open question, which our work directly addresses.* \n\nOur goal was to build a strong, general-purpose audio-language foundation model, not only by innovating in isolated components, but by designing the entire system holistically—from *novel data creation paradigms* and stage-wise training strategies to the unified encoder architecture -- thereby finding the answers to the unknown and revealing it to the community. While we acknowledge that individual elements may not be entirely novel in isolation, that was not our aim. We believe that audio intelligence still benefits greatly from scaling both data and parameters, and we demonstrate that doing so meaningfully enhances performance. \n\nGoing forward, scaling reinforcement learning and reward-driven training will be a key focus to further improve grounding and reasoning capabilities. Importantly, AF3 is fully open—including code, data, and checkpoints—offering the community the first powerful and transparent model for audio understanding and reasoning.\n\n> It would have been interesting to analyze when and how the model fails after training with proposed well-curated datasets.\n\n**Ans.** Thank you for the question. Our proposed training method has undergone extensive ablations, where we developed a 5-stage training recipe that contributed to AF3’s performance. Key factors include: a) Audio windowing, b) Dataset weighting, c) Stage-wise freezing/unfreezing of model components, d) Hyperparameter tuning, e) Unified audio-text encoder design -- these design decisions make AF3 robust. *However, despite these, AF3 has some known limitations:*\n\n- **Training Data Biases**:  In rare stress-test cases, AF3 produces responses that include spurious or memorized tokens unrelated to the input. These cases often stem from biases in the training data and are a known phenomenon in LLM/MLLM fine-tuning [1]. We plan to explore reinforcement learning (e.g., RLHF) to mitigate such issues in future versions.\n\n- **Limited Instruction Robustness**: AF3 tends to perform best on instructions seen during training, and its generalization to novel or out-of-distribution instructions is still limited. Increasing instructional prompt diversity during training is a key direction for improvement.\n\n- **Hallucinations in Long-Form Generation**: While AF3 performs well for short, factual responses, we observe occasional hallucinations in longer, open-ended outputs. This is a known challenge in MLLMs [2]. We aim to improve audio grounding and incorporate retrieval-augmented generation techniques to enhance long-form fidelity.\n\n- **Multi-Turn Dialogue Degradation**:  AF3 maintains coherent multi-turn conversation up to ~8 turns, after which hallucinations may increase. We plan to investigate dialogue-aware training methods to extend robustness over longer interactions.\n\n- **Coverage Gaps in Specific Capabilities**:  AF3 currently underperforms in: i) Multicultural music understanding, ii) Multilingual speech comprehension, iii) Noisy and overlapping speech scenarios. These gaps primarily arise from the lack of such data in the training corpus. Addressing this through targeted additional data is a priority for future releases.\n\nWe will add these comprehensively in the revised version of our paper.\n\n------\n#### References:\n\n[1] https://proceedings.mlr.press/v235/ghosh24a.html.   \n[2] https://openreview.net/forum?id=3PRvlT8b1R"}, {"review_id": "SAKkRH1WUb", "reviewer": "Reviewer_o9GS", "summary": "This paper introduces Audio Flamingo 3 (AF3), a fully open large audio-language model (LALM) capable of comprehensive reasoning and understanding across speech, sound, and music. AF3 advances the field by introducing several key innovations:\n\n- AF-Whisper, a unified audio encoder trained via a novel captioning-based representation strategy across three audio modalities.\n\n- Support for on-demand thinking (inspired by chain-of-thought), multi-turn multi-audio chat, voice-to-voice interaction, and long-context audio reasoning (up to 10 minutes).\n\n- Four newly curated high-quality datasets---AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat---designed to improve task-specific reasoning.\n\n- A five-stage curriculum-based training framework, progressively increasing context length and task complexity.\n\nAF3 and its chat variant (AF3-Chat) achieve state-of-the-art performance across over 20 benchmarks spanning audio QA, reasoning, ASR, captioning, and multi-modal dialogue. The model is positioned as the most capable and transparent open-source LALM to date.", "questions": "1. How does AF3 handle real-world noisy or overlapping audio in multi-source environments?\n\n2. Were any metrics (e.g., latency or memory) collected on the streaming TTS system in deployment settings?\n\n3. Is there any evidence of LLM hallucination or bias in AF-Think or AF-Chat, and how are such cases handled?\n\n4. Are there plans for multilingual extensions or use of Whisper’s multilingual capability in AF-Whisper?", "limitations": "AF3's performance on noisy or real-world conversational audio is not deeply analyzed despite model scope suggesting applicability.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n- Excellent empirical results across a wide range of tasks, including long-context QA, sound/music reasoning, and speech understanding.\n\n- Significant open-source release, including trained models, code, datasets (13M+ audio-caption pairs), and demo interface.\n\n- Well-grounded training methodology, including dataset-specific prompting, skill-specific QA generation, and multi-modal data balancing as detailed in the appendix.\n\n- Streaming TTS design is thoroughly described in the appendix, with a detailed architectural diagram, codec specs, training pipeline, and latency optimization.\n\n- Rich appendix clarifies dataset composition (e.g., Table 7 with hours and pairs per dataset), training hyperparameters (Table 8), and prompting strategies (Figures 11–46), enhancing reproducibility.\n\n- Evaluation on MMAR and AIR-Bench (Appendix Table 9) further strengthens the claim of general-purpose audio reasoning ability.\n\n**Weaknesses:**\n\n- Closed LLMs used for synthetic data generation (GPT-4.1, Gemini) reduce full replicability despite an open model release.\n\n- Only English supported, limiting multilingual utility, though data and tools could support extensions.\n\n- AF-Think's effect on model robustness is described but ablations are shallow.", "was_revised": true, "final_justification": "The work is technically solid with comprehensive evaluation; All my concerns have been appropriately addressed.", "rebuttal": "Dear Reviewer o9GS,\n\nWe thank you for your thorough review and constructive feedback. We're grateful for your recommendation to strongly accept our paper. We have tried to address each of your concerns point by point.\n\n-----\n### Weaknesses\n\n> Closed LLMs used for synthetic data generation (GPT-4.1, Gemini) reduce full replicability despite an open model release.\n\n**Ans.** Thank you for the question. To ensure full reproducibility, we will release all datasets used to train the model—including AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat—alongside the model checkpoints, training code, and data processing scripts, as stated in the paper. While we used closed LLMs (GPT-4.1, Gemini) for some synthetic data, we are committed to releasing the final generated datasets to ensure end-to-end applicability.\n\n> Only English supported, limiting multilingual utility, though data and tools could support extensions.\n> Are there plans for multilingual extensions or use of Whisper’s multilingual capability in AF-Whisper?\n\n**Ans.** Thank you for the suggestion. We fully acknowledge this limitation and have noted it in the Limitations section of our paper. Expanding to multilingual speech understanding is a key goal for the next version of Audio Flamingo. Interestingly, despite not including any multilingual speech data during training, AF3 exhibits some emergent multilingual capabilities, which we attribute to the inherent strengths of Whisper and Qwen2—both of which are multilingual models. This further motivates our plan to incorporate explicit multilingual supervision in future work.\n\nMultilingual extensions would only require adding multilingual data to the mixture, and we plan to follow the same process as AF-Whisper and AF3 training with appropriate weights to multilingual data in the mixture.\n\n> AF-Think's effect on model robustness is described but ablations are shallow.\n\n**Ans.** Thank you for the question. While we included a basic ablation in the main paper, we provide additional clarification below to address the depth of AF-Think’s effect on model robustness, answering the most important questions:\n\n- *Does the model always generate long, CoT-style answers after AF-Think training?*: No. The model only engages in reasoning when explicitly prompted (e.g., with the thinking prompt shown in Fig. 3). Without this, it defaults to concise, task-appropriate responses. This demonstrates robust control over reasoning behavior.\n\n- *Does training on AF-Think hurt foundational performance?* No. We evaluate Audio Flamingo 3 on MMAU-test without the thinking prompt after Stage 3.5 (i.e., post AF-Think training) and observe no drop in performance, achieving scores of 73.11 and 73.0 on MMAU-test and MMAU-test-mini, respectively.\n\n- *Does thinking mode always improve performance?* Not always. For simpler tasks like ASR or sound event classification, thinking mode may not help—and we do not expect it to (e.g, we maintain an almost similar score of 78.9 in NSynth Instrument Classification and 1.92  on SPGI ASR). The thinking mode is designed to enhance tasks that require multi-step reasoning, such as those measured by MMAU and MMAR, where it shows clear gains.\n\n------\n\n### Questions\n\n> How does AF3 handle real-world noisy or overlapping audio in multi-source environments?\n\n**Ans.** AF3 is able to handle real-world noisy and overlapping audio to *some extent*. We attribute this to training examples from our Speech-in-Sound Captions and QA data, which include speech with naturally occurring background noise, interruptions, and mild cross-talk from AudioSet and YouTube8M.\n\nHowever, we do not expect AF3 to perform well in extreme noise or dense overlapping speech scenarios, as it was not explicitly trained on such data. Prior work has shown that robust performance in these settings requires specialized training data and objectives [1]. **Handling multi-speaker, multi-source, and noisy audio is a key focus for the next version of Audio Flamingo.** We plan to include datasets such as Chime, LibriheavyMix, etc., in training and evaluate on benchmarks designed for these challenging real-world conditions.\n\n> Were any metrics (e.g., latency or memory) collected on the streaming TTS system in deployment settings?\n\n**Ans.** Thank you for the question. Yes, we report latency metrics in Table 3. \n\nSpecifically, for a 10-second audio generation on an A100 GPU:\n- Text-to-audio token generation: 5.94 seconds\n- Waveform synthesis: 0.02 seconds\n- Total time: 6.68 seconds for a 10-second audio clip\n\nIn a streaming TTS setup, we observe:\n- Time-to-first-token: 0.15 seconds\n- Inter-token latency: 0.06 seconds (both including synthesis)\n\nThese metrics demonstrate that AF3-Chat can support low-latency, real-time text-to-speech generation suitable for deployment settings.\n\n\n> Is there any evidence of LLM hallucination or bias in AF-Think or AF-Chat, and how are such cases handled?\n\n**Ans.** Yes, we did observe instances of LLM hallucination and bias during the creation of AF-Think and AF-Chat. To mitigate this, we implemented the following safeguards:\n\n- In AF-Think, some responses in the thought process referred incorrectly to elements like “the caption” or “meta-data,” which were not part of the input. We used keyword-based filtering to remove such hallucinated examples.\n\n- In AF-Chat, early generations occasionally reflected hallucinations stemming from inaccurate captions or transcripts. We addressed this through iterative caption refinement, which significantly improved data quality.\n\nWe will add these details to the final version of our paper. That said, we acknowledge that the datasets are not entirely noise-free. *For future versions of Audio Flamingo, a key focus will be on developing and integrating automated data filtering techniques to further reduce hallucinations and improve supervision quality at scale.*\n\n\n------\n\n### References\n\n[1] He, Xinlu, and Jacob Whitehill. \"Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio.\" arXiv preprint arXiv:2505.10975 (2025)."}, {"review_id": "skzNLCwRZL", "reviewer": "Reviewer_7pz4", "summary": "The paper proposes Audio Flamingo 3 (AF3), an audio-language model that unifies speech, sounds, and music understanding within a single framework. AF3 combines a AF-Whisper encoder with a lightweight “think” module to support tasks from transcription and captioning to expert-level reasoning, multi-turn dialogue, and voice-to-voice interaction. The authors curate four big datasets (AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat) and use a five-stage curriculum learning to progressively enhance encoder alignment, reasoning, context extension, and conversational fine-tuning. Results show taht AF3 achieves state-of-the-art performance on over 20 audio understanding and reasoning tasks.", "questions": "1. Can the model start reasoning without explicit prompting? \n2. What are AF3’s latency (RTF) and total parameter count? It doesn’t seem to run in real time, right?\n3. How do the cut-offs when stitching segments affect performance? Have you tested different overlap or boundary settings?\n4. The voice-chat sounds robotic—have you explored ways to capture natural prosody and interruptions?\n5. What are the architecture details of the \"audio adaptor\" and the \"streaming TTS\"?", "limitations": "Although section 7 mentions limitations in the title, a dedicated limitations section is missing. The authors should include such a section and provide a systematic overview of all known constraints - including language coverage, compute requirements, pipeline latency, and prosody limitations - alongside an explicit discussion of potential societal impacts (e.g. demographic biases, privacy risks).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Model and data will become publicly available.\n2. A single encoder (AF-Whisper) handles speech, sounds, and music.\n3. State-of-the-art performance on 20+ benchmarks \n\nWeaknesses\n\n1. English-only training.\n2. High computational cost. Training and inference need substantial compute.\n3. The cascaded voice-chat pipeline strips out prosodic nuances - so conversations feel flat and robotic, with no natural interruptions or emphasis. \n4. Method-wise the contributions are incremental.", "was_revised": true, "final_justification": "I thank the authors for their detailed response and for addressing my concerns point by point. The clarifications on reasoning behavior, latency, overlap handling, TTS design, and the distinction between foundational and thinking checkpoints were very helpful.\n\nSome issues remain such as English-only training, high computational requirements and the lack of natural prosody in generated speech. Nonetheless, I find the overall contribution significant: the paper demonstrates a strong, integrated audio-language framework with state-of-the-art performance across a wide range of tasks.\n\nIn summary, despite some limitations, this is a technically solid and impactful paper that advances the field of general-purpose audio-language models, and I maintain my original score.", "rebuttal": "Dear Reviewer 7pz4,\n\nWe thank you for your thorough review and constructive feedback. We're grateful for your recommendation to accept our paper and address each of your concerns point by point.\n\n-----\n\n### Questions:\n\n> Can the model start reasoning without explicit prompting?\n\n**Ans.** Thank you for the question! No, AF3 does not reason without explicit prompting. *This is by design*—our training methodology and the AF-Think dataset were created to support a novel on-demand thinking paradigm. Unlike domains such as math or coding, many audio tasks (e.g., ASR, captioning) do not inherently require reasoning. Therefore, the model is trained to reason only when explicitly instructed to do so, e.g., via a thinking prompt like “Please think and reason before you respond.”\n\nWe also take this opportunity to clarify the distinction between our two checkpoints:\n\n- AF3 foundational checkpoint (after Stage 3): Suited for core audio tasks such as classification, captioning, and ASR.\n\n- AF3 thinking checkpoint (after Stage 3.5): Trained with an additional LoRA adapter using AF-Think, targeted at tasks requiring deliberate reasoning and long audio comprehension.\n\nImportantly, the thinking checkpoint also behaves like the foundational model when the reasoning prompt is absent. However, due to its exposure to reasoning-style supervision, it may be biased toward that style and perform slightly worse on basic tasks—this is why we keep the checkpoints separate (as also noted in Table 7 in the Appendix, where Stage 3 data is not fully reused in Stage 3.5). **We will open-source both checkpoints upon acceptance.**\n\n> What are AF3’s latency (RTF) and total parameter count? It doesn’t seem to run in real time, right?\n\n**Ans.** AF3's total parameter count is 8.2B (Qwen2 LLM + Whisper Encoder), similar to most existing LALMs (Qwen2-Audio, GAMA, LTU, etc.) AF3's latency (RTF) is 0.14 computed on the test set of LibriSpeech-clean. \n\nReal-time performance depends heavily on the inference infrastructure and GPU used. While AF3 supports real-time low-latency streaming outputs via TTS (see Appendix Section I for details), it does not yet support streaming audio inputs, which, together with the model size, may limit its real-time applicability, particularly on edge devices (real-time can still be achieved on larger GPU settings). *That said, most existing LALMs also lack real-time input support on edge devices.* Our current focus is on achieving general audio intelligence through scaling data and model capacity. Real-time performance is an important future goal, and we plan to release smaller, efficient versions of the next AF versions to support deployment on edge devices.\n\n> How do the cut-offs when stitching segments affect performance? Have you tested different overlap or boundary settings?\n\n**Ans.** Thank you for the question. *Our initial small-scale experiments showed a drop in performance when segments had boundary overlap.* Unfortunately, we do not have full-scale results across different boundary settings, as we were constrained by compute resources. However, since our windowing strategy is inspired by Audio Flamingo 2, we’d like to share results from AF2 comparing different overlap settings:\n\n| Model | CMM | MMAU-Sound | MuchoMusic |\n|----------|----------|----------|----------|\n| 4 sec overlap   | 55.0     |  45.6   | 31.0     |\n| 2 sec overlap   | 73.0     | 57.8     | 43.8     |\n| no overlap   | 82.0     | 65.1    | 56.5     |\n\nWe observed that overlapping segments in long and structural audios (like music) may confuse the model, causing it to misinterpret boundaries and lose coherence. This likely happens because, in the early stages of training, the model sees mostly short, single-segment audio, learning to treat each segment as a coherent unit. The concept of overlapping segments is only introduced later (during long audio fine-tuning), making it harder for the model to generalize. Additionally, for audios like music where the structure develops over time, the overlapping model confuses the model by losing coherence. \n\n> The voice-chat sounds robotic—have you explored ways to capture natural prosody and interruptions?\n\n**Ans.** Thank you for the question. Capturing natural prosody and interruptions was beyond the scope of Audio Flamingo 3, as it requires training a dedicated TTS model on expressive speech data (or just simply integrating external TTS systems into AF3 as a cascaded setup).\n\nOur current TTS implementation serves as a functional demonstration, showing that low-latency voice-out is achievable with LALMs in a simple and unified manner. *With appropriate data, it is relatively simple to achieve this (although most of such data is not open and would defy our open claim if we used)*. As noted in the caption of Table 1, voice-out is not the core novelty of AF3—but our streaming TTS implementation itself is novel, as described in Section I of the Appendix.\n\nIt's also worth noting that existing voice-to-voice systems (e.g., duplex-style models) primarily focus on voice interaction, not on audio intelligence, which we define as holistic perception, understanding, and reasoning over audio. Achieving audio intelligence with natural prosody remains an open challenge and is a key priority for future versions of Audio Flamingo.\n\n> What are the architecture details of the \"audio adaptor\" and the \"streaming TTS\"?\n\n**Ans.** Thank you for the question. The audio adaptor consists of a two-layer MLP with a GeLU activation in between. We will include these architectural details in the revised version of the paper. Details of the streaming TTS setup are provided in Appendix I, Figure 10. It outlines our lightweight, low-latency design used for generating audio in real-time from model outputs.\n\n------\n\n### Weaknesses\n\n> English-only training.\n\n**Ans.** We fully acknowledge this limitation and have noted it in the Limitations section of our paper. Expanding to multilingual speech understanding is a key goal for the next version of Audio Flamingo. Interestingly, despite not including any multilingual speech data during training, AF3 exhibits some emergent multilingual capabilities, which we attribute to the inherent strengths of Whisper and Qwen2—both of which are multilingual models. \n\n> High computational cost. Training and inference need substantial compute.\n\n**Ans.** Thank you for the question. We acknowledge the high computational cost, which is common to most foundational models. AF3 is designed as a general-purpose, intelligent audio-language model capable of handling complex reasoning over diverse audio inputs. That said, AF3 is compute-efficient relative to its performance:\n\n- Our intelligent data curation and training design allow AF3 to outperform more powerful models like Qwen2-Audio [1], Qwen2-Omni [2], and LTU-AS, while using significantly less data.\n\n- As a result, AF3 requires less compute for training compared to these models.\n\nAt inference time, AF3 has a comparable cost to other leading models in the literature, including Qwen2-Audio, Qwen2-Omni, and LTU-AS.\n\n> The cascaded voice-chat pipeline strips out prosodic nuances - so conversations feel flat and robotic, with no natural interruptions or emphasis.\n\n**Ans.** Thank you for the comment. We addressed related concerns earlier in the rebuttal, but we’d like to add a specific clarification about the cascaded voice-chat pipeline.\n\nWhile end-to-end voice-to-voice models (e.g., using codec inputs/outputs) may better preserve prosodic nuances, implementing such a system in AF3 presents challenges. In our initial experiments, training AF3 with codec-in/out led to catastrophic forgetting, causing the model to lose its core audio intelligence—its ability to holistically perceive and reason over diverse audio types.\n\nWe acknowledge that models like Moshi [3] handle natural prosody well, but they are not designed for audio intelligence. These models focus on translating textual conversations into speech using LLM capabilities, without reasoning over the audio content itself (e.g., sounds, background noise, speaker variation). *Bringing together natural prosody and deep audio understanding remains an open challenge, and a key research direction for future versions of Audio Flamingo.*\n\n> Method-wise the contributions are incremental.\n\n**Ans.** Thank you for the comment! We would like to take this chance to highlight some contributions: The core novelty of our work lies in unlocking and demonstrating advanced capabilities in audio-language models—including long audio comprehension, multi-audio and multi-turn dialogues, on-demand reasoning, and emergent thinking—capabilities that were largely absent from the literature, and how to enable them remained unknown in open-access models, which did not expose or document such behaviors, just open-sourced weights. As a result, how to effectively elicit and train these capabilities remained an open question, which our work directly addresses.\n\nOur goal was to build a strong, general-purpose audio-language foundation model, not only by innovating in isolated components, but by designing the entire system holistically—from novel data creation paradigms and stage-wise training strategies to the unified encoder architecture -- thereby finding the answers to the unknown and revealing it to the community. While we acknowledge that individual elements may not be entirely novel in isolation, that was not our aim. We believe that audio intelligence still benefits greatly from scaling both data and parameters, and we demonstrate that doing so meaningfully enhances performance.\n\n------\n\n> Although section 7 mentions limitations in the title, a dedicated limitations section is missing.\n\n**Ans.** Thank you for the comment. We will add this to the final version of our paper, describing the limitations broadly. \n\n------\n### References\n[1] https://arxiv.org/abs/2407.10759.  \n[2] https://arxiv.org/abs/2503.20215.  \n[3] https://arxiv.org/abs/2410.00037."}, {"review_id": "Fq1fCWk4ZR", "reviewer": "Reviewer_P65i", "summary": "The authors present a large audio-language model for general audio understanding across speech, sound, and music. The core architectural contribution is AF-Whisper, a unified audio encoder that can process various audio types within a single module. This encoder is combined with a Qwen-2.5-7B LLM and a streaming TTS module to form the overall architecture. The authors also introduce a multi-stage training recipe using a self-curated dataset across multiple tasks, including general/long-form audio understanding, conversational audio, audio reasoning tasks, and voice-in/voice-out scenarios. The model is evaluated on over 20 standard audio benchmarks and demonstrates strong performance. The authors also commit to open-sourcing the code, checkpoints, and data (with partial data already provided in the submission) upon paper acceptance.", "questions": "Please see the weakness above; the reviewer has posed several actionable questions.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Commitment to open-source: If fully released, the code, model, and dataset would be a valuable contribution to the community given that there are limited fully open-source ones in this field.\n2. Unified audio encoder (AF-Whisper): The introduction of a single encoder that works well on diverse audio inputs simplifies the overall system design and is a great contribution.\n3. Strong performance with 7B backbone: The model achieves competitive results across a broad range of benchmarks despite using a relatively compact 7B language model.\n\nWeaknesses\n1. Experimental Setup\n\t- Missing baselines (e.g., GPT-4o-audio): The authors note (line 279) that GPT-4o and Gemini were not included due to rate limits. While understandable, it would still be valuable to report results on a small subset of tasks, especially since GPT-4o-audio is currently a key reference model in the field. \n\t- Audio Captioning (Table 2): This task appears to show the smallest gains. It would help to understand why results are relatively weak here. Also, why aren’t results for Audio Flamingo 2 on AudioCaps reported? More recent LLM-based evaluation metrics [1, 2] could improve the analysis here.\n\t- Evaluation model bias: Many results in Table 2 (reasoning tasks) and Table 3 are evaluated using GPT-4o. Can the authors add comparisons using other LLMs to rule out evaluation bias?\n\n2. Other Points\n\t- Ablation details (Table 4): (1) It seems that AF-Whisper is based on Whisper-Large-v3 (1.5B). Were parameter sizes controlled for in the ablation when comparing dual-encoder setups (e.g., with CLAP)? (2) The line “+ w/o AF-Whisper” is unclear. Does it mean with or without the encoder?\n\t- Licensing issue: The authors mention that licensing information is provided in Appendix G, but the appendix currently lacks the promised details, especially for those that the authors collected.\n\t- Context length and token cost: It would be helpful to specify the maximum context length supported by the model (is it inherited from Qwen-2.5-7B as 128K?) and how many tokens per second are required for different audio types.\n\n[1] CLAIR-A: Leveraging Large Language Models to Judge Audio Captions\n\n[2] MACE: Leveraging Audio for Evaluating Audio Captioning Systems", "was_revised": true, "final_justification": "Thanks for the clarification. The reviewer was satisfied with additional explanations, clarifications, and experiments. Therefore, the score was maintained as Accept (5.0).", "rebuttal": "Dear Reviewer P65i,\n\nWe thank you for your thorough review and constructive feedback. We're grateful for your recommendation to accept our paper. We have tried to address each of your concerns point by point.\n\n------\n\n### Weaknesses\n\n> Missing baselines (e.g., GPT-4o-audio): The authors note (line 279) that GPT-4o and Gemini were not included due to rate limits.\n\n**Ans.** Thank you for the question! We would like to clarify a potential misunderstanding: we do compare AF3 with GPT-4o-audio and all Gemini versions across all benchmarks—classification, captioning, and QA—except for ASR benchmarks. The reason for this exception is that ASR datasets are large and often exceed API rate limits for these closed models. That said, to address this concern during the rebuttal, we ran limited-scale evaluations on four ASR benchmarks, and present the results below:\n\n| Model | Libri (clean) | SPGI | Vox |\n|----------|----------|----------|----------|\n| GPT-4o-audio    | 2.90     | 4.91    | 8.42     |\n| Gemini 2.5 Pro   |  2.04   | 4.30     | 9.79     |\n| Audio Flamingo 3    | **1.51**    | **1.86**     | **5.66**     |\n\nWe will include these results and clarifications in the final version of the paper.\n\n> Audio Captioning (Table 2): This task appears to show the smallest gains. It would help to understand why results are relatively weak here.\n\n**Ans.** Thank you for the question! AF3 shows relatively smaller gains on AudioCaps and Clotho because of inherent limitations in current captioning benchmarks and evaluation protocols. Below, we outline key factors:\n\n- Captioning is a perception task—not a reasoning task: Audio captioning primarily evaluates descriptive understanding, not complex reasoning. As a result, foundation models like AF3, which are optimized for reasoning and general audio intelligence, show modest improvements in this task compared to reasoning-heavy benchmarks.\n\n- Limitations of current evaluation metrics: Metrics like BLEU, CIDEr, and SPICE are biased toward style matching, not semantic correctness. For example: Clotho may label a sound as “a forest,” -- while AudioCaps might describe it as “animals among rustling leaves.” -- Both are correct, but style-biased metrics reward only the phrasing closer to the reference.\n\nSince AF3 is trained on diverse real-world audio, its captions are often semantically accurate but stylistically different, leading to under-rewarded performance. *We intentionally did not fine-tune to match benchmark-specific styles, as that would constitute benchmark hacking (e.g., training with prompts like “in Clotho style”).*\n\n> Why no Audio Flamingo 2 scores on AudioCaps?\n\n**Ans.** AF2 was evaluated zero-shot and was not trained on AudioCaps. It reports a baseline CIDEr score of 0.58. In contrast, models like Pengi are trained on AudioCaps, making them fairer points of comparison in Table 2.\n\n> LLM-based evaluation metrics for Audio Captioning\nThank you for pointing us to [1, 2]. We agree that these can provide better insights into caption quality. Below, we provide results using MACE [2]:\n\n| Model | AudioCaps | Clotho |\n|----------|----------|----------|\n| Gemini 2.5 Pro   |  0.59    | 0.56     | \n| Audio Flamingo 2    | 0.47     | 0.42     |\n| Qwen2.5 Omni    | 0.51     | 0.55    |\n| Audio Flamingo 3    | **0.59**     | **0.60**    | \n\n> Evaluation model bias\n\n**Ans.** Thank you for the question! To address concerns around evaluation model bias, we evaluated AF3 and baselines using additional LLMs beyond GPT-4o. Specifically, we report results, using the same prompt, with Qwen2.5-7B-Instruct:\n\n| Model | LibriSQA | LongAudioBench |\n|----------|----------|----------|\n| Gemini 2.5 Pro   | 8.0    | 68.5     | \n| Audio Flamingo 3    | 8.1  | 79.1    |\n\nAnd Llama-3.3-70B-Instruct below:\n\n| Model | LibriSQA | LongAudioBench |\n|----------|----------|----------|\n| Gemini 2.5 Pro   | 9.2    | 72.3     | \n| Audio Flamingo 3    | 9.2  | 78.6    |\n\n> Ablation details (Table 4): (1) It seems that AF-Whisper is based on Whisper-Large-v3 (1.5B). Were parameter sizes controlled for in the ablation when comparing dual-encoder setups (e.g., with CLAP)? (2) The line “+ w/o AF-Whisper” is unclear. Does it mean with or without the encoder?\n\n**Ans.** Thank you for the question!\n\n- On parameter control in dual-encoder setups: We did not perform strict parameter control in the ablation. Our intent was to demonstrate that a dual-encoder setup (e.g., CLAP + Whisper-Large-v3) inherently increases model size and leads to less stable training, without a proportional performance gain. To the best of our knowledge, there are no standard methods in the literature for parameter-matched comparisons in dual-encoder architectures with heterogeneous backbones. Our goal is to highlight the advantages of a unified encoder, as introduced via AF-Whisper, which handles speech, sound, and music in a single, efficient module.\n\n- On the line “+w/o AF-Whisper” in Table 4: This refers to the dual-encoder baseline, where we use CLAP for non-speech audio (sound/music) and Whisper-Large-v3 for speech. This setup is used to compare against the AF-Whisper unified encoder, which processes all audio modalities. We will clarify this in the final version of the paper.\n\n> Licensing issue\n\nWe apologize for the oversight—Appendix G will be updated in the final version to include full details on licensing. All datasets introduced in this work will be released under a non-commercial, research-only license. This includes the datasets we collected and curated specifically for training AF3. This is due to the licensing terms and data sources used; all assets will be distributed under a research-only license upon acceptance. (Please note that by “fully open,” we mean that the model weights, training data, and code will be publicly released, along with transparent documentation of the training methodology. We do want to strictly abide by the licenses and terms of the datasets provided by the community for training, for our release, most of which are research-only.)\n\n> Context length and token cost\n\nThank you for the question! AF-Whisper encodes a 30-second audio segment into 750 tokens, and our model is trained with a context length of 16K tokens, corresponding to up to 10 minutes of audio, as stated in the paper.\n\nHowever, as correctly noted, the model inherits a maximum context length of 128K tokens from Qwen-2.5-7B. That said, we do not guarantee performance beyond 16K, as: i) The model was not explicitly trained or optimized for longer contexts. ii) There are currently no available benchmarks to evaluate such long audio comprehensively. We will clarify these details in the final version of the paper."}]

Author rebuttal:

Meta-review: This paper presents Audio Flamingo 3 (AF3), a fully open large audio-language model with a unified encoder, new large-scale datasets, and a staged curriculum enabling long-context reasoning, multi-turn/multi-audio dialogue, and voice-to-voice interaction. AF3 achieves state-of-the-art results on 20+ benchmarks and commits to releasing all models, code, and data, making it a significant community resource. While some methodological contributions are incremental and limitations remain (English-only training, reliance on closed LLMs for synthetic data, limited prosody in TTS, and incomplete robustness analysis), the work was judged technically solid, well-evaluated, and impactful. Reviewers’ concerns were addressed convincingly during rebuttal, with clarifications and new experiments provided. Overall, I recommend acceptance (spotlight): the openness, breadth of capabilities, and strong empirical results make AF3 a valuable and timely contribution to audio-language modeling.
Decision: Accept (spotlight)