Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
Authors: Sreyan Ghosh
Zhifeng Kong
Sonal Kumar
S Sakshi
Jaehyeon Kim
Wei Ping
Rafael Valle
Dinesh Manocha
Bryan Catanzaro
Venue: ICML 2025 poster (2025)
Keywords: audio
sound
large audio-language model
Primary area: deep_learning->large_language_models
TL;DR: A parameter-efficient Audio Language Model that achieves sota on several audio understanding and reasoning benchmarks.

Abstract: Understanding and reasoning over non-speech sounds and music are crucial for both humans and AI agents to interact effectively with their environments. In this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM) with advanced audio understanding and reasoning capabilities. AF2 leverages (i) a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves state-of-the-art performance with only a 3B parameter small language model, surpassing large open-source and proprietary models across over 20 benchmarks. Next, for the first time, we extend audio understanding to long audio segments (30 secs to 5 mins) and propose LongAudio, a large and novel dataset for training ALMs on long audio captioning and question-answering tasks. Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed LongAudioBench, an expert-annotated benchmark for evaluating ALMs on long audio understanding capabilities. We conduct extensive ablation studies to confirm the efficacy of our approach.

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
Sreyan Ghosh 1 Zhifeng Kong'
Sonal Kumar S Sakshi" Jaehyeon Kim
Wei Ping'
Rafael Valle
Dinesh 2anocha² Bryan Catanzaro
Abstract
AudisFlamingo
Enteilment AudioCape
Understanding and reasoning over non-speech sounds and music are crucial for both humans and Al agents interact effectively with their environments. this paper, we introduce Audio Flamingo 2 (AF2), Audio-Language Model (ALM) with advanced audio understanding and reasoning capabilities. AF2 leverages (i) a cus- tom CLAP model, (ii) synthetic Audio QA data for fine- -ee--griindd audio reasoning, and (iii) a multi- stage curriculum learning strategy. AF2 achieves state-of-the-at performance with only 3B pa- rameter small language model, surpassing large open-souree and proprietary models across over 20 benchmarks. Next, for the first time, we ex- tend audio understanding to long audio segments (30 secs to 5 mins) and propose LongAudio, a large and novel dataset for training AL.S on long audio captioning and question-answentte tasks. Fine-tuning AF2 LongAudio leads exceptional performance gAudioBench, an expert-annotated benchmark for evaluating ALMs on long audio understand- ing capabilities. We conduct extensive ablation studies to confirm the efficacy of our approach.
Mediey-Solos DB
Entaliment
MMAU Music
ClothoAQA
MNAUU
AudioCaps
40%
60%
025
200
MuchaMusic
CREMA-D
OpenAQA
LongAudioßench (owrs)
Figure 1: Audio Flamingo versus previous SOTA ALMs dio understanding and reasoning benchmarks (values normalized), AF2 outperforms all baselines and has smaller model footprints.
impaired. While Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities through lan- guage, extending these systems to comprehend audio is key to building intelligent systems capable of reasoning with contextual auditory cues (Kong et al., 2024). Verbal speech, inherently tied to language, benefits significantly from (L)LM advancements (Watanabe et al., 2018; Chen et al., 2024a); however, the potential enhance perception and reasoning non-verbal audio remains largely under- explored (Ghosh et al., 2024c), Audio-Language Models (ALMs) extend language models with audio understand- ing capabilities. Contrastive Language- Audio Pre-training (CLAP) (Elizalde et al., 2023a) was among the first encoder- only ALMs to bridge audio and language with contrastive learning. Building on this, subsequent efforts introduced Large ALMs (LALMs), which integrate audio encoders with pre-trained decoder-based LLMs, enabling open-ended Audio Question Answering (AQA) and free-form response generation (Gong et al., 2021; Tang et al., 2024; Ghosh et al., 2024c). However, despite these developments, even the most advanced LALMs continue to underperform on expert- level reasoning tasks compared foundational tasks like
proposed Lon-
1. Introduction
Understanding non-speech sounds, non-verbal speech, and music (collectively referred to "audio" in this paper) is essential for real-world applications such detecting anomalies in industrial environments, recognizing emo- tional cues, and improving assistive technologies for the
Work done during an internship at NVIDIA. 'VVIDIA, Santa Clara, CA, USA 'University of Maryland, College Park, MD, USA. Correspondence to: Sreyan Ghosh <sreyang edu. Zhifeng Kong <zkong @ com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
event classification. For example, Gemini-1.5-Pro (Team et al., 2024), one of the most advanced models, achieves only 54.4% and 48.5% on the MMAU sound and music subsets (Sakshi et al., 2024), benchmark for evaluating expert-level audio reasoning. This underscores the chal- lenges of improving an LLM's ability to understand and reason over audio, which we attribute to the lack of high- quality training data and robust audio representations.
2. Related Work
Audio-Language Models: ALMs can be classified into two broad categories: 1) Encoder-only ALMs: Encoder- only ALMs are a class of Multi-Modal Language Mod- els (MILM) that learn shared space between the audio and language modalities with an encoder-only LM and an audio encoder. CLAP, pioneering encoder-based ALM inspired by CLIP (Radford et al., 2021), showed state- of-the-art performance on audio-language tasks like re- trieval, zero-shot classification, etc. Following this, sev- eral attempts have been made to improve CLAP by scaling data (Wu et al., 2022b), incorporating additional training ob- jectives (Ghosh et al., 2024d), or with synthetic data (Ghosh et al., 2025). Other notable works include Wav2CLIP (Wu et al., 2022a), AudioClip (Guzhov et al., 2022) and CoL- LAT (Silva et al., 2023). 2) Decoder-based ALMs: With the advent of L.MMs, Pengi (Deshmukh et al., 2023), a pi- oneering decoder- based ALM, achieved SOTA results on variety of audio classification tasks. Following Pengi, large number of Large ALMs were introduced, includ- ing fully open-source models like LTU (Gong et al., 2024), LTU-AS (Gong et al., 2023a), SALMONN (Tang et al., 2024), AudioGPT (Huang et al., 2024), GAMA (Ghosh et al., 2024c), Audio Flamingo (Kong et al., 2024), and open-access models like Qwen-Audio (Chu et al., 2023) and Qwen-2-Audio (Chu et al., 2024). A majority of the advances have focused on scaling model size and datasets, with very little advancements in data quality or audio en- coder representations. This has eventually translated to performance advancing on foundational tasks like classifi- cation and captioning but madee-performing on expert-level reasoning, the skill required for advancing towards Artificial General Intelligence (AGI) (Morris et al., 2024).
Main Contributions: In this paper, we propose Audio Flamingo 2 (AF2), a parameter-efficient ALM that com- bines 3B- -a-rammmeter small decoder LM with a 203M- parameter audio encoder, achieving state-of-the-art audio understanding and reasoning capabilities. There are three major innovations in our method: (1) Data: Recent stud- ies highlight that improving data quality can rival or even surpass the performance gains achieved by scaling compute and model size (Abdin et al., 2024). To this end, we propose AudioSkills (Section 4.1), large-scale, skill-specific AQA training dataset featuring complex, reasoning- iintoniinnittesiveeeeeeee ques- tions paired with each audio. We design questions that target seven distinct skills, with the primary aim of improving fine- grained reasoning capabilities in ALMs. (2) Audio Encod- ing: We propose AF-CLAP (Section 3.1), where we scale CLAP training to over 8M audio-caption pairs, incorporat- ing synthetic data, and propose an improved contrastive loss for better representational quality and robustness. (3) Training Strategy: We propose a novel 3-stage curriculum training strategy (Section 5) for improved performance.
Additionally, for the first time, we extend audio understand- ing to long audios, moving beyond 30-second clips to audios lasting up to 5 minutes. To enable the model to comprehend and reason over long audio, we introduce LongAudio. a novel dataset comprising over 260k carefully curated AQA instances with audios ranging from 30 seconds to 5 minutes. LudgAudio spans 10+ audio categories and supports 6 tasks, including captioning and 5 reasoning- we introduce LongAuudiBennch, an expert-annotated bench- mark for evaluating ALMs on long audio understanding. AF2, trained on LongAudio, significantly outperforms our baseline. In summary, our main contributions are:
Long Audio Understanding: Current ALMs are limited to perceiving at most 30 seconds of audio (Chu et al., 2024), with a majority confined to at most 10 seconds (Ghosh et al., 2024c; Gong et al., 2024). This can be attributed to the fact that these models employ audio encoders that support only short audio encoding and datasets where majority of the data is only at most 10 seconds. While long speech has received some attention (Chiu et al., 2019; Arumugam et al., 2023), and the field on long video understanding has seen ad advancements recently (Weng et al., 2025; Xue et al., 2024), to the best of our knowledge, no current work or datasets exist for long audio understanding and reasoning.
QA tasks. Next,
1. We present Audio Flamingo 2, a SOTA ALM with ad- vanced audio understanding and reasoning capabilities.
2. We propose innovations in data generation, architecture design, representation learning, and training strategies.
3. We introduce the long audio understanding task and create dedicated training and evaluation datasets to drive progress in this area.
3. Audio Flamingo 2 Architecture
Fig. summarizes the AF2 architecture, consisting of four primary components: i) AF-CLAP: CLAP-based audio encoder with sliding window feature extraction, ii) audio representation transformation layers for additional capacity, iii a decoder-only language model, and iv) gated cross- attention (XATTN-Dense) layers for audio conditioning.
4. Audio Flamingo 2 outperforms larger and proprictary LALMS across over 20 benchmarks, despite being smaller and trained exclusively on public datasets.
5. We conduct systematic ablation studies to demonstrate the impact of each design choice.
2

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
The unssul agect dee combisation of the misk and the mecturiions
2) Pre-training
23 Fine-innigg
2) Loug Fine-tuniee
Gated XATTN-Dense
LM Block
Gated XATTN-Dense
LM Block
111 audioo music and hammur beating
AF-CIAP
Transformation Layers
Gated XATTN-Dense
ge4
Stiding Window
What umasual labout this audio?
Tokenizer
Figure 2: Overview of Audio Flamingo 2's cross- aross-attention architecture and three- stage curriculum training.
3.1. AF-CLAP Audio Encoder
ensure diversity and reduce redundancy, we filter out seg- ments with audio-visual similarity above a threshold p. We then prompt GPT-40(2024-05-13) (Prompts 23 and 22) to generate audio-centric captions that exclude visual attributes and emphasize sound events. Using this approach, we col- lect approximately 5.5M new audio-caption pairs, including 4M from unlabeled short audios and 1.5M from long videos. Detailed dataset statistics are provided in Table 9.
CLAP, pre-trained with contrastive loss on audio-caption pairs, shows strong audio understanding and natural lan- guage alignment (Wu et al., 2022b; Elizalde et al., 2023a). These make CLAP a suitable choice for building ALMs. However, CLAP is less favored in prior works (Tang al., 2023; Chu et al., 2023) due to its under-performance com- pared to SSL pre-trained audio encoders (Gong et al., 2023b; LI et al., 2024; Ghosh et al., 2024c). We hypothesize this is due to the limited availability of high-quality audio-caption pairs, which causes CLAP representations to struggle with compositional reasoning (Ghosh et al., 2024d) and linguistic variations in captions (Selvakumar et al., 2024).
3.1.2. AF-CLAP TRAINING OBJECTIVE
Compared to the standard contrastive loss in audio-languee pre-training (Wu et al., 2022b; Elizalde et al., 2023a), we improve the training objective for better robustness lin- guistic variations and compositional reasoning abilities.
In this section, we address these issues and introduce an im- proved version of CLAP called AF-CLAP. Specifically, we focus on: i) constructing a large-scale, high-quality training dataset, and ii) improving the training objective to for better representational robustness.
Improving Linguistic Invariance: CLAP-like models struggle to generalize to linguistic variations in captions that humans easily understand (Selvakumar et al., 2024) (e.g., failing to equate helicopter and chopper). To address this, for every caption in the dataset, we generate M 1 linguistically varied captions with identical semantics and composition. These variations, along with the ground-truth caption, are treated positives.
3. AF-CLAP TRAINING DATASET
We scale the training dataset for AF-CLAP over 8M (10- second) audio-caption pairs. We collect these data from open-source audio and video datasets, with an emphasis on audio diversity and caption accuracy.
Improving Compositional Reasoning: Captions with dif- ferent word orders or structures often convey distinct re- lationships between acoustic events, such as temporal se- quencing or attribute binding. However, CLAP-like mod- els struggle to capture these nuances (e.g., differentiating whether one sound follows precedes another) (Ghosh et al., 2024d). To address this, we introduce composition- aware negatives, For every M positive captions, we generate N variations with modified temporal or attribute composi- tions and use them as negatives. An example below:
Existing models like CLAP Wu et al. (2022b) and MS-CLAP (Elizalde et al., 2023b) rely heavily on single- label audio classification datasets for captions. This limits their ability to generalize to complex, real-world audio with diverse compositions. Automated captioning efforts (c.g., Yuan et al. (2024)) yield only about 1.4M pairs, lack diver- sity, and under-represent critical domains like home sounds.
Inspired by the recent success of training vision LMs on im- ages from long videos (Venkataramanan et al., 2024), we col- lect audio from open long-video datasets. Specifically, we
Original Caption: A dog barking followed by the sound of train approaching.
select 100k diverse videos from MiraData (Ju et al., 2024) and Video Recap (Islam et al., 2024) (see Appendix H.1 for selection details). We segment these videos into 10-
Positive: A dog barking followed by the sound of a railcar approaching.
Negative: A dog barking preceded by the sound of railear approaching.
second clips and generate video captions using Qwen2-VL-
IB-Inntruct and audio captions using Qwen2-Audio. To
3

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
RoPE (Su et al., 2023) with base of 4096 to encode tem- poral information into the features.
Long Shoet Valcos
Representation Transformation Layers. To expand model capacity, we apply three self-attention layers to the audio feature representations, each with 8 heads and an inner dimension of 2048 (Kong et al., 2024; Vaswani et al., 2017).
Coveny &
Aedio and Video Capties
Addi Casiic
Yadeo 2aplaes
Gated Cross-Attention. Following Audio Flamingo, we use gated cross-attention dense KATTN-Deese) layers from Flamingo (Alayrac et al., 2022) to condition audio represen- tations on the LLM. Each layer consists of two blocks: (1) a residual block with cross-attention and tanh gating and (2) a residual block with a dense layer and tanh gating. These layers are inserted before each LLM block.
Ni J. Segusinn
Captic
Dats
AF-ELAP
Figure 3: Illustration of AF-CLAP training process. Wc collect long and short videos, segment it into 10-second clips, caption it, and prompt an LLM to generate audio captions. The data is then used to train CLAP with a modified contrastive loss.
The XATTN-Dense layers reduces the quadratic attention complexity in prefix tuning to linear complexity. For in- stance, let 1 = 80 be the text token length and /2 = 30 x 64 be the number of audio embeddings. Prefix tuning requires self-attention complexity of (4 + +12)² = 4 x 10s, whereas our cross-attention complexity is around 4 x x ~ 1.5 x 10S.
Contrastive Loss. Each sample x in the training data X now has M positives {P(x) ) and M N negatives (m) M,N
m= Let A(x) be audio embedding from the HTSAT lange audio encoder (Chen al., 2022) with MLP projection, and T(x) be the text embedding from Flan-T5 text encoder (Raffel et al., 2020; Chung et al., 2024) with MLP projection. Let su, v) exp(u v//) be the similar- ity function with temperature T. Our training objective is:
Frozen Language Model. Our architecture uses Qwen2.5- 3B (Yang et al., 2024a), a decoder-only causal LLM with 3B parameters, 36 hidden layers, and 16 attention heads. We find this model to have the best cost cost-performance ratio, as it has enough capacity for audio understanding and is light enough (see Section 6.6 for a comparison of LLM sizes).
C =
1 B
Σ
log Swgg (i) + 1>EE,=sS.. S(G.))
1(),
where
4. Audio Flamingo 2 Training Data
8(j,i== = Σ S (m)
Table 23 summarizes the datasets used to train AF2. We first convert common benchmark datasets used in Audio Flamingo AQA format (see Appendix F.1). In addition, we propose two datasets: AudioSkills for audio expert rea- soning and LongAudio for long audio understanding.
m
Snag (1)
Σ s (T m.n
where B the batch size. Our approach encourages CLAP to learn both linguistic invariance and compositional reason- ing, aligning its capabilities more closely with human-like understanding and thus providing more human-aligned rep- resentations. AF-CLAP achieves SOTA performance on various retrieval and audio classification benchmarks. Since comparing CLAP performance is beyond our scope, we compare results in Tables 7 and 10.
4.1. AudioSkills: An Expert Audio Reasoning Dataset
Expert-level reasoning requires mastery of diverse and spe- cialized skills (Ericsson, 2003; Huang & Chang, 2022). However, most existing audio datasets focus on surface-level properties, such as acoustic events or category classifica- tion. This limitation extends to QA pairs derived from these datasets, which often fail to require expert-level reasoning.
3.2. Audio Conditioning Architecture & LLM
To address this gap, we introduce AudioSkills, a high- quality, skill-specific synthetic dataset designed to prioritize the development of reasoning and problem-solving abili- ties. This dataset is carefully curated to ensure diversity and relevance, grounded in the hypothesis that expert rea- soning emerges from the mastery of various relevant skills and world knowledge. Specifically, we use combination of open-source sound and music datasets and synthetic au- dio and prompt GPT-4o, along with any available metadata, to create QA pairs. AudioSkills focuses on audios V 30 seconds long to enable effective scaling on open-source
Audio Feature Extraction. For each 10-second segment, we extract dense audio features i E R64x2018 from the penultimate layer of AF-CLAP. This approach yields higher- quality features compared mean-pooled representations from the final layer (see Appendix B.2). For longer au- dio, we use non-overlapping sliding windows to compute and concatenate audio features. The maximum number of sliding windows varies across training stages, with up to 30 windows (5 minutes) when training on LongAudio. Once the sliding window features are obtained, we apply
4

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Other AQA Datasets
Temporal Relationship Idearification
General Reasoning
Counting
Foundational: What is the geure of
Order: In what sequence do the
Question: How does the melody in the audio contribute to the hypnotic effect the music?
Level 2: How many times did the dog burk?
the music?
sounds finst uppear the audio?
Answer: Rock
(A) Rain (B) Humaa voice
Auswer: Two
ClothoAQA: Are there people having
Attribute: How does the sound of thunder change over time?
By changing frequently maintain interest. (B) By maintaining consistent and repetitive loop ...
Level 3: How many times did the second sound occur in the entire audio?
conversation?
Yes
Gitts louder Gets softer
Answer: Four
OpenAQA: What kind of sound do the mechanisms make?
Grounding: When does the human voice appear the audio?
Attribute Identification
Information Extraction
Answer: Mechanical sounds
(A) Beginning (B) Middle (C) End Referring: What sound appears in the sudio?
Question: Which the loudest event in the audio?
Question: Which instrument peimarily
SALMONN: What happening
provides the bass line that complements the female vocalist's melody?
vehicle?
Auswer: The loudest event the audio is the sound of dog barking.
Answer: Vehicle accelcrating.
(A) Rain (8) Human voice ....
(A) Piano (B) Tuba (C) Guitar
Contextual Speech Event Reasoning
Contextual Sound Event Reasoning
How does the repetition in the spoken utterance influence the interpretation of sequence of actions taking place?
Question:
Considering the correlation the man's speech the music, infer his possible connection music playing the seene. Answer: The man is likelly guitarist instructer who explaining the process of tuning a guitar.
Question:
Answer: The repetition suggests urgency or frustration in an effort to manage ongaing situation, possibly highlighted by background sounds of movement.
Figure 4: Examples from AudioSkills. Compared to other AQA datasets, questions in AudioSkills require deliberate reasoning.
datasets. It spans seven distinct skills selected for their plausibility, relevance, and ablation insights as follows:
other sounds or events.
6. Information Extraction: Focuses on understanding characteristics of the audio beyond just surface- properties, detailed content analysis, and the applica- tion of external world knowledge when necessary.
1. Temporal Reasoning:
Understanding temporal rela- tionships between acoustic events in the audio. We generate 4 types of MCQ-type QA pairs: i) Temporal Order: Understanding the sequential order of events or sounds; ii) Temporal Attribute: Understanding how sound attributes change over time; iii) Temporal Re- ferring: Answering questions referring to sounds at specific temporal locations in the audio (e.g., start, mid- dle, end); iv) Temporal Grounding: Identifying the temporal location of specific acoustic events.
7. General Reasoning: This category encompasses ques- tions that do not fall into the specific skill types above but require a combination of multiple skills or unique abilities, such as identifying relationships between mul- tiple events or interpreting complex scenarios.
In total, we generate <4.2M QA pairs. Figure 4 compares AudioSkills to other open-source AQA datasets, highlight- ing that these datasets lack the complexity required for delih- erate reasoning. While training on such QA pairs is useful for alignment (Zhou et al., 2024; Wolf et al., 2023), it does not equip models with the specialized skills needed to han- dle complex questions (Sakshi et al., 2024; Ghosh et al., 2024a). Additional details, including statistics, metadata, and prompts, are provided in Appendix D.1 1 and Table 14.
2. Attribute Identification: Recognizing attributes of events in multi-event audio, such sound character- isties (e.g.. loud bang) or gender (e.g., male speech). We generate QA pairs using attributes extracted via the method proposed by Kumar et al. (2024).
3. Counting: Counting occurrences of specific sounds in an audio. We use the Synthio TTA model (Ghosh et al., 2024b) to create QA pairs at 3 difficulty levels: i) Level I: Single sounds concatenated; count the event occurrences. ii) Level 2: Multiple interleaved sounds; count the main sound occurrences. iii) Level 3: Same as Level 2 but referenced by their temporal position.
4.2. LongAudio: A Long Audio Understanding Dataset
We construct LongAudio, the first large-scale long audio un- derstanding dataset, which is comprised of over 80K unique audios and approximately 263K AQA pairs. Audios are sourced from open long-video datasets, including a sub- set of MiraData(Ju et al., 2024) (featuring diverse content from natural scenes to gaming) and the entire Video Re- Cap(Islam et al., 2024) (egocentric videos of daily activi- ties). To ensure diversity, videos and carefully selecting samples from each cluster (see Appendix H. Fig. 5 (left) categorizes these videos and topics across several domains. We generate captions for video and audio segments using Qwen2-VL VL--VLLL-B-nnttee and Qwen2- Audio, respectively. GPT-40 is then used to
4. Contextual Sound Event Reasoning: Identifying the purpose of a sound acoustic event in the context of other sounds and events in the audio. This skill requires audio understanding, temporal reasoning, world knowl- edge, and various other skills. Inspired by CompA- -R in GAMA (Ghosh et al., 2024c), we expand the dataset from 2200 to =350k QA pairs.
filter MiraData by clustering
5. Contextual Speech Event Reasoning: Similar to Con- textual Sound Event Reasoning but focused on identi- fying the purpose of spoken utterances in relation to
5

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Video Categories
Question Categories
Duretion Distribution
Entte taliment & mence
Captbrinng
Other
Temporal Q6
Foucrtional Informative
Pitt Q4
15.8%
Datly the tome
Complex Q4
Adeenture Deskeration
25,4%
25.7%
Goncrul QU
34.5%%
Sports Sacreation
Subeorne QA
Creative Artstic
15.7%
fatorical Culturll
Autamisive Transpo tabinn
015
Craepency
23.4%
13.6
3.33
13.1%
Figure 5: The proportion of video categories (for sourcing audios) (left), question categories (middle), and distribution of durations (right) for the LongAudio dataset with 262,928 unique AQA and 80k unique audios. We target captioning and reasoning tasks.
CUP Seg
Video Caption
Audio Caption!
Vertfoontion
CUP Seg
Aati-Sag 2
Audo
Video Caption
Audio Cuption
MILL
Vidoo Caption
Audio Caption
LLMI
Qhitin
Longaoda
Eapert
Legindsshec
Revee
Segnuvtr Capuions
Metadare
Long Video
Figure 6: The pipeline for generating LongAudio. The process begins by segmenting the long vidco into short vidco and audio clips, each =10 seconds. These clips are individually annotated with captions. Subsequently, LLM is employed to generate question and-answer pairs based the captions of these clips. A subset of the data goes through expert review construct LongAudioBench.
generate questions that ask for captions, challenge models to reason, or extract information from long audios. Fig. 6 outlines the Lodgoudio creation process. Fig.5 (middle) shows the distribution of these categories as outlined below:
Fig.5 (right) shows the distribution of audio lengths, and Table 15 shows examples of each category.
LongAudiioench: We sample 10% (=:3k) from LongAu- dio, proportionally across QA types, for expert human ver- ification. Annotators manually verify and correct these in- stances, discarding any incorrect ones. Subsequently, three authors conduct quality check on the corrected samples.
1. Captioning: The task is to generate detailed and accu- rate descriptions of long audio, focusing on capturing the essence of the entire audio.
The full annotation process is detailed in Appendix E.7. The final version of LungnnuiiBenchh has 2,429 instances. For evaluation, following a wealth of prior work (Ghosh et al., 2024c; Yang et al., 2024b), we use an LLM-as-a-judge framework (Zheng et al., 2023): we prompt GPT-4o with the model response and ground truth answer using prompt 25 and score the response on scale of to 10.
2. Plot QA: The task is answer questions related to the overarching narrative storyline of the audio, requir- ing understanding of temporal and causal relation- ships between individual acoustic events.
3. Temporal QA: The task is to identify the temporal positions of acoustic events and their relationships, such order or overlap within the audio, including how certain attributes change with time.
5. Audio Flamingo 2 Training Strategy
4. Needle QA: The task is to locate and reason about a specific "needle" segment embedded within a longer audio "haystack," ensuring the response is tied explic- itly to the needle.
We train AF2 using a 3-stage curriculum learning strategy, progressively increasing the audio context length and im- proving data quality: Stage 1: Pre-training. This stage focuses multi-modal alignment to align audio representa- tions with the LLM. For training, we leverage large-scale and plausibly noisy classification and captioning datasets. Inspired by (Cheng et al., 2024), we include a small amount of high-quality QA data to improve pre-training. During this stage, CLAP and LLM layers are frozen, and only the audio representation transformation and gated cross-attention lay- ers are trainable. The audio context is restricted to 3 windows (30 seconds). Stage 2: Fine-tuning. This stage
5. Subscene QA: The task is to answer questions about specific subscene within the audio, requiring identifi- cation and understanding of localized events.
6. General QA:
The task ended questions about the audio that may span multiple events, themes, or contexts, demonstrating overall com- prehension and reasoning.
address broad, open-
6

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table I: Comparison of Audio Flamingo 2 with previous SOTA LALMs on foundational audio understanding benchmarks. CAP
Captioning, AQA Audio Question Answering. CLS = Classification, Ti-Zero-Shot Also, see note in Section C.
Dataset
Task
Previous SOTA
Ours
Dataset
Tass
Previous SOTA
Ours
ClothoAQA innnincss
AQA - ACC
74.9%
Qwen2 Andio
86.9% 112.00
Clotho-v2
CAP- CIDE
0.45
Qwen2 Audio
0.46;:001
DothoAQAAmm-in
AQA ACC
49.5%
Audio Flamingo
52.6% 42.6%
AudioCups
CAP- CIDEr
0.46
Audio Flamingo
0.58
AVsicAVQA..do
AQA-ACC ACC
72.1%
Owen Audio
72.3% 10.2%
CREMA-D<5
CLS- ACC
26.3%
Audio Flamingo
36.6% (10.156
NonSpeech7k
ACC-ACC
83.9%
Audio Flumingo
84.3% +0.4%
Ravdes:
CLS- ACC
20.9%
Audio Flamingo
26.3% 45.4%
CochiScene
CLS ACC
91.6% %
Pongi
82.1%
GTZAN2S
CLS ACC
65.2%
Audio Flamingo
69.1 1.11%.... 43.5%
NS
CLS ACC
60.1%
Pengi
62.0% % 11.0%
sol-ey-solo--DB's
CLS- AACC
85.6%
GAMA
85.8% 10.2%
NSmmmtmee
CLS ACC
78.8% %
Owen Audio
71.7%._77.9 -..7%
USSK2s
CLS ACC
71.2% Audio Flamingo
68. 0%. -3.2%
FSD50k
CLS mAP
47.9% GAMA
49.2% % 4..8%
ESC502S
CLS ACC
80.6% GAMA
83.9% 43.35%
focuses on improving audio understanding and reasoning by learning skills. For training, we use high-quality short- audio classification, captioning, and QA datasets. This stage is often referred to as instruction-tuniin (Chu et al., 2024; Ghosh et al., 2024c). During this stage, only the LLM lay- ers are frozen, and the CLAP model, audio representation transformation layers, and gated cross- trainable. The audio context length is increased to w = windows (1.5 minutes). Stage 3: Long Fine-tuning. This stage focuses on context-length extension and teaching skills specific to enabling reasoning on long audios. For training, we employ our proposed LongAudio dataset, keep the au- dio representation transformation and gated cross-attention layers trainable, and increase w = 30 windows (5 minutes).
6.2. Smaller but Better
We employ standard benchmark datasets for evaluation (Ta- ble 24). Foundational benchmarks include ClothoAQA, Mu- SicAVQA (audio-only), NonSpeech7k, NSymth, CREMA- D. Ravdess, GTZAN, Medley-solos-DB and USD8K and Clotho-v2 and AudioCaps. Reasoning benchmarks in- clude MMAU (sound and music subsets), Audio Entail- ment, ApenAQA- MuchoMusic, CompA-R-test, Mu- sicInstruct (Long subset), MusicQA, CMM (audio-language subset) and LongAudioBench.
layers are
Foundational Audio Understanding: Table I presents the performance of AF2 on foundational audio understanding benchmarks. For evaluation, we follow the similarity- retrieval approach using our CLAP model, proposed by Deshmukh et al. (2023) and widely adopted. While founda- tional audio understanding is not our primary focus, AF2 shows competitive results against SOTA LALMs while hav- ing half of their size (e.g., Qwen(2)-Audio, GAMA, and LTT are equipped with 7B L.MSs and large audio encoders).
6. Experiments
6.1. Experimental Setup
We train our model using 128 NVIDIA A100 80GB GPUs. During pre-training, we use an effective batch size of 1024, the AdamW optimizer (learning rate = 10-4, weight decay= = 0.1), and 6 with automatic mixed precision for efficiency. For fine-tuning and long fine-tuning, we adopt dynamic batching based on audio length, ensuring batch sizes are multiples of 2, with effective batch sizes ranging from 128 to 1024 (see Appendix H.2). We benchmark our model against recent SOTA LALMs, including GAMA, Audio Flamingo, Qwen-Audio, Qwen2-Audio, LTU, LTU-AS, SALMONN, AudioGPT, and Gemini (Flash v2 and Pro v1.5), GPT-4o- audio and report results for the best model. For zero-shot evaluations, we exclude corresponding datasets from all training stages, consistent with prior work (Gong et al., 2024; Ghosh et al., 2024c). For LongAudioBench, as cur- rent LALMs (except Gemini) do not support audio inputs of length 330 seconds, we adopt a two-step cascaded ap- proach. First, we generate captions for short segments of the input audio, and then we prompt the same LALM to answer the question using these captions. Our experiments with passing the original long audio consistently outperformed our cascaded approach. For Gemini, we prompt it with the entire original long audio.
Table 2 compares the performance of AF2 on audio reasoning benchmarks. We employ the orig- inal evaluation strategy and metrics. With a much smaller LLM, AF2 outperforms all LALMs by large margins. We provide fine-grained results and failure cases of AF2 on LongAudioBench in Table 13 and Table 22.
Expert Reasoning:
Table 2: Comparison of Audio Flamingo 2 with previous SOTA LALMs on audio reasoning benchmarks, all AQA-based.
Dataset
Previous SOTA
Ours
MMAU Sound
61.7%
Gemini Flash v2
65.1 1% +3.4%
MMAU Music
56.5%
Gemini Flash v2
72.9%
+16.4%
AE Clotho
83.3%
Qwen Audio
92.5% % +0.2%
AE AudioCaps
64.2%
Qwen Audio
93.3% +29.1%
CompA-R-test
80.0% GAMA-IT
96.4 4 +16.4%
MuchoMusic
51.4%
Qwen Audio
56.5 % +%5.%%
OpenAQA
80.0% GAMA-IT
86.0% 6.00%
MusicInstruct (Long)
86.1%
MusiLingo
90.2% 19.
MusicQA
90.0%
MusiLingo
93.0% 43.0%
CMM Hallucination
76.0%
SALMONN
82.0% 4.6.0%
LongAAdioBentt (ours)
45.3% - Gemini Flash v2
64.2 % +18.99
7

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
6.3. Enhanced Audio Features Boost Performance
6.6. Effect of Scaling LLM
Table 3 compares the performance of AF2 using AF-CLAP against various SOTA CLAP models on benchmark datasets. The MMAU score is averaged across the sound and music subsets. "AF-CLAP 630k" refers to AF-CLAP trained using the same strategy but with Laion-CLAP's 630k audio-text pairs. "AF-CLAP w/ con." represents our model trained with the same data but using the standard contrastive loss formulation. The results show that replacing AF-CLAI with other CLAP models results in performance drop across benchmarks, highlighting the importance of robust audio representations for improving performance.
Fig. 7 illustrates the performance of AF2 across various LLM sizes, ranging from 0.5B to 7B, trained with and with- out AudioSkills. The results demonstrate that data quality often surpasses the performance gains achieved by simply scaling compute. Training with AudioSkills not only de- livers superior performance overall but also significantly boosts reasoning capabilities, even at smaller LLM sizes. In contrast, when AudioSkills is excluded, reasoning perfor- mance heavily depends model size, with performance scaling more gradually parameters increase. This high- lights the critical role of high-quality, skill-specine data like AudioSkills in driving robust reasoning capabilities, regardless of model size.
Table 3: Benchmark results various CLAP as audio encoders.
Model
AudioCaps
GTZAN
MuchoMusic
MMAU (avg)
CLAP
0.51
65.2
$2.3
63.8
MS-CLAP
0.55
65.8
53.1
64.3
AF-CAAP 630k
0.51
66.0
54.6
63.9
AF CLAP w/ con.
0.53
66.2
54.4
66.8
AF-CLAP
0.58
69.1
56.5
69.0
6.4. High Quality Data Boosts Reasoning Abilities
Table 4 compares the impact training data composition on performance. "w/ 1/2 data" and "w/ 1/3 data" refer to experiments using random subsets comprising half and one- third of the total instances from each dataset. Key findings include: (I) AudioSkills significantly enhances AF2's s rea- soning performance and shows notable improvements when combined with other datasets. (2) For challenging tasks like reasoning and zero-shot classification, data diversity is crucial, as using 1/2 of the data outperforms using only Ope- nAQA. (3) More data consistently improves performance.
NYUu (ang iy Auciatils
renoo (Pe) with cboci
MucheMlall Addiaad's
MuchoMeate ers Audidadla
0.5
Figure 7: Performance comparison of AF2 on different LLM sizes, w/ and w/o AudioSkills. More results in Table 19.
Table 6:
Overview of 10 training schedules detailing whether CLAP, XATTN, and LLM components (in order) are frozen or unfrozen during cach stage and their impact on performance.
Table 4: Comparison of training different data compositions.
Trining Suges
Benchmanks
AudiooCpp
CIZAN
MMAU
LongAnciol
Model
AudioCaps
GTZAN
MuchoMMusic
MMAU (avg)
0.50
56.7
58.8
5.1
stage
w/ 1/2 data
0.48
51.3
46.7
49.3
1/3 data
0.41
47.7
43,9
45.1
0.56
68.9
68.0
5.1
w/o AudioSkills
0.58
68.8
42.6
48.6
0.59
70.9
w/ OpenAQA
0.55
19.3
39.8
38.2
@
0,59
72.3
60.7
577
*AudioSkills
0.55
22.6
51.3
57.5
0.52
65.7
63.4
533
Original
0.58
69.1
56,5
69.0
stage
2
66
880
d
Siv
0.45
0.58
62.6
69.1
59.1
6,2
800
@
0.62
76.2
6.5. Cross-Attention Outperforms Prefix-Tuning
stage
d
$60
0.99
74.0
62.0
6.3
Table 5 compares GAMA, a SOTA 7B LALM using prefix tuiing, trained on the same data and strategy (excluding stage 3), with AF2, AF2 significantly outperforms GAMA, attributed superior audio representations and the shift cross-attention-basee conditioning.
6.7. Effect of Training Schedules
Table compares 10 training schedules and their impact on AF2 performance. For 1- and 2-stage training, data from later stages is combined with earlier stages. Additionally, we evaluate 4-stage curriculum where stage 2 is repeated with LLM fine-tuning, and long fine-tuning is shifted to stage 4. Key findings include: (1) Fine-tuning the LLM improves classification and captioning tasks due to the style memorization effect (Ghosh et al., 2024a), which benefits
Table 5: Comparison of AF2 with GAMA. SOTA LALM,
trained on our same data and same training recipe.
Model
AudioCaps
GTZAN
MushoMusic
MMAU (avg)
GAMA (
0.67
13.8
33.7
38.1
GAMA [ours]
0.41
54.7
40.3
52.8
AF2
0.58
69.1
56.5
69.0
8

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
retrieval-based evaluation (see Appendix C) but reduces performance on reasoning tasks. (2) Pre-training alignment
ling the emotion dimension in voice generation systems. arXiv preprint arXiv:/806.009: 2018.
is essential before fine-tuning, (3) Gradual context-length extension is critical for effective long-audio understanding,
Agostinelli, A., Denk, T. I., Borsos, Z., Engel. J., Verzetti, M., Caillon, A., Huang. Q., Jansen, A., Roberts, A., Tagliasacchi, M., Sharifi, M., Zeghidour, N., and Frank,
7. Conclusion, Limitations and Future Work
C. Musiclm: Generating music from text, 2023.
In this paper, we introduce Audio Flamingo 2, an ALM designed for long audio understanding and expert reasoning. Our model leverages a custom CLAP, trained on large- scale dataset with novel objective function, and is trained on synthetic reasoning AQA data to develop unique skills es- sential for real-world reasoning. Audio Flamingo 2 achieves SOTA performance in audio understanding and reasoning despite being small in model footprints. Additionally, we propose LongAudio and Becc advance the field of ALM reasoning over long audio contexts.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et Flamingo: a visual language model for few- shot learning. Advances in nearal information processing systems, 35:23716 1137166237666 2022.
Arumugam, G. P., Chang, S.-Y., Prabhavalkar, N. S. R., Wang. Q., and Bijwadia, S. Improved long-form speech recognition by jointly modeling the primary and non- primary speakers. In 2023 IEEE Automatic Speech Recog- nition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.
For future work, we aim to address current limitations, in- cluding: (1) enhancing speech content understanding ca- pabilities, (2) scaling AudioSkills to include more diverse datasets, and (3) developing audio encoders inherently ca- pable of processing long audio.
Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., and Wermter, S. The omg-emotion be- havior dataset. In 2018 International Join! Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2018.
Impact Statement
Bittner, R. M., Salamon, J.. Tierney, M., Mauch, M., Can- nam, C., and Bello, J. P. Medleydb: A multitrack dataset for annotation-intensive mir research. In ISMIR, vol- ume 14, pp. 155-160, 2014.
Audio Flamingo 2 represents a transformative step in ad- vancing ALMs by addressing the dual challenges of expert reasoning and long audio understanding. By integrating a small yet efficient architecture with novel synthetic datasets like AudioSkills and LongAudio, AF2 bridges critical gaps in reasoning and auditory comprehension, setting new state- of-the-art benchmarks. Beyond technical achievements, AF2 holds significant potential for real-world applications, including assistive technologies, anomaly detection, and multimedia analysis. By open-sourcing both the model and datasets, this work aims to democratize advancements in audio-language modeling, fostering innovation and collabo- ration across the research community.
Bogdanov, D.,, Won, M., Tovstogan, P., Porter, A., and Serra, X. The mtg-jamendo dataset for automatic music tagging. ICML, 2019.
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
Cartwright, M., Mendez, A. E. M., Cramer, A., Lostanlen, V., Dove, G., Wu, H.-H., Salamon, J., Nov, O., and Bello, J. Sonye urban sound tagging (sonyc-ust): A multilabel dataset from urban acoustic sensor network. 2019.
References
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08900 2024.
Chen, C., Hu, Y., Yang, C.-H. H., Siniscalchi, S. M., Chen, P.-Y,, and Chng, E.-S. Hyporadise: An open baseline for generative speech recognition with large language models. Advances in Neural Information Processing Systems, 36, 2024a.
Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B., and Vijayanarasimhan S. Youtube- 8m: A large-scale video classification benchmark. arXiv 2016.
Chen, C.,, Peng, P., Baid, A., Xue, 7.,, Hsu, W.-N., Harwath, D., and Grauman, K. Action2sound: Ambient-awan generation of action sounds from egocentric videos. arXiv preprint arXiv:2406.099 042722 2024b.
preprint arXiv: arXiv:
Adigwe, A., Tits, N., Haddad, K. E., Ostadabbas, S., and Du- toit, T. The emotional voices database: Towards control-
Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vg- gsound: large-scale audio-visual dataset, 2020.

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpaatrckk T., and Dubnov, S. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2022.
Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE Inter- national Conference on Acousties, Speech and Signal Processing (ICASSP). pp. 736-740. IEEE, 2020.
Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. Clap learning audio concepts from natural language su- pervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023a.
Cheng. D., Gu, Y., Huang, S., Bi, J., Huang, M., and Wei, F. Instruction pre-training: Language models are supervised multitask learners. arXiv preprint arXiv:2406.1444 2024.
Chiu, C.-C., Han, W., Zhang, Y., Pang, R., Kishchenko, S., Nguyen, P., Narayanan, A., Liao, H., Zhang, S., Kannan, A., et al. A comparison of end-to-end models for long- form speech recognition. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pp. 889-896. IEEE, 2019.
Elizalde, B., Deshmukh, S., and Wang, H. Natural lan- guage supervision for general-purpose audio represen- tations, 2023b. URL https: http://axxvvvv... 2309.05767
Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., and Norouzi, M. Neural audio synthesis of musical notes with wavenet autoencoders, 2017a.
Chu, Y., Xu, J., Zhou, X., Yang, Q.. Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models. arXiv preprint arXiv:2311.07919, 2023.
Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., and Simonyan, K. Neural audio synthesis of musical notes with wavenet autoencoders. In Interna- tional Conference on Machine Learning, pp. 1068-1077. PMLR, 2017b.
Chu, Y., Xu, J., Yang, Q. Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J.. Lin, J., al. Qwen2-audio technical report. arXiv preprint arXiv:2407. 10759, 2024.
Ericsson, K. A. The acquisition of expert performance problem solving: Construction and modification of mediating mechanisms through deliberate practice. The psychology of problem solving, pp. 31 31-83, 2003.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling ciatructionffintturn language models. Journal of Machine Learning Research, 25(70):1-53 2024.
Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X.
Fsd50k: open dataset of human-labeled sound events.
Defferrard, M., Benzi, K., Vandergheynst, P., and Bresson, X. Fma: A dataset for music analysis. arXiv preprint arXiv: 1612.01840, 2016.
IEEE/ACM Transactions on Audio, Speech, and Lan-
guage Processing, 30:829-852, 2021.
Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X. Fsd50k: An open dataset of human-labeled sound events, 2022.
Deng, Z. Ma. Y., Liu, Y., Guo, R., Zhang, G., Chen, W., Huang, W., and Benetos, E. Musilingo: Bridging music and text with pre-trained language models for music captioning and query response. arXiv preprint arXiv:2309.0877 08730, 2023.
Font, F., Roma, G., and Serra, X. Freesound technical demo. Proceedings of the 21 ACM international conference on Multimedia, pp. 411-4114111 2013.
Deshmukh, S., Elizalde, B., and Wang, Audio re- trieval with wāvtext5k and clap training. arXiv preprint 10Xi::2209.4425.. 2022.
Foster, P., Sigtia, S., Krstulovic, S., Barker, J., and Plumb- lcy, M. D. Chime-home: A dataset for sound source recognition in domestic environment. In 2015 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 1-5. IEBE, 2015.
Deshmukh, S., Elizalde, B., Singh, R., and Wang, H. Pengi: An audio language model for audio tasks, 2023.
Deshmukh, S., Han, S., Bukhari, H., Elizalde, B., Gamper, H., Singh, R., and Raj, B. Audio entailment: Assessing deductive reasoning for audio understanding. In The 39th Annual AAAI Conference on Artificial Intelligence, 2024. URL www.vpreprintazaaaa.....a
Gardner, J., Durand, S., Stoller, D., and Bittner, R. M. Llark: A multimodal foundation model for music. arXiv preprint arXiv:2310.07 160, 2023.
2407. 000...0..66.2 18062.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and numan-labeled dataset for audio events, 20/7 IEEE international conference on
Doh, S., Choi, K., Lee, J., and Nam, Lpmuusicaaps: Lim-based pseudo music captioning. arXiv preprint arXiv:2307.16377 2023.
10

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Gong, Y., Luo, H., Liu, A. H., Karlinsky, L.,, and Glass, J. R. Listen, think, and understand. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpp:://openeee//////e//// /wppnreveeeeeeeeeeeeeeeeeeeeeeee wwt.wwiwwwweeeeeeeee.m... net//oruntt id=nRZBPXdJ1C
Ghosh, S., Evuru, C. K. R., Kumar, S., S, R., Aneja, D., Jin, Z., Duraiswami, R., and Manocha, D. A closer look at the limitations of instruction tuning. In Forty- first International Conference Machine Learning, beteenreviewnee//oooeeeeee nerrfo
Guzhov, A., Raue, F., Hees. J., and Dengel, A. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
2024a. URL id= XGQ.
Ghosh, S., Kumar, S., Kong, Z., Valle, R., Catanzaro, B., and Manocha, D. Synthio: Augmenting small-scale audio classification datasets with synthetic data, arXiv preprint arXiv:2410.02056. 2024b.
Heittola, T., Mesaros, A., and Virtanen, T. Tau urban acous- tic scenes 2019 openset, development dataset.".
Hershey, S., Ellis, D. P., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., and Plakal, M. The benefit of temporally- strong labels in audio event classification. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 366-370. IEEE, 2021.
Ghosh. S., Kumnr, S., Seth, A., Evuru, C. K. R., Tyagi, U., Sakshi, S., Nieto, O., Duraiswami, R., and Manocha, D. GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6288-6318 Miami, Florida, USA, November 2024c. Association for Compu- tational Linguistics. doi: 10.18653ww/20044emmmm. 361. URL https: http//acaaatoooooooo//////0 emn 1 p-ma in
Huang, J. and Chang, K. C.-C. Towards reasoning in large language models: survey. arXiv preprint arXiv:2212.10400 2022.
Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z.,
Wu, Y., Hong, Z., Huang, J., Liu, J., et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 23802-23804, 2024.
Ghosh, S., Seth, A., Kumar, S., Tyagi, U., Evuru, C. R., S, R., Sakshi, S., Nieto, O., Duraiswami, R., and Manocha, D. Compa: Addressing the gap in compositional rea- soning in audio-language models. In The Twelfth In- ternational Conference on Learning Representations, ew, forum?
Islam, M. M., Ho, N., Yang. X., Nagarajan, T., Torresani, L., and Bertasius, G. Video recap: Recursive captioning of hour-long videos. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18 18198-18208, 2024. doi: 10.11099CVPR52333.2004 01723.
2024d. URL id= 8 WN.
Ghosh, S., Kumar, S., Evuru, C. K. R., Nieto, O., Du- raiswami, R., and Manocha, D. Reclap: Improving zero shot audio classification by describing sounds. 2025.
James, J., Tian, L., and Watson, C. An open source emo- tional speech corpus for human robot interaction applica- tions. Interspeech 2018, 2018.
Gong, Y., Chung, Y.-A., and Glass, J. Ast: Audio spec- trogram transformer. arXiv preprint arXiv:2104.0177 2021.
Jeong, I.-Y. and Park, Cochlscene: Acquisition of acoustic scene data using crowdsourcing. In 2022 Asia-Pacifer Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 17-21. IEEE, 2022.
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., and Glass, J. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU), pp. 1-8, 2023a. doi: 10. 1091SS944 2023. 1038974..
Ju, X., Gao, Y., Zhang, Z., Yuan, Z., Wang, X., Zeng, A., Xiong, Y., Xu, Q., and Shan, Y. Miradata: large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407. 063506388,, 2024.
Gong, Y., Rouditchenko, A., Liu, A. H., Harwath, D., Kar- linsky, L., Kuehne, H., and Glass, J. R. Contrastive audio-visual masked autoencoder. In The Eleventh In- ternational Conference on Learning Representations, 2023b. URL htthtts/// ptttpp:///penreviiece htt.:/eettrourtt LM.yk5b....
Kim, D., Kim, B., Lee, H., and Kim, G. Audiocaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
of the Association for Computational Linguisties: Hu- Language Technologies, Volume (Long and Short Papers), pp. 119-132, 2019.
Speech and Signal Processing (ICASSP), pp. 286-290. IEEE, 2024.
Livingstone, S. R. and Russo, A. The ryerson audio- visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)=0019639,, 2018.
Koepke, A. S., Oncescu, A.M.,, Henriques, J. F., Akata, Z., and Albanie, S. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Mul- timedia, 25:2675-2688 2022.
Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., and Catanzaro, B. Audio flamingo: A novel audio lan- guage model with few-shot learning and dialogue abili- ties. In Forty-first International Conference on Machine Learning, 2024. URL https: httph:htt///pppe /ttp:////peeeeeeeeeeeeeeee net./ forum? WYWK.....................
Lostanlen, V., Cella, C.-E., Bittner, R., and Essid, S. Medley- a cross-collection dataset for musi- cal instrument recognition, February 2019. URL https: //dpi.org////.. 5 2w2////eeooo..o/ www.eeeooooo........ 1344103.
Lotfan, R. and Busso, C. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing. (144):471-488 2017.
Kulesza, A., Taskar, B., et al. Determinantal point pro- cesses for machine learning. Foundations and Trends@ in Machine Learning, (22-3):1332223 2012.
Kumar, S., Seetharaman, P. Salamon, J., Manocha, D., and Nieto, O. Sila: hignalto-langueee augmentation for enhanced control in text-to-audio generation. arXiv preprint arXiv:2412.0998% 2024.
Martin Morato, I. and Mesaros, A. Diversity and bias in audio captioning datasets. 2021.
Mei, X., Meng, C., Liu, H., Kong, Q.. Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing. 2024.
Law, E., West, K., Mandel, M. I., Bay, M., and Downie, J. S. Evaluation of algorithms using games: The case of music tagging. In ISMIR, pp. 387-392. Citeseer, 2009.
Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W. Ny-embed: Improved techniques for training lims as generalist embedding models. arXiv preprint arXiv:2405.1 17428, 2024.
Melechovsky, J., Guo, Z., Ghosal, D., Majumder, N., Herremans, D., and Poria, S. Mustango: Toward controllable text-to-music generation. arXiv preprint arXiv:231 2023.
Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., and Hu, D. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108- 19118, 2022.
Morato, M. and Mesaros, A. Maes multi-annotator captioned soundscapes, July 2021. URL https:// doi org/10. wtt.:ooo.......eeeeeea. 5281/2enooo .1roodd..1111111 5114771.
Morris, M. R., Sohl-Dickstein, J., Fiedel, N., Warkentin, T.,
LI, Y., Yuan, R., Zhang, G., Ma, Y., Chen, X., Yin, H., Xiao, C., Lin, C., Ragni, A., Benetos, E., Gyenge, N., Dannenberg, R., Liu, R. Chen, W. Xia, G., Shi, Y., Huang, W., Wang. Z., Guo, Y., and Fu, J. MERT: Acoustic music understanding model with large-scale self- supervised training. In The Twelfth International Confer- ence on Learning Representations, 2024. URL ht tps: //apenreview. net /eret/formmme forum? d=w 9MS1Bu.
Dafoe, A., Faust, A., Farabet, C., and Legg, S. Position: Levels AGI for operationalizing progress on the path to AGI. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, (eds.), Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 36308 36308-36321. PMLR, 21-27 Jul 2024. UtIhttps:: //te::///rrrooee/ proceedoeeeeeee ngs ml m.w..rsss.////////mmmmmmmmmm.. 4b. html.
Lipping, S., Sudarsanam, P., Drossos, K., and Virtanen, T. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pp. 1440-1144. IEEE, 2022.
Oncescu, A.-M., Koepke, A., Henriques, J. F., Akata, Z., and Albanie, S. Audio retrieval with natural language queries. arXiv preprint arXiv:2105.02192 2021.
Liu, S., Hussain, A. S., Sun, C., and Shan, Y. Music un- derstanding Ilama: Advancing text-to-muse generation with question answering and captioning. In ICASSP 2024-2024 IEEE International Conference on Acoustics,
Pichora-Fuller, M. K. and Dupuis, K. Toronto emotional speech set (tess). Scholars Portal Dataverse, 1:2020, 2020.
12

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Piczak, K. J. ESC: Dataset for Environmental Sound Clas- sification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pp. 1015-1018. ACM Press, 2015. ISBN 978-1-4503-3459-4 doi: 10.1145/2739373. 2806390.
Silva, A., Whitehead, S., Lengerich, C., and Leather, H. J. CoLLAT: On adding fine- fine-grained audio understanding to language models using token-level locked-language tuning. In Thirty
Conference on Neural In- formation Processing Systems, 2023. URL https: / //openreview. net forum? 2n
acm. wtt.:/a........t.ii.....o
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint
Sturm, B. L. The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use. arXiv preprint arXiv: 1306. 1461, 2013.
2018.
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023, httpt:ttt///aaeeee org orogorg/ae. / 2104. 09864.
Radford, A., Kim, W., Hallacy. C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learming, pp. 8748-8760 8763. PMLR, 2021.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.1320 2023.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. International conference on machine learning, pp. 28492-28518, PMLR, 2023.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W. Lu, L., MA, Z.,, and Zhang, C. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:: bettps://openreveee wwtp//ppeeeeeeee////e net /frrmm? 1d=14 24=14mn7HpKVE
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Team, G., Georgiev, P., Lei, V. L., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, 7., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05 055330, 2024.
Rafii, Z., Liutkus, A., Stöter, F.R.., Mimilakis, S. L., and Bittner, R. Musdb Musdb18-he 18-hq-an uncompressed version of musdb18. (No Title), 2019.
Rahman, M. A., Hakim, Z. I. A., Sarker, N. H., Paul, B., and
Fattah, S. A. Sonics: Synthetic or not-identifyng coun-
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need, Advances in neural information processing systems, 30, 2017.
terfeit songs. arXiv preprint arXiv:2408. /4080, 2024.
Rashid, M. M., Li, G., and Du, C. Nonspeech7k dataset: Classification and analysis human non-speech sound. IET Signal Processing, :7(6):e 12233, 2023.
Venkataramanan, S., Rizve, M. N., Carreira, J., Asano, Y. M., and Avrithis, Y. Is imagenet worth video? learn- ing strong image encoders from long unlabelled video. The Twelfth International Conference on Learning Representations, 2024. URL htwps:ttt/////et /otp://v/eeeeeeeeeeeeeeeee n -ennummddd---eiiiien Yen11Gns2..
Sakshi, S., Tyagi, U., Kumar, S., Seth, A., Selvakumar, R., Nieto, O., Duraiswami, R., Ghosh, S., and Manocha, D. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19)) 2024.
Salamon, J., Jacoby, C., and Bello, J. P. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041-1044, 2014.
Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., Soplin, N. E. Y., Heymann, J., Wiesner, M., Chen, N., et al. Espnet: End-to-end speech processing toolkit. arXiv preprint arXiv:1804.0000 00015, 2018.
Selvakumar, R., Kumar, S., Giri, H. K., Anand, N., Seth, A., Ghosh, S., and Manocha, D. Do audio-language models understand linguistic variations? arXiv preprint arXiv:2410. 16505, 2024.
Weck, B., Manco, I., Benetos, E., Quinton, E., Fazekas, G., and Bogdanov, D. Muchomusic; Evaluating music un- derstanding in multimodal audio-language models. arXiv preprint arXiv:2408.0133: 2024.
13

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Weng, Y., Han, M., He, H., Chang, X., and Zhuang, B. Longvlm: Efficient long video understanding via large language models. In European Conference on Computer Vision, pp. 453-470. Springer, 2025.
Zheng, L., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang. Y., Lin, Z.,, Li, Z.,, Li, D., Xing, E., al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Wolf, Y., Wies, N., Avnery, O.., Levine, Y., and Shashua, A. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.1108. 2023.
Zhou, C.,, Liu, P., Xu, P., Iyer, S., Sun,, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Wu, H., Seetharaman, P., Kumar, K., and Bello, Wav2clip: Learning robust audio representations from clip. In ICASSP 2022, 2022a.
Wu, H.-H., Nieto, O., Bello, P., and Salamon, J. Audio- text models do not yet leverage natural language. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023a.
Wu, Y. Chen, K., Zhang, T., Hui, Y., Berg- Kirkpatrick, T., and Dubnov, S. Large-scale con- trastive language-audin pretraining with feature fu- sion and keyword-to-caption augmentation. ICASSP 2023 2023 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pp. 1-5, 2022b. URL https: / api. wew.:esssaatiicccho.ooo.o org/CorpusID:2555 II:::DDD:233111 0826..
Wu, Y., Chen, K., Zhang, T., Hui, Y., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and seyword-to-capee to-caption augmentation. ICASSP 2023-2 -003 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023b. doi: 10. N0.110//CCASS449355. 2023.10095960
T.,
Xue, F., Chen, Y., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., et al. Longvila: Scaling long- context visual language models for long videos. arXiv preprint ar Xrv: 10188, 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu. B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15/15, 2024a.
Yang. Q. Xu, J., Liu, W., Chu, Y., Jiang, Z., Zhou, X., Leng. Y., Lv, Y., Zhao, Z., Zhou, C., et al. Air-bench: Benchmarking large audio-language models via genera- tive comprehension. arXiv preprint arXiv:2402.07729, 2024b.
Yuan, Y., Jia, D., Zhuang, X., Chen, Y., Liu, Z., Chen, Z., Wang, Y., Wang, Y., Liu, X., Kang, X., et al. Sound- vecaps: Improving audio generation with visual enhanced captions. In Audio Imagination: NeurIPS 2024 Workshop Al-Driven Speech, Music, and Sound Generation, 2024.
14

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
A. Table of Contents
from complex real-world audios, which represents only significantly small portion of Laion-CLAP.
1. B AF-CLAP
Table 10 highlights the performance of AF-CLAP on zero- shot audio classification benchmarks, where consistently achieves SOTA results. We make the same conclusion for Laion-CLAP as previously stated.
2. C: Limitations of CLAP-Retrieval-Bass Evaluation
3. D AudioSkills
4. E: LongAudio
5. F: Prompts
We emphasize that benchmark datasets do not holistically evaluate CLAP model's capabilities, highlighted in several works (Ghosh et al., 2024d; Wu et al., 2023a; Sel- vakumar et al., 2024; Ghosh et al., 2025). While AF-CLAP outperforms baselines on benchmark datasets, its audio fea- tures are significantly more robust and enhance the audio perception capabilities of (L)ALMs (see also Table 3).
6. G More Results and Design Analysis
7. G.I Effect of RoPE
8. G.2 Effect of Audio Transformation Layers
9. G.3 Effect of Cross-Attention Frequency
10. H: More Details
C. Limitations of CLAP Retrieval Based Evaluation
B. AF-CLAP
B.I. Training Hyper-parameters
We attribute the suboptimal performance of AF2 on some datasets the limitations of the CLAP-based evaluation method, which often fails to retrieve the correct label cor- responding to the model's open-ended generation response. we follow the evaluation scheme introduced by Deshmukh et al. (2023) and widely adopted in prior works (Gong et al., 2024; Ghosh et al., 2024c). This approach uses a CLAP model to retrieve a label from the label set by comparing the model's 's open-ended generation and assigning the label with the highest similarity as the predicted output. We show some examples below where even correct prediction by AF2 leads to incorrect retrieval and therefore lower accuracy:
We train AF-CLAP on 8 A100 80GB GPUs. We train it with a learning rate of Se-4, M=N=3, and an effective batch size of 256 for 12 epochs. This batch size is smaller than that in the literature, but we do so due to computational constraints.
B.2. Feature Extraction Layer
As described in Section 3.2, unlike Audio Flamingo which uses CLAP audio embeddings directly, AF2 discards the CLAP head and rather leverages dense features from the final layer of the audio encoder (HTS-AT our case). Ta- ble 8 compares the performance of AF2 trained with CLAP head features versus dense audio features from the encoder's last layer. The results clearly demonstrate that using dense features significantly improves AF2's performance.
Dataset: GTZAN, Correct Label: Rock, Predicted
Label: Punk Metal, Retrieved Label: Metal
Dataset: ESC50, Correct Label: Pouring water, Pre-
dicted Label: liquid, Retrieved Label: Water drops
B.3. Training Datasets
Table 9 provides detailed statistics of datasets used for train- ing AF-CLAP.
However, fine-tuning the LLM leads style memorization, which favors this evaluation method. We show some ex- amples of prediction shift and how this leads to increase in accuracy:
B.4. Comparison with prior-art
Dataset: GTZAN, Correct Label: rock, Predicted
Table 7 presents the performance of our CLAP audio-to- text and text-to-audio retrieval tasks using the Clotho and AudioCaps datasets. "AF-CLAP 630k" refers to AF-CLAP trained using our proposed strategy but with Laion CLAP's 630k audio-text pairs. "AF-CLAP w/ van. con." represents AF-CLAP trained with the same data but using the stan- dard contrastive loss formulation. "AF-CLAP w/o noise red." represents AF-CLAP trained using the same data and method but without our noise reduction step. Our final proposed AF-CLAP achieves state-of-the-ar performance across all metrics. Overall, AF-CLAP 630k does not lead to any improvements over Laion as our data augmenta- tion and cleaning are only applicable to captions obtained
Label: rock, Retrieved Label: rock
Dataset: ESC50, Correct Label: clock alarm, Pre- dicted Label: clock alarm, Retrieved Label: clock alarm
D. AudioSkills
D.I. Dataset Statistics
Table 14 presents detailed category-wise statistics on our proposed AudioSkills dataset. We also list the meta-data used for data generation. For meta-data, transcripts were
15

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table 7: Performance comparison of AF-CLAP with baselines on Text-to-Audio and Audio-to to- TexText retrieval on AudioCaps and Clotho.
AudioCaps
Clotho
Model
Texi-to-Audio
Audio-to-Tens
Texioto-Audio R@5
Audio-to- Text
R@D I
R@5
R@10
R@1 1
R@5
R@ 10
R@1
R@ R@10
R@1
R@5
R@ 10
MMT
36.1
72.0
84,5
39.6
76.8
86.7
6.7
21.6
33.2
7.0
22.7
34.6
ML-ACT
33,9
69.7
82.6
39,4
72.0
83.9
14,4
36,6
49.9
16.2
37.6
50.2
CLAP
34.6
70.2
82.0
41.9
41.9
84.6
16.7
41.1
54.1
20.0
44.9
58.7
CompA-CLAP
72.6
81.6
45.2
86.7
16.8
43.5
19.7
45.2
55.6
Laion-CLAP
36.1
71.8
83.9
46.8
82.9
90.7
16.1
38.3
51.1
22.7
48.5
60.8
AF-CLAP-630K
36.3
71.8
83.9
46.3
83.0
90.9
16.1
38.3
51.4
22.7
48.7
60.8
AF-CLAP van. con.
36.8
72.0
84.0
45.2
83.5
90.0
16.5
41.9
22.9
51.0
63.2
AF-CLAP w/o noise red.
35.4
70.7
81.6
45.0
82.6
91.0
16.3
40.5
52.7
22.8
51.2
59.4
AF-CLAP (ours)
37.3
72.9
84.0
46,9
84.1
91.9
17.3
43.9
56.8
23.2
51.2
63,5
Table 8: Performance comparison of AF2 with different audio feature extraction methods from CLAP. Head refers to audio fea- tures extracted from the CLAP head, and Dense refers dense features extracted from the last layer of audio encoder.
E. LongAudio
E.1. Detailed Statistics
Table 11 presents detailed statistics of LongAudio and Lon- AudioBench, categorized into the various types of QAs.
Model
AudioCaps
GTZAN
MushoMusic
MMAU
Hcad R1x256
0.50
54.7
42.6
47.0
Dense R64x2048
0.58
69.1
56.5
69.0
E.2. Examples
Table 9: Statistics of audio-caption datasets used for CLAP train- ing. indicates the dataset was collected by us. Furthermore, datasets with captions were made to go through our cleaning stage to reduce noise, as mentioned in Section 3.1.1.
Table 15 shows category- -wise examples from LongAudio.
E.3. Comparison with other datasets
Dataset
#Audio-Text Pairs
Table 12 compares the duration statistics of LongAudio with other AQA datasets. LongAudio stands with the longest average audio durations.
You (Abu-El-Haija et al., 2016)
3,947,057
Sound-VHCaps (Yuan et al., 2024)
1,657,029
MiraData' (Ju al., 2024)
748,320
Action2sound" (Chen et al., 2024b)
306,602
NSynth (Engel et al., 2017b)
289,205
E.4. Fine-grained results for AF2
Freesound (Font al., 2013)
256,695
AudioSet Strong" (Hershey et al., 2021)
216,622
Table 13 presents category-wise fine-grained results of AF2 LongAudioBench. While AF2 demonstrates strong performance, fine-tuning on LongAudio further improves scores across all categories, particularly in tasks unique to LongAudioBench, such as NeedleQA and SubsceneQA, which were not encountered during AF2's two-stage train- ing. These results emphasize the significance of our pro- posed LongAudio dataset effectively extending (L)ALM context length and improving long-audio reasoning.
VGGSound (Chen et al., 2020)
185,161
FMA (Defferrard et al., 2016)
106,412
Video Recap' (Islam et al., 2024)
64,627
CochlScene (Jeong & Park, 2022)
60,855
FSD50k (Fonseca al., 2022)
40,966
MACS (Morato & Mesaros, 2021)
31,675
Laion630kgseseendttee (Wu al., 2022b)
31,201
MagnaTag/ATune (Law et al., 2009)
25,863
SoundDeses (Koepke al., 2022)
23,085
Clotho (Drossos al., 2020)
19.195
TAU-Urban (Heittola et al.)
14,400
MusicCaps (Agostinelli al., 2023)
5,479
E.5. Success and Failure Cases on LongAudioBench
WayText5K (Deshmukh et al., 2022)
4.347
Table 22 presents success and failure cases of AF2 on Lon- gAudioBench.
SONICS (Rahman et al., 2024)
1,602
HavCaps-SoundBiiie
al., 2024)
935
MUSDB18 (Rafii et al., 2019)
276
MedleyDB- Pitch (Bittner et al., 2014)
103
E.6. Metadata
Total
8,037,712
For MiraData, metadata includes visual captions (e.g., main object, background, and style) detailed in the original paper and shown in prompt 9. For Video ReCap, metadata focuses on action captions describing activities, as shown in prompt 15.
obtained from Whisper Largev3 (Radford et al., 2023). We generate all audio captions from Qwen2-Audio and visual captions from Qwen2-Vi VL.
16

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table 10: Performance comparison of our CLAP with baselines on Zero-shot Audio classification benchmarks.
Model
ESC-50
VGGSound
FSDSOK
TUT
AudioSet
NSynth
Wav2CLIP
41.4
40.4
10.0
3.0
28.2
5.0
5.9
AudioClip
69.4
65.3
9.9
6.6
29.5
3.7
6.8
CLAP
29.6
9.9
Laion-CLAP
88.2
74.1
21.2
58.4
20.8
11.8
CoLLAT
84.0
77.0
19.0
29.0
9.0
CompA-CLAP
86.5
88.1
21.9
19.6
56.7
21.6
11.8
AF-CLAP-630k
74.3
21.2
57.5
19.3
12.0
AF-CLAP w/ van. con.
91.0
91.8
23.5
28.1
63.0
21.8
17.4
AF-CLAP w/o noise red.
91.1
91.3
23.9
26.9
63.2
22.8
15.5
AF-CLAP (ours)
91.3
92.3
24.1
27.2
63.2
23.7
15.9
Table 11: Dataset statisties for LongAudio and LongAudioBench.
Category
LongAudio
LongAudioBench
Audio QA Review
Captioning
67,498
917
Plot QA
61,511
237
Audio:
Complex QA
50,315
Subscene QA
7,908
176
Temporal QA
66,836
253
Question:
General QA
8,860
485
Ohsose the corrett cotton omona the cations below
Total
262,928
2,429
(A) ckkng metooy, women speaking English
Footiaps outdoors, speaking, melady
(G] lute mexdsy soman speating bnplish, foonstips outcoors meildy
Table 12: Comparison of duration statistics LongAudio with other AQA datasets. LongAudio stands out with the longest aver- age audio durations.
Method
Min
Max
Avg
Answer:
Clotho-AQA (Lipping et ul., 2022)
15.00
30.00
22.53
(A) clock roking a flus meloyy, a woman speaking English
AudioEntuilnent (Deshmukh et al., 2024)
15.00
30.00
22.49
CompA-R (Ghosh et ul., 2024c)
0.47
10.01
9.87
OpenAQA (Gong ct al. 2024)
0.06
180.00
12.01
MU-LLAMA (Liu et al., 2024)
29.12
29.12
29.12
Salmonn (Tang et al., 2024)
0.47
1069.04
10.62
Validation:
LongAudio (ours)
5.00
1797.71
117.08
Pass
Fail
Table 13: Fine-grained scores AF2 on LongAudioBench w/ and w/o fine-tuning on our dataset Long
Submit
Category
AF2
AF2 w/o LongAudio
Figure 8: Snapshot of the annotation tool used for LongAu- dio annotation.
Captioning
63.75%
46.02%
Plot QA
68.02%
44.15%
Temporal QA
62.61%
51.00%
Needle QA
63.13%
35.92%
each other. Thus, each QA pair was at least annotated by 2 individuals. Their primary role in both stages was to an- notate the data while ensuring accuracy, consistency, and adherence the predefined guidelines (mentioned next). We build annotation tool for this purpose, and Fig. provides snapshot of the tool. This multi-stage process, involving annotation followed by detailed review, was de- signed to enhance the reliability and depth of the annotated dataset, leveraging the combined expertise and experience of the annotators.
Subscene QA
64.03%
33.61% %
General QA
63.61%
42.39%
Avg
64.19%
50.22%
E.7. Human Verification for LongAudioBench
The human verification process has been approved by our institution's Institutional Review Board (IRB).
Annotation Method. The annotation process for LongAu- dioBench was carried out in two stages by group of experts. In the first stage, experts corrected and verified each QA pair, including discarding completely erroneous pairs. the second stage, experts reviewed QA pairs corrected by
Annotation Guidelines. The authors of the paper put for- ward the following annotation guidelines for annotating and verifying Long/uuioBBench:
Accuracy and Consistency: Ensure all annotations
17

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table 14: Detailed statistics of AudioSkills, categorized into individual reasoning types, together with details on open son-co datasets, additional meta-data, and prompt used for QA generation.
Reasoning Type
Size
Datasets Used
Meta-Datta Used
Prompt Reference
Temponil
188, 799
Sud MusicBerch
Time Stamped Events (CT)), Cipccon
Fig. 9, III
Armibute Mentification
201,45
Sud
Amribute values from Kumar (2024), Caption
Fig. 26 pythonic Fig. 13,27, 28, Fig.
Counting
50,493
Synthetic
Transcript, Caption
Contestial Sound Event Reasoning
982.847
AudioSet-S SL. MusicBench, FMA
Time Stapped Events, Caption
Contectual Speech Event Reasoning
1,272.782
AudioSet-S SL
Transcript. Caption
Information Extraction
858,639
AudioSet-S SL. MusicBench, MTG-Jameede Jamendo, MusicNet
Time Stapped Events, Caption
Fig. 31, 33, 31,37,3 Fig. 33, 35,33
Gencral Reasoning
704,040
AudioSSe-SLL MusicBanch, MusicNet, FMA
Time Stampeal Events, Caption
are accurate and consistent, strictly adhering the predefined standards and guidelines.
AQA format for training AF2. For the answers, only the la- bel was retained, excluding any additional text. Initially, we experimented with adding prefix to the labels to convert them into sentence format, but this approach did not yield favorable results.
Two-Stage Process: Ensure all QAs go through the
2-stage verification process mentioned above.
Listening Requirements: Listen to the complete au- dio before annotating to ensure the QA pair is contex- tually accurate.
F.2. Prompts for other tasks
Table. 26 and 26 list prompts we used to convert founda- tional audio classification and captioning datasets into AQA format for training AF2.
QA Pair Validation: i) Ensure the audio in the QA pair is valid, not corrupt, and corresponds to the question. ii) Discard or flag QA pairs containing irrelevant or ambiguous content.
1. Prompt 9: Prompt used for generating Temporal QA for LongAudio from MiraData.
Question Format: i) All questions must be in En- glish. ii) Check if the tagged QA type (e.g., captioning, PlotQA) is correct and relevant. iïi) Avoid mentioning identifiable information about the audio, such as names or metadata.
2. Prompt 11: Prompt used for generating Plot QA for Lodgoudio from MaraData.
Prompt used for generating Caption QA for Longo from MiraData.
Prompt 12:
Answer Format: i) All answers must follow the MCQ or open-ended format, pre-defined for each question category. ii) Additionally, annotators should ensure answers are comprehensive and free of ambiguities.
4. Prompt 13: Prompt used for generating Contextual Sound Event Reasoning QA for AudioSkills.
5. Prompt 14: Prompt used for generating Caption QA for Video-ReCap.
Tool Usage: i) Use the annotation tool (Fig. 8) for all corrections, reviews, and annotations. ii) Document and report any technical issues encountered during the process.
6. Prompt 15: Prompt used for generating Subscene QA for LongAudio from Video-ReCap.
7. Prompt 16: Prompt used for generating General QA for LongAudio from Video-ReCap.
Quality Assurance: i) Each QA pair must be reviewed by at least two individuals to ensure reliability and
8. Prompt 17: Prompt used for generating Subscene QA for Long from MiraData.
adherence to the guidelines. ii) Maintain detailed logs of discarded QA pairs and reasons for exclusion. iii) Escalate ambiguous or complex QA pairs to the team for collective review.
9. Prompt 18: Prompt used for generating Temporal Reasoning QA for AudioSkills.
10. Prompt 19: Prompt used for generating linguistically variant positives for short-audio captions for CLAP training.
Collaboration: i) Maintain open communication with other annotators to clarify task-specific doubts or re- solve disagreements during review. ii) Share feedback on ambiguous QA pairs to iteratively improve the an- notation process.
11. Prompt 20: Prompt used for cleaning and removing noise from synthetic short-audio captions.
12. Prompt 21:
Prompt used for generating composition- ally different negatives for short-audio captions for CLAP training.
F. Prompts
13. Prompt 22: Prompt used for generating synthetic short-audio captions from Video ReCAP.
F.1. Prompts for foundational datasets
Tables 26 and 27 list the prompts used to convert founda- tional audio classification and captioning datasets into the
14. Prompt 23: Prompt used for generating synthetic short-audio captions from MiraData.
18

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table 15: Category-wise examples from LongBench.
Category
Question
Answer
How would you describe the scene from the audio?
Two women converse in car with ambient bird chirping and occasional mechanical noises background.
Captioning
What is taking place in the audio?
The audio features ambient wind, distant motor vehicle hums, and occasional background noises creating an eerie atmosphere.
What is the predominant mood conveyed by the background music and sound effects in the audio segments?
Sad and suspenseful.
Plot QA
What is the primary theme conveyed by the audio in the initial segments?
The audio conveys solemn and spiritual theme with classical music and religious references.
What the correct order sounds in the audio? Choose the correct option among the options be-
low:
(A) clock ticking, flute melody, woman speak-
ing in English
(B) Footsteps outdoors, a man speaking, piano melody
(A) A clock ticking, a flute melody, woman speaking in English
(C) A flute melody, a woman speaking in English,
footsteps outdoors
Temporal QA
(D) A clock ticking, a man speaking, a piano melody
When can the sound of a horse galloping be heard? Choose the correct option among the options be-
low:
(A) At the beginning
(B) In the middle
(B) In the middle
(C) Towards the end
What audio event indicates an interruption during the conversation between the and woman?
The background sound of phone ringing from 0.53 seconds to 7.40 seconds.
Needle QA
What audio element creates a sense of suspense in the dark room scene?
The cinematic strings playing a scary tune along with a church bell and distant horror scream.
What happened between the sound of ocean waves and the sound of mechanisms functioning?
The sound of phone being dropped on table followed by mechanical fan noises.
Subscene QA
What sounds occur between brushing teeth and using phone?
Music playing continuously with occasional run- ning water and shower sound.
What continuous background noise heard while the person using the laptop?
The sound of an engine idling consistently.
General QA
What was the continuous background noise throughout the audio?
Engine idling.
15. Prompt 24: Prompt used for rewriting audio captions and removing implausible acoustic elements.
17. Prompt 26: Prompt used for generating Attribute QA for AudioSkills.
16. Prompt 25:
Prompt used for evaluating responses for questions in Bunnee..i
18. Prompt 27: Prompt used for generating Complex QA for AudioSkills.
19

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
19. Prompt 35: Prompt used for generating QA for Au- dioSkills using FMA.
Table 18:
Comparison of performance with various cross-
attention conditioning frequencies.
Frequency AudioCaps
GTZAN
MuchoMusic
MMAU (avg)
20. Prompt 36: Prompt used for generating QA for Au- dioSkills using MusicNet.
0.50
56.7
50.1
61.7
0.58
70.0
54.4
65.7
0,58
69.1
56,5
69.0
21. Prompt 37:
Prompt used for generating QA for Au- dioSkills using MTG-Jamendo.
22. Prompt 38: Prompt used for self-verification gener- ated QA pairs for MiraData.
G.4. Results on different LLM Sizes
23. Prompt 39:
Prompt used for self-verification gener- ated QA pairs for ReCap dataset.
Table 19 compares the performance of AF2 on various LLM sizes, ranging from 0.5B 7B.
G. More Results and Design Analysis
Table 19: Results for AF2 different LLM sizes, ranging from 0.5B 7B.
G.1. Effect RoPE
Dataset
AF2-0.5B
AF2-1.5B
AF2-3B
AF2-7B
Table 16 compares the performance of AF2 with and with- out applying RoPE in the audio transformation layers. We a drop in performance in LongAudioBench and MMAU without the application of RoPE, which highlights its impor- tance in long-context and reasoning capabilities.
AudioCaps
0.46
0.50
0.58
0.58
GTZAN
65.6
66.1
69.1
68.8
ClothoAQA
80.4
82.7
86.9
85.1
MMAU Sound
61.0
65.0
64.4
65.0
MMAU Music
68.0
70.9
72.9
73.1
MushoMusic
49.7
52.4
56.5
57.2
Table 16:
Performance comparison of AF2 with and without RoPE in the audio transformation layer.
CompA-R-test
82.4
92.4
96.4
95.1
Frequency AudioCaps
GTZAN
LongAudioB
MMAU (avg)
RoPE
0.56
w/ RoPE
H. More Details
H.1. Topic-wise Video Segmentation
G.2. Effect Audio Transformation Layers
To segment videos into topic-specific clusters, we adopted multi-step clustering approach utilizing both textual of video captions. Below, we detail the methodology:
Table 17 compares the performance of AF2 with and with- the audio transformation layers. We see drop in per- formance without the audio transformation layers, which highlights its importance in expanding audio representation learning and adaptation capacity.
1. Feature Extraction:
We extracted semantic feature embeddings from video captions to represent each video. To achieve this, we employed NV-Embed (Leee et al., 2024).
2. Clustering: Next, we applied K-Means Clustering to group videos based on their feature representations.
Table 17: Performance comparison AF2 with and without the audio transformation layer.
Frequency
AudioCaps
GTZAN
MuchoMusie
MMAU (avg)
3. Cluster Selection and Diversity: After clustering, analyzed the clusters to ensure relevance and diversity. For each cluster, representative subset of videos was selected using a combination random sampling and Determinantal Point Processes (DPPs) (Kulesza et al., 2012), a diversity- based scoring method. This ensured that selected videos captured the range of topies present in each cluster while avoiding redundancy.
transform 0.55
64.3
51.8
59.0
transform 0,58
69.1
56.5
69.0
G.3. Effect of Cross-Attention Frequency
Table 18 compares AF2's performance across different cross-attention conditioning frequencies. As reminder, conditioning the audio representations using cross-attention after every LM layer (frequency of I) yields the best per- formance. However, conditioning every 3rd layer performs competitively, with a noticeable performance drop when conditioning reduced to every 6th layer.
4. Manual Review and Refinement: To further ensure the quality and relevance of the selected videos, we con- ducted a manual review of a random sample from each cluster. Videos with ambiguous or low-quality content were discarded or reassigned as needed.
20

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
H.2. Data Loader
far. If adding the next audio clip violates either max max_tokens or max sentences, the current batch is finalized, and new batch is started. Additionally, if the batch size is multiple of bsz_mult, we consider finalizing the batch early to meet this alignment requirement. Any remaining examples at the end of the data list form the final batch.
AF2 trained on audio datasets with durations ranging from 0.5 seconds to 10 minutes. Using generic data loader would result in excessive padding within batches, leading to unstable losses. To address this, we implement dy- namic batching scheme that groups audio samples of similar lengths, minimizing padding. This approach is further con- strained by a predefined maximum duration for each training job. The algorithm is detailed in Algorithm 1 and 2, with its two main components explained below:
This two-stage approach first weighted bucketed blending, followed by dynamic batching-
Multiple datasets contribute examples proportionally to their desired weights.
Weighted Bucketed Blending. We begin with several datasets, each assigned a specific weight that determines the relative number of items sampled from the dataset per epoch. For each dataset, audio clips are divided into buckets based on their duration-e.g, short clips are grouped one bucket, medium-length clips in another, and so on. This bucketing process minimizes padding overhead by ensuring that only audio clips of similar lengths are grouped together.
2. Effective shuffling to prevent overfitting.
3. Efficient grouping of audio clips by length, reducing wasted padding and speeding up training.
Effectiveness. Our dynamic batching scheme reduces the percentage of paddings in each batch from 58% to 16%.
During training epoch e, the number of items to sample from each bucket is determined by the following indices:
H.3. Computational Analysis
Training Cost. We trained Audio Flamingo 2 on 128 NVIDIA A100 80GB GPUs for each stage and variant. For the original version with the 3B model, stage takes about days to train, stage about 5 days, and stage about 12 hours. We provide detailed cost analysis in Table 20.
start idx= = Le x bucket size x weight % bucket size, end_idx [(e + 1) x bucket_size x weight % bucket_size.
These indices define "slice" of the bucket for the current epoch, ensuring that each dataset and bucket contributes a controlled number of items. If the index i completes an entire bucket (i.e., when i mod bucket_total = 0), the bucket is shuffled deterministically using a seed derived from the dataset name and the current epoch. This approach maintains randomness across epochs while ensuring repro- ducibility.
Table 20: Final model training cost comparison of Audio Flamingo 2. Costs are estimated using AWS (e.g., https: and
wawaancetttte/pppeeeee excludes LM training costs and other failures. The cost of experi- mentation is about 5x the final model training cost.
//aws. amazon www/ec2/
LLM Size
Total Params
GPU Hours
Estimated Cost
0.5B
1.0B
104
RUSD 55K
Dynamic Batching. After blending all datasets and gath-
1.5B
2.5B
160
sUSD 84K
cring their examples into a single list, we apply Dynamic Batching to form more efficient mini-batches. The primary goal is to group audio clips of similar lengths to minimize the total number of padded frames per mini-batch. Each audio example's 's token count (e.g., frame count or another relevant measure of length) is used to incrementally add examples to batch, subject to the following constraints:
3B
4.7B
228
RUSD 120K
7B
11.0B
510
UUSD 268K
max sentences: The maximum number of examples allowed in a batch.
max tokens: The maximum total frames/tokens permitted in batch, calculated as batch size x maxio
bsz mult: An optional alignment constraint requiring that the batch size either be smaller than this multiplier a multiple of it, often for multi-GPU efficiency.
At each step, we compute the total frames/tokens in the ten- tative new batch, considering the maximum frame length so
21

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table 21: List fine pre-training and fine-tuning datasets together with their training composition.
Dataset
Audio Length
#Audio- Text Pairs
Pretraining Epochs
SFT Epochs
AudioSkills
4200K
2.0
CompA-R
159 hrs
350k
2.0
MusieBench
115.5 hrs
686k
2.0
Mu-LLAMA
62.9 hrs
70k
2.0
Salmonn AQA
800 hrs
270k
1.0
ClothoAQA
7.4 hrs
9.7K
1.0
OpenAQA
693.2 hrs
1.0
Clotho-v2
24.0 hrs
19.2K
1.0
MACS
10.9 hrs
17.3K
1.0
FSD50k
80.8 hrs
41.0K
1.0
CochIScene
169.0 hrs
60.9K
1.0
NonSpeech 7k
6.2 hrs
6.3K
1.0
Chime-home
5.0 hrs
4.5K
1.0
Sonyc-UST
34.9 hrs
27.9K
1.0
Emov-DB
7.8 hrs
6.8K
1.0
JL-Corpus
1.4 hrs
2.4K
6.0
Tess
1.6 hrs
2.8K
2.5
OMGEmotion
3.0 hrs
1.7K
3.0
MusicAVQAnaulaan audio-unly
77.1 hrs
5.7K
3.0
MusicQA
62.9 hrs
70K
3.0
AP-MusicCapSMMD
5805.7 hrs
1331.8K
0.026
D-MMusiCaaSMITT
126.4 hrs
46.9K
1.0
LP-MusicCapSMc
7.4 hrs
7.9K
2.0
MusicCaps
7.4 hrs
2.6K
6.0
NSynth
321.3 hrs
289.2K
1.5
MTG-Jamendo
3768.9 9 hrs
55.6K
1.0
MusDB-HQ
29.1 1 hrs
10.2K
2.0
FMA
860.7 hrs
104.2K
1.0
OpenAQA
693.2 hrs
1959.8K
1.0
1.0
Laionn6O0ggSSoondEfffers
456.9 hrs
15.1K
5.0
Laion630k
2494.8 hrs
306.5K
1.0
SoundDescs
749.7 hrs
23.
1.0
WayCaps
3793.3 hrs
402.6 K
1.0
AudioSet
2617.8 hrs
950.8K
1.0
T
23.8 hrs
4.3K
3.0
MSP-Podcast
73.9 hrs
45.
3.0
MELD
8.7 hrs
32.9K
3.0
MusicAVQA.. Anndii-viiiii
142.4 hrs
17.9K
3.0
MusicQA
62.9 hrs
70K
1.2
I.-MMusiCaaSSMSD
5805.7 7 hrs
1331.8K
1.0
MTG-Jamendo
3768.9 hrs
55.6K
1.0
22

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table 22: Examples of Highest and Lowest Scores for Different Reasoning Types
Reasoning Type
Prompt
Output
Model Response
Score
Give description the atmosphere in the audio.
Continuous background noise with occasional sounds of objects being moved and water running in apartment.
Continuous
background
noise with occasional sounds of water running and objects being moved.
Captioning
Give a general description of the au- dio content.
Background noise and oc- casional speech are heard as someone moves around house, decorates, watches television.
man speaks in English with neutral tonc, accom- panied by background noise occasional mechanical sounds.
What can be heard at the beginning of the audio? Choose the correct op- tion among the options below: (A) A
(A) A man speaking in En- glish
(A) A speaking in En- 10 glish
man spcaking in English, (B) Clas- sical music with drums, (C) Instru- mental electronic piece, (D) A pop song.
Temporal QA
What indicates a transition to an in- tense and momentous scene during the group of men riding horses in the desert?
The wide brass melody over powerful string section cre- ates a cinematic and epic at- mosphere, suggesting im- portant moment.
The sound train horn honking followed by train passing by.
What do hear between the piano playing and guitar strumming?
hear a male voice speaking in English with sad mood.
A male voice speaking in En- glish with sad mood.
10
What happening between sound of game music and the idling engine?
The scene consists of per- son playing with the re- mote control with back- ground whirring and clicking sounds.
man speaking En- glish.
Subscene QA
What sudden auditory change oc- that might indicate shift in the person's physical state while navi- gating the snow?
A significant increase breathing and panting sounds, suggesting physical exertion difficulty.
The sound of breathing be- comes prominent, suggest- ing the person may be exert- ing effort or experiencing comfort.
Needle QA
What does the woman mention about the view from the bedroom?
She mentions seeing the red and yellow cliffs from her bedroom in the morning. Sad.
She mentions that the view from the bedroom is amaz- ing.
2
What is the mood conveyed by the male voice singing in the audio?
The male voice singing in the audio conveys a deep and res- onant mood.
10
Plot QA
What innovative feature does the man highlight about the Pulse smart watch?
It can without using a phone.
The man highlights the smartwatch's ability to monitor heart rate and sleep patterns.
What natural sound is consistently present throughout the entire audio?
Birds chirping lightly.
Birds chirping lightly.
10
General QA
What sound is heard consistently throughout the audio clip?
Continuous
background
The sound of rooster crow- ing.
noise.
23

---

Audio Flamingo 2: An Audio-Languaae Model with Long-Audio Understanding and Expert Reasoning Abilitics
Algorithm Weighted Bucketed Blending & Dynamic Batching
Require: all_data: A dictionary of datasets, each with contents (bucketed examples) and weight
Require: epoch: Current epoch index
Require: IsBrokenFile()): Checks if an audio file is unreadable
Require: ShuffteDDictFixeRRandd/ seed): Shuffles the bucketed data deterministically
Require: batch size size.fn()): Function implementing dynamic batching by size (Algorithm 2)
function (aAEN epoch)
self _dta + (data: : (), total_mn _num 0}
dataset_name all_data
contents 4- all data dataset name)..cntent contents
weight all_data [ataset_name) weight for (bucket_idx, bucket_data) contents do bucket_total <- |bucket_data]
if bucket. = 0 then
continue
Skip empty buckets
end if
start_idx lepoch x total weight] mod bucket_total
end_idx + (epoch + 1) x bucket_total x weight mod bucket_total
for idx +- start_idx to end_idx
do
if (idx > 0) A (idx mod bucket. total 0) then seed <- SumOOCharCodees (daaaaassedd(dttttttt.iee daatasttttaaneee || "epoch" || Lidx ftdxx/bucke bbcckettttaal)
bucket_data <- ShuffleDictFixeRRandll htwpe//eeednndduueebeeeeeee (dacket_data, seed)
end if
key + idx mod bucket_total
item < bucket data[key]
if (BBROKENFLLEiiemm(nama then
continue
D Skip broken files
end if
self data["data"]]ssae data[" totall num"]] <- item
_numm]]]_mm]]] t- dell_ddta[[totallnnn ""ttttttllumm"" +
end for end for end for return self data end function
function DYNAMICBATCHINNC(niies num tokens fn, max tokens, max sentences, bsz mult) High-level wrapper for dynamic batching
batches + batch by size fn(indices, num tokens fn, max tokens, max sentences, bsz mult)
return batches
end function
24

---

Audio Flamingo 2: An Audio-Lange Aaniu-La Model with Long-Audio Understanding and Expert Reasoning Abilitics
Algorithm 2 Dynamic Batching by Size (Pscudocode for batch.by.size.ff
Require: indices: Array of example indices
Require: numTokensVeclle Number of tokens (or frames) for indices
Require: max_tokens, sentenee:::::: Batch constraints
Require: bsz mult: Required multiple of batch size (e.g., for multi-GPU alignment)
function BATCH_BY SIZLSSSIEEEECiiiIe numTokens Vec, max mak_tokens, max_sentences, bsz_mult)
batches + []
D List of subarrays of indices
batchStart <
tailMaxTokens <
batchMaxTokens t
for pos to lindices| 1 do
tailMaxTTkeens max Max Vec/oo)
newBatchEnd + pos -
newBatchMaxxToees < max(bacchMaxTToens, MailMaxTokens,
new.BatchSeeneences < EnwBatchEnd - batchStart
new.BatchNumTToens + newBatchhSentnness new.BatchMaxTokenn
overflow + (new BatchSentences max_sentences > 0) (new:BatchNum Tokens > max tokens >> 0)
sizeOk + (newBatthhSentenness bsz_mult) V (newBatchSentences mod bsz_muil = 0)
if overflow then
Finalize the current batch and start
Bacchss..PEENDOnniccc///llaaaia pos])
batchStart + pos
batchMaxTokens <- numTokens VecTTooensVeepppooooo [po]pos]
tailMaxTTokens +- numTokens Vec [pos]
else if sizeOk then
D Optionally finalize batch size is multiple of bsz_mult
batches.APPEND banidiicebsacchtarrtt...
batchStart: newBatchEnd))
batchStart newBatchEnd
batehMaxTokens <-
tailMaxTTkkens <- 0
else
> Continue accumulating examples in the current batch
batchMaxTokees <- new.atchMax Tokens end if end for
Add any leftover examples in the final batch
if batchStart lindices| then
:))
end if return batches end function
25

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table 23: All datasets used to train our model. We mark the datasets used in Stage 1', Stage 2 Stage 3 or multiple stages of training 1,2 Datasets marked with * were added to AF2 over the ones already present in Audio Flamingo.
Audio Type Task
Datasets
#Audio Text Pairs
WavCaps' (Mei al., 2024), Macs (Martin Morato Mesaros, 2021),
CAP
SoundDescs' (Oncescu et al., 2021). Clotho-v2' (Drossos et al., 2020).
829
WavText5K' (Deshmukh al., 2022), Laion-630k (Wu et al., 2023b)
General Sound
Clotho-AQA (Lipping ct al.. 2022), Open-AQA? (Gong et al., 2023a),
AQA
Salmonn AQA* (Tang al., 2023), AudioEntailment" (Deshmukh et al., 2024)
~990 K
CompA-R* (Ghosh et al., 2024c), AudioSkills 1.2" (ours), LongAudio" (ours)
AudioSet (Gemmeke al., 2017). FSD50K (Fonseca et al., 2021),
CLS
CochIScene (Jeong & Park, 2022), NonSpeech7K "(Rashid et al., 2023),
~ 1091 K
Chime-Home" (Foster et al., 2015), Sonyc-UST' (Cartwright et al., 2019)
CAP
LP-MusicCaps' (Doh et al., 2023), MusicCaps' (Agostinclli al., 2023)
1388 K
MusicQA (Liu al., 2024), MusicAVQA (Li al., 2022)
Music
AQA
MusicBench" 2* (Melechovsky et al., 2023), Mu-LLAMA" ?" (Liu et al., 2024)
~94 K
AudioSkills (ours), LongAudio (ours)
NSynth² (Engel et al., 201 MTG-Jamendo (Bogdanov et al., 2019),
CLS
~559 K
FMA (Defferrard et al., 2016), MusDB-HQ - (Rafi et al., 2019),
MSP-Podcast' (Lotfian & Busso, 2017), Emov-DB' (Adigwe et al., 2018)
Speech
CLS
JL-Corpus (James et al., 2018), Tess (Pichora-Fuller & Dupuis, 2020),
~92
MELD² (Poria al., 2018), OMGEmmoion" (Barros al., 2018)
Table 24: All datasets used for evaluation in AF2, with additional usage/contexttual notes for foundational datasets.
Dataset Type
Audio Type
Task
Datasets
CAP
Clotho-v2 (Drossos et al., 2020), AudioCaps (Kim et al., 2019)
UrbanSound8K (USD8K) (Salamon al., 2014) (environmental sound classification),
ESC50 (Piczak, 2015) (environmental sound classification),
CochlSeene (Jeong & Park, 2022) (acoustic scene classification),
CLS
Sound
FSD50K (Fonseca et al., 2021) (sound event classification)
CREMA-D (Cao et al., 2014) (emotion classification), Ravdess (Liv- ingstone Russo, 2018) (emotion classification).
Foundational
NonSpeech7k (Rashid al., 2023) (non-speech audio classification)
AQA
Clotho-AQA (Lipping et al., 2022) (audio question answering)
NSynth (NS) source (Engel et al., 2017a) (instrument classification). NSynth (NS) instrument (Figell et al., 2017a) (instrument classification.
CLS
Music
GTZAN (Sturm, 2013) (genre classification), Medley-solos- DB (Lostanlen et al., 2019) (instrument classification)
AQA
MusicAVQAAudio (Li et al., 2022) (music audio question answering)
OpenAQA (Gong et al., 2024), MMAU Sound (Sakshi al., 2024), AudioEntailment (AE) Clotho (Deshmukh et al., 2024), AudioEntail- ment (AB) AudioCaps (Deshmukh al., 2024), CompA-R-test (Ghosh et al., 2024c)
Sound
AQA
Reasoning
MMAU Music (Sakshi et al., 2024), MuchoMusic (Weck et al.,
Music
AQA
2024),
MusicInstruct (MI) (Long) (Deng al., 2023), MusciQA (Liu et al., 2024)
26

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Table 25:
More examples from AudioSkills.
Category
Example
Temporal Relation-
Order: In what sequence do the sounds appear in the audio? (A) Car engine, honk,
ship Identification
music (B) Music, honk, car engine
Attribute: When does the breathing sound change over time? (A) Disappear (B)
Get louder (C) Get soft
Grounding: When does the sound of bird chirp occur in the audio? (A) Beginning
(B) Middle (C) End
Referring: What sound appears first in the audio? (A) Music (B) Car honk (C)
Dog bark
Order: In what sequence do the footsteps appear in the audio? (A) Slow footsteps, running, door slam (B) Running, door slam, slow footsteps (C) Door slam, slow footsteps, running
Attribute: How does the volume of the rain sound change over time? (A) Increases (B) Decreases (C) Stays the same
Grounding: When does the sound of a baby crying occur in the audio? (A) Beginning (B) Middle (C) End
Referring: What sound appears immediately after the thunder? (A) Wind blowing
(B) Rainfall (C) Car alarm
Contextual
Speech
How might the playful interaction between the boy and the goat be affected by the tone of the speech and the background sounds?
Event Reasoning
What can be deduced about the environment and the relationship between the boy and the goat from the audio events?
Contextual Sound Event Reasoning
What might the presence of the music and its continuous play suggest about the overall atmosphere of the scene?
Based on the timing and sequence of impact sounds and male speech, identify the possible work environment depicted in the audio.
Counting
Level 1: How many times was the hammer sound heard?
Level 2: How many times did the first sound occur in the entire audio?
Information
Extrac-
What key is the main melody of the audio played in? (A) major (B) major (C) A# major (D) G# major
tion
How does the melody in the audio contribute to the hypnotic effect of the music?
(A) By changing frequently (B) By maintaining consistent and repetitive loop (C)
By using multiple key changes
General Reasoning
Considering the chord sequence Bm7, D, Bm7, D in the audio, what is the role of the melody in relation to these chords? (A) follows the progression (B) It decorates the progression
How does the melody in the audio contribute to the hypnotic effect of the music? (A) By changing frequently (B) By maintaining consistent loop
Attribute
Identifica-
Which event has the highest pitch in the audio?
tion
Which is the least loud event in the audio?
27

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Proept
tou are a helpful al uss assiatant. You nead to act as questoonranse generator for long audios that yenerate poescionkkooeer paizo that zequice at sndacarancing of the andez sound avaese, These cuastions reter to typas af queations that woold sax the neda; about the coder of particciar sound eventa in the codio. The queation ahould be of Ho2 tope. You can flexible with the narbur optiens. The qusatiens should reqaiza texprrai zeaanting tinal acurced
queation infermation provided suisiple typus
abllition, pairs:
qusezate
Video Blosk Cuptioon aloeet euption sumatixing shat the untian video is abock, Liss a visuel pezspectioe.
Vituoo Dettae Cuptions detielled caption datailing all somia Lox uniicw vitie, Izom vixual prespentivn.
degect Eapeson; capeitn deserbbing
video, vizusl parspaccize.
Video Style Caption: A short captias conearizing the style, in an abstract sarnar, in which the entire video in thot, feom viausl parspactive.
Captions caption deseiiiing the
unglen
untire
Lius visuel perspeetive.
Video and kadta Segaent Captions:
this te the nost ixportant inforration which you stould focus on, this lett of llate. Rvery inser 11st bas eapsione that deseribe secand segrast of the long video, The fizet caption describen the video vaile the ascend caption dasczides the audic the 10 second sagnent,
zeguens,
secranta cheenslegical codes,
fires
segeen:
Genscace queatien-annve pair with the following zeqriceranta:
2. Toe queatiens should be the dolloving topear Ordes, Attribute, Democzul Ratercing and Desporal Grounding.
3. Wetcher the proupt nor the reeponce shoold be sere thar 15-30 words,
"acoording the detella"
pesepta tuspenssst you include
you hearing
yourseli.
cupluens
Linestanos please
they use molay.
queations
eotho autte captionns ray be netey and not correct, You need to vertiy using the useus! eagritone and the other uteust cose before franing your Oh patr, kaditionatly, use the other steal detatie inderatand the matn ieporeacce of the broader scese and asice oo paine accoringgly. tsb Mort trgartantly, only genarate questione and
perepactivn, Loh., you heazing, to agent
only by hearting the
not
requice visusla.
sere in an exaxple with just the ospciona 1200 will be procided with more Information, as scated carllerl:
"captiene": EEMA in black aust in atanding in rom. le in holding spoon in his right hand and tork in his left hand. He in looking the apoon and dask. The mas then pata the apeakl slectrente kayboard haraony and nanacing basd on the callo. addicionall,, thare da a make apaaking in Razdacin, who awara to be batwean 16-25 years old and oenorgs feartel -lod.*l, (1) nan sleeing in room with opeen his rand.", *In this audio, there waste playing fioe the begtertng fng the end. frexte estes is rpeaking in Rnghtich (eA "), buckgsound
and tezk
nouts. ", "Ihe codio containa backquound of oninous instrurntte rusic with slow texpo deaturino
eseriig zad nood, exying "hey and bend the spocnt There's alse the sound breathing in berweer the spoken looking agaun.. the
speaka holding spaon seying Lhey appeas audio:
playing
speakezts
sype)
(Ai Konan spesking for the diras tims
12) child and speaking about
for dunsbets
12) zan shouting 'May" Anaver (Drdez typat: (3) (J (D) (2) Thie quastizo eac. alzo be pirazed as "AAzzagee the zounds according to thair sequence" and then you ozovide the optioon and the anivers.
Question /Tengoral Referring type) Khat heard towordo the beginning of aodio?
(ki wan shouting Ney?
12) Gunabeta
spesk
13 k chsld and van talk abont spoon knewer (Reeporal Referntnng rypels (0) & child and a iann TATR sbout egoon Thie questiin osn 5150 be prrssed 25 "Shat haggene 5*
Qusation (Tagueall Grounding Lype''s
sudiinly ganabocs lourd?
[Ai the begtn
131 the aiddle
IC and
Jazwez (Zemporal Goounding typel: (C), the and
Queation (Atteibute sypel:
intenetey of change thecugheut
IAI .t becomsa aore Londes and incenue
St. reraina unchanged
[C) St bacemsa more zoftez and cain
Anawex: (Atceibute type) (AI it becoses sore louder and intense
par type.
["tespors] Badezring Quastion": Question,
Anavec";
"Zemssrul Grounding Anavez:: Anavez, Quastios?; question, Ansvart: Anaves, Queatimn:: quastien, "Atcribute Anavect: Asswee].
onky retuan the 2008 and nothing slas. Runarbur that the sounda abould distinpulatable feom supprral parapective and say attanzion that cosations not. giestion type not porstble, e, don't sorpor and on'y cutprr
overlapping sounde eseporal order or conparison to input
Figure 9; Prompt I used for generating Temporal QA for LongAudio from MiraData.
28

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
: went you to ast us complex queation/anaver qunazator, 2 will provide you with on audee exption that describes the order in shich milkipla acunda oceus hasing vords Like "wisd", "atter', "followe by", win.ts aetitienally I vill provide a List ef sound avents with disulien information (aug,, Jais 2.000- 20,020: the react Lime La 0.000 auconda, and the Las 20.000 secomda, au diselim
Lime atuct Lime 10.000 ausondal- Your lask generave oi cupplax mulliplertiiiin queations Megu) with fous optisos. how procided the queztion lipe and souple quu Lisn, souwura buloe
Demporal Evant Reasoning
[inderatanding the zequanes and sining of eventa in the aodio. For the givan audio, idaatity which the following sounds haard the Senpeat I
cutio cassion: tx zire tzuckla sizen can be hasrd over tzattic and basan veicea."
Met of sudio events: "I (Prre engine, fire track (stnen)- 000-13.0000 0000,,, i, "ctrafete rofee, madiay net nefise-b.co con- 0,030y,, " "joosnn volosed 226-5
Pype 1. Queativnz
longest
the collowing Reazoning far Queation Genaratiion: this quaation focuses sodia inderatanding of model. oy:ions:
hoard longeat
5. Fize ancice, fize treck
2. Bazan voice
3- Rind
Necwtigg
opetons
Pizze engtie, fine trick
Rolas for generating options
skip the questiona Zor wiich the Longeat nolse la background sound soise
2. You ssculd keep some incorsect ogtiona Cxom the sample event list like busan voice and others sound event which are not present In the List of audio events Wind, eat Meoking
Reaconing:
caloutution:
Fire englne, fire trick (etren)-d -d, 200-10.000 b00 -> 15.003 0.000 10.000 lend time timee
"tuille auise, coudhey noive +> 10.000 - 0.000 12.003
Ruzan vatce =a 9.221 1,546 5.66
2. Mutch sith options only rize engine and busan volce is present in the option, do the cosrect ansvex in tire engine, fire crack
Type 2, Identlffy
doracions
the rotlowing Seseple for foestion Generation: this quertson fecuses andta underatanding of andel.
autto,
heard shortest duration.
Opriona:
Fise umgane, fizw Lzuck
2. Baran voicH
3. Wind
Mecwing
Opeion:
Vaile dazigning the optiions, unauze they contreat clearly with the correct choine, naking vazinz for sodal to recsening akilla idvatity the Lous cause of the chsiddee esotioral reepones.
Gsidelinwa:
to Risaye ast about evarts with signtticant duration. lhe the start and end tines provided to determine the doration after each event. Iynore instgsifiant eventa, such backgroudd stacio noise, which confus liaceners due their anbigueus nources,
touroes, like ieeriiiabbe tesying, elapping arival Darkinng neowing)
doean't inclade Eech quaztiona rust prepended by phessez libe 'Beadd the given sodio', "he the given cutio aupple', "Gives the eadio sumple' Ouspat Fornal:
andio desoription and list audio
guestions optiond
The ouspas Joamet aus be joen, I vank you su gunurate queation for euch csteprinn, Buru ia us unapple:
("Gonession type's equestion types, "Guserse"" ogenerate gueetienes, 'Optiene': (schoice'a,, schalcelo, cchsteels, ochoscedol, 'nnowere's : ccorrect optiens, Reason: cressen for zelecting a options, Oategory: "Terparal"s i.. Make sure quastion, options and answere ruet not exceed 20, 15, 1b wards. esprion:
Figure 10: Prompt 2 used for generating Temporal QA.
29

---

Audio Flamingo 2: An Audio- Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
4 Proapt
You are helprui AI axxtatant. You reed to act as grantsoe-anewe genacater for long sodies that gensrate Goertton-Koaee: paize, want you to generate quest-ci-anover pairs that queation broades plot the autios questions should requize aufficient underotanding of contents the audio reasoning based the understanding to boouracely answered, My final objective to train audio agent these question-anoee lenaraodio both the sodie and video to generate the pairs:
paiza
sultiple typea
'nformation
Video Sheas Deptiens A abozt caption auraacicing what the untioe video is diess, from a visual parspretive.
Kideo Denaie Duptiens detailed caption deteiling all
the uniice viduo, from viaual perapeatoww.
video Bain Sbjoct Captions A cuption ducriiiag the main Zooas of the untire video, frux visual perspective,
video Style Daptions A sbozt caption sumsaciaing the atyle, an abatraot nanner, in which the estize video is shet, fron visual perapective.
video Canera Captions A caption describing the casera angles in which the eatize video is shot, from visial perspective.
Viden and Audio Segrant Captiona: This in the rest inpestant information which you ahould tocus on. This is list lista, Every inner List han 2 copt that describe secend segrent
long video, The firee oxgtton describes the video esdae,
second csptton describes bodio the secuant,
aeguenta in the avcund 10 aegaunt
that fizat
Uenezate queation-nnveer pait with the following requicamnta:
1. The answers should be abstract and cripp and not desoribe each detail,
Neither the prospe for the responce should be more chan 15-20 words,
3. to not shrazes like "sccerdine to the decallst in both the occepts or zumoonaea: you ahould ank and anaver 1d you ware hearing the audio youreelr. alss, If the capttors have tirestarge please Ignore thes as they are noragy to not inolode then in the guestions or answers.
rrth andio capiiion zay noiay and not correct You need to varify wsing the viauel captions and other visual cusa bufore feaning your CA patn, aecteionally, use the other visual detatlz to understand the
exportance the broader zcens make pasis accordingly..
5. Nost tepartantly, only geserate the quaetionz and ansuace troe an andio perepactioe, sie., what you are hearing, An agent should be able to anever it only by hearlng the soundo. not ask questions that zequize visuals.
Zaze in axarpie with fuat the cantiona tyes will be provided with nace infermation staced marlinc): ""captionat:s 1LA man in blacs suit in standing in a roor, Re to holding a speon in hie right hand and a fork in hie left hand, Re is looking at the spoon and fork, the van then pots the apeon and fock in his nouth.", "The cocio containo backaround of criinus inavrmmetal maie wits a slov teapo Leaturing a apocicy slactzenic hayboard sazaony and ausacing bun on the cullo. addilienaly. chuce is u msle apeaking in Sundaran, sho sour to be buteeen 16-25 yours old and conveys Coazful soud, thr (*A oa in sitling in a zo0n with apeon in his band," by tin this andio, these is muis playing fros the beginiing to the und. A fezain voice ss spaskinng in Regiish with sad maod, saying "tay and band the spson", , There's aiso the zound of breathing
between the spaken vorda. ".. 1"A nas in black suit sitsing in roon with a shibe chair and wooden casinet. Be in hoising apcen his handa and Looking as it.". backgroond contains quiet ruelo playing throughout the diration of audio, A volce speaks in tnglish caying 'shat truche
apean." The apeakez's ao sone austenl. Ih 1t ran is aissing as table with spoon his hand.", "the andie consaina gh pitch
aynch pad playing speech
nevert
woice can wiile ahimscring bella Soding
saying
only youresir..
oulivured Eoglish arbient teack
singus, "), (%A
looking
belind
"shi codio
torale voice speaking in Kandarin can almo be heaed saying "yas2atesbbb Lonliihh plach clong
aynch playing
anaeraaareeeee weweeeettttttteeeeee te telibieee The speach has Barbin th, 1"A nan tocking hisselr in slvror. "In this
seiae alona with intermistant 36-yaa--old ferale ruzzenaniiiinue quoes aounda smmmoking surface contact at intareela (0.58 to 1.37, 2.43 2.54, 2.85 to 3.97, 2.47 3.64, 3.07 to 1,05, 1.46 to 4,61, 6.8 to s.55, 5.43 to 5.6,, 5.66 to 6.04, 6,43 to 6.67, 6.05 to 7.03, 1.06 to 7,62, 1,86 to 3.00, 1.0 to R.61, 8.35 9.03, 9:42 tath, 1Th seean in a blace hoodle ie holding gmn and pointing to at ran in a black zute, the san ie standing in front of window The venan is your lng aunplannes and has has hait pulled back. The san is sour aug black ouit and has a black bell. Yhu musu in holding a black gan and zas har hate polled back, The nan is weating black suie and has a black beit. The weran in haiding biack zan and has her base pulled back. The zan is wearing a black suit and has a black belt. the sonan is Selding a black gun and has her haie palled back. The ran is wearing", "the audio concains cunahota, video cuza asckquoand neiee,
Handarity delivered
there continunos baokground
qusazic imsset sounda, axplesions, and qusticn. Aoditionally, there in rais voice abouzing "Bey" in English with a fearial mood."), 1*A ran in black aais atanding in soen with qun in his hand, de poinsing the qun at sao own who ase standing in front, his, the aut the sult is veacing a black hut and aunglamnu.. the soom in duck and tiuss rmm the suit. la hulding a qun his hand is pointing seiding gune.
les araks is she eis, The also containe theoughout 10.00-10.00) astoratio 16-25 the audio. Ansvers mante
the beo in faont, his. the teo ous sladdin froms of weasing (0.00-0.70. 3.26-4.69, antitionally,
artillery
6,72-10.001, video Randazin
background
6.7995579), 18), wot. weteee Werrr (ndee totoeteeeeeeeeeee wew.eeeeeeeeeeeeeeeeeeee wuseeeedei tr (5et (11 gnestion beceribe insrogeecive jexsney wish apeon shifta chila's voice Sints auleooaae,v,,y, unding intsmne qurficu and sonfrentation.
saying
Generate guextion and snswer pate. Return 3909 in the tollowing format ("Ooeettont: Conetion, "Ansvent:; Annoee).
only return the ssex and nothlnn else. heee la the Zaput inforration:
Figure 11: Prompt used for generating Plot QA for LongAudio from MiraData.
30

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Prorps
You are i helppeo AT ascietant, You need to now aot as an audio captiion writer for tong audios that desoribes the soene free an auditory perspeat" we or the scunds that can be haurd. By finel abjuctiiv is to Leain en andio aqunt with these andioreaption pacta to undoe is with lone andio underatandann and capliening abilition, the walle is sourced feom u viduo and you will by provided wish milliple lppes of inforaction about beth Lie undio and vidho 20 genarate the OR paizes
video short captions short caption sunnaziaing what encire video about, foor visual perspective.
Video Dense Captions A detailed capcion decailling all evente in the escize video, from a visual peropective,
Video Main Object Captlion caption describing the rain focus of the encire video, fron visual perspective.
Video: Style Caption: A abort caption sumarizznn the abyle, in an abstsast ramner, in which the antiee video is shot, from a visual pezapective.
diides Oisera dxpttin:: A coptiio describing the cosera angles in whithh the eettrr video to shet, froe vieus) perepertioe.
Video und Audio suepant captiona: this is the mest ixpectant infermation which you ahould focms on, this la Liat of lista, Every inous list has 2 capticne that describe a second regsact of the tong uiseo, the tiret cageton decentire the wideo while the recond caption desoribes the aysto of the second aeguint. The os auquants use in chesociopioul urdus, whichh aaann that Ui ficst list is of the fiist sucund sugrant, the sucond List of the second 10 second segrent and so on,
Generace audio caption with the following requizenents:
1. 7os nestion abould be abstract and criap and dascribs the broadee:
cxpttinn should not more than 20-25 words.
3. Do HCZ sheaaus Like "according to the deteila" oz "acund of of in the qunerated caption. Alse. if the suquant capticnn have Limwatanpa placau ignare then ar they are notey, to not includs they in the question ons or answers.
segrent-icten audto captiionz way notay your Go pate, nontiinally.
vertiy aring to inderatand the natn seportance the capeions the paiie ancondingly.
cornect,
Franing
other
5. Mort importantly, only generate the queations and anrwers feen, an audio perepective, tyb., shat you are hearing, An sgect should be able to anaver st only by beazing the sounde. do not ask queations that zequize visuals.
Do not cust return as coption with come-cogaaated andio events and rather wrice Higatatioalll ooherent caption in AudioCaps style, Generate one caption, Return a 3909 In the foll tolloking rorasts ("Ogecoont degeton).
only retorn the 2004 and nothing elze, Sere the input Information:
voupple with just cuptioon (yor be provided with inforaction stated uarlissts "cuplimm"" atanding in rooe, and terk in his nouth.", "The audio contains background of oninour instronents: rusis with slov and nenacing bass on the collo, todlcionally there is nale speaking in Sandarin, who deens to be betveen 16-25 years old and conveys fearful
black wall then the
hoiding
his
hand and fark his laft hand. le looking at the spean and fort, festuring sposky elactrancc kerboard tarsony
slod."), (*A man in actsing in roce with a apean in his hand.", ", "In this andis, thare is maie playing fim the beçlaning to the and, A denala veice ia apacking in Enaliah with a and aood, aaying "Tyy and band the spean', i thace's slso the sound of breaching in betsamn the spokan verds."), i) [*X zon in blank cutt be titting a noon with weite chate and wooden cabtrae, le te diiding spoen stz hands and looking at itif, the baceground containe guiet murse playing throughout the domation the audia, male vaice speste in Englise saying thaa truch? There se no speen,
spevisets tome in meuteall "), ("6 can silting at Cable with a spocn in handif, thee andiy coiteina vide synch put playing a melium torliiii pitch reouth by
speech with att Sttevering bells tading towards the end. ferale volce speaking in Sanderin, delivered by Hevear-old fusale ran looking at himeelr in vieror," ", *in this aottle there te contiemon background notie along with teterstttent rouroe-anb sounds cosediling our Cace esstant as internals (0.58 Lo 1.37, 2.40 to 2.66, 2.85 to 3.21, 3.0) La 3.54, 3.87 to 4.05, 4,46 to 4,83, 4:85 to 3.06, 5.43 to 5.62, 5:66 to 6,34, 6.43 to Suth, 6:85 to 7,05, 7:44 to Tef2, 1.35 to 8.06, 8:42 to 8,61, 6185 to 5.03, 9.42 willy 1%% musan in black houdle is holding a ges and pointing it as a man in bisce aust. The zan is acanding in trom of window. The weman in weazing aungiazess and has har hafe pulled back. the ran in vasring a black aaks and has a black bait. The venan in holeing a bisck qun and has har hale pulled back, the nan in vearing a black wits and has blank belt. the woran is hot holding holding black qun and has has hiss pulled buck. The zan is reasing", "the audio contaian quaalata, video que bachazound anios, apuscenene quiqqu guneric impact sounds, explosium, and qunficw. idiitimmlly, these standing
Loop. synth aingss."), redtor-to-hio pirch
beaed sayang
only yoursels.
Eogliss
also
speater, paying 115522 beeddha
speech
blank gon and has her halr pedled back. The nar is wearing a b black not and has black telt. The woran
male vaice shocting "Buy" Eoglish wish i fear slod."), ("A mas black standing holding gun
gin
pointing the gun
mingianees,
hand
pointing
hin sunciazass The roos dack and there lot mnake in that, "The audio containa sounds qusahats, acti-lery fize intervals 10.0006.70, 3.26- 4,05, and 6. 6.72-23.00), video quza backeround soine throuchous 18.00-20, pesitionsllyy lyy there is speecs in Handarte by a vale aged 16-25 with a neutral ased esying
the
in front
also holding guas,
black hat
autoaatis qundice (0.75:5.75). and qunaric imaac sounda (4.56-4.83)-
"uuSseS (us52e 20 he aetaattt2tteee weeeeeeeeeeeeeelleee af aedee a5700 sat u "III. quistion: Describe the plet of the audio, Ansvezs A nan' a
inteopestive ve Jossney with a apon. shifts an a chibdfa voice hista at sell riinoovyyy wnding in intome gunfice and cnnrociation.
Do not fuxt zaturn a caption with comms socto evente and cather weite Hingatatioally coberant caption in AudioCage styin. Cenarate one caption, seturn a J909 in the following foowsts ("oapcion": eaptionl.
only return JeoN and nothingg else. scre the Input. Information
Figure 12: Prompt used for generating Caption QA for LongAudio from MiraData.
31

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Peompt
vi11 provide you with 2. differect types of informtion abost 10-second andio dlip:
1. A 11st where each comsa-oeparated slexent indiostes the individual eventa coourting in the audio at varieus time segrents. For exarple, "Sspeeoh- (10-0.60) would nean huran speect between 0.0 second to 0,64 second.
2. A exption of the audio describing in a brief and sbetract naster the acene in which the a takes place.
I vant you to act as frorpt Gcserator, Rccording to the event informtion and the caption, design some instructions and correspanding reapaases. The instruction abould be desiqned in way such that can be ansvered only iron susio withoct the caption and any other detail provices. the instruction abould invoive one or nore hapa comslex knovledge and carplex reasoning based or the scene created the audio and the correspoding caption, Enauze thst the knocledge and reasening cacino in the instructiona are precise and sufficiently chaliencing, to the catent that only vell- educated people experta the respective field provide adequate zeapeisos.
instructiens must plasses like 'according the caption! beth questiona answersy you ahoald ask and yourself.
following conditiona:
you observing
questiens knowledge,
abould diverss
sirple questions about the intamaity
or gender apeaking the ucterance; your questions
answered directly based the image and your instructtone. Tostead, regiiies the test-tater to carefully obeerve fu I2 questior camot be answered, please not ask,
Your inseruction
image have deep knovledge
image in order to snaser connectly.
Cone up with diverse instructiors for the edge topics abous with diffrnent language etyles and accurate answers, The iestructions shoold contain intorrouutive aentencsa and declarative sentences. The answers should be less than 30 nozda.
Outgut Zoznat, shooh is 1ist of 1sons:
Ee"Tnstructoons restruction exarple, 'Ansver's answer exapple, "Rhowledge teptots The spectiii Inowledge tepto), I ['Inetruction'' instrucctor exaplee "Aaswez"; ansvez example, 'Knovledge topic"; The spesific inovledge topiel, ... Boro are sone exarples of inpars and outpuras:
Isout Has of audio evanta: (' "1 64),
251-05.... 0.0.000000-.66 1150--2.65-3.2222 ' (DcQ-3.444-3.597 i, (000-,.0066 236--6.663339) "Geeeris iupact souode-s, 200-5, by (1tg--..77655.99 972) ', 1BB-,cs-5.99666.700 ", (2102-66222-56.600 1000--6.117- b.256) ', (Ganerii impact sounce-1. 849-8.2711 ", [aaugheee- 201.0.......88)))
"Genezie imsect scunda-0.
1..29) i, (1i.-00.9-11.98)) ', "120-11.322 653-1..555",, (Spaach- 551-449.01111
:0oe-1,,21-2,11911
4.:233 5,
(.o0-9..67-10.11
Captioni A baby orles while vonen laughs, creating a joyfut and lively atnosptere in donestio setting.
Ootgot Het of joons: Ii'nntrrction's tanalyre the sounds in the aodia and deternin the rest iikely eause of the lsughter heard towerds the and of the recording. Consides the potestial interactiors bebsen the dittecent results Iron the playful inseraction between the dogs and the baby, as indicated by the overlapping sounda of dogs and the baby's presence,', "Khowledge toplo"s "Huran and Aninal Behavler Interpeetation" i, ('Instruciloe") 1Frm the giver audio, infer the type of dorest:o setting deploted in the scene, Baze your inference the veriety and sequence sonns, parciestarly fonusing barking, other sackground rolxes that may thare.', "anzver's "The zetting is likely sone with active festly envireeent, evidenced by continucus presence of
acuroex and their terporal everiaps.' ' 'Anaver": "The laugher iskely
the interactior between the hunan speaking, the
apeech, and everyday Sousehold sounds.", " 'Knowledge topie'; "Bviitmmentaal icoustios and Doreatio Boundacages" (*Instruction's "Considering the duration and placement speech laughter in the audio, Infer the poseible exotional dynancos speakers, Rox do these elerente irteract shape the scene's stroughere?", 'Aaswer'; "The finally to
likely transltion fror
cheotic or lively nood
joyful and relased scnespiaze.'))
Ioput
(":taaest- 1241", (Kecchaniame- 0.9...7771 17711, 122mast-2.37665.dd ca-d, contact-é 603-9. (54))
speech, vonan speaktinge,, (31-1,430)', "Iaued (Forraee soeach, apeaking 3865--5590. "(Iaact-(,99-5222 "120--1.00608.781"
"[Surdace
peacerel ontdoor setting." Holely in7.' '2naver'; aucceata natura., outsoer envirommet, possibly during eazly norning or evenizg when such vi-diite typicsily activa.', ', 'Koovledge topie': "Soviconmantal Sourd Analyaia and Wildlife Behavioe"), I'Inassuctonns ; "Analyze the prssance and timing the doe's backing in the laster part of the audio. Considering the proceding soanda and infer the dog'a possible esaction or behavisz in this contaxt. ', "Aasweet: "the doo's ls bazking folloving peacetal nature sounds and speecs could indieate a responze to new stimute,, possibly a vixitor or anina: in the area,", "Xnoxledge tople's "Animal Bebavior Ansiynis in Diverss Sound tavi rormants"), ('Inatrationns "Dedice the saran's iskely activity or purpose in tass satting,', "Aoeeer: "the woran night be engaging in ar out activity like gardening or bird-watohing. i 'Rnwweedge topic') "Bunan cativity recogaction through soene aystys I!
Caption:: extrp
diatance dog day
creating tively acroeprare
Ostgut Hist at grone: [l'Iattrcction' 'that
presence
sounds
Iaput lis; of aod.o evanta: ( Masiico000-00 0--10.01., 'Male Aingleq-0.D-10.00) 1,676-10.0)", 100-1--7.6
INile spezol, naa
"Male spocch, zan opeakinge
Captions: '2 bastetbell bounces while zustc plays, and a van spects in an Indoor stage enviromnett''
Outgut, List of 1305a: Ii'nattvetionn'; "conaldeting the presence of ccord aounds tovarda the end of the aucio, daduce the nature of the event taking ploos. How do the clesanca of maie, singing, and spaach aupeest the type of event and audience involvesene",, "Anowaz"; ; "hhe event seara to be live partoraance er concert, with the crows's reaction indicsting an engaged and resporrive susiance, typical in aues settings,', i, "movledge toplet: rgvent Atnosphaza Analysia",, ('Instrction": "Siven the contimunta presence at suzic and main zinging throughout the autio, anslyze the rale et the man's soeach in shapine the atscaphere of the ncens. Bou does his apeecs, interapersed with music and singing, contricute to the everall anvizonennt"", 'Ansvee'; "The nas's speech libll serves commenary ez narration, adding personal oz intersetive clemen to the susical perforsance, erbancing the audience's copapasant," ', "Koolldde cople's 'Macio and Spcesh Dynazics"], I. ("Lnstrustinn's envirmerntt. "Anawer's 'hee
"Identily the gense of rusic being played and caplain hos is Hosely spoest energetis, unhancing the lively arbiance
it comp Lepenea the stanspheze et the indoer aporta ez peztoraace event in indoor aesting.", "Kookiedge topie': 'Xusie Gence Datacsion and Scana Analyaia';)
Ingent Has of andio eventes (tlnestong eventes
Captionn: leaption:
Outgut. 1ist of Jsons1
Figure 13: Prompt used for generating Contextual Sound Event Reasoning QA for AudioSkills.
32

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
You are a helptul az asstatant. You are halptul AZ azsistant, You need to now act as an audia capcion veiter for Long audios that describas the apese frow auditory perepective the counds that can he heard, You will be proutded with a "iet or Hers, vhere nach fnner Hist will have different kinds of ratardasa for aboct 4rassood augaest of Erzinate audio clip, All lista combared supeesent zaterdata for all sucend seqrants of the Erninate undio clip.
exaple of un iumes List, auturddaia of browordd some ios (' drives cart, 'the aallo custaino sound panchiing buioy feued in oll, tociosed by the sound that nane dtan being auaced a papse towsd, There sa no speech or music present in this clsp.', 'A parson is cooking a kitthen, They are stirring i pot
stove and adding ingrediante to st.'l
The firat slment in an actionn caption of the auzond acune. "M2 ot la pezzon deing the action tie video conaz Exen equresntzic video, The caption le generated fron sepeiontnn rodel and sight be rolay and prone to errors,
caption
second scene. The caption ts generated
aodio-nsgt tontig model right notey prone
The third element in video caption of the second scane, the caption ie generated tron a visto-capsiening model and night be notey and prone sc ezzor9.
Seyond this, will also provide you with action desoription that desocibes the actions that take place In entire egooencrie video.
All inpes Lista are cisonelogically asxanged in Grascond aucuenta Erom. begianing to und. You night cake uite account the cencrussencs of evanta intac Intra-liet
netey nera-dsta.
Oenezatee the cuption with the follosing sequitementr
:. You asd to season and gunerale simple aene caption about the untine oudio file in about. 15-20 worda, You do not need to discr.be unall audio event in eaption, veite caption that describes the broases toor auditosy pacapectiie..
not usa the phease 'Soand s".
the aimple verta for the capsiens and only qenerate the audio caption fron auticnry pezapective in Autintapa atyle.
1. not jest return caption with ocee separated andio events and rather write a Hingut eyeolly coherent pepticnn regentoxise audio captiors may soiay and not cossent. You need to varlly using the viauel cuptima before fxaning the final cuptica."
Only return the captiion and nothing eize.
Figure 14: Prompt used for generating Caption QA for Video-ReCap
Table 26: Prompts designed for specific audio processing tasks, used to transform foundational audio understanding datasets into QA pairs.
Task Name
Prompts
Instrument Classification
What instrument is playing in the audio?, Identify the instrument in this audio clip., Classify the instrument heard in the music., What is the primary instrument in this track?, Provide the instrument tag for this audio., Which instrument is most prominent in the clip?, What instrument is featured in this music?
Audio Captioning
Caption the input audio., Describe the sounds in the audio., Provide a caption for the audio., What is happening in this audio clip?, Summarize the audio content., Describe the events in the audio., What sounds can you hear in this audio?, Give detailed description of the audio scene., Caption the sounds in the audio., What is the main action or event in the audio?, How would you describe the sounds in this audio?
Music Captioning
Describe the music in the audio., Provide a caption for the music., Sum- marize the characteristics of the music., Summarize the music content in a sentence., Caption the input music.
Speech Emotion Classification
Identify the emotion in the utterance., What is the emotion of the ut- terance?. Describe the emotional tone in this audio., What emotion is expressed in the audio?, What is the primary emotion in this recording?, Identify the feeling conveyed in the utterance., How would you describe the emotion in the audio?, What emotion stands out in this audio clip?, Describe the mood of the speaker., What sentiment is present in the au- dio?, What is the dominant emotion in this audio?, Classify the emotion expressed in the clip.
33

---

Audio Flamingo 2: An Audio-Languag Model with Long-Audio Understanding and Expert Reasoning Abilitics
Prompt
helptti
genarates captianing
sudio capcioning questions rafer evente. The abjective in train audio with these csnatioc-anmee paics to andov wita leng-asdia inderatanding and gA abilities. You will ded with list 1iste, where inner seet wil" have di reenent kinds veta-data for (-secrnd segrent Seatngge aodlo clip- All lizta cumuued eeprement all auterdata for all aecund sequanta of the Fexinate andio clip.
questions would
audio befere and particular
exceple ot an inner List with rata-data of d-second scene te: (Ted detves the cant, "the wodto containe the sound coneching beinn fesed wil, folloned by the sound of that sum itm being placed a kitchen, they stirzing pot the stove and adding ingredlacta to st,"]
lowel. there in speech maie present in Uhis clip.', 'A pecson is cooking
The firat slmeet in an action caption of the suzond acane. *M2 ot La pezzon deing the action video conaz Eron eqerasntric video, The caption qunerated Erom. aapritming model and night bu noiay and to ezcoes.
the aecood slamas is an audio caption of aussod acane. the auption in qunerased fron captioncg modal and night be monay and peons easues.
Sh Chitd slasmn in video cuplion of Uhe awcund scmne. the puplion is generated feor videorapp aodel and night. be uoiay and yeome Le ezzozs.
seyond this, will also provide you with action deseription that desoribes the actions that take place In entire egoeescric video.
All innez Lista aze cheonolozioally accanqed in Grascond aucmenta from. begianing to und. You right cake inte account the concrurrencs of evants intac intra-list to be cobrat to aciay mesardata.
Generrate: gusttien-anrver paire kitt the folloxing requieeente::
the queatioon anever should be sub-esene exptiening. The qusstion ssould nection avants, reparated particular signiticant the middle soich can captioned. soase should not randon and can noderate lengto.
Xention the starting and anding asciening events or actiona abatouet way tbov bazan sould quastien an At agasti and not directly, The zodel should reason evente accions being referred and cspttinn ocene ie between acpordingly.
The queatioca ahould have level of corplexity and cecsize ressoning and detailed understanding of the autio.
Neither the proupt nor the response shoold be sore than 13-20 worde.
1ile "assurdin
deaalls"
the peoepla us rusponsess you should
you
aadie
yourselr.
Mast irpartantly, only generate the quastions and antvare frox audia perepective, sie., shat you are hearing, agent should be able to antre 1t onl by hasring the acunde. not ans questions that requize visuain.
Securn a 2004 in the foll toxing forasts ("guestion 1t; Question, "Anever Anower, "Question 2"; Questior, "Ansver 2"1 Answeri, *1the zegrent-slte sudio cupcisne may be solay and not corsuct. You need so varify using the visual captioms befeze fraring the Zinal caption.
only rutean the sson and mothing slow.
Figure 15: Prompt used for generating Subscene QA for LongAudio from Video-ReCap.
34

---

Audio Flamingo 2: An Audio-Languag Model with Long-Audio Understanding and Expert Reasoning Abilitics
I Proupt
helprts aestatant.
greattoe-annve genenater aodies. preatde you detastz long audio, yeu fins dejective tuain list sill have different kinds of rstardata for
infersttionatteet antaratitttttoo it with lomyradiio abilitioa, sill shart f-saoon regaect s-rinate audto cltp, All liets comined repensent rata-data for all secend eegrante the s-ninate audio clip.
list of share
exapll of an inner Hes veta-data of 6-second isi tree drives the cart, 'the aodio contains sound soneshing being fried oll, loxed by sound kitehen, They stiering por
fren being and adding ingrediente to st.")
towel, there speech
present in this olip.', 1A person ie ooekting
The firet alasent is actiton captioon of second icane. "ed of ts perseon deting the action viides conen ego-centric video, The caption ganezated reon capsiening model night notay and to ecrars.
The sacond alament is an audio caption ot the sacond acase. The caption is genarated fron an sodie-captioning nocel and right be nosay and prene to errors,
The thirdd elesent is video coptioon of the second the orgition is genersted fron video-cagtioniin vodel and night notey prone
Seyond this, will aleo provtide you with one actioon deseripttin that describes the actions that take place in the entire egocentnte video.
All immes Lists are chaonologioally exsanged in d-secand sogaanca Zrom. bezianing to und, You night Lake iate aesvant th conerutsonee of eventa intes or intra-liat to be rabust to netay neta-dats.
Generate qusstion-anvver palie kitt the tollowing requizenents
The proupt should reguize the vodel to underetand the sodio, should wodel nexeon, mate inforration night signi rientt event in the entiire andio, change, event, Interesting aspect. You also meddle-to-the-bsyree
content, and extract information from the audio, opcionally,
gueetion about partiestan sceet the sodio, The questione should have sone lavel. of corplexcty..
Nesther the pronpt nor the response shoold be more than 13-20 worde,
like "ascurding
decails"
the proepta or resposses you should
you beasing audie
youranir.
Enly generate the queationns and anavece rron, an audia pacapectiive, d.e., shat you are beazing. An agent shoulc be sble unsver it only by bearing the counde, to not aok quest that requiine visusls.
glve wany hiitts about other evente in the question, abstract posetibee.
5. Use sirple verda.
qovation fioz, 3nd parsun parapection.
aufice "Vot anyshure the queation caihee cules "pucoon" 10
3. Iutuun a 2008 in the following Louaat: ("quation 1"; quostion, "Ammmes Life Anovas, "yuu din 2"; guestion, *Auuwuz 2": anowaa) reylue aeppenertio sudio capcions may be soiay and not correct, tou need c0 verify usiny the visual captions before fearing the cisal caption.
only raturn the as0n and nothing nive.
Figure 16: Prompt used for generating General QA for LongAudio from Video-ReCap.
35

---

Audio Flamingo 2: An Audio- Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
I Proept
catio gunuralia cusposing Anteez puica, subracsne audio suptioning quiilion calut Lo apper ques Lioua that sould ask tie sodd to cuplinn avlevent pazt loog andie that iv proseded and sucoodded pacticular aut or eventa. The qusations stould requre reasosing be accurately anseredd, Ny Linal otjectiive La to Lroin audio opent with these Ranstiorransee pairo to undow it with Long cogroudio understanding and UA Liclitiess The autio is souseed lrem Video and you will be provided with nultiple tapes of Informaton about both the audio and vidko to generace The U6 pairos
belptul
captiosing Roaation
Video Short Capesion: A sbort eapsion scmaarizing what the ensice video is about feee visual pacapectioe.
Video bense Otgcions A detailld coption detailing all eventa in the estire video from visuel perspective.
Video Nain Obfect coptions A exption deaceibing the rain focus of the antize video fron visual parapactive.
Video style Otgcions stort exption someriziig the style in abstract watner, whies the estire video shot from visual perspectine.
Vi deo Daeera Caption: à coption deseribing the oseera angles in which the entire video In shot from visusl perspeotive,
Video and Audio Sequans Captiona: This is the masn irpoctant information which you abould focus this Lint of lists, Evezy innez Liat has 2 captiona that dascribe a 1broacend deqract of the song video, the firat capsion deaneibas the video, vaile the secand paption desceious the audio of the 10-swcood seaset. The Liat et augannaa ure in channologinal oedee,, vhico awans thes the fless list is of the flest 20-sucond sugaunt, the sucund l:at of the avcond 10- aecond augaent, end so on.
quverton-sneeer patn
following regutteeents:
2. Thu quus Line and comees puic ahould bu foe subrsome cuptiosing Thu question should muntion tao wrusta, supazeted by a pacticulas and signicioani some in the rickle shich can be cuplioned The somne ahould not be canden somt and can be of asturraee Lungth, '12, Montion the stazting and unding enclosing eventa or actions
abatiant may thoe hann would que in apenti not directiy, the nedel shouid ressen wiish wventa or actionn are being rellered to 15-20
captian
sccardingly. 103. Seither
proept
ceapones ahould
1. Do Not use pheases Like "acoording to the details" is both the poorpts or responsesy you should ask and ansver as if you sere heazing the audio by yoursalf, Also, of the exptions have tizastioss plessn iqnore than as they are ceisy. Do net inciude then the quastions or snavare.
1. <<the codio exptiona nay be aoday and not corrent. You seed variey using the vinaal captiona and the other vinasi oves batere tearing your oA pair. Adticimally, cas the etaer vinial datails to undesstand the asin ixpostance of the brcadez scene and assa oA paica sccoedingly. t
f. Hest ixpertantly, by heartig the swrnde. io not
generate the questions and answers from austo persgentive, fier, what that regitie vnsusle.
heaeing, in agent should able to anower it only
Fene io an exseple with juct the captiions tyno will be grose ded with none information ten ar stated nattterss teape aneet: EISS man in a black sutt ie standing in coco. the is bolding a apssa in his sights hand and a Fosk in sis buct hand. the is Looking at the apean und lork. The san then pais the speen and look in his routth.", "the audie containa batkgsound of caisour ins cnanttaa masee with aloe Lamgu Deslurang spenky sisntaunic beyoourd hutrosy and ranacing bess on the calloe, addi domilly there ua
specking Randurin, who moun
between 18-25 yours and consaga featiul wood.") ("A ran is ailling higliah
playing
with white duration of the audie, A asse voice spaska in Inaliah aaving "hhae trutar thare da no spcon." The apaczerta a tone in nncozal, "). ITA man La sisting at table with socon in his hand.", by "The cntio conteina a vida aynch pad playing at a medias aedeartto bothiihh pitch with serues arrect feom abort to finish of the loep, a chalets voice can be beard maying 'Se,,
saying "Toy and band the spoon". There's chair and woodas cabines. it holding the sound of breathing between the spoces his hands and looking st.", "Tee Sachground containa
zan black mit in aitting
maric playing thrnschect
only yoursel..
apeach La deliveced in Eoglish
anod tussie ainqur ("A buid bay ia looking at alaees, syoth pad playing at andtoe-to--- gh plech slong suylng
standing Tuuss2oudbbbbsoe sosteedyysteeee aatelhhhteeeeeeee
corraing
apeahlang
in Mandarir, deltowrdd by 3d-pear-old fecale epeaker, "i (9h, man ie yuous sounds reserbiing outlice contact at intervaie 10.20 to 1.37, 2.02 to 2.66, 2.00 to 3.07, 3.47 1,64, 3-07 to 4.05, 4.46 to 4,61, 4.00 to 5.56, 5.43 to 5.63, 3.80 to 6.54, 2.63 to 1.62, 6.05 to 7,03, 1.46 to 7,62, 10 1.24, 5.42 to d. 1.85 9.53, 1.42 tat), (TA wonan in black hoodse in halding a qun and pointing 15 asn in black auit. The ran atanding an doont of sindow. The waman veacing aunqisaass and has ber puiled back. the wan is waazing black auit toloiing sparing", "The aidio batk. The
Barbin
looking at himell in nizeos, by tin this autlo Chaze in insous rockgronnd moise along with internd ctent.
gin and
the and
grnohatt, video thane ie a rale volce shouting 'Ray! i- Reg with teactul rood, 190 nan in blace onte standing in reom with a gm in his hand. it is poiening the gun at two eas who are standing in foont of him. The aan in the mite be vearing a blace har and sunglaze axzee. The roes in dart and these in sot of zeake in the air. The amn in the rute in halding gun his hand and pointing it the mo aad front Sis. beo nen are standing in front hin anoke in that, rhaa autio centains aze alse holding quna. The nan in the ause waazing black bat and augglarnss The coon dark and thare is throughous 10.20-33.011 sounda qunabeta, generio
mackymoond
couros-antl
generdn
counde, exploction, and ginfire, adstesonslly,
quoe backgrmund 16-25 with neutral heard batseen the chile starting rpaas shout a spoon and the susten gunft me tn corgur Ansver in to nan and on 1d speak about spoor, and der orcheeersl scunde, to lowed by wonnn speak and the backgroond awind gettingg rense."
10.00-9.70 addi cinaliionlll 75-6, aa sadeedhhudddi weeee al audt4 wodedea wew.eeawaooeeeoooooo wwwee uudder "I11. output Gosation is "Whet "Kutteblud uet2eetttt2eeeeee
quatize
soundo 26-1
Mandarin
nale
Setumn 2008 in belliwing Recmate ("Quaation it;s qusatims, "Anzreus 2ffe Anowee)) 2E Liuzw nu akgriticant avent thas can be captiened es zuch, jus. column "Nume" for buth gastiim und Auseus, malas pixcae discequrd any acnuliic woun, in the undio caplisming camlioning Hartarin apauch. Thit ia baklucination
satio tupliioning modal. agiueingg wounis, nube paic acscodingiyy"
only return the 2508 and nothing aize. Hare to the snpur information:
Figure 17: Prompt used for generating Subscene QA for LongAudio from MiraData.
36

---

Audio Flamingo 2: An Audio-Langnn Aungu-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Pcoupt
should
the
caption that describes the audio. auditory perspeative, describes A coma--appazated list, whare sound aucunda in the 10rsweend cuilo, the third the unding time of the sound that sound auson the 10rsacond andio. You swed Higrlape cocaliorrmmnrunne paicy Lugotal undrstandiin. An sauple bilows
of
informatio 16-secood audior audio
the relation between sounde, the list sa 2 staments: the first is the the sound the aucund the atarting time that
aapul Cup-ions "hhe blow, vases splathes fostatopo wilo sogparinng sormone salking near netural outdoos aco..
Copur Cinastarp list: "I' dtsik, tosterepa-D. parreepp-b.cd...e 2-0,2781%, oolndd nosse (eteroshone) -5.0-7,1431 (BRidd-.---0..0)
1otsk,, tootstega-D (:30pp---............ 499-0.79911,
"halk, toocatopa-l 105---p---.0--111..11 (84-1,3171 "04ll, footstepo-) (oogo----i--i- (1-11,1-111 61-1,8561', 20id--2.11422.1
melll 2ootatepe-2 2.277--2222-2222.- 227-2. 626) "Mhist1ing22 2.89-3..12411 "walk, foctatapo-d, 35:-----.555---800 "criok-1, 1111--1,399-5,,00 959-5.0099 "walk, tootstepe-5.1e8
(Kalk, tootsteps-2. (e0ep--22.8993... 881-3,3111 3111" 10813, tootstepe-3.en 687--1 228- 5.551) ", (Maak, foststepans. p.tpsapaa......e... 244-6.281', (Mack, footacapar 5. (27-1.225""
"[crack") (23--52295) ", "Naik, foctatapa 2ctaatpp-77.55-11 7,355-1.978 9,66)"
(asis, footstepa"
2 9.339 y, thak, Ecotatapa 8.550-888813130 318-9.1111",, (2ick-9.201 201-1, 1522
Ourput Anunent (Question 1 (arder): tin shat zequence do the acunde tiret agpear in the aodto?:
1A
181 Caint Cioking sound appears.
I0 Pootscops
IDI crack aound", Anavez Iterporsl ordse): ((A) (CI (B)
Queation (tanpozal cofezzincls "Mhat acund appearи carlisst the audio?
JAi trecking
[Ri Brack sound
ICi Xind
101 dhiatting", Ansess 2 (caspesall oufussingles ((C) Wand moiae"
groundingtss "allen does Licking sound audior
10 The beginning
18I
1o The end", Anawer ttevporal grornding): *(8) The eldste
Queation: (barporal atizibatel: "Mhat happena se soias procreamust
JAI becrzes Louder and invense
181 zeraino uachanged
[C) nottar and Anewer (beeporal attributel: (D) unshanged)
sivilar style exarple.
2. Kany counds may be overlayping. and forsolate seguence questions accordingly.
You different questione like *hhat sound appeare last audio?t, followed the
14 cutation type cannot be rada feas catuan "Mas"" for both Queasion and Ansver.
queatimn
optiona.
Je.uzn only the dEeN and anthing slow.
the Zogust detailut
Figure 18: Prompt used for generating Temporal Reasoning QA AudioSkills.
37

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Prompt
Ae: Like an a audio esption geserator, 1 will provide you with an aodio caption thas desoribes the acene of on audio, you need to generate "Inuusticallly rephrased captions, The rephrased cagt shculd sound sixilar to the original, fron an aodlo and acoustic scene perspective, You can change one os nere worda to qsoezata thene eaptions. the chanqed worda night corcespend accustic simeetta, atiributss, acunda ez describs the actusl scunds behind the spoartic elenencs nectloned foned and more vividiy.
Tere ADD aote exarples:
osiginal Captions bieds chicy and custle
backgroond wille spocka, possibly off.ve sutting, followed the sound of lallisoguss
sepheased Capt.cins [sixds chitp and tualle in the backgroudd shile a ran speako, pussibly in a clesed roon settiog, followed by the sound of a chapper., nigh-pliched birdeorg echoes softly over faint rostingg leaves, as wante voice breaks the cain, followed by a distest heloogpee's shycheso thzux., Birda shipp and leaves suatie acitly in the background an a nan soeaks, libely indoeza, folliwed by halscmptar's hux.]
ozigine. Caption: vonan plays quitaz and ainaa to group of people in occastorally.
while ran's voice is heard in the backgeound, and baby crisa
Sephoased Exptiene: IA veean struxe a guitee and singe samsly to gathared crosd, as a monte vosice vumerr eaftly and a baby's occastoral crter add beief, poignant incate expplimme.. A kami alaum quilas and sings cour whih quoup people, un tun voise musmas the buokyaound and baby by crles.r is vonan singe and playa guitar audience in ross, sich a nan's voice in baskground and baby's intervition: czies.]. only
negutive captiona aimilar tashson.
Zezu the nev captions:
Figure 19: Prompt used for generating linguistically variant positives for short-audio captions for CLAP training.
Proupt
Aes an audio caption ceveiies. 2 will provide you with audio caption that duscribes the acuns of aodio. Duseice the caption by tust recouing snaersl elemente caption. keep reseriiten ty coharent. change any other
contribute desoribing
detallis Keep the rest cape consrant, you ray only y change eleseet prezent, jost
original
seze are dose exarples:
original Captions: Birds chiep and rostle in the baokground whiTe a nan spacks, possibly in office or police setting, with a hazy atnosphere.
Sesritten Capticas Birda chicp and curtie in the background wails a ran aoeaia.
oziginel Caption: aoterboat apoeding folloved by thee avund of wind and wind moize, shile zeratercomtzollle bosta coeate zipplea en the lake inder a partly cleudy sky.
Sossition Cupt.caz moissboat apooting by, followed by the sound of wind and vind mine, shile othur beats cveate eipples on the laku.
original Caplives Leuck puzfoseu a busnuu on zool, unilling large pluw of bluck wnoku, shile aanta voloe can be loazd as Jhe sun siows sate in the cleaz say.
Saxzitten Captuens truck parterzz a burnout on road while a man's vasiee can be beard.
only return the rewrittten exptten and noching else, Artnltere ia the new cspetions
Figure 20: Prompt used for cleaning and removing noise from synthetic short-audis captions.
Proupt
Aes iske sudio captiion generator, sill provide you with audio caption that describes the ecen of the sodio, You need to generate corpasitienslly different exptiens. The corportsianslly by different caprione should sound dirferent from the criginal fron an sudio and sccuatic acens pesspestive, rllp one or more verdo the ouption to generate the captions. You oay change one or roze vordd related order, atutib sounda caption stanit.
change the coppasitienelly. may eschange
captice
Busu usu aceu exarplezs
original Captions Birds chicy and zus tie in the baokground ville ran upacka, pouiibly in offico uz polise willing, with a bazy alssophuee.
Begalive Captionas [Birds chisp and taille efiss vosin speaks, pouilly is an office or polioe westing with basy aimophare. firds chisp and ruatle the backgrourd while zan speaks, possibly in an catdoor or parden setting with a hasy acrospaare. Bog barks befoce san speaka, possibly or pelice setting,
stramptere.]
Origins. Captions: venan plays a qustar and sinas to a group of people in rom,, while a ran's voice is heard in the backgeoune, and baby criea eccastorally.
Regarive Coptions:: IK plays plane sings to grorp peoplee eccaetorally.. sorso playe gutter and singe to group people cutstoone, baby playu
voean
backgroond, bayy crles in the bacaground, bily langia
has ot ysoup occasiorally.. Return Litt negative captionn stmslar fastion.
while voice
the captions
Figure 21: Prompt used for generating compositionally different negatives for short-audio captions for CLAP training.
38

---

Audio Flamingo 2: An Audio-Langan Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Proapt
You three exptions:
Audio Captions denerated solely fron audio Ingut by an aodio language sodel, it ray contain inacourantes due to sisilar sounds produced by different objects.
2. Vinusl Capticas Enscribez the acane frem vinionrenly pazapective and be used greund teush to undezatand the
Actton Exptions: Bescribes the actions from an egsoentrio [rlrst-pereon) perepeative,
Task:
using
captiion.
Cooract any accors in the sodio caption based on the visusl intormation,
Combine insights from all three captions to generate coherent and accurate audio-vieus) caption that includes all the given information.
Sncerseave the addio and visuel Information together Instead of venticning vieual informctio first and then aodio
Soatouctiona:
Onit any nandaria sulated inforration
One the vrtens captiion groind trots verity evecta the audio caption; diecard nisintergreted evente.
Ailgn and inbegrate information foom all exptions to create a cobasioe nxcrative,
.ncezaix she soundo actiona the audiorsioud captions avoid nencioning sounds and actions separately.
snouce the findl caption accurazely reflects the sosse and is shoct, crisp, and to the point,
Include additionnl explanations.
ourpot Porrati Return only an audto-steust csptton and nothing else.
Snumple 1-
Audio cupcions undio cmntaing audimiaa auch us sulchet puel, clock Licking, buckeround nocse, and beief Lone coseabling boup us bloup. pressos sasry apencs seconda). there
apeaking Mandacin papeo tovel. recording 10.00
scesthing being present this clip.
tallioedd by eocrstopceds
continues from begioning
Vieusl Captions A person is stirring a bool of food with chopstioks, a person is cocking in a kitchen, They are stirring a pot on the stove and adding inqeedienca to it. A pesson is audersog pot of food on steve.
the bossle. cooks, pot choascikk wdllng ingrodiunta, leying crscille, brist buupa, agulated sound the buskgtound.
Captions
apsan plastis
Snample
Jucio Caprion: The sodio contains sounee of tzings and rachaniass throughoct its eccize duracion fter start to rintah 1e.00 to 10.00 secenda) - the sudio concains dorestic counds, including frying food on a sizzle and the sound of diches being clanked together. there la also a sale voice speaking in Kandazin neving 11A11188 , whach tramilate to "Don't you cut baced alemet" The speakes hes an açe babress 16-25 years old and apeaka wish a neutral rond. the sadte containe sechanieme such as rateter paol, cloct clcking background notee, and brier rone recem try beep or bi eeg. Shezu in alao augrent share aersome apeaks in Mandatin suying 'AN 7
Vizunl Capeiion: parson cooking Good
litzzen, They are sesoring on the stove, person cooking in Ailahon.
food the
ix cooking sinchen, They attiering
Accion Capeions 8C stixe the food, 10 C stire food, PC C stize food in the trying pan with the spoon,
Audio-pt suad -opption incecrupted by brind buepa, Ganerate the camult foz: Audio Capsien: (), Vinaal Cestion: 1,, Action Captions
cooks in the kitchen, ectrring food
dlenling and olanking diahes nix with tieking oceasionally
Figure 22: Prompt used generating synthetic short-audio captions from Video ReCAP.
39

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
PProppt
csgeson genarator.
ouption peosiiee you the
captione 10-secund cltp: 31
caption Licully
caption with the follwwing culest it dais any scousti elarent that nandazin speech anything sulateds it is miilly ba sallocintton 21 the visuel caption as ground teuth to verify events in the aodlo captions discard any wisinterpreted events, correct any acroes in the aodio caption based or the visual inforration, 3) combin insughts feee woth castiona to qurecate the Zinal caption in Audiotaca style, 4) to not une the carra "in the audio", "in heard", etc. Desczion the scena dicectiyy
exaeples:
holding spoon
right
lonking at spoon
and tork. The then the and feok in nouth.
kudio Caprtans The audio contains background of ceinous instrumental maric with zlov teape featuring sposky electrence keyboard hamony and ranacing bass on the cello. additionally, there la a sale speaking Sandarin, the soceo to be betwein 16-25 years old and conceys feaztul acod. ourpot Oxpesons ontnous instruennce moslo plays with elon, cerie olectronic harkory and deep bass.
cegriton:
driving
talking amother
sireing
paseenger
4.05 7.38 awcundt, and limilly Zeon. 9.11 to 20.00 uscomda. vdlition to the ails apwwch, there slav sound of langhts cccatzing bulmoon 4.05 4.96 secussta.
ind
1.00
Outpot Capcion; car engine hume softly az twn nen talk, with bctef lnoghter in between,
only zetuen the audio paption and nothing slas.
toger capeione Video degriton:
you to generate caption.
Autio Ruprions 1)
Ourpot Oxpesion:
Figure 23: Prompt used for generating synthetic short-audio captions from MiraData.
the goound Lruth cuptien deare the watio. captioon and cenove acourtie elseants rastributes, evants, that placatble decn audic alona and zay have been herrowed foom video caption while gererating she geound teath sodio eapcion.
libe
audio
Beze aze auss exarplea:
Grcund Zeuth Captions ran valking on queza.
sound arrolling enpty ganden. inditionsl vides exptions
grass
Re-weistan sodia exptions A vallinn
only zeturn the cosslctes caption and neching else.
Beze aze the naw captiora:
Figure 24: Prompt used for rewriting audio caption and removing implausible acoustic elements.
#Proxpt
I kant you to sot as a Responce Evstustor, I vi11 drovide you with response to question generated by an AI agent that underetands audio, The aqunt was 4sked to qunezate a cespanns to queation abous lena audio that rasqua besseen 30 aucanda and misutaa, bo evaluata the casponae, i xill siso propeide you with the question and the ground truth anzser that is timen esaluated. Being the questiio and the ground eouth anewer, assign zcone of 1-10 to the reepenze, where 1 is the lncest score and 10 ie the hisnest score. Your ansiustion shoold be baxed on the detailedrate cocceatneso, and bl tncataco of the sesponse. Retura a J80B with a single key foooe", whexe the value of the key is the sccre, Mtze is the ceapasses
Figure 25: Prompt used for evaluating responses for questions in LongAudioBench.
40

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
EPronpt
tasked with giestlotaanser 198) help underatand attribated of rile. insightfsl specteic than allov
learh about londoees, reverb, duratiio tinestarped in ausio, Salor are desello of the itputa provided and the sapected ouigut forral:
Input Informcioo
1. Rodio Inferastioss cutio containa multiols timsscareed evanse, with uach event describing activity ceouerence the audio.
Revezurs The perzislanes of aften reseumed
Sitch: the trequency
adto charanterislig
Doration: the Length of time (In recsede) for which the event occune,
3. Event Information:
L1at timattaeed evesta with thair cornespanding caprions. Exarple
"audio_id"" audto ident
"Lansslanp"; Ning backing, 0.0-2.40,, a tean Leaghing, 2.4r6.car), " oug backing in the distance, colloved by san Leghing" "atisibules"s
*Lnodrens"s 135, 243,
*pitabn;
*duration")
Cagriton: datatled text decoription the entire audio or spact avent in the audie.
Genorace pairs
should that
following requireennss
(losdness,
dusation) specifle acdio.
2. queationn abould explors:
Cespesative attributes le.e.. "Mhish avent
hichast pitch?"!
Desporel culesiocahiga lo.e.. "Mht is the coutness of the even 4s 30:12:0551)
Deseriptive altributeu leve., "aoe long dou the event 'dog baskiny" last?")
3. Ensure the 66, patie are tasighrful and specitis to the provided austo detatle,
@A paize abould be returned in 2009 forrat. liace exippl of the expected atructuze:
("aodio Lef; Yeocio_id> spp_paiiees
"guestien";
tinestarp
"Maish avent bas the longeat ducasioo2", Renewort) the event non laughing with duration seconde."
"queation":
"gueation":
dag barktng?
"queation": "Bow much ruvech is associated with the event at cineatarp 20:2:051", Renewortt
"queation"s "Miich
Manaenet; MMh lnoghing londect cound in
7ay
eventa,
A appppiii qussies
lostmesa, pitcis
Torgoral quesies evena at upeciale Linslappa)
deliva quarias (e-s., zeoping dascriptions attstsurati.
Comparacive or anslytical quarias (a.5., "Mhich evant has lscast zeverb?")
3. ruitiple avents abaze similar actributes, rantion then axplicitly anavaca.
tted
Zare te the Sepot:
Figure 26: Prompt used for generating Attribute QA for AudioSkills.

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
PPronee
Nent you to uct us complex avotioonamaee yunaaator, i wilk procide you with un undeo expizon that dspcribe the ocder in shichs aulkiplu acundts oceue luning woads Like "ellt,, "stvue", "Lollored by", wieds. notitimallyy I will provide List of sound wounta with daselien informution (euger dais 1.000 20,0201 the suect Lite ia 0.000 ausombs the Law genarace copplex suitiple-chsten queations (822)) with optiass.
sucomia, au daselion in and Lime stact Cite 10.000 sussmbll your Laok
Bvent-Based Scund Reazoning
iinferring fois--nd-eeffet relationahige between different counds and events, For exorple, audior ty orash oollowd a child orylngh)
Example Quastion:
What conld have caused the ceotionsl rerponze obeerved in the childd's crytigg from the sodto?
Seusoning for govation Gemeration:
quaction Suuserandree.ee ressoning,
understanding causalsty
eoents.
rade by the strer corectionally onteted hightight
sudden impact or collision tollowed by the ohild gecting burt.
2. A anothing iuilaby playing sottly naazby
3- Isiandly cocvaraation betveen adolta in the backeround
C. quntie apisah of watsr from aearby tountain opeion:
follosed by the child gatting
Waile designing the optioon, anture they contrast clasrly vith correct chotoe, nakingg easter for modal to reasening skilla idestify the true cause of the childts exotioral responso.
Guideliness
Io Al Alys ask about everts with signifionn duration. Uhe the start and end times procsded to detereine the doration after each event, Ignore irsigniffiant evunta, auch baskground stacic nolan, shich could coafuse icacaners dus to thair uniigcus avurcus. that
generated
coursse,
leeying, clapping) arive! (hrrking, neowing)-
Goserppiio
nos gunazate quatissa uptions
Outpot Fornst:
thu curgut Josmel aus be jaon, thee is exeples
( ("uuaasiorr' ; agararate quastiones, "Optione"; : (echoloata, schoteads, cchatends, schaicatol, "nnveeee' : scorcnct options, Reazon: cceascn for selecting a opsionn). Make sure cuastion, cotiona and snavern mast not exceed 20, 19, 10 verda pespectively.
Figure 27: Prompt I used for generating Contextual Sound Event Reasoning QA for AudioSkills.
Irroppt
complex quastion/anaver cunerator,
individust
cenurace infamation--ssain ruitiple-mee queationa aosista, Each clase, factusi inforration ruleted to acunda iz the audio, absizees about natutal
eateacting
Guideliner for Genarating greations: Pocux on informtiitt-enette taske:
questions should oxtracting spesitio underatalding
veatier condiciora, time day, type sound, or aeason based sounda represent?", ia likely sepresentadi"
questions like *shet kind Sncore olarity ioestion fommlattion
The oon most direatly ask for inforeation that osn be dediced free the sodto desortptton, vichour requtring complex intergretentinn for ssuaple:
Oorrect: "Mazed on the andio, shat season Hetly represented"*
ancorrect: "Bured un andin, shit somario oculd Usee soundaa roprosentt"
Qseation
pradez:
"Shat natural phenseenan is indicated by the given auzuence?"
*shat woather cordition is suggesced by the counds
*ghat caszon to like'y y reprosented by the sudiot"
*ghat cime of day ie suppested baeed on the comndes
Avoid aberrsee neasontag:
requise undestaading instead, fooss direct correlations Audio dascripsion docus:
celationship difiseent llike "aildfiae extanguiatedd lisary rainf), sounda known
Focua on the peasence of cartais cypes of scunda te.a., shandes, rain, hird cailat to qucezate queations about catural facta, the b descriptton Isoks fiable aotlvttles relared to sesther, sescon, or nntural prenoretoo, do not generate a qusetton,
Seapple
Serseture:
"question'ss <penorate questions>,
"optione") (c cerotesls, conolosls, sehelostbl, "Ahowor"i scorrent opt oon, "Beazoo": faussen for aulecting the cusract optioo>
kudio Cageions *Altarnating prssence of shardar zain sounds"
Quaation: "Bazed on the audio, what wasthar condition is likely accurring?" Opcionnl: rrmunderatorst, "Clear sklest, "Light deiasle", "Herry snow*] Anawes: "Shunderatsur"
prezence Cuplises
siggare rrunderstor,..
chipping
Ozipal Fornals
The ourpor dorast wast be fecn. Hare is an extopla:
(iqueation i egenerate questions,, toptions": i (ccholee.s,, <choloeis, ccholceio, schoice(s), 'ansversts scozreet options, Heasoss sceasen for selecting a coniono). Hake reze cuastion, cptiona and ansvera nust not excsed 20, 10, 10 veeda. bese ia the audio cuptioo: Kap>
Figure 28: Prompt used for generating Contextual Sound Event Reasoning QA for AudioSkills.
42

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
PPronge
want act grection/ansver generator. will protide with Heet of andto caption that staree indtetdoet sounde that can be heared in cutio, Your Look is Lo gemurate cupplex sulliplershoion queatioo (1SQu) with four splious. how pauvaded the quaation Lype vouappl quis List, uuSANEE belov
Svent-Risse Bound Reasorings (sserrinn coste-andeatfect relationships betweer different sounds and evence, For exarple, andior "AA crash tolloved by a child orylngg))
ceosioe:
fellowed
ceying"
Questions
the
chilat's
Seazoning for queation Gunsrasionttil queztio focus undesstanding coumility beimeen wonta. The sxart sound rade the child stoue the coeah anmniionily usitied to hightiigtt the comerent utteet semming.
Ogeions:
t. sudden tepact or colliston fellowd by the child gatting hurt
2- seothing iuilaby playing sottly naazby
3. friendly corversation ion betwen adulta the background
water
seatby
cornect opeions
A audtan Lepact collision folloved by the shild quiting burt
Vaile designing the optiion, ensuze they conteaat cleatly with the cocrent choioe, scking easier for a aodel to the reasoning skills idiatity the Lrue ouidelines:
esoticrll responss.
Vaile gasezating the quaation aske suza thas you hide the acund, Pez scmple:
kodti onr i pertoon 1s putt ting sllverasee out of the dishsester
incarment generatian: Froo the glvwn sodts idecrity why ie the parzon polling a allowneree our of the dizhwshhe
Oornect genaration: Pron the greenn ausio sometsyy shy is the person doing the activity that can be heard from the andto Reazon: As the pereon is pulling the asivencare nut of the dieheshhrr, the wart posssble activity is ba/sha night be doing te scoking ansses: cooking
Fere
giees
1. Ignore indigni fientt everto, such as background or statle notse, which con'd confese 'lateners doe to theiy arbiguocs coureee.
2. Pheus on events that can be generated by ieneiriis' e courone, like honan activietes
elapping or srivel sounds (hrkking, neout ng)- 5. 22 the sudio cescription and List of audio aventa docan's inciate such idectitiable activisies, de nes penecate quitiioi or optiona for it.
4. Each quaztions sust be propasited by phezaus Libe "Burd in the qiven wodio', "he the given cntio auppis", 'Gives the collo suppl'', utc.
Ouspal Formals
ourpor cauzerate questionss, "Gpticne"s Iechelzata,
fecn.
exspli::
schoscatol, "nnvezee: reoseect optiens, Ceeason selecting
sprioons. Hake cuartion, cotiona ansvers rust not exceed 20, 10, veeda.
sere lo the blst of audio osgeion caps
Figure 29: Prompt 3 used for generating Contextual Sound Event Reasoning QA for AudioSkills.
Prrorgit
will provide you with kende of Inforeation abort 10 second audioi
1I coptton for the audio decortbing the scunde and the andio cveers taking place ir. the andto. Thile oxptiion te siso vteton-taepited and conetrocted with conbanation clauel and andio cuea. Lemmesspp
apoken
lil esy) in
Liutuesip is noiay and
You seed genarate question ansver-patea feem this information for the aucio. Heaee are the rules you should abide geserate this quartion ansuer-
pate:
:. qusations should requiza ccsplex peasening fres an At acent to pzovida a paapotan, Flasss do nes provida aizple queation-anzvez paica, Your quastiona
choold not be answered di vesrry ltased on the todio and the queetton, Instead, it shorld regutie the at agent to conetully ly perostve the autto and hame a deep
kooxiedge the content. to anaves cxrrectdy.
queation shoold tnelide reasontng sbeut both the trancoript
the counde, tiei the qu 15ke "acooeding to the ospesinn both the quaettion and anovere; you should ask and annves you were perceiving audio
wake the agent resson aboot
3. bo Rin
yoursell.
4. The questions and answers should be as diverse as possible.
5. The question should not refer to the contents of the ceansoript or the caption directly as the All agent will just be exposed to the audto as shote and
it is the aquzta duty to undesatand and perceive the spencs, and sounds fron the sodio.
up
queatie ahould
cifferent lunquequ acylea accurata ansvess. queations centain. intezeoqative sentences and
declstelive sentundea. The
Ourpot forest should ba a 2s08 of cocile of the tollowing forast:
(*Gunasiont;; The geterated quastion", "Manver"; "The qunarated anawee for the queation", "Kackiedgs captet: "The specitie knoviedge topto"), ("Queatsen": "Asower"s yor. "xooxiedge coplot: cocrcly aed
Base aza axarsies:
apeakine, searthing for tie tile, srider the backgroard counds comporerts fila exploner.
clicking acceiling,
naviqutes sheovah dicectory,
input. Teamuription:: instaliation disentury is sorive. dobble click sixas and un sxcuuu ny nistake, You Lu Saaan Lamn coraon.
ouiput queations uiven the sound of the coguter mouse clicking and serolling, hox does the specker's cossection in the xialle of lie apesch affect the saserpretation of the task is partoming?
Ousput Answars The speabes's coctection ausqesta is acrez tiis naviqution, highiicheed by the zomee intarantions, indicating an adjuatment in his zearch atseteqy.
Topat degriton: A man speake about nevere thonderstorve productig dasiging scinds in wertors counties in Geongta, while weather wtert ware people take precuutions on cartair dste. Sngut teanecription: misckezeat, odee, Patterson, Sersein, Ressees, Jeceap, fr., nown and Doace Pend. Dogelar ratar tae indicated coes weak ratation with the ouiput gostionn: What gould be inferzed about the potestiel suverity of the watter cond.tions given the locationn contioned by the passon and the sounds in the audio? ourput Andwers The audio soggests severity is high, as the locations listed are past an argent weather alext, enphaslaing the need for presautions.
:t quaation cazaut bu anzvezee, please not ank. 26 queation is not posainlle, plaaze return "Jena". Flease do not zutuzn queationa for zon-ingllab or non-senetble trancenseee. For all these cases, just rettirn "Mone" for both queettions and anewers.
Ioput Eaprionsh, vense pareionstely delswere a spench, ber votoe clear and condident, in an contennnees narked by tenale opeech and narratiion Japut teansoript.on: at the und of the yoar. And this your the snake bringu tosth such noze clarity and resolve and decib.omrakiingg
Selov is the Ingets
Figure 30: Prompt 4 used for generating Contextual Speech Event Reasoning QA for AudioSkills.
43

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
EEreupt
Your task is to genarate diverte set of misso knowledge quactione and thetr carreggending anewere for each soe In. the datadet. Rach guaction and snaser patr should foresteed se 33E8 object.
Shese questions vill be used test Language Models, ensure they are directly relecant to the audio concest described in captiona, sdentlosl rerpontes,
approprlace diftieulty lewels
vary different rodela night gesesate sixilaz
question wiltiple-choice giestion (100).
You will be paocided wath::
1. "he expeicne: mzsic exprios and an alternate caption. Both contain decatie about the mostc in the sudio and their characterlsttio.
preapt chi control ventence shicrahing Lhe ahead seqanoe.
pecapt_kkyys
key.
concelned cheads Dissatarea the desected chorda. this
cevining mustango,
inous for training Kustange,
Thee all of thess infersstior free thess cageione s the ground writh to greate vel velepple guestlons for each aodie, focusing questtone sbout the Maaamny and Chord Progreset shate Io the audte,
.nsteuctionss
+ Each queation should redar to the audio file as "tho audie'.
Enause the quaationa axe diveras and tocua on teacing knowledge of bazscny and chorda in the audio.
Jenld grestiow Hike *ahst the chord progreaston in the audio?", thise queation 16 very dirser and lase. Rother rake gueetiens whiich can nake you think cxpesons proended harrony and
Co inslude
quastions about scantitying inateumecte.
rorrat your output precisely specified. Enly raturn the 3508 sbrect, aething eiss
The expacted ase8 structure Laz
taudie sel: (pathl, "quarciar'; equastions,
"opticne": pcoptionis, coptients, coptionls, coptionel) lanover's conmect options,
"dl-flonley") /"exei//"nediuxx ('nediur ("hard"h, *catagory' staggtt terpo!s
selov is the Inputs
Figure 31: Prompt used for generating Information Extraction QA for AudioSkills.
Prronge
generate
inowtedde questione correepending ng
dataent. question
okjacs.
Toue queatiins will be undd to bust Autio lumpuge Rodela, unaue they disenlly relsont a the andio content diseribed in tie cuplima., you abould axxign exch quaction an agpropetane direimiey savel: "essy', "redlus", ar thard", If an antver ray vary bacause dietarent rodels night genecate stvilar but sor adentioal rusporues, teare the question as ailtipterdto.ce question amsor.
pravidad Information free these expeiion ground truch create wiltiple questlane for each forniing qusetions about the characteriat cerriittien **Hellddss the sodio.
captiona:
caption.
.nstructiions
Fosuve the quartions
knowledge of music quality and teatoie in sodio.
Kahe queatiens shich nake think about zow knoebedge in tumm tucture of the muxie.
- Potte texture giartione can be related to knoel of gialsty of the sadto, testure, chanactertos of a votos in the audio,
and other luua celeted to beturre and quility Auk susse bettare quistion in un ind.sucl may.
lake nse expeions provided to find the anneer.
queaticn
stroeptace
only output precissly Joon
me expeoted JB3 structuve 1o:
faudio ipathl,
"optiene": pooptienis, coptien2s, ceptionls, coptionés), tanaver's courzecs option>,
"cicricuisy's <' c'eaty" "raedburt /"hard">, "catequcy": ettuae Tuxtupe">
Selov is the Ingeti
Figure 32: Prompt 3 used for generating Information Extraction QA for AudioSkills.
44

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
Frzeppt
Your task is to generate diverre set of musto knokledge questions and their corresgond:ng answers for each You In the datadet, Rach question on and answer pair should be forsateed a6 a dSE object.
Thase queations will bu used tust Autio Lunguage Rodala, so unaze shay use diruclly relccunt oa the andio ocniant described in she cuptions, You abould "audius",
quesiion appeopecaio dillicult sdentieni rerpostee, teane the guastion weltipla-cheice guestion (NOD).
vayy becuus ditloreni.
night gesuzale windlaz
You will be providad with tve exptions: a musec caption and an alternace caption. Both concain decasta about the maste in the audio and thei: charscreri lis.. lhee the Information free these expesons as the ground truch to create wiltiple quest! and for each audio, focosing quaetions about the Wellodye the sodio.
inateucLienos
Each queation should refer to the audio file as 'the audio',
Ensuze the cusatizna are diveras and forua on tasting knowladge of shythe and terpo in the audio.
avold quastions like "Mhtt is the saape the audiot", this queation vasy diract and iaza. Bather rake queationn which cun aado you think about 2u36 sargo,
Rhythi and indirect
greations
be relamed inoeledge teeph, beate, rhythr, tine eignature and other reaturee inllated to rhytha, Ask thythe and
guections
- Kabe one of cxprions provided to find the anreet.
to not usk qoeation dbout sood and muitions.
Forat your output precisely as specified.
Only reture the 2008 dog e0t, sothing clse
espected
30
faudio ie': ipatil, 'cuarsiae'; equastions, "optiene": Iroptienia, ceptiends, ceptiendo, copticnbs) koozpect_optteee
"di"Fionlty") cleany"/"nediuee i/ "catogory' strngetn and
Generace the rescit forsecorp string autio_id pats:
Figure 33: Prompt 4 used for generating Information Extraction QA for AudioSkills.
FPronpt
will previde you with an audto exprite that deczriber the order in which vniriple rounds oscur (uxtng wanda 1ike "eith", latherr, tdellowed by", ato.). Aatitimmalyy will poovide a list of sound avena with dazation infosmation (aog Hais. 0,0001000000 the stazt Liew 0.200 aucanday and und cime stazt queation type exaple question, secsedo), your taok generace complex noltiple-cteioe questions (e50s) with four
15.000 secenda, so duration la
opcions.
Souzce information Extraction
tidenticying a souzce of a sound in sueplex acouric aqsos)
eutio expcions ta 2ire Lzuckfa sizes cus be heard oves tratlic and basm voicea."
Mitt of soddo evente: "It (Pore engine, fire trock (stoen) 000-133.00000000 ', "Crrattia rolee, readay nclee-0.c00-1a.. 0000-1,000. 'juuan volct=3. 3566-=2566--..22)1
Pype Questions
coeplax given idencily
siren.
Oosiona:
Fire trunk
2. Ambulance
Poltos Car
Boa-
Cosrec- Oplioos
Ralas generating options
Skip the quastiona for which the sound background sound oz scine
2. Ioan cenarating the opcionn, ideacidy the acitiple scurcea that can peoduce the sira scund as incereect optioon. For exuplee sicen can be qenacated by an anbul snon or po' loe car, but based on the overall aessstic sceno described by the andio caption and the Hett of sound evento, it can be Inferred that the nost likely webiccle to a fire truck.
Reseoning:
shatloee
uscursing the
describe
souns, and desernine Lype
gesezated
Vaile designing the optionn, ensure they contrast clearly with the cocrent cholos, aaking it easier for a rodel to use reasoning skills to intatity the true cause of the chsidts asoticcal casponzs. osedelined
i. Alsaya azk about evanta vith similiiant disetion. the the stazt and end tines parvided to datarmine the disetion after each event. Iqnoce iniiuitiant anbiiiions given auple'. 'Gives watio ausple'
moch backyrordd stario
propested by plesses libe
outpost Forast;
The forast wast fien. vant
genarata quastion each estegorint. Bera an azaeple:
(*Gusasion type": equestion types, "Gusecioe"' <aazerate queationss, "optiene": : fecholcnsa, schatea2s, schateals, cchoscedsl, "Annverets ccotrect options, season: cresson for seleocing a options), Sake suce question, options and ansvers must not exceed 20, 18, 10 wordo.
zere le the audio ospetoon <ap> time aligred events
Figure 34: Prompt 5 used for generating Information Extraction QA for AudioSkills.
45

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
rusic history, ruslo and you audio of quaations music, you hete is described in clok-fomacted capeion below oe, describing andio elip you ltatening Ansser all suaring the audio slip. This caption previded asss last the doxm: []facas kay": "asme yalus". Tochas kay": "othes yaiualll, share the tays and yaloez supessuct assedata about the mvae clip.
expert Al assiatanc that is inoeledyeatle about rusic prodaction, rusical
the 200M ray contain the folloving lislda:
optional usep-grovided
"albun, tagels optional sser-poovided tage aesoclated with the track sibis. "astias, tagets sptional unsr-prorided tags azsocisted vith the track astiat. "eraok.genne genre top's the top genne for the track teset freguent as decoreined by user voresi, "eraat.geeree, a all genre labele for the track. "trask, informtiooot "Leack. Laqquau code's i the Lungarge of the txuck.
optionel userspracided Information sbout the track,
teepo_tn, httpps_t__et____e_ Ser, staute the
teepo sony,
per JaPM,.
("Liau")
beat ("beat numbee zue ouagle.. indiettes the tine signatuze instarce, song
cadisases in 474 tire)
beat every
song,
cherds: lsst of the shords of the sone, containing their ataes tinn, and time, and the chord being played.
bay: the bny of the aonc-
ancwering the question, Aak you parzon asking abort rusto,
converaation
tone an acecarane hearing
quactione and give correeponding ancwers.
Anx factial quaetions about the rusical charatterietion and coorest of the presence of various Instrumnns and vosals, conpo, gente, reletive ordering of events in the clips etc,
including the and exstione, audio charasteriattie, hamonis atructure,
only Inolude quactions that have deffnite snspers based or the provi ded secadats or your bankground knowledge of this specstie rustc se an lngell:gent axsistant. Meike as nany qusation as you can using the provided isputa, zay to include rixture et simol cuastions (tia there sucphhne in the asag?" clipge
agerastente terpo asotiona conveyed the sung artiet the
clip queations
per nlmae 1n) peasible,
coeplex giestione aupects
you decortbe
overall castion the nane posssblle
rany
Agata, do not ant about uncertain decadia, deoeids datailed answare shen annwering coeplex queatiens. For sxapple, sive datadled axamples or reazoning staps to aske the content noce convincing and unll-orqanized zed. Exstain any masical conoapta that would be untesttier to aonveussctess You can inclods mileiple pazaqrapha necessary, Kabe waze that the ganerated quss tistins contain queations astina about the musical charactaciatica and content the aune, theze are nultiple placaible answers you are heartng
qusetion, nake sure vention rong and reportiin facte about you
the plassible cholces. Do not spect ficalll reference the proet ded retadeta the
reeponsty fieeess, respond
"NEORTNN:: not the word "ratadata" snyshers in the anewere to the queetione, to sor discinze that metadata about the eong se prevsided to you, Alaaye whu in listaing to the andiie.
ccuree you
securn a single Jaon List object containing the queat ioneaoone palco, Kach elewent in the usen List, should be Joon object that has the tollowing structura: ("cuaaionnt *egunszzcn TEx7 cans muse> "anaves"; "CAMEMER TEXT Gons nESE>";
Figure 35: Prompt used for generating QA for AudioSkills using FMA.
You are an expert AT statattnt that to tnooledgeabbe about voste production, rustcal strrezore, rusic Metorg, and nusto styles, and you are heariigg audio of x chore sp of nostee that you hear is deserthed in the ioot-foraated surpore below, deseribing the ssee andtte clrp you are lettnning to Answer all suections as you are hearing the austo sitp. This description ie previded in 3808 diertonary. shere the kegs and valose represent evente in the rusto clip.
usUM dintimmary contains the following buysr "cospoaer", "compenition", "rovermnn" "unsenble", "notee".
The main compenact of the 350H La "cotas" flald, siich in neated 2908 dietienncy, The kays in "notex" rescensct individusi inateurants, and the valuez sn asos 1sse reprisenting all of the natas played by that inssrarant in the rusic clip. Each elemans in the valus 3508 1sas repcesenta one zote plaved in the mase, and includes the following baya:
abasez the abass time of the sece, in aucenda undr the and Lime of the zobk, in ausonds pitchh the piach
addition these fielde, the 3S08 sizn contatos the folloxing spactal annotations:
conpo in beats per sisuce vadiow: the targo of the track in beste per ninute 13P90
downbaats nadhori a 31st of the acenbeats in the song, containing their cining ("tive") and their asscosaced best ("beat, ninbertt, for extogl best raxber Indiicated the firet best of every ressure of the song. The maxisce best rimber indi the Line aignature (ioz instance, aung with beat number 4 will be in 4/4 tira).
chasdee List the chorts of the song, custaining sheis avuct timw, and tim,, and the shord buing played.
descriaing the clip bear it betng played. Kabe eare to descritu the arduring of the diffurent, instrumnts (eluch plays first, shich playa at the undt, theae shyllem, aspagiios, churdaa, repuating patterns, etc.
Provide dotatled musical deceription of the cltp, feon the perepective
muttesl
The anavara abould be in tone that an AI azsiatant is heazing the maie and dascribing is to
only proetdo detallle that are based on the proat ded vetadets or your backgroond kmoxlndge moste se Intelll-gett gent acstatant, Aesune that there snclude
or instrrmente in the iupllaple puragraphur if cemuuarss do not upecitinally referomes Lae provided rataoiate in the supparout inabaad, axsgend it ase boaring the and roporting a eich deses appion of shit you heat. the deseriplions ssculd keep in aind that this may only be an excerpt or part of song, and set the complete sorg.
basider these in *notes" data, Riplsin any mustcal condegte that would be intantlsas notemueletae,
disclose vetadata sboot the eong is prouided to you, higgh- deaceaption andii and do strply Hist wores playeds spentfio noter ("ub. 05 or P10) should only nentloned they partieularty ieportant are wxpect who is Lisisming to the catio. to not suntine ue ank the Lruck cilly, actiat,
*eetadats" snyatere to ansvers to the quest one, say
specitioally
Focus
the description of the song. Always unsoss sibun.
Figure 36: Prompt used for generating QA for AudioSkills using MusicNet, borrowed from Gardner et al. (2023).
46

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
ktreupt
tou sou an sapeet Al unsiatiant that is knoolelguatlee about rusic production, masieal uuee music hiskoryy and case stylus, and you are beacung uudao of shore cllp nusse, What you hake is described in the 2tol-femarted caption belov, descrsbing the zare audie clip you are Listaning to. Anaser all questions
you aze hearing the audio slip, This caption is provided use Liat the forms [1"aose key"t "some value", tocher keght fother valce"t's shere the keye and values sepresent retadata about the ruste olip.
The 2008 ray contain the folloving flekea:
List of
ansocieted with the mana.
ansteumers: List of isa inaceuaania koow to be on the song. phiee instrumenta ast Listed beze may wlso be possuns, 2f Lou veng contains vocals, dlyy vi11 not be sentloned here.
acod/thhne:: 1ist moeds or theses axeociated with the zong. par
teepo basteeee__eeeeeeeeee_eee paddtte the tempo doanbesta gradeon:
beat ("beat suwoer"). For exaeple, beat nimber irdieates the firet thair timing (*sias")
asanziated
the
naxinus beat nimber Indicated the tive signature itor instancby dong with beat humber I kill be in 4/4 tine). chondes a Tiet of the chernds of the cong, contatning their stsrt tiee, and cine, and the chord being played.
bay: the key of the astig.
Dezign commesation bulnoun you pacson usking about this music, The aammu should be in tone that un AL uniiaiant in hussing Ub auike and anzwering the queation. diveres quastiona give correrpending snssess.
Only cuassiora that requice complex zazzoning about the conrest in the rasis, poraibly conbined with other background Ineviedas. Baza zeso exarplea complis queations that you could ask:
Aahk about background kaneledee about the rusic.
for eongs or artiete with steilar style.
sbout the order of evente in the autie, for exople, "Maat scase, firet, the dern break or the vocsis? to the plano and the guitar play at the case guserior, only
are present the track.)
type masie.
heard this
aakk about box change the mail In specifie vay, for exuplle, to make it better, happiet, nore danceable, oz to sound Like another genze.
-ask how nusic professor would desoribe the track,
- Aaak about any cuitizei, hastoricst or poocles zaferaness selazed to this track, in tarra that the qunazal public vould use.
Aakk co desezioe the acmnarin in which peopie would Listen to tada tzack, açuin in tumru that the çenssal public vould use.
-dat inateactioon that could be provided to an Al in codes to genarace muzic that ia surdles to this acng, without wiing the sord sinilex or ruiscenos to
partlenlar rong.
Do Hot bazts quaatioons that ba anewered with stngle astetbute the 300N ruch
-Mhat key is the soni lay
- Khat in the gazza of this aonq?
etc.
only Milke rany
gusetions
snewers detuila,
providedd
or your bankyround edge ortats spectitic possitie, usapll,, gove detuiled surple secaminng
liger difoerent
you can coveinziny and will-uzgnieed. Explain
isgula,
supss lo rake Us ouitan
susical emsoupta
that would unfaxillar to inimani You include mut ipple pazigrapha if necmiary. if Shere are miltiple plassible anseezo to queation, nake
size to sention all of the placeible chelcss. be not speci fieslly zedezance the provided ratadata in the rasponse; instead, cespand as it you aze hascine the song and repoating facts about what you bass. the peoasaa:: Hake sure the provided ansvezn not contain the pirases *the ratadsta" *based on the provided nesedass". i woo diaclosn that zatadata about the aune provided: alwaya staves as i2 you are an expert vha in liatening to the audio.
Hake zurs that the queationn and coeples, and that the detafled production, mustcall semsoture,
reffece experttiie
expert AT anslerart that te Inowtedgabble about nueto
histary, and rusto styles listening the cltp.
Please return a single coon test abject contatning the quatsion-saave paize, Each aterent io the 2S0R 11at should be a as08 sbywat that has the following structures ("quesciontt "equesrzio stor unee Huggs "unovez"; "opoeveet TEXT use5 meauee
Figure 37: Prompt used for generating QA for AudioSkills using MTG-Jamendo, borrowed from Gardner et al. (2023).
You AI ansiatant. will provida with:
3508 objest castainine:
The pata to the audio file.
qeettin about the sodio.
An sazver to that greetien.
(1ixe) capticns, pase captiara coreappands 10-secand regsson sudia clip, every list.
astry contatoe: caption
csprise resperrively
consider only caption
Your task 1s to validate this JR0N based on the following oriterial
Cozsecinaaa of Anawse: The snavez aass cotrectly adicasa the given quaation in the context of the aucio.
3. Relevanos of Consttor: The goestion rost be related the autte coneart,
3. Zocuat Soth the queatsio und the ansees abould be abous the audio, nok about the varion coppoaat.
queation
relared mame:
ayy parzon is sppacing spectiie.
awa, bu. question
contean viauel cuuay plasae rephease
question and anever
visusi coss.
IE all thess conditions are rat, you suss return a 3508 abject with Tchazet; "paza". Ochersise, return the sara 3508 object but with "chaco": "dasi",
Fie the telloiing ootpot forests
*oyath of mndionn,
"pzcepits Liqua, *outguchs heanswerp", *chasof; tepaan or fallie
16 the tnpets
Figure 38: Prompt used for self-verification of generated QA pairs for MiraData.
47

---

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitics
PPrompt
You an sepect AI agstatant. will provide you vith:
1906 covest coctainings
giestion that question,
audio.
bn
h. sering illsti of capeions, where each entry containg: an acctio captson about the action being perforeed by the user, an sodts copttion and siation
captiion respectimly.
Eech puiz of cuptions coceespands oo a fruesond segaont of the audio clip.
100r task validate JSEN
following exiteriat
Audic-Focus Focust Both the custiio the anaves abould about the eadio, about the vinson ccmponent.
4. Ko guestion or answers shoold be related to shat any person la speaking in spectric.
5. Dusove visusi cues fron the queation and sasest if tae sbove critusies sout, but the queacion and utancra contein visusi pou,, plaaau puplicue the queetion and anever to reroze vienst cone,
eat, you
sbject
sbject "checz";
tellowing output forrat:
"oase"; "opash of the audioon, *pzoepoft: "oqutrilou""
*outguifs "Caasteuu"", "egase_cs oor_
Selov 1s the Ingect
Figure 39: Prompt used for self-verification of generated QA pairs for ReCap dataset.
Table 27: Prompts designed for specific audio processing tasks, used to transform foundational audio understanding datasets into QA
pairs.
Task Name
Prompts
Audio Event Classification
What are the unique sounds in the audio?, Provide a comma separated list of all sounds you hear in the input audio., What sounds can you hear in the audio?, List the sounds present in the audio., Identify the distinct sounds in the audio clip., What specific sounds are detectable in the audio?, Give list of sounds you can identify in this audio., What types of sounds are included in the audio?, List the audible elements in the audio., What are the main sounds you notice in this audio?, Provide list of all recognizable sounds in the audio., What sounds are prominent in the audio clip?, List all distinct sounds in the audio., What are the different sounds occurring this audio?
Sentiment Classification
Identify the sentiment of the utterance., What is the sentiment of the utterance?, What is the primary sentiment of the utterance in this record- ing?
Music Understanding and Classification
Generate music tags including genre, instrument, and mood, Identify the genre, instrument, and theme of this music, What the tags for genre, instrument, and mood for this track, Describe the genre, instruments, and theme of the audio, What is the genre and mood of the music, Provide tags for the genre, mood, and instruments in the track, What music tags best describe this audio, including genre and theme, Generate tags for the genre and atmosphere of the music, What genre and instruments are featured in this track, What are the primary genre and theme of this audio, Describe the mood, genre, and instrumentation in the music, Identify the genre and musical style of the audio
Genre Classification
What is the genre of the music, Identify the genre of this audio clip, Classify the genre of the music in the audio, What musical genre does this clip represent, Provide the genre tag for this music, Which genre best describes the audio, What the genre label for this music clip, Classify the genre based on the audio content, What category of music does this audio belong to, Determine the genre of the audio
48


Reviews:
[{"review_id": "m271duf09I", "reviewer": "Reviewer_Y32n", "summary": "This paper introduces a state-of-the-art audio understanding LLM, with a focus on long and complex acoustic scenes. Audio understanding has so far been limited to superficial captioning of individual sound events often generating artificially inflated captions that try to give an illusion of complexity while having no more substance than a simple classifier. Thus, developing models that show an actual ability to analyze complex audios in depth, and provide precise sound scene analyses is of wide interest to the audio community. This paper significantly improves over the previous sota in that regard, being competitive with powerful baselines such as Gemini. However, my main concern is about the nature of this papers' contribution: while a few interesting training ideas are introduced, the framework and architecture are almost identical to the original Audio Flamingo, while the training data brings most of the improvement in performance. As a reader, the main outcome of this paper clearly appears to be the proposed training datasets and the long context benchmark. Thus, I would like to ask the authors to explicitly explain their plan in releasing the training data and the evaluation benchmark during the rebuttal period. Moreover, a potential concern is the appropriateness of such a dataset paper for publication at ICML. I think the performance gains that this dataset provides, and the very welcome introduction of a benchmark that goes beyond superficial acoustic labeling make it worth for publication if this data is released.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.\n\n> Thus, I would like to ask the authors to explicitly explain their plan in releasing the training data and the evaluation benchmark during the rebuttal period.\n\n**Ans.** As stated on Page 1 of our paper, we will open-source all code, data, and model checkpoints. We have all approvals and plan to release the dataset publicly at the time of ICML notifications (if accepted). This will include all QAs and audios released on GitHub and HuggingFace.\n\nFor the purpose of this rebuttal, we are providing access to a part of the QA data at this [anonymous link](https://github.com/anon-sub-openreview/af2). We kindly ask that you do not share this dataset for now.\n\n>  I believe that a clearer narrative would have focused on the dataset and the evaluation, while benchmarking several existing models on this data, which would leave even more room for discussing in depth the challenges of creating audio understanding datasets.\n\n**Ans.** Thank you for the suggestion. We did benchmark the strongest fully open-source LALM with available training code—**GAMA**—on our dataset. As shown in Table 5, simply introducing new data is insufficient to significantly improve audio understanding performance.\n\nOur results highlight that **it is the combination of better audio perception** (via stronger audio encoder), the **cross-attention architecture**, and **curriculum learning** that leads to meaningful improvements. This holistic approach is critical for the unique challenges posed by long audio understanding.  Moreover, our architecture choice of cross-attention also supports our first exploration of **tackling long-form audio reasoning** . Section 3.2 of the paper details how our cross-attention mechanism offers a more efficient and effective alternative to prefix-based architectures, avoiding the quadratic time complexity that arises from increasing context length in standard attention mechanisms.\n\n> On contributions beyond just data (AudioSkills)\n\n- **Efficiency in long audio modeling:** Section 3.2 explains how our AF2 module acts as an effective alternative to prefix-based architectures for long audio inputs. By leveraging cross-attention instead of increasing context length, we avoid the quadratic time complexity associated with standard attention mechanisms\n\n- **Dynamic batching for efficient training:** As described in Appendix H.2, we introduce a dynamic batching strategy based on audio length. This significantly reduces padding, improves training efficiency, and yields faster convergence with better model performance.\n\n- **Superiority of cross-attention over prefix tuning:** Section 6.5 demonstrates that cross-attention outperforms prefix tuning on the same dataset, indicating its effectiveness for audio-text alignment and long-context understanding.\n\n- **Curriculum learning strategy:** In Section 6.7, we compare 10 training schedules and show that our proposed curriculum learning strategy consistently achieves the best performance. We further highlight a counterintuitive insight: fine-tuning the language model weights degrades performance, and curriculum design is crucial for reasoning over long-form audio.\n\n- **High-quality data over large models:** Finally, Section 6.6 shows that training smaller LLMs on high-quality audio-text data results in performance that matches or exceeds that of larger LLMs trained on lower-quality data—underscoring the importance of data quality over model size for audio reasoning.\n\n- **Long audio modeling:** (also acknowledged by the reviewer) We explore long audio reasoning for the first time and make attempts to build training and benchmarking datasets for this purpose.\n\n- **A proof that better and robust audio representations can improve performance:** Table 3 highlights that AF-CLAP improves Audio Flamningo 2 performance across benchmarks. Beyond the chosen benchmarks (primalrity taken from literature), as also mentioned in Section 3.1.1, AF-CLAP may help Audio Flamningo 2 performance in various real-world scenarios by expanding its breadth of audio understanding (e.g., home sounds). Finally, robust representations also help our UnusualQA task (in demo) as the representations capture fine-grained information about the input audio.", "claims_and_evidence": "Claims are clearly supported through extensive experiments.", "methods_and_evaluation_criteria": "The paper comes with some interesting modeling ideas for contrastive training: a \"weak\" audio captioning model and a visual captioning one to produce rich audio captions makes a lot of sense. Similarly, using captions with correct content but incorrect chronology as negatives during contrastive training is also a simple and interesting trick.\n\nYet, the improvement from this methodology is marginal wrt the gains obtained from the introduced synthetic dataset.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "Experiments cover a wide range of audio domains and several baselines among the strongest models for audio understanding LLMs. What is a bit unusual with this paper is that it is both presented as a model paper and a dataset paper. I believe that a clearer narrative would have focused on the dataset and the evaluation, while benchmarking several existing models on this data, which would leave even more room for discussing in depth the challenges of creating audio understanding datasets.", "relation_to_broader_scientific_literature": "I believe the main impact will be the dataset and the long audio benchmark. The contrastive training tricks will probably also become standard, however they are likely more marginal in terms of impact.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "L168,C2: \"to condition audio representations on the LLM\" -> \"to condition the LLM on audio representations\"", "questions_for_authors": "Update post-rebuttal: I appreciate the authors commitment to release a dataset that will be a significant contribution to the field of audio understanding by allowing for more complex and realistic scenarios. I thus increase my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e4n6cPvPOV", "reviewer": "Reviewer_s3Xm", "summary": "This paper introduces Audio Flamingo 2 (AF2), a small yet powerful Audio-Language Model (ALM) with advanced audio understanding and reasoning capabilities. AF2 leverages a custom CLAP model, synthetic AQA data, and a multi-stage curriculum learning strategy to achieve state-of-the-art performance across 20+ benchmarks. It extends audio understanding to long audio segments (30 secs - 5 mins) and introduces LongAudio, a novel dataset for long audio captioning and QA tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.\n\n> Data Accessibility: While the authors mention open-sourcing code and data, providing a clear timeline or repository link would enhance accessibility.\n\n**Ans.** As stated on Page 1 of our paper, we will open-source all code, data, and model checkpoints. We have all approvals and plan to release the dataset publicly at the time of ICML notifications (if accepted). This will include all QAs and audios released on GitHub and HuggingFace.\n\nFor the purpose of this rebuttal, we are providing access to a part of the QA data at this [anonymous link](https://github.com/anon-sub-openreview/af2). We kindly ask that you do not share this dataset for now.\n\n> Limited discussion on the model's ability to understand speech content.\n\nAs mentioned in the Abstract and Introduction, Audio Flamingo 2, like its predecessor, primarily focuses on sounds and music. Speech content understanding is intentionally out of scope for AF2, aligning with other models in this family such as Pengi [1], GAMA [2], and LTU [3].\n\nThat said, extending AF2 to handle speech is part of our ongoing and future work. We would like to share some preliminary insights:\n\n- **Minimal changes needed for speech support:** Incorporating speech understanding simply requires adding speech data to the training corpus.\n- **Stage-wise improvements:** We observe that including ASR data in Stage 1 and Speech QA data in Stage 2 significantly improves AF2’s performance on speech tasks.\n- **Replacing CLAP with Whisper:** Using a customized Whisper model in place of CLAP enhances AF2’s ability to process speech alongside music, improving performance on ASR and speech-to-text translation tasks.\n\nWe are actively working on these extensions and will explore them more fully in future work.\n\n> Potential over-reliance on synthetic data for training, which may affect generalization.\n\n**Ans.** Thank you for the thoughtful question! Most of the audio used in the *AudioSkills* and *LongAudio* datasets is sourced from real-world recordings (as mentioned in the paper). The only exception is the *Counting* skill, which accounts for less than 1% of the total audio data (see Table 14 for detailed statistics). While the question-answer QA pairs are synthetically generated, this modality is inherently artificial—even in human annotations. Importantly, our use of real-world audio ensures there is no sim-to-real gap during training or inference.\n\n> However, the reliance on synthetic data and LLM-as-a-judge framework for evaluation could introduce biases or limitations in the results.\n\n**Ans.** Thank you for raising this concern. We adopt the *LLM-as-a-judge* evaluation framework following widely used practices in long-form video and multimodal benchmarks such as Video-MME [5], as well as other recent LLM evaluation benchmarks for audio understanding such as CompA-R [1] and AIR-Bench [5]. Prior work [4] has demonstrated that this framework provides a more robust and semantically grounded measure of generation quality, especially for open-ended tasks. It also mitigates common issues in traditional evaluation approaches, such as overly strict regex matching or limited coverage of reference answers.\n\nTo reduce potential bias, we carefully designed our evaluation prompts (see Fig. 25) to be neutral, consistent, and faithful to the context of each QA pair. We acknowledge that synthetic evaluation frameworks have limitations, and we view this as an active area of improvement for future work.\n\n\n### Citations\n\n[1] GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities (https://aclanthology.org/2024.emnlp-main.361/).   \n[2] Pengi: An Audio Language Model for Audio Tasks (https://openreview.net/forum?id=gJLAfO4KUq).   \n[3] Listen, Think, and Understand (https://openreview.net/forum?id=nBZBPXdJlC).   \n[4] AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension (https://aclanthology.org/2024.acl-long.109/).   \n[5] Video-MME The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (https://arxiv.org/abs/2405.21075).   \n[6] Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs (https://openreview.net/forum?id=3PRvlT8b1R)", "claims_and_evidence": "The claims in the submission are generally supported by clear and convincing evidence. The paper presents Audio Flamingo 2 (AF2) and provides extensive experimental results across 20+ benchmarks, showing state-of-the-art performance.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the problem of audio understanding and reasoning. The introduction of LongAudio and LongAudioBench provides relevant datasets for evaluating long audio understanding, making the evaluation criteria comprehensive and appropriate.", "theoretical_claims": "The paper primarily focuses on empirical results and architectural innovations rather than presenting formal theoretical proofs. It describes the model's design, training strategies, and performance on various benchmarks without delving into theoretical derivations or proofs.", "experimental_designs_or_analyses": "The authors conducted extensive ablation studies to confirm the efficacy of their approach, including comparisons of different CLAP models, data compositions, training schedules, and LLM sizes. They also introduced LongAudioBench, an expert-annotated benchmark for evaluating ALMs on long audio understanding. However, the reliance on synthetic data and LLM-as-a-judge framework for evaluation could introduce biases or limitations in the results.", "relation_to_broader_scientific_literature": "Not applicable.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: \n1. Proposes innovative data generation, architecture design, and training strategies.\n2. Achieves SOTA performance across 20+ benchmarks with a smaller model.\n\nWeaknesses:\n1. Limited discussion on the model's ability to understand speech content.\n2. Potential over-reliance on synthetic data for training, which may affect generalization.", "other_comments_or_suggestions": "**Data Accessibility**: While the authors mention open-sourcing code and data, providing a clear timeline or repository link would enhance accessibility.", "questions_for_authors": "What are the potential limitations of using GPT-4o for generating QA pairs in the AudioSkills dataset, and how did you mitigate these limitations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B4Sj7chlFL", "reviewer": "Reviewer_VWKg", "summary": "This paper propose Audio flamingo 2, an audio language model with advanced audio understanding and reasoning abilities, demonstrated by state of the art performance on several benchmarks. The authors develop a custom clap model, a dataset called LongAudio to extend ALMs to 30s-5 minute audios, and another called AudioSkills with a focus on reasoning, and demonstrate the usefulness of curriculum learning. Also introduced is AF-CLAP, a novel method of doing contrastive pretraining, and 3-stage curriculum learning. The paper shows the use of LLMs to label synthetic data and has a demonstration of high-quality reasoning in the audio domain, all at a smaller size than competitor ALMs.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging review. We are happy you liked our paper. We'd just like to clarify that, in addition to the dataset contributions, our paper also presents several modeling insights that we believe are novel and impactful:\n\n\n- **Dynamic batching for efficient training:** As described in Appendix H.2, we introduce a dynamic batching strategy based on audio length. This significantly reduces padding, improves training efficiency, and yields faster convergence with better model performance.\n\n- **Efficiency in long audio modeling:** Section 3.2 explains how our AF2 module acts as an effective alternative to prefix-based architectures for long audio inputs. By leveraging cross-attention instead of increasing context length, we avoid the quadratic time complexity associated with standard attention mechanisms.\n\n- **Superiority of cross-attention over prefix tuning:** Section 6.5 demonstrates that cross-attention outperforms prefix tuning on the same dataset, indicating its effectiveness for audio-text alignment and long-context understanding.\n\n- **Curriculum learning strategy:** In Section 6.7, we compare 10 training schedules and show that our proposed curriculum learning strategy consistently achieves the best performance. We further highlight a counterintuitive insight: fine-tuning the language model weights degrades performance, and curriculum design is crucial for reasoning over long-form audio.\n\n- **High-quality data over large models:** Finally, Section 6.6 shows that training smaller LLMs on high-quality audio-text data results in performance that matches or exceeds that of larger LLMs trained on lower-quality data—underscoring the importance of data quality over model size for audio reasoning.", "claims_and_evidence": "1) The authors claim that improved data quality can trump scaling of compute/model size in terms of performance. They show this by developing two high quality datasets (LongAudio and AudioSkills) and comparing scaling with and without the high quality datasets, which shows considerable performance differences, and the claim is therefore well justified.\n2) The authors claim that their AF-CLAP method is superior to other contrastive language-audio pre-training methods. They justify this claim by training other types of CLAP methods on the same data as their proposed method, and show that the resulting performance does not measure up to AF-CLAP.\n3) The authors claim that curriculum learning is helpful and justify this by running detailed comparisons against different training schedules and showing results on a variety of benchmarks.", "methods_and_evaluation_criteria": "The methods are well justified, the evaluation criteria is correct and comprehensive.", "theoretical_claims": "The paper is primarily empirical, there are no real proofs to check.", "experimental_designs_or_analyses": "The experiments are set up thoughtfully. Various training strategies are considered, different architectural decisions are ablated, dataset design and benchmark creation is well-done. Experimental design is excellent overall.", "relation_to_broader_scientific_literature": "The paper builds upon Audio Flamingo, it's predecessor model, and is directly inspired by contrastive language-audio pretraining. It uses several methods from prior work, including HTSTAT for generating audio embeddings and T5 for text embeddings. It relies on benchmarks that are well-suited to the tasks of reasoning and audio understanding that it is attempting to do. The authors also extend the ongoing discussion in the community around data quality via detailed data collection and ablations that specifically prove it's usefulness. Broadly the paper engages deeply with the existing audio understanding literature.", "essential_references_not_discussed": "I did not find major references missing", "other_strengths_and_weaknesses": "Strengths:\n1. The breadth and comprehensiveness of the experimental work done here is impressive and is a clear indication of the strength of the results.\n2. The scaling and data quality experiments are compelling, and the invention of new datasets are a welcome addition to the community.\nWeaknesses:\n1. The section explaining the training objective could be improved by giving slightly more intuition to how the loss directly contributes to the goals of improving linguistic invariance and compositional reasoning.\n2. The novelty of the paper is in the connection of various components and the invention of new datasets, not necessarily in new insights into the modeling method apart from the loss function. This is, however, a minor weakness and somewhat universal in large-scale pretraining focused work.\n\n## update after rebuttal\nI am satistied with the authors responses to my question and will keep my score", "other_comments_or_suggestions": "None", "questions_for_authors": "I do not have any questions of the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DmSojGRAG9", "reviewer": "Reviewer_bZqQ", "summary": "This paper introduces Audio Flamingo 2 (AF2), an advanced Audio-Language Model (ALM) designed for long-audio understanding and expert-level reasoning. AF2 leverages a custom CLAP model, synthetic Audio Question Answering (AQA) data, and a multi-stage curriculum learning strategy to achieve state-of-the-art performance in audio understanding and reasoning tasks. The model outperforms existing large-scale models despite having only a 3B parameter language model. Key contributions include the proposal of a new dataset, LongAudio, for long-audio understanding, and a benchmark, LongAudioBench, for evaluating ALMs on long audio segments. The paper also details innovations in data generation, architecture design, representation learning, and training strategies. AF2 demonstrates superior performance across 20+ benchmarks, including foundational audio understanding and reasoning tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.\n\n> Will the dataset be released? It seems to be a huge project if the followers of this paper want to reproduce the dataset.\n\n**Ans.**: Yes, absolutely! As stated on Page 1 of our paper, we will open-source all code, data, and model checkpoints. We have all approvals and plan to release the dataset publicly at the time of ICML notifications (if accepted).\n\nFor the purpose of this rebuttal, we are providing access to a part of the QA data at this [anonymous link](https://github.com/anon-sub-openreview/af2). We kindly ask that you do not share this dataset for now.\n\n> Why the authors filter out segments with audio-visual similarity above a threshold? Why a high relevance on audio and vision is not a good signal indicating a good data? How to filter out the video whose image are not quite relevant to the audio (e.g., a video with background music added)?\n\n**Ans.**: We would like to clarify a potential misunderstanding. As stated in Lines 158-160 of the paper, audio-visual similarity is **not** used to discard low-quality segments—it is used to discard **redundant** or highly similar consecutive segments. Our goal is to promote diversity in the training data. Here's how the method works:\n\n1. We segment each video (and its corresponding audio) into 10-second clips.  \n2. For each 10-second segment, we compute a **visual embedding** as follows:  \n   - We extract the middle frame from every 1-second interval (10 frames total).  \n   - Each frame is passed through CLIP to get an embedding, resulting in $\\mathbb{R}^{10 \\times 768}$.  \n   - These are mean-pooled to obtain a single visual embedding of shape $\\mathbb{R}^{1 \\times 768}$.  \n3. We compute the **audio embedding** of the 10-second audio using LAION-CLAP, also resulting in $\\mathbb{R}^{1 \\times 768}$.  \n4. We then average the audio and visual embeddings to obtain a single **audio-visual embedding** per segment: $\\mathbb{R}^{1 \\times 768}$.  \n5. Finally, we compute the cosine similarity between consecutive audio-visual embeddings and discard those with high similarity. This ensures diversity in both content and modality for CLAP training.\n\nTo address the last part of your question: if a video has low audio-visual correspondence (e.g., background music over unrelated visuals), the embeddings tend to be **highly similar** across segments. As a result, such segments are **filtered out**, since they lack useful diversity.\n\nWe will rewrite this line to make it clearer in the final version of our paper. \n\n> The audio feature seems to have a length of 64*2048, how to use it to calculate similarity with T5 feature? Mean-pooling or a linear?\n\n**Ans.**:  Thank you for the question. We would like to clarify a potential misunderstanding.\n\nWe follow a similar setup to LAION-CLAP for training our model, using **HTSAT-Large** as the audio encoder. The original HTSAT-Large model produces a feature embedding of shape $64 \\times 2048$ for any 10-second audio clip.\n\nFor CLAP training, these embeddings are **mean-pooled** and passed through a **linear projection layer**. Specifically:\n- Audio representations of shape $n \\times 2048$ (after pooling) are obtained for a batch of size $n$.\n- These are then compared to text features from CLAP, also of shape $n \\times 2048$, to compute similarity.\n\nThis linear layer is trained jointly with the rest of the CLAP model.  \n\nTo integrate our trained CLAP with the LLM and build Audio Flamingo 2, we **remove the final linear layer** and instead use the raw $64 \\times 2048$ embeddings from the last layer of HTSAT as input to the LLM. This provides the model with richer temporal representations for downstream multimodal understanding.\n\n> Currently the AF2 seems to be an expert model on audio understanding. Can it be extended to 1) multiturn, 2) speech scenario (i.e., accept a spoken question and then answer)?\n\n**Ans.** Thank you for the question. Yes, absolutely!\n\n1) **Multi-turn Extension:** Just like the original Audio Flamingo, extending to multi-turn interaction is straightforward. We simply include the dialogue history in the text context and condition on prior audio inputs. These are separated using the special `<sep>` token, which is already a learned component in AF2.\n\n2) **Speech Scenario Extension:** To support speech input, we add speech data to the training corpus. Empirically, we found that:\n- Incorporating **ASR data** during Stage 1, and\n- Including **Speech QA data** during Stage 2 significantly improves AF2’s speech understanding capabilities.\n\nFurthermore, we observed additional benefits when replacing CLAP with a **custom Whisper-based encoder** capable of handling both speech and music. This modification specifically enhances performance on ASR and speech-to-text translation tasks. Extending AF2 to better handle speech is an active area of our ongoing and future work.", "claims_and_evidence": "The claims made in the paper are well-supported by clear and convincing evidence. The authors demonstrate that AF2 outperforms existing state-of-the-art models, such as GAMA, Qwen-Audio, and Audio Flamingo, across multiple benchmarks. The performance improvements are attributed to the use of high-quality training data (AudioSkills), an improved audio encoder (AF-CLAP), and a novel three-stage curriculum training strategy. The paper provides extensive ablation studies to validate the impact of each design choice, including the use of different audio encoders, training data compositions, and training schedules. The results show that the proposed methods significantly enhance the model's reasoning capabilities and long-audio understanding.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed in the paper are well-suited for the problem of long-audio understanding and reasoning. The authors introduce LongAudio, a large-scale dataset with over 260k AQA instances, specifically designed for training and evaluating ALMs on long audio segments. The benchmark LongAudioBench further validates the model's performance on long-audio tasks. The evaluation criteria include both foundational audio understanding tasks (e.g., classification, captioning) and reasoning tasks (e.g., AQA, entailment), providing a comprehensive assessment of the model's capabilities. Popular benchmarks are also utilized.", "theoretical_claims": "This is not a theoretical paper.", "experimental_designs_or_analyses": "The experimental designs and analyses presented in the paper are sound and valid. The authors conduct extensive experiments to compare AF2 with existing state-of-the-art models across various benchmarks. The use of a three-stage curriculum training strategy is validated through ablation studies, demonstrating the importance of gradual context-length extension and data quality. The paper also explores the impact of different audio encoders and training data compositions, providing insights into the factors contributing to the model's performance. The results are consistent and highlight the effectiveness of the proposed methods.", "relation_to_broader_scientific_literature": "The key contributions of this paper are closely related to the broader scientific literature on audio-language models and multi-modal learning. The work builds upon previous efforts in contrastive learning for audio-language alignment (e.g., CLAP, Wav2CLIP) and extends them by introducing a custom audio encoder (AF-CLAP) and a large-scale dataset for long-audio understanding (LongAudio). The focus on expert-level reasoning aligns with recent trends in developing models capable of complex reasoning tasks (e.g., GAMA, Qwen2-Audio, SALMONN).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\nOriginality: The paper introduces several novel contributions, including the LongAudio dataset, the AF-CLAP audio encoder, and the multi-stage curriculum training strategy. These innovations collectively enhance the model's capabilities and set it apart from existing ALMs.\nSignificance: The focus on long-audio understanding and expert reasoning addresses a gap in the current literature. The proposed methods and datasets are likely to drive further progress in this area.\nClarity: The paper is well-written and provides a clear description of the methods, experiments, and results. The ablation studies and detailed analyses help readers understand the impact of each design choice. The paper contains enough details.\n\nWeakness:\nSee Questions.", "other_comments_or_suggestions": "See Questions.", "questions_for_authors": "1. Will the dataset be released? It seems to be a huge project if the followers of this paper want to reproduce the dataset.\n\n2.  Why the authors filter out segments with audio-visual similarity above a threshold? Why a high relevance on audio and vision is not a good signal indicating a good data? How to filter out the video whose image are not quite relevant to the audio (e.g., a video with background music added)?\n\n3. The audio feature seems to have a length of 64*2048, how to use it to calculate similarity with T5 feature? Mean-pooling or a linear?\n\n4. Currently the AF2 seems to be an expert model on audio understanding. Can it be extended to 1) multiturn, 2) speech scenario (i.e., accept a spoken question and then answer)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Audio Flamingo 2 (AF2) describes training and evaluating an audio LM with improved synthetic language data, as introduced via LongAudio, LongAudioBench, and AudioSkills. The former addresses gaps in long-context evals, the latter a training dataset for reasoning over audio they shared with submission. While the architecture and curriculum are less novel (R1, R3), the datasets and focus are, plus results are SOTA over many experiments (all reviewers). Improvements due to arch choices, data quality, and scaling are analyzed. Speech content is out of scope (R2) but this is declared. Thorough experiments and new datasets (albeit synthetic w/ synthetic filtering; R2) are a benefit to machine learning, so with the reviewers I support acceptance.

(PS: Though not visible to authors, R4 writes the QA sample and author rebuttal addressed their concerns.)
Decision: Accept (poster)