FLAM: Frame-Wise Language-Audio Modeling
Authors: Yusong Wu
Christos Tsirigotis
Ke Chen
Cheng-Zhi Anna Huang
Aaron Courville
Oriol Nieto
Prem Seetharaman
Justin Salamon
Venue: ICML 2025 poster (2025)
Keywords: open-vocabulary sound event detection
sound event detection
audio language model
multimodal model
audio representation learning
Primary area: applications->everything_else
TL;DR: An open-vocabulary sound event detection model via logit-corrected frame-wise contrastive objective and large-scale synthetic dataset.

Abstract: Recent multi-modal audio-language models (ALMs) excel at text-audio retrieval but struggle with frame-wise audio understanding. Prior works use temporal-aware labels or unsupervised training to improve frame-wise capabilities, but they still lack fine-grained labeling capability to pinpoint when an event occurs. While traditional sound event detection models can precisely localize events, they are limited to pre-defined categories, making them ineffective for real-world scenarios with out-of-distribution events. In this work, we introduce FLAM, an open-vocabulary contrastive audio-language model capable of localizing specific sound events. FLAM employs a memory-efficient and calibrated frame-wise objective with logit adjustment to address spurious correlations, such as event dependencies and label imbalances during training. To enable frame-wise supervision, we leverage a large-scale dataset with diverse audio events, LLM-generated captions and simulation. Experimental results and case studies demonstrate that FLAM significantly improves the open-vocabulary localization capability while maintaining strong performance in global retrieval and downstream tasks.

FLAM: Frame- Wise Language-Audio Modeling
Yusong Wu Christos Tsirigotis 2 Ke Chen
Cheng-Zhi Anna Huang? Aaron Courville 24 Oriol Nieto Justin Salamon
Prem Seetharaman
Abstract
2022; Liu et al., 2023a; Evans et al., 2024). In the audio domain, ALMs like CLAP (Elizalde et al., 2023; Wu et al., 2023) learn instance-level alignments between audio and text. However, these global embeddings cannot precisely localize the temporal boundaries of specific acoustic events,
Recent multi-modal audio-language models (ALMs) excel at text-audio retrieval but strug- gle with frame-wise audio understanding. Prior works use temporal-aware labels or unsupervised training to improve frame-wise capabilities, but they still lack fine-grained labeling capability to pinpoint when an event occurs. While traditional sound event detection models can precisely lo- calize events, they limited pre-defined cat- egories, making them ineffective for weal-orld scenarios with out-of-distribution events. In this work, we introduce FLAM, an open-vocabulag contrastive audio-language model capable of lo- calizing specific sound events. FLAM employs a memory-efficient and calibrated frame-wise ob- jective with logit adjustment to address spurious correlations, such event dependencies and la- bel imbalances during training. To enable frame- wise supervision, we leverage large-scale dataset with diverse audio events, LLM -LM-generated cap- tions and simulation. Experimental results and case studies demonstrate that FLAM significantly improves the open-vocabulary localization capa- bility while maintaining strong performance in global retrieval and downstream tasks.
Enabling local alignment between audio frames and text would significantly benefit applications like audio content search and event detection, allowing users to pinpoint ex- actly when a described sound occurs. However, unlike the image domain whhre large datasets (Schuhmann et al., 2022) and granular segmentation labels (Kirillov al., 2023) are more readily available-aute audio frame-level an- notations paired with text are exceedingly scarce. Existing Sound Event Detection (SED) datasets (Serizel et al., 2020) often have limited, fixed vocabulary and remain relatively small in size due to the significant human effort required for annotation. Although self-supervised approaches (Xu et al., 2021; 2024; Li al., 2024) attempt to learn local align- ment from audio-text pairs, the overall data volume remains modest relative other domains, limiting their scalability.
In this paper, we present FLAM (Frame-Wiss Language- Audio Modeling), an ALM model designed for frame-level open-vocabulary SED. Unlike standard ALMs, FLAM's au- dio encoder provides both global sample-level embedding and sequence of frame-level embeddings. By matching each frame embedding text embeddings, FLAM can de- tect not only whether sound event is present but also when it occurs within the clip.
1. Introduction
We train FLAAM with a frame-level contrastive objective and incorporate logit adjustment techniques (Menon et al., 2021; Tsirigotis et al., 2023) to deal with the spurious correlations of frame labels during training. Then, we propose memory-effcient training strategy, which handles large batches of frame-wise data without compromising computational feasibility. To overcome the lack of frame-level audio-text annotations, we build a large-scale data augmentation pipeline that synthesizes 10-second audio mixtures from text-labeled acoustic events. This approach automatically re-labels event boundaries, creating one-mo cee-mmmllooo-mppll dataset of diverse, open-vocabulary SED examples.
Multi-modal contrastive models, such as Vision-Language Models (VLMs) (Radford et al., 2021; Zhai et al., 2023) and Audio-Langae Language Models (ALMs) (Elizalde et al., 2023; Wu* et al., 2023), effectively learn open-vocabulary repre- sentations that enable strong retrieval, understanding (Liu et al., 2024), and text-conditioned generation (Ramesh et al.,
'Adbee Research "Mila Quebec AI Institute, Uni- versité de Montréal 'Massachusetts Institute of Technol- ogy *Canada CIFAR AI Chair. Correspondence to: Yu- song Wu <wu.yusonger mila.quebee>, Justin Salamon <sala- mon @addob.comm..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FLAM: Frame-Wiss Language-Audio Modeling
Our key contributions are follows:
Given these instance-level embeddings, CLIP models clas- sify pairs of observations in batch B by minimizing an InfoNCE (van den Oord et al., 2018) objective, LCLIP
Frame-Level Open-Vone Vocabulary SED: We introduce
FLAM, extending ALMs to produce both sample-level and frame-level representations for open-vocabulary event detection.
1 2B im1 2j=1 1 Cari + log
eaee-e -e
eaef
log
LI=1 of
De-biased Frame-level Contrastive Learning: We develop a frame-level contrastive objective that in- cludes a bias correction term and an unbiased event classifier, effectively handling label imbalance in SED training (Figure 1).
(1)
where a is a logit scale, a = ea controlled by a trainable parameter a'. We would like the dot product between em- beddings, ef ej,, to be high for positive pairs, where i j. and low for negative pairs, for which i * j.
Another form of instance-level contrastive representation learning method is explored by Zhai et al. (2023, SigLIP). In their work, pairs of observations are directly classified as positive or negative. Instead of an InfoNCE loss, Zhai et al. (2023) minimize a binary cross-entropy loss ESigLIP
Scalable Data Augmentation: We propose pipeline that synthesizes audio mixtures with precise event boundaries from text-labeled corpora, yielding large- scale ( (IM samples) open- open-vocabulary SED dataset.
State-of-the-Ar Performance: FLAM outperforms prior self-supervised approaches on both traditional closed-set and open-set SED (Fig. 2, Table 1), while preserving the strong retrieval (Table 2) and zero-shot classification (Table 3).'
B B
log (2n,5 (a o/ + B)),
(2)
1=1j=1
where 23 = =1 if and 21, = -1 1 if not, and a is the logistic function. As before a > is trainable logit scale, and is a trainable logit bias. Proper initialization for the logit bias is crucial for effective model training, mitigating the effects of 1 : B - 1 label imbalance between positive and negative pairs. If an appropriate logit bias did not exist, a model could trivially predict negative values for ef ee for all i E [B] thus impeding learning in large batch sizes. In contrast, offsetting logits with an appropriate bias, which captures the training artifact of the marginal statistic on & labels, enables learning high ef ee values when 2t, = 1 and low values when و, = 1. We include details in I.
2. Preliminaries
As a multi-modal representation learning paradigm, con- trastive learning methods introduce instance-level classifica- tion tasks using paired observations found in batches each training step. Specifically in CLIP (Radford et al., 2021), which is also adopted by ALMs such as CLAP (Elizalde et al., 2023; Wu* et al., 2023), a contrastive learning task is formed so that individual samples from one modality can be classified against alternative samples in a batch using their associated observations from the other modality, and vice versa. In more detail, for audio-language contrastive learn- ing, suppose that we sample batch B- = {(x,, Y) from the training dataset, which contains audio samples, X,, paired with text descriptions, Y. An audio encoder fer maps an audio sample x to a d-dimensional embedding, which is afterwards normozallieed to produce ea R.d. Likewise, text encoder f' maps text descriptions y em- beddings which after Iz-normalization The encoders are based on feature extractors, E and E' re- spectively, followed by Multi-Layer Perceptrons (MLP). In brief, we get instance-level embeddings for an audio sample x and a text description y by
3. Methodology
Open-vocabulary SED aims to locate text descriptions of acoustic events in an audio signal. Figure 1 provides a high- level comparison between traditional contrastive ALMs and our proposed FLAM approach for frame-level event de- tection. In this work, we train our model directly on a labeled open-vocabulary SED dataset. At each training step of our method, we sample batch from the dataset BSED = [(x,, {Yr,,, 2 ( {)=>) The batch contains audio clip samples X, along with a variable number K₃ of positive text descriptions Yi, that correspond to the acous- tic events present in each audio clip. Each text descrip- tion is accompanied by frame-wise labels Z E { (-1.1}' indicating when in the audio each event occurs. When Zi0e 1, it means that the acoustic event described by YE. is present in X frame I E [Z.] of audio clip X, Con- versely, Z = 1 indicates that the acoustic event is not audible at frame X,,-- We construct SED data by synthesiz- ing audio mixtures from a larger audio-text corpus, and we describe the data augmentation process in $4.
call e' Rdd
[a(x) (
fa(y) ll
ee ('()) =
c'(v)
and we type e.- eX,,) to mean the embedding of an au- dio X; in batch, and similarly of = e' eYYY) the embedding of text description Y,
Detailed results with real-world sound event detection exam- ples are shown at: https: : // f1 am -mmoell titb. i//
2

---

FLAM: Frame-Wiss Language-Audio Modeling
cegtion-
Tramerrra Contartiva
Instance Contrastive
[Girds chirping quiet foreet)
crest OCCESIONS
caption-
Test Frorder
cvatt coocee
sodao corsodding
[Gzeon wores crashing on the ahore]
salls
tec embodding
poctiin par
[Soft footstipps on woeden floor)
caption
<4
regalise pai.
det preskae
prodact
evvnt Instance
event instance
event instance
Audio Encoder
Text Encoder
Simulation Angmortation
Auitoo bimhokinng
[Gontle sandy shore]
Audio Mistaadoo
Audio Incoderr
Audio Trackes)
Text Caprionns)
Test brnokinp
FranccAttc Audio Entheddgg
Traditional Language-Auu Modeling
Framooiie Objective
Insarsc-niic Objaolive
FLAM: Frame-wise Language-Audic Modeling
Figure 1. Comparison of traditional audio-language contrastive modeling (left) with the proposed Frame-wise Language-Audio Modeling (FLAM, right). Traditional models generate global embeddings for audio-text pairs and optimize an instance-wiss contrastive objective. In contrast, FLAM integrates both instance-wise and fineigrained frame-wist wise contrastive objectives. Audio mixtures are synthesized from event instances, paired with text descriptions, and processed to produce frame-wise embeddings. Frame- trame-wise contrastive learning explicitly aligns individual audio frames with textual event descriptions using event-specific scales and biases, enabling precise temporal localization of sound events.
Traditional SED targets a fixed set of classes, predicting propose
trastive ALMs often produce temporal representations, e-) in the second-to-the last layer, which can use obtain global representation, en (+) 7. EiELy e9. (+), by averaging across the frame dimension (Chen et al., 2022). Thus, our formulation can be built upon current ALMs with minimal computation overhead on model inference. Second, our formulation is computationally efficient at inference time. Compared to an alternative formulation which di- rectly outputs events matching given both audio and event text query, our formulation allows to precompute tem- poral audio representations ea.loc() and only compute text embeddings when a new prompt is given.
binary label per time frame. In contrast, open-vocabulary SED in this work as detecting the temporal boundary of any event described by some text y. To this end, we propose that, given an audio sample x, the audio encoder fa.lo outputs frame-level representations in Lxd which we consequently nornoaale (Figure 1) along the embedding dimension.
(x)) ea)oo(()); [fa.)oc))))) E Rd, LE [4]
(3)
consisting of d-dimensional embeddings for each frame out of total L frames. Unchanged from the instance-level set-
Figure 1 illustrates the key differences between traditional audio-language modeling and our proposed FLAM frame- work. Traditional approaches (left) produce global embed- dings and optimize instance-level alignment between audio and text pairs. In contrast, FLAM (right) leverages both global and frame-level embeddings to enable precise tem- poral localization. Specifically, FLAM synthesizes audio mixtures from diverse events, creating temporally aligned frame-level embeddings. These embeddings are explicitly aligned with textual event descriptions through frame-wil wise contrastive learning, augmented by event-dependent scaling and biases. This dual objective structure allows FLAM to not only detect whether events occur but precisely localize when they occur in audio clips.
ting, the text encoder fe yields a single embedding e' (y)
f'(y) ( E Rd for any text query y. For notational conve- nience in the batch setting, we write e represent the embedding of frame of audio sample X,.
alooc = eoe (X,); ())( to
Unlike contrastive ALMs which are usually concerned only with instance-level rankings via comparison of dot products of text query and audio clip embeddings, open- vocabulary SED requires 1) detection with unlimited open- set language prompts, and 2) calibrated probabilities for each frame and event. This because each frame can con- tain a variable number of active events, including none. To detect temporal occurrence of acoustic events, we propose to construct classifier which takes a temporal audio em- bedding and text embedding, and detects whether an event occurs in frame E [L] of audio I.
3.1. Robust Training and Inference
We propose to formulate the open-vocabulary SED as a con- trastive objective based on binary classification. Given
The reason for proposing this formulation is twofold. First, can efficiently leverage current contrastive ALMs. Con-
3

---

FLAM: Frame-Wiss Language-Audio Modeling
Ground Truth Labels
FLAM
FLAM (mw cosine similarity)
MGA-CLAP
breaking bones
books being thrown
mechanical heep
65
whoosh short
racquetball bounce
troll scream
0.0
10
Time
Time (6)
(6)
Ground Truth Labels
FLAM
FLAM (mw cosine similarity)
MGA-CLAP
1.0
0.1
0.2
0.5
0.5
-0.1
0.0
0.0
0.0
Time
Time (6)
Time (6)
(4)
breaking bones
books being thrown
mechamical beep
whoosh short
racquethall bounce
troll screum
Figure2 2.
Example outputs from FLAM and MGA-CLAP on open-vocabulary sound-event-detectio example (top row) with detailed
value each frame (bottom row). FLAM, trained with event-dependent logit adjustment, converts raw cosine similarity into calibrated predictions. By contrast, the unsupervised MGA-CLAP model produces less accurate results and cannot be calibrated from its output.
batch BSED randomly sampled from the training set, we collect all the event descriptions found in the batch y = -iie(ra] (Y (t.)) J&=, with 3 = K elements, and re- define frame-wist labels on the union of description for all i E [B), K/ E [ E [Z., as 2,,0'1 = 1 if Di = (1.k) for some & E [K,]. otherwise 21, -1
3.2. Logit Adjustment for Event-dependent Imbalances
From the binary classification perspective of our objective LSED ($3.1), labels =i,,,I are highly imbalanced, as the large majority of frame-text pairs are negative. Beyond the dependence on the batch size, which increases the number of negative frame-text pairs as we have discussed in $2 about SigLIP, we need consider dependencies of positive frame- text pairs on the event descriptions present in our training dataset. For example, "thunder" may occur infrequently and have short duration in our dataset, whereas "rain falls" may appear more frequently and persist for a longer time. It is important to counteract such idiosyncrasies of our training dataset, if we would like our final model to perform robustly across event descriptions in the open-vocabulary case.
and /
In essence by computing the union of event descriptions, we deduplicate potential overlapping acoustic events present in different audio samples in the batch while maintaining con- sistency with the dataset about positive audio frames with respect to acoustic event descriptions. In addition, we as- sume that all frames of an audio sample X; are negative with respect to an acoustic event description y which does not appear in the dataset for that audio sample, y $ ) We propose to minimize the open-vocabulary SED objective LSED
K.
k=1
To avoid overfitting to dataset-specific priors and the par- ticular choice of batch size, we introduce text-dependent logit bias term (3(()) in the logit function of Eq. 5. Prior work on long-tailed multi-class classification (Menon et al., 2021), and more recently on training classifiers under spuri- ous correlations (Liu et al., 2023b; Tsirigotis et al., 2023), has developed logit adjustment techniques to handle distri- bution shifts due training data biases. Here, adapt logit adjustment to binary classification in order to deal with the aforementioned label imbalance challenges.
KBK
B
log (Zi,k,1 ((X,,
(4)
BKL
1=1 km 1=1
where Mz,1,,) 1. is a logit function we model as
1,,,,,) a'( e eo (x)) e')) + (w)) (5)
with a((-)> > 0 being a text-dependent logit scale and (+).
a text-dependent logit bias. Note that we slightly abuse notation so that et T'
(')))) in Figure
Inference Our goal is to robustly determine whether each frame (z, I) contains given event y. We seek a classifier that treats positive and negative predictions equally regard- less of y. As we show in $C.3, the Bayes-optimal classifier, under our working hypotheses, is given by
Compared to LIILL in Eq. 2. our loss in Eqs. 4-5 is still a binary cross-entropy classification objective, which acts at the frame-level of an audio instead of the instance-level, and aims to learn Pdata (z 1 x, y) via a (h(x,1, I, v)). Moreover, in contrast to SIgL.P, our logit scale and bias are conditioned on event descriptions. As we show in the following sections, this allows us to model label-imbalance structures on a finer level which we argue it is important for open-vocabulary SED.
2"(,,,,y) = arg max Panta x, (1,,,)) =E(-1.1) Pdata (2 y)
(6)
Since 2 E {-1,1}, taking the arg max corresponds to checking whether Pdata = x, x,,,y) > Pdata(z
4

---

FLAM: Frame-Wise Language-Audio Modeling
Table 1. Sound event detection performance on synthetic open-vocabulary SED (Held-out, ASFX-SED and traditional closed-sct SED
dataset (DESED. MAESTRO, Audioset-siie strong. UrbanSED). FLAM produces more accurate sound event detection compared with existing models on both vocabulayy and closed-set set SED datasets. Bold numbers shows the best performance across all models. MGA "GA-CLAP"
MGA-CLAP trained on our data.
Held-out
ASFX-SED
DESED
MAESTRO
Audioset-S
UrbanSED
Model
AUROC
AUROC
PSDS
AUROC
MPAUC
PSDS
AUROC
PSDS
AUROC
FLAM-Global
67.76
65.14
7.09
85.52
51.13
1.11
82.54
0.82
67.39
FLAM (proposed)
91.0
81.23
9.37
91.66
56,97
11.16
94.76
29.52
93.62
MGA-CLA -GA-CLAP"
74.17
69.56
14.72
89.28
52.50
1.24
79.12
6.42
78.22
MGA-CLAP (reported)
26.4
10.1
8.7
1 y). In other words, p(z = 1 y) acts as the classification threshold for the posterior p(z x, y). To achieve a fixed boundary at 0.5, independent of the event description y, we define
computational overhead and leverages the language under- standing of the text encoder. To prevent SED objective interfere with bias estimation, we stop gradients propagate from the LSED in Eq. 4 to the MLP of classifier 33. We train the text-dependent logit scale a' in similar manner where another MLP appended to text feature extractor, giv- ing a a'(y) MLP" (E" (y)). Different in per-text bias, we update ML.P via LSED in Eq. 4.
Pdata (2 = x0, x.1.y)
,(x,1,)) = Pdaaa = 2,4,) + Pdata y)
A value of s(2,, (,) 0.5 indicates a positive detection. (data(==1lp)
Furthermore, if (y) = log Paau(3==1(p)) is sufficiently negative, the ratio in Eq. 7 can be approximated by
Finally, we have experimentally found that adopting a per- text logit scale, which trained by minimizing CSED of Eq. to be additionally beneficial for downstream open- vocabulary SED.
s ((x,1.)) x a [og Pdata y 2.1)
(8)
Pdata (3)
3.3. Memory-Efficien Training
Logit Bias Training Assuming that training and model hypothesis are able to attain the minimum of our LSED loss in Eq. 4, our model learns
Computing the full LSED across all B K I. frame-text pairs can be prohibitively memory-intensive To address this challenge, we adopt a chunked approach inspired by SigLIP (Zhai et al., 2023) that avoids gathering all embed- dings on a single GPU. Specifically, each of the NGPU GPUs processes its local subset of audio frames and text prompts compute pairwise losses. Since each audio clip may contain varying number of events, we allocate text slots equal to five times the number of audio clips in batch, padding any unused audio or text entries with placeholders. Next, we pass the text embeddings (and associated masks) to the next GPU in a ring. After N-PU transmissions, each GPU has accumulated all cross-device loss terms without centralized data collection. This strategy enables large-batch training while respecting single-GPU memory constraints.
hr" (xx 1. y) = log Pdata (y x.1) + (y)) Pdata(y)
as we show in SC.2. We observe that if we have the logit bias 3 (v) in Eq. 5 approximate 3 (y), then we effectively absorb the spurious train-time statistical relationships and enable a' (y) ga.lo0(x), e²(2) in Eq. 5 to approximate the robust logit log (asta(yla.d) of the classifier in Eq. 8.
Pyataly)
However, computing (*) for every possible text prompt y is generally intractable. We therefore approximate this quantity via independently training (3' auxiliary classi- fier, implemented via a lightweight MLP appended to the text feature extractor, 3 (y) MLP" (E'v))) (v)). We train the
4. Dataset and Data Augmentation
MLP to minimize the loss Lp =
FLAM is trained on two data sources: (1) large-scale au- dio-text corpus, similar to those used by contrastive ALMs, and (2) an open- apen-vocabulary SED dataset synthesized by in- serting one or more sound events (and their captions) into 10-second background clips. Below, we first describe the audio-text corpus ($4.1), then explain the augmentation procedure ($4.2).
ms log a (3 (1 - Σx) Exglog (=
K
(10)
where b = BL te 0.5 (z + 1) E [o, 1). the average label for prompt 3x across all frames and clips
i in the current batch. This approach introduces minimal
5

---

FLAM: Frame- Language-Audio Modeling
Table Recall performance of to audio (T2A) and audio to (A2T) retrieval. FLAM has comparable Audio-text retrieval performance evaluated on ASFX, Clotho and Audiocaps datasets. Bold numbers shows the best performance across models trained on our dataset, while underlined numbers indicates best performance among all models. MGA-CLAP" is -LAP trained on our data.
ASFX
Clotho
AudioCaps
T2A
A2T
T2A
A2T
T2A
A2T
Model
Dataset
R@1 1
R@5
R@1
R@5
R@1 1
R@5
R@1
R@S
R@1
R@5
R@1
R@S
FLAM Glohal
4.4
14.8
4.0
13.8
14.3
35.8
17.9
39.8
36.0
70.5
46.1
78.6
FLAM
FLAM-Collection
4.4
15.2
3.9
13.9
13.8
33.2
16.7
42.2
32.1
64.8
43.3
75.0
MGA-CLAP
3.9
14.8
3.9
13.8
13.4
30.3
18.7
39.1
36.7
69.9
47.2
78.3
Reported Performance from Prior Studies (but trained on different datasets)
LAION-CLAP LAION-Audo--60K
2.0
7.6
1.6
6.0
16.1
38.3
22.7
48.5
36.1
71.8
46.8
82.9
CompA
CompA-Collecttoe
16.8
43.5
23.9
50.7
36.1
78.6
47.8
83.5
MGA-CLAP
WavCaps
2.3
8.3
2.0
7.4
20.4
46.0
25.3
51.2
41.8
76.1
54.4
83.6
4.1. Audio-Text Dataset
We then randomly place each event into random position, allowing a maximum of concurrent sound events. When mixing audio, we apply a 10 ms fade-in/out for each event before mixing to ensure natural onsets and offsets.
We gather a large mix of licensed proprictary sound effect datasets and publicly available CC-licensed general audio datasets, consisting of approximately I.IM audio samples with corresponding metadata. Sound-effect data typically contain a single clean event accompanied by tags captions, while general-audio datasets are noisier and may contain multiple events that are not fully described. Because the sound-effect clips are generally clean, they are well-suited for insertion into diverse backgrounds without significant interference. We prompt Mixtral (Jiang et al., 2024) with the file name, tags, and any available textual descriptions to generate captions of lengths in 2-13 words.
AMS-Based Boundary Correction To reduce label noise from silence in events, we compute the A-weighted RMS loudness over each event and classify frames below -70 dB as inactive. We also include a smoothing step to avoid rapid label fluctuations (see C.4).
5. Experiments
We pursue four main research questions in this work: (1) How does FLAM perform on both vocabuucayy and closed-set SED tasks? (2) How does FLAM perform on stan- dard audio-text retrieval benchmarks? (3) How does FLAM perform downstream tasks typically used to evaluate con- trastive ALMs? (4) How effective is proposed design of combining frame-wise and global contrastive learning?
4.2. SED Data Augmentation
Event and Background Audio Filtering We divide the corpus into two categories: sound events, which are shorter than 10 seconds and do not contain the keyword "ambiance", and background audio, which lasts at least 10 seconds and includes the keyword "ambiance".
5.1. Training Setup
Event Selection and Placement To create 10s training mixture, we randomly pick one background clip and sample N ~ (11, 10) events. In 80% of cases, events are drawn from sound effect datasets; in 20%, from general audio datasets. Each event is placed at a random start time, with at most three events overlapping.
Our model architecture follows the LAION-CLAP frame- work. The audio encoder (.()) is an HTSAT network (Chen et al., 2022), while the text encoder E=(-) is RoBERTa model (Liu, 2019). The HTSAT model takes 10-second audio inputs, and, after an MLP projection layer, outputs 32 frame embedding sequence e g0))0()) E RLxd global audio representation e (xx) is obtained by averaging these frame-wise embeddings over time.
Splitting and Repetition A naive placement procedure risks biasing the model to single instances of each event whereas real-world scenario often seek to detect events that are repeated or fragmented in time. To reflecting realistic event occurrence, each sound event is split into (2, 3) seg- ments (with each segment at least 0.5 long) in 10% of the cases, and repeated 22( 3) times in other 10% of the cases.
We first train an ALM baseline using only the global con- trastive objective from $2. This baseline, F'AM-GGloba is conceptually similar to a CLAP-style model (Wu* et al., 2023), and is trained on the audio-text corpus described in $4, along with Clotho (Drossos et al., 2020) and Audio- Caps (Kim et al., 2019). We train on 10 second audio clips in 48kHz. Further training details can be found in $ C.5.
Random Loudness and Mixing An offset is sampled from 30) dB for each event relative to the background.
6

---

FLAM: Frame-Wist Language-Audio Modeling
5.2. SED Performance
Table Zero-shot classification accuracy. FL.AM consistently out- performs the baselines across three benchmark datasets, underscor- ing the advantage of Traee-level alignment for robust zero-shot performance. MGA-CLAP" is MGA-CLAP trained on our data.
Table summarizes results on closed-set datasets (DESED, MAESTRO, AudioSet-Strong, UrbanSED) and two syn- thetic open-vocabulary test sets (Held-out, ASFX-SED). Acoustic events and audio backgrounds of both Held-out and ASFX-SED datasets are unseen during training. We report PSDS, and MPAUC depending on each dataset's 's stan- dard metric, and additionally use a frame-based AUROC for all dataset (details of evaluation metrics in $ C.7). For open-vocabulary SED datasets, calculate AUROC by only compute true positive and false positive rates over the events that actually occur in an audio clip.
Model
ESC50
US&K
VGGSound
FLAM-Global
81.6
65.4
38.9
FLAM
86.9
75.6
39.3
MGA-CLAP"
72.6
69.9
38.6
LAION-CLAP
89.1
73.2
29.1
CompA
89.1
85.7
29.5
MGA-CLAP
94.9
83.7
31.8
Our final system, FLAM, initialize from FLAM-Global and trains additionally on a frame-wise contrastive objective LSED and a prior loss 4p- The total loss is:
FLAM achieves substantially better frame-level alignment than both FLAM-Global and MGA-CLAP on nearly every metrie. On DESED, MAESTRO, and UrbanSED, FLAM yields improved or comparable AUROCs, while also scoring better PSDS values except DESED dataset. We hypothesize that the PSDS on DESED mainly results from its limited scale, containing only 692 real-world annotated samples, thus could be more prone to higher variance and incomplete coverage of acoustic events. FLAM's performance on the synthetic tasks (Synth, ASFX) is particularly strong, indi- cating effective generalization to unseen audio events. In contrast, FLAM-Global fares reasonably well on retrieval but underperforms fine- fineggrinedd detection, highlighting the need for explicit frame-level training. The MGA-CIAP is indeed better than FLAM-Global on SED metrics, but it performs poorly compared to FLAM, which proves the self-supervieed local alignment is less robust than FLAM's direct frame-wise objective. To further validate that FLAM's performance gains result specifically from improved frame- level alignment rather than merely enhanced classification, we introduce an alignment-specife metric based on Spear- man correlation, which shows FLAM achieves substantially better temporal alignment compared to baselines (see Ap- pendix C.1
L = CLIP LCLIP + +g SED LSED + 7P Epp
where we set CLIP 1, ySDD 200, and 7p 1. We set CLIP and ySED such that LCLPP and LSED are in same scale, while we found setting 7p 1 is enough to make Lp converge. This approach allows FLAM to integrate both global sample-level alignment and fine- fine-grained frame-level alignment.
We generate 1 Million mixtures for training using the aug- mentation procedure where each mixture has a length of 10 seconds. We hold out 5k backgrounds and 10k events, and make 10k mixtures from the held-out events as our primary test set (Held-out). For additional evaluation of general- ization, we create another 10k test mixtures, ASFX-SED, using sound effects from the Adobe Audition SFX Library (ASFX)(Wilkinss et al., 2023) that were entirely unseen dur- ing training.? Since our frame-wise contrastive method accommodates closed-set SED, we train FLAM not only the synthetic open-vocabulary SED dataset ($4.2) but also on Audionet-Strong (Hershey et al., 2021), DESED (Serizel et al., 2020), and UrbanSED (Salamon et al., 2017). We retain the global contrastive objective so that FL.AM sees the same data as FLAM-Global, thereby reinforeing robust sample-level alignment during the second training stage. Further details of training FLAM is included in C.6.
5.3. Accurate and Calibrated Output
FLAM's robust training with logits correction and inference using an unbiased classifier enables it to produce intuitive, calibrated probabilities for event detection. Figure 2 com- pares FLAM's detection output with MGA-CLAP, with ad- ditional examples in $ A. FLAM effectively transforms raw cosine frame-text similarity into interpretable probabilities, whereas MGA-CLAN fails to detect certain events and lacks an effective output calibration mechanism.
We compare FLAM to MGA-CLAP (Li et al., 2024), which follows a similar architecture (HTSAT + BERT (Devlin, 2018)) and uses only a global contrastive objective. MGA- CLAP adds shared codebook for text and audio embed- dings, adopts contrastive loss that upweights hard nega- tive samples, and modifies the HTSAT model to be more temporal-aware. We re-train MGA-CLAP on our dataset for direct comparison with FLAM and FLAM-Global.
To demonstrate the effectiveness of our bias-corrected objec- tive, we trained ablated models without per-text logits bias correction and per-text logit scaling. As shown in Figure 3, FLAM consistently achieves higher Fl scores across vari- ous thresholds the ASFX-SED dataset using the same inference formulation. The precision of all models increases
We publicly release the ASFX-SED dataset for future bench- marking: http://flan fttp:///faammmmmel....a model. gi thub. 1 aa//ffxxass sed. html.
7

---

FLAM: Frame-Wai Wise Language-Audio Modeling
broad classification performance.
Prociion
6. Related Works
15
Sound Event Detection Sound Event Detection (SED) seeks to detect which sound events occur in an audio sig- nal and where they occur temporally (Mesaros et al., 2021). Conventional SED systems typically framed as a multi-label, multi-class classification problem over discrete time frames, with a binary cross-entropy (BCE) objective applied at each frame (JiaKai, 2018; Cornell et al., 2024). SED datasets often remain small in both duration and number of classes as temporal annotation of acoustic events is expensive and time consuming. Synthetic methods, e.g., Scaper (Salamon et al., 2017) address label scarcity by mixing isolated sound events with background audio, yielding accurate boundaries at scale. Our approach departs from traditional SED in that we target an open- open-vocabulary setting, where any textual description of a sound event can serve as a query. In con- trast to traditional SED data synthesis, our work extends this strategy to an open-vocabulary regime, enabling event detection for any textual query.
4.1
X5 Tlreshold
66
31
04
31
62
03
64 Theesolé
65
16
63
FLAM
BLAM wis peooedtaale
FLAM wis partesbbi
PLAM with prosect & scale
61
Figure 3. Framewise F1, precision and recall on ASFX-SED dataset. FLAM achieves higher F1 under various threshold com- pared to ablated models without rext-dependent bias and scale, showing the effectiveness of our logit-correction objective.
with the threshold, indicating that FLAM's output is polar- ized due to the unbiased classifier transforming raw outputs into near-binary probabilities. Additional qualitative results in $ B show that FLAM provides more accurately calibrated detection outputs, whereas ablated models produce either overconfident or poorly confident probabilities.
Fine-Grained Multi-modal Alignment Pairing local and global information in the same sample has been success- fully explored in the image domain through contrastive representation learning approaches such as Deep Info- Max (Hjelm et al., 2019) and distillation like DINO (Caron et al., 2021). These techniques enhance performance across general and downstream image tasks. More recently, VLMs including GLIP (Li et al., 2022) have incorporated CLyP-style extensions into object detection pipelines to facilitate zero-shot open-set object detection and grounding. Similarly, Mukhoti et al. (2023) introduce Patch-Aligned Contrastive Learning (PACL) as an unsu- pervised extension of CLIP for open-vocabulary semantic segmentation. PACL leverages large-scale captioned im- age dataset and includes a "variational" CLIP objective that softly classifies image patches through softmax over their local alignment with the associated caption.
5.4. Retrieval Performance
We evaluate text-audio retrieval using recall at ranks and on Clotho (Drossos et al., 2020), AudioCaps (Kim al., 2019), and the ASFX dataset. Table 2 shows that FLAM and FLA.M-Global achieve strong performance on ASFX dataset, indicating that our training corpus generalizes well to the sound effects domain. Existing ALMs like LAION- CLAP and CompA perform better on Clotho and Audio- Caps but only moderately on ASFX, which highlights the limitations of noisy web-scraped audio data those models trained for specialized sound effects. Additionally, MGA- CLAP trained on our dataset matches FLAM's performance on Clotho and AudioCaps, further demonstrating the sig- nificant impact of training distribution. Notably, FLAM's frame-wise training results in minimal degradation of re- trieval scores, demonstrating that enhancing local alignment does not compromise sample-level performance.
methods
In the audio domain, neither labeled local alignment datasets nor large-scale unsupervised datasets with up to a billion samples are available. Some audio-language models im- prove temporal reasoning (e.g.. "a car crashes before man cries") by incorporating temporal compositional examples into contrastive objectives, enabling the learning of complex temporal relationships (Ghosh et al., 2024; Wu et al., 2023; Yuan et al., 2024). However, these approaches only achieve instance-level or partial alignment without explicit event localization. Few studies address local text-audio align- ment. Text-Audio Grounding (Xu et al., 2021; 2024) aligns word-audio pairs by mapping 527 AudioSet events (Gem- meke et al., 2017) to phrases in the 46k-sample AudioCaps
5.5. Zeroshot Classification Performance
Table 3 reports results on ESC-50, US8K, and VGGSound under zero-shot classification, where each dataset's class names serve as the text queries. FLAM surpasses FLAM- Global and MGA-CLAP when all are trained on our corpus, showing that frame-level supervision also refines global representations. Relative to larger-scale ALMs like CompA, FLAM remains competitive, particularly on VGGSound. These results suggest that explicit frame-level alignment enhances overall discriminative ability without sacrificing
8

---

FLAM: Frame-Wiss Language-Audio Modeling
captions, but it is limited by its ontology and dataset size. MGA-CLAP (Li et al., 2024) represents a more closely re- lated approach that builds model with shared codebook to regularize embeddings and trains ALM with a temporal-aware audio encoder using contrastive loss with hard negatives, enhancing temporal correspondence yet still relying on sample-level labels. In contrast, our work leverages ground-truth temporal labels for -ocabuauu, frame-wise SED, integrates synthetic and closed-set data to enhance localization, and enables training on a million-scale supervised dataset.
enhance open-vocabulary localization.
audio-language contrastive
Impact Statement
This work introduces FLAM, a model for frame-wise audio- language alignment to improve sound event detection using natural language queries. Our goal is to advance the field of multimodal learning by enabling fine-grained and inter- pretable audio understanding. FLAM may benefit applica- tions such content iidexiig. accessibility, and multimedia retrieval. While we do not foresee significant ethical risks, we encourage responsible use of the model in real-world scenarios.
7. Conclusion
In this paper, we introduced FLAM, a frame-wise au- dio-language modeling framework for open-vocabulary sound event detection. Our approach augments standard audio-text contrastive learning by incorporating a frame- level contrastive objective and a text-dependent logit bias correction mechanism to address severe label imbalance. Through large-scale data synthesis in which we mix labeled sound events into diverse 10-second background clips, we created an extensive training corpus that enabled robust frame-wise supervision. Experimental results demonstrate that FLAM significantly outperforms models trained solely at the clip level in terms of fine the-graindd event localization, while preserving strong retrieval performance and reliable zero-shot classification. By aligning text-based queries to localized acoustic frames, this work extends the versatility of multimodal audio languag models and opens the door to open- vocabuacyyu sound event detection. The improved tem- poral precision of FLAM not only benefits open-vocabulary sound event detection but also offers more interpretable global and local representations that could strengthen down- stream tasks requiring temporal alignment.
A Acknowledgment
We would like to express our gratitude to the following indi- viduals for their insightful discussions and valuable advice on the project: Yuanbo Hou, Samuel Lavoie.
This work was done while Yusong was interning at Adobe. The contributions of the first three authors are as follows: Yusong Wu was responsible for the model design, code im- plementation, data pipeline and augmentation, experiment training, and paper writing. Christos Tsirigotis contributed the logit adjustment loss design and the inference classifier, as well as authoring the derivations and proof section in the paper. Ke Chen participated in figure creation, experiment training, model design, and provided technical mentorship throughout the project.
References
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In IEEE/CVF Inter- national Conference on Computer Vision, 2021.
FLAM represents an initial step toward large-scale open- vocabulary sound event detection, but several aspects remain to be improved. The current training corpus, while diverse, is still limited in scale; future work could explore larger and more diverse corpora, potentially by synthesizing additional labeled mixtures or leveraging web-scale audio. The model architecture itself is relatively lightweight, suggesting that scaling up the encoder or introducing more expressive ar- chitectures may yield further gains. Additionally, FLAM uses fixed 10-second audio input and coarse frame reso- lution, which constrains its ability to handle longer more temporally nuanced recordings. Future efforts could focus on supporting variable-length audio and adopting encoders with finer temporal granularity. Beyond architectural and data improvements, future work could explore the use of real-world frame-level annotations, better evaluation proto- cols, KL penalty to align frame-level outputs with global model, and generative augmentation strategies to further
Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., and Dubnov, S. Hts-at: at: A hierarchical seken-semantic audio transformer for sound classification and detection. In ICASSP 2022-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-650. IEEE, 2022.
Cornell, S., Ebbers, J., Douwes, C., Martin-Morató, I., Harju, M., Mesaros, A., and Serizel, R. Dcase 2024 task 4: Sound event detection with heterogeneous data and missing labels. arXiv preprint 2xXi:24406080566 2024.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv: 100.00000, 2018.
Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio

---

FLAM: Frame-Wii Language-Audio Modeling
captioning dataset. In ICASSP 2020-2020 IEEE Inter- national Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pp. 119-132, 2019.
Ebbers, J., Haeb-Umbach, R., and Serizel, R. Threshold independent evaluation of sound event detection scores. In ICASSP Acoustics, Speech and Signal Processing (ICASSP), pp. 1021-1025. IEEE, 2022.
IEEE International Conference
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead. S., Berg. A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. Clap learning audio concepts from natural language pervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
2023.
Li, L. H., Zhang, P., Zhang, H., Yang. J., Li, C., Zhong, Y., Wang, L., Yuan, L.,, Zhang, L., Hwang. J., Chang, K., and Gao, J. Grounded language-image pre-training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Evans, Z., Carr, C., Taylor, J., Hawley, H., and Pons, Fast liming-conditinn latent audio diffusion. In Con- ference of International Society for Music Information Retrieval (ISMIR), 2024.
Li, Y., Guo, Z., Wang, X., and Liu, H. Advancing multi- grained aligmment for contrastive language-audin pre- training. Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7356-7366665 0366-736,777 2024.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and buman-labeled dataset for audio events. In 2017 IEEE international conference on acousties, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. AudioLDM: Text- to-audio generation with latent diffusion models. Pro- ceedings of the International Conference on Machine Learning, pp. 22450-21474 2023a.
Ghosh, S., Seth, A., Kumar, S., Tyagi, U., Evuru, C. R., R., Sakshi, S., Nieto, O., Duraiswami, R., and Manocha, D. Compa: Addressing the gap in compositional reason- ing in audio-language models. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Liu, S., Zhang, X., Sekhar, N.. Wu, Y., Singhal, P., and Fernandez-Granda. C. Avoiding spurious correlations via logit correction. In The Eleventh International Confer- ence on Learning Representations, 2023b.
Hershey, S., Ellis, D. P., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., and Plakal, M. The benefit of temporally- strong labels in audio event classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 366-370 IEEE, 2021.
Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv::197..11 1692, 364, 2019.
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, 2019.
Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. Long-tail learning via logit adjustment. In International Conference Learning Representations, 2021.
Mesaros, A., Heittola, T., Virtanen, T., and Plumbley, M. D. Sound event detection: A tutorial. IEEE Signal Process- ing Magazine, 38(5))67-83) 2021.
JiaKai, L. Mean teacher convolution system for dcase 2018 task 4. Technical report, DCASE2018 Challenge, Tech. Rep, 2018
Mukhoti, J., Lin, T., Poursaeed, O., Wang, R., Shah, A., Torr, P. H. S., and Lim, S. Open vocabulary semantic segmentation with patch aligned contrastive learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401. 04088, 2024.
10

---

FLAM: Prame-Wise Language-Audio Modeling
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Xu, X., Dinkel, H., Wu, M., and Yu, K. Text-to-audio grounding: Building correspondence between captions and sound events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), PP. 606-610. IEEE, 2021.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text- coxt-conditional image generation with clip latents. arXiv preprint arXiv:2204.06//11 1(2)3, 2022.
Xu, X., Ma, Z., Wu, M., and Yu, K. Towards weakly supervised text-to-audio grounding. arXiv preprint arXiv:2401.02584 2024.
Yuan, Y., Chen, Z., Liu, X., Liu, H., Xu, X., Jia, D., Chen, Y., Plumbley, M. D., and Wang, W. T-clap: Temporal- enhanced contrastive language-audia pretraining. arXiv preprint arXiv:2404. 78006, 2024.
Salamon, J., MacConnell, D., Cartwright, M., Li, P., and Bello, J. Scaper: A library for soundscape synthesis and augmentation. In 2017 IEEE Workshop on Applications of Signal Processing Audio and Acoustics (WASPAA), pp. 344-348. IEEE, 2017.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings the IEEE/CVF International Conference Com- puter Vision pp. 11975-11986, 2023.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278- 25294, 2022.
Serizel, R., Turpault, N., Shah, A., and Salamon, Sound event detection in synthetic domestic environments. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 86-90. IEEE, 2020.
Tsirigotis, C., Monteiro, J., Rodriguez, P., Vazquez, D., and Courville, A. C. Group robust classification without any group information. Advances in Neural Information Processing Systems, 2023.
van den Oord, A., Li, Y., and Vinyals, O. Representa- tion learning with contrastive predictive coding. ArXiv, abs/1800.037488, 2018.
Wilkins, J., Salamon, J., Fuentes, M., Bello, J. P., and Nieto, O. Bridging high-quality audio and video via language for sound effects retrieval from visual queries. In 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 1-5, IEEE, 2023.
Wu, H.-H., Nieto, O., Bello, J. P., and Salamon, J. Audio- text models do not yet leverage natural language. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 1-5, IEEE, 2023.
Wu* Y., Chen K., Zhang T., Hui*, Y., Berg- Kirkpatrick, T., and Dubnov, S. Large-scale contrastive language- audio pretraining with feature fusion and keyword-to- caption augmentation. In ICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.

---

FLAM: Frame-Wiss Language-Audio Modeling
A. Sound Event Detection Results
A.I. ASFX-SED Dataset
Jub
humer ommo
loc
Oopphg
lunar
dro
aoc dog
copke
parte
wak dvy
matd
TLAM
TEAM colia darlatee
cobe dickee
MCA-CLAP
decnnk
pa 25
cati Fargec
(b)
Figure 4. Sound event detection results of FLAM on ASFX-SED dataset
12

---

FLAM: Frame-Wiss Language-Audio Modeling
A.2. Synthetic Held-out Dataset
ACA
paet cherme
1o0
coviligg.
1d4
YLAM
cidival lie
and repostr, chernee.
196' camoie
bell youe anciaal sdll.
la lite dlent
cais Alyillaiia
ILAM
FLAM
onk oegisas ieraei. rndicarry canuse gioeat dralldde and preaaaeee -dull.. lans lase cuactigg rat horry dbytherke ao-
ILAM
rosk.
MSS-CLAP
cal rigites dkd chermet.
comi-ccr pa
conS camiil gooad oor.. drateedid prommaen sdilt lego hot as cavetagg ras berry, rtyltern
MGSALLAP
fraa 20-8 ctll. Sant
FLAM
FLAM UIE CoRNe umderay
Teeld
NICACTLA
sccan vidu part.
FLAM
MANCLA
camiil grostito rike dact cleckgg
diaile Lndd pro mataaal aid cat most deteet sou arrencc or
Gsi Batac sbc dest vibie gralt beroue
snak busilyy strr kindig gra
a
(b)
Figure Sound event detection results of FLAM synthetic held-out dataset.
13

---

FLAM: Frame-Wir Language-Audio Modeling
A.3. Audioset-Strong Dataset
Fat:
T-AH
TLAM
HAMIn weller
ILAM
mal spont me sptclii
ende caleppodd
Figure 6. Sound event detection results of FLAM on Audioset-Strong dataset.
14

---

FLAM: Frame-Wiss Language-Audio Modeling
B. Ablation Models Output Comparison
GroedTack Lank
Graa Likk
endia coallal
abauk pwn
Tarkh
Tockt TLAM
TLAM
fimee
eodirior trp
snd drp
rallcces
boder dirve
raquuiiall
nasd lapac
TLAM poraaaatil
bruscing leeu
dry-
wet
valionss
1t AM wosieie has
kana unars
exturee top
wok drg
A4
25
14
vhral skat
(b)
Figure 7, Sound event detection results of FLAM on ASFX-SED dataset compared with ablation models.
15

---

FLAM: Frame-Wiss Language-Audio Modeling
Table Sound detection performance synthetic open-vocabulary SED (Held-out, ASFX-SED) and traditional closed-set SED
dataset (DESED, MAESTRO, Audioset-strone UrbanSED) among ablated models.
Held-out
ASFX-SED
DESED
MAESTRO
Audioset-S
UrbanSED
Model
AUROC
AUROC
PSDS
AUROC
MPAUC
PSDS
AUROC
PSDS
AUROC
FLAM
91.0
81.23
9.37
91.66
56.97
11.16
94.76
29.52
93.62
w/o FLAM-Global Init
91.35
80.34
6.37
89.7
52.62
10.56
94.3
29.53
92.35
w/o Closed-set SED data
86.91
74.94
9.02
89.7
53.92
2.42
84.37
6.51
76.12
w/o Global Loss
91.33
81.15
10.25
91.98
32.45
10.77
93.89
30.68
93.08
Uomen Lonk
dtey proahe
tas da duciiiil
cuntall,
leary
mis rythebe
one sury conte groahir
Kays iaee
EAM poree
repire doment
a8
Figure 8. Sound event detection results of FLAM synthetic held-out dataset compared with ablation models.
C. Appendix
C.1. SigLIP Objective
SigLIP (Zhai et al., 2023) treats contrastive alignment as a binary classification problem LsigLiP
1 B 1=1==1
log (zi,) (a of of + B))
(11)
where zıj (+1) is +1 if A, and Y, match, and -1 otherwise. Here, 3 is learnable bias, and >> 0 is again trainable logit scale factor. This can be viewed as synthetic binary classification with label zi trained with binary cross entropy loss.
In original SigLIP paper, the authors initialize the learnable bias as 10 to compensate the fact that most labels are negative labels in the binary classification problem during training. In this paper, we provide perspective that is derived from
16

---

FLAM: Frame-Wiss Language-Audio Modeling
Creaa Land
mak rocck quidig
spott spoking
cale sport can spoiiing
ILAM mnspeiee
mik
FLAM
TLAM
FLAN Meeertt
gever
ILAM
FLAN
FLAV posedd
TLAM
Trngo
speck 2LI gpildngg
spends speiiing
4BA speck (oodig
a
(b)
Figure Sound event detection results FLAM on Audioset-Strong dataset compared with ablation models.
the logit adjustment for the label-imbalanced binary classification problem, which ultimately leads to the formulation of the objective in $3.
Consider the binary classification objective SigLIP objective (Eq. 11) written equivalently as follows, for binary labels = = 0.5 (zi.j + 1) E {0, 1):
Σ iE [
24.5) 2 log (h(X,, Y3)) + (1-21,)))) log (-)(X,
(12)
B
where (()) is the sigmoid function (ι)) 1)) and h(:,)) is the logit function, which is defined as:
Mx. ax) (3) +
(13)
Thus, in expectation and at the minimum, we can view the SigLIP model as learning
(()(,,)) Rs Panta (z = 1 x,
(14)
can see, however, that this conditional distribution is sensitive to the marginal distribution of the labels during training
17

---

FLAM: Frame-Wist Language-Audio Modeling
P2 1):
Pdata (z 1,
Pdata (2 |x, (. 1)
Pdlata 1, + Pdata
86, y)
1
Pdatal (= 1, y) Pdatal (2 = 11, 21 y)
log
1 Pdata (== 1,5.9)
Rdata(==1,z.))
a log Pdata (2, ====0 1)
Palata (2 1)
+ log
Priata (2. y = -1
Paata (z = -1)
Pdata Pdata(
a log
Pdlata Peeitta + log B-1
B
d log Praaaayle
log( - 11
(15)
Pdata(y)
The classification problem that SigLIP learns is label-imbalancee due to the outnumbered negative samples compared to positive samples. In the derivation above, we can observe that for very large batch sizes, the term log 1) can effectively dominate the logit making Pdata (z
x, y) 0 as B + co. If model logit bias 3 = 0 in Eq. 13, then the dot product has to learn to be negative for all and y, thus impeding the learning of positive associations. However, by setting it directly so that 3 = log B 1) we find that
e (I)) el e'()) = log Prlata (u) log(B=T),
(16)
Pdataly)
and the dot product of embeddings effectively learning the same log-density ratio as CLIP. To verify our reasoning, we note that the optimal batch size and 3 combination in the SigLIP paper was B 32768 and 3 = -10. Indeed, the optimal according to our reasoning log 32768 10.399 Rs -10.
C.2. Optimum LSED
Similarly to the derivation of the model at optimum in expectation for SigLIP in SC.1, we have the following for our proposed loss in Eq. 4
Pduta (z 2,,
Pdata (z x,1
Pdata (a 1, 2, 1. y) + Palat = x. 1,
1
Pdata (= 1, x, y)
log
Paaca Pdara aan=(2==11.r...p 1.,....))
Pdtta 12 1xx, 1 1)
Paatal Pánca(==1,0.1p)
log Pdata (x,1 8, = =1
Pdata (x, 2 -1) + log Pdata 2 = 1)
Pilaaa (2)
log Pdata l 3)
Pduta (2 3)
+ log
Pdatal (ar, 1)
Printa (z -1 Patnta (T)
Pdata x.1) + (0(w))
σ
log
(17)
Pdata (y)
C.3. Derivation of Robust Classifier
As we have argued in $3.2, we recognize that our proposed LSED in Eq. 4 introduces an imbalance in the labels 2 E {-1, 1} between positive and negative tuples of frames in audio samples and event descriptions y. Furthermore, we recognise the possibility that our training dataset may impede learning an open-vocabulary detector as may introduce spurious correlations among the labels and the contained event description y. That is the event "barking" might be more rare in terms of positive frames contained in the dataset, than the even "meowing". However, an open- vocabvocabulary detector should not prioritize learning some events more than others, we would like it to be able to detect audio frames of relevance regardless of event descriptions provided.
Mathematically, this means that our training and testing conditions of our model may differ. In particular under the training set, Pdata (z 11) o Paata 32) for different event deseriptions y/ * y2- However at test-time, we deploy out
18

---

FLAM: Prame-Wist Language-Audio Modeling
model under distribution where Ptest (z 31) Ptest (= 122 Ptest (z = 1) for y1 A y2- That is during test-time equally interested in detecting
assume that label and event description are marginally independent. Moreover, events, well as their absence, correctly. This is reflected in uniform test-time condition across possible labels. In other words, Ptest 1) = (2 1) = 0.5. Finally, we consider that the mechanism, that generates positive or negative frames given a certain event, is invariant between train-time and test-time conditions, P 1 y, =) Prest (2r, y, y,,
Following Menon al. (2021) and Tsirigotis et al. (2023), the Bayes-optimal robust classifier is given by
= (xr, 1 v) arg max =é(-1,1}
Prest (,,,y)
(18)
We use the Bayes rule and the assumed invariance to express an equivalent classifier in terms of the training distribution Pdata
Pteat (2,,1)
Prest (z x1,,8)
Prest (2 x.1, L, v)
xs
Os Prest (xr, (. = Pdata (z, 1 y, 2)
U1- (2-1,1)(2)
Prest 3)
Paata(= Ir (1,,,1)
(19)
Pdata (2 y)
So, would like find classifier for which
Pdata (z xr, 1,)
= (z. 1.y = arg max = Pdata 2 y)
(20)
In our case, there are two possible outcomes for the classifier 2 = 1 and 2 === 1. Taking the argmax involves comparing the two density ratios. Specifically, we decide that = (xr,,,)) if
Pdata
L,2)
Pdata - xr, 12,,,y
Pdata (z
y))
Pdata (z -1 1/)
Pdata = Ir, 1,,
1 - Prlata (z = x2,1,1)
(21)
Pdata (z y)
Pdata (2= = 1 y)
Pdata = 1 x,, 1,,1,)) > Pd (z = 111)
Essentially, (2 = 1 y) determines a threshold for the classifier Pdata = 1 xr, xx,.,) = o (h(x, (,)) to robustly
decide whether frame (zz, I) can be described by y. Observe that Pdsta x, I, y) has different decision boundary for different prompt y. To fix this, we define new function s(x: 1, y) that has unified decision boundary of 0.5 for all y: Pdata (2 = xr, 1,
8(x, (. Pdata 1 x. 1, v) + Pdata 1
(22)
Notice that (,,, > 0.5 + (2 = 1 x. y) > Pd = 1 y).
In practice, we can use ( Pasta (w] to get approximate value of s(x,1,)) y) when 8 (y) is negative enough for all This is Panta
because:
Pdata = x. v.
s(I, 1, =
Prasa (= = x. + Pdlaa = w)
1
1 Pauta(==11) (== 119) 119)
1 log
a ((8"(1)) (11)
Pasta (== x.y.)
Peiata (ulz.))
Pausa (v) + +8
(23)
(log Edata (wla,)) (w)(
log Peasa (V iz Pesta (M)
+ 14c
1tc 1 -0 (g)
Pdsta (M)
1
a (log Pdata (y 2,()
as
3
00.
Pdata (y)
In practice, observe 8 to be near 8, which result in neglectble difference before and after the approximation. By approximating s(x, 1, v) with olog Patta (vlxsd)) we save the compute for estimating 3 (y) during inference.
(anaa())
19

---

FLAM: Frame-Wise Language-Audio Modeling
C.4. Loudness Relabel
We compute the RMS curve using a window size of 2400 and hop size of 1200 (50 Hz 48 kHz sample rate). Post- processing proceeds follows: (I) remove negative segments shorter than 10 frames (200 ms) if they lie between positive segments (mark them as positive), (2) remove positive segments shorter than 2 frames (40 ms) if the total event exceeds 10 frames (200 ms).
C.5. Training Details for FLAM-Global
Following LAION-CLAP (Wu* et al., 2023), we initialize HTSAT and RoBERTa from pretrained checkpoints. We use batch size of 768, a learning rate of 10-1, and an Adam optimizer with 31 = 0.9, 82 = 0.99. The learning rate schedule employs cosine warmup (3200 steps) followed by linear decay, for total of 50,000 steps.
All sampled captions are converted to lower case, with a maximum text input length of 77 tokens. We sample data from our (1. MM) dataset, AudioCaps, and Clotho weights of (1.0, 0.1, 0.1). During training, 30% of the time we randomly downsample audio 16 kHz or 32 kHz, then upsample back 48 kHz. For each audio, we randomly choose one of its captions and one of its tags, forming the training text as "keyword, tag". We also removed all samples from evaluation datasets from the general audio training data.
C.6. Training Details for FLAM
We train FLAM with batch size of 512 and learning rate of 10-4, using Adam (31 0.9, 32 0.99) and the same whrneeathen-decay schedule with 3200 steps of warmup and train 120,000 steps. When sampling, we use our dataset, AudioCaps, and Clotho with weights (1.0, 0.1, 0.1). Additionally, we sample from synthetic SED data, AudioSet-strong, UrbanSED, and DESED with weights (0.5, 0.5, 0.1, 0.1). We again apply the same audio downsampling strategy as in FLAM-Global
To improve event-caption generalization, we replace LLM- LLM-ggenrated captions with random tags 50% of the time. We initialize the last bias layer of the per-text bias MLP to -8 (matching the average converged logit bias in Lp) and the last bias layer of the per-text scale MLP to log( (10) following CLIP (Radford et al., 2021) and SigLip (Zhai et al., 2023).
C.7. Details of SED Metrics
Following MGA-CLAP (Li al., 2024), we apply median filter of size frames to the SED output before evaluation. We _eval (Ebbers et al., 2022) to compute the following:
PSDS, (or refereed as "PSDS1" in other works) with DTC=0.77 GTC=0.7, aSt 1, oCT 0, and 6max 100.
MPAUC, the mean partial AUC segment-level ROC curves (I segments), capped at FPR of 0.I.
AUROC, the full ROC AUC on segments of length 0.3125 s (the HTSAT frame length).
C.8. Training Details of MGA-CLAP
We follow MGA-CLAP (Li et al., 2024) hyperparameters and architectures except that: (1) we use input text tokens instead of 30, (2) we train for 50,000 steps match FLAM-Global.
C.9. Details of Ablation Models
The ablated models are defined as follows:
FLAM without per-text scale uses a scalar scale initialized to log(10), which is updated by the SED loss.
FLAM without per-text bias employs a scalar bias initialized to 100, following SigLIP's default, and this bias updated by the SED loss.
FLAM without per-text bias and scale incorporates both scalar scale initialized log(10) and scalar bias initialized to - -10
All ablated models utilize the same hyperparameters as the original FLAM model.
20

---

FLAM: Frame-Wise Language-Audio Modeling
C.10. Ablations on SED Training
We conducted ablation experiments investigate the impact of three factors: (1) removing FLAM-Global initialization (i.e., initialization from pre-trained HTSAT and RoBERTa), (2) removing the global loss during SED training, and (3) removing the closed-set SED dataset. Table 4 presents the results. While removing FLAM-Global initialization and global loss leads to only minor drop in performance, removing the closed-set SED dataset causes significant performance degradation, underscoring the importance of closed-set data in our SED training.
C.11. Alignment Correlation Metric
To robustly quantify the alignment quality between audio frames and textual descriptions, we propose an biigmment-specifee metric based on Spearman's 's rank correlation coefficient (p). This metric directly measures how well the model's frame-level similarity predictions correspond to the actual temporal occurrence of audio events, independent of absolute similarity magnitudes or decision thresholds.
Formally, for each captioned event, compute Spearman correlation between:
The model's predicted output for each frames.
The binary ground-truth event presence labels for corresponding frames.
higher Spearman p indicates more accurate temporal alignment of predictions.
Table 5 shows FLAM significantly outperforms baseline models (MGA-CLAP, FLAM-Global) open-vocabulary datasets (ASFX-SED and synthetic Held-out), confirming that the improved SED performance of FLAM indeed arises from enhanced frame-level alignment.
Table 5. Spearman rank correlation (p) measuring alignment quality. FLAM achieves significantly higher alignment correlations than
baseline methods both datasets, indicating superior temporal alignment. All reported correlations have a p-value < 0.01.
Model
ASFX-SED p Held-out p
FLAM
0.409
0.600
MGA-CLAP
0.256
0.352
FLAM-Global
0.197
0.262
C.12. Additional Ablation Experiments
We present further ablation experiments to clarify the impact of the global loss component and temporal granularity (number of frames, L).
Global Loss Ablation Removing the global contrastive loss (FLAM no global loss) resulted in slight improvement in SED metries but a significant drop in retrieval and zero-shot classification performance. As shown in Tables 6, and 8, the global loss is essential maintain robust global alignment required for effective retrieval tasks. This indicates critical trade-off between local alignment objectives for SED and global representation quality necessary for retrieval and classification.
Temporal Granularity Ablation (L 128) Increasing temporal resolution from L 32 to L 128 (FLAM - L 128)
yielded minor gains in SED performance but adversely affected retrieval and zero-shot classification accuracy, suggesting that finer temporal resolution may encourage overly specialized frame-level embeddings at the expense of globally coherent audio representations.
The detailed ablation results are summarized in Tables 7, and 8.
These additional experiments reinforce the necessity of joint global-local training, highlighting trade-offs inherent in temporal granularity choices, and underscore the importance of balanced frame-level and global objectives.
21

---

FLAM: Prame-Wist Language-Audio Modeling
Table Sound event detection performance on synthetic open-vocabulary SED (Held-out, ASFX-SED) and traditional closed-set SED datasct (DESED, MAESTRO, Audioset-strong, UrbanSED).
Held-out
ASFX-SBB
DESED
MAESTRO
Audioset-S
UrbanSED
Model
AUROC
AUROC
PSDS
AUROC
MPAUC
PSDS
AUROC
PSDS
AUROC
FLAM-Global FLAM (proposed)
67.76
65.14
7.09
85,52
51.13
1.11
82.54
0.82
67.39
91.0
81.23
9.37
91.66
56.97
11.16
94.76
29.52
93.62
MGA-CIAL
74.17
69.56
14.72
89.28
52.50
1.24
79.12
6.42
78.22
FLAM no global loss
91.33
81.15
10.25
91.98
32,45
10.77
93.89
30.68
93.08
FLAM
128
92.6
82.51
9.52
92.45
49.79
11.3
94.83
36.0
95.1
MGA-CLAP (reported)
26.4
10.1
8.7
Table 7. Recall performance of text to audio (T2A) and audio to text (A2T) retrieval.
ASFX
Clotho
AudioCaps
Model
Dataset
T2A
A2T
T2A
A2T
T2A
A2T
R@D 1
R@5
R@1
R@5
R@I
R@5
R@1
R@5
R@
R@5
R@D 1
R@5
FLAM Global
4.4
14.8
4.0
13.8
14.3
35.8
17.9
39.8
36.0
70.5
46.1
78.6
FLAAM MGA-CLAP"
FLAM-Collection
4,4
15.2
3.9
13,9
13.8
33.2
16.7
42.2
32.1
64.8
43.3
3.9
14.8
3.9
13.8
13.4
30.3
39.1
36.7
69.9
47.2
FLAM - global loss
FLAM-Collection
0.8
2.8
1,4
5.0
5.4
13.5
8.1
23.0
3.9
20.3
7.1
23,6
FLAM 128
4.4
14.8
3.8
13.4
11.6
29.4
16.0
38.0
21.8
53.1
28.9
64.0
Reported Performance from Prior Studies (but trained on different datasets)
LAION-CLAP CompA
LAION-Audio-630k
2.0
7.6
1.6
6.0
16.1
38.3
22.7
48.5
36.1
71.8
46.8
82.9
CompA-Collection
16.8
43.5
23.9
50.7
36.1
78.6
47.8
83.5
MGA-CLAP
WavCaps
2.3
2.0
46.0
51.2
41.8
76.1
54.4
83.6
Table 8. Zero-shot classification accuracy.
Model
ESC50
US8K
VGGSound
FLAM-Global
81.6
65.4
38.9
FLAM
86.9
75.6
39.3
MGA-CLAP
72.6
69.9
38.6
FLAM no global loss
70.4
18.5
FLAM L 128
83.1
79.0
33.3
LAION -LAP
29.1
CompA
89.1
85.7
29.5
MGA-CLAP
94.9
83.7
31.8
22


Reviews:
[{"review_id": "kgHsTMeTHL", "reviewer": "Reviewer_bqpP", "summary": "This paper introduces FLAM, an audio language model, which incorporates a frame-level sound-event localisation loss along with a contrastive learning objective to produce frame-level representations aligned with natural language.\n\nBy using a custom augmentation pipeline to combine multiple sounds in a single sample, this paper obtains a training dataset composed of audio, multiple captions, and a binary mask indicating the presence of the captions in the sound at a given frame.\n\nFLAM also compensates for the label imbalance on the dataset (note not all classes have the same duration!) by estimating a per-caption bias and logit-scale.\n\nExperiments on text-to-audio and audio-to-text retrieval show FLAM being competitive with baseline audio-language models (eg. ~86% R@5 for both text-to-audio and audio-to-text retrieval on Clotho compared to LAION-CLAP), while showcasing very strong sound event detection on both open-vocabulary and closed-vocabulary sound event detection tasks.\n\n**Post-rebuttal update**\nThe main concerns identified during the review process regarded the training set, authors clarified the questions and committed to releasing a validation dataset. Additionally, the authors added experiment ablating the number of frames and the global loss, providing a better understanding of FLAM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer **bqpP** for their detailed and thoughtful review. We appreciate your recognition of FLAM’s contributions and have revised the manuscript to address your comments and questions. Below, we respond point-by-point, reordered for clarity. Due to space constraints, we include additional results and responses on the supplementary webpage:  \n**https://flam-model.github.io/response_reviewer_bqpp.html**\n\n---\n\n### Questions Related to Dataset\n\n**Dataset size and audio length**  \nAs noted in Sec. 4.1, the training set includes 1.1M samples. The 1M augmented samples are 10 seconds each, matching FLAM’s fixed-length input (line 317). Original text-audio clips are variable-length; we sample 10-second segments during training.\n\n**How many types of events are covered in the captions?**  \nThe captions span most categories in the [Universal Category System (UCS)](https://universalcategorysystem.com/). These include nature sounds (e.g., thunder, rainfall, bird calls), urban and human-made sounds (e.g., car engines, speech), and professionally designed sound effects (e.g., gunshots, lightsabers).\n\n**License of dataset**  \nAll proprietary datasets are fully licensed. We clarified this in the revised opening of Sec. 4.1:  \n> “We gather a large mix of licensed proprietary sound effect datasets and ...”\n\n**Will the training dataset for FLAM be made available?**  \nDue to licensing, the full dataset cannot be released. However, we will release the ASFX-SED dataset generated by our augmentation pipeline. We hope this will provide a valuable benchmark for future research in open-vocabulary SED.\n\n**How effective are the dataset augmentation techniques?**  \nPrior to our work, no training dataset existed for open-vocabulary SED. We show how to scale the construction of such a dataset via data augmentation techniques. This enables frame-level training, since we know frame-level event labels by construction. This makes a crucial difference for significantly outperforming MGA-CLAP (Table 1), which is trained without explicit frame-level supervision.\n\n---\n\n### Additional Technical Questions\n\n**How large is FLAM? How long does it take to train?**  \nFLAM has ~150M parameters. FLAM-Global trains in ~12 hours; full FLAM in ~24 hours. It shares LAION-CLAP backbones but adds:  \n1. A 1024-dim projection head  \n2. Two MLPs predicting the per-text bias ($\\beta^t$) and per-text scale ($\\alpha^t$)\n\nIn comparison, CompA is larger, using HTSAT-large and FLAN-T5. We updated line 400 to clarify:  \n> “Relative to larger-scale ALMs like CompA, FLAM remains competitive, particularly on VGGSound.”\n\n**Effect of varying L (number of frames)**  \nWe thank the reviewer for this suggestion. Due to limited time, we trained a variant of FLAM with L=128 and observed a trade-off between SED performance and retrieval/zero-shot performance. See the supplementary link above for results.\n\n**In table 1, is the gap between MGA-CLAP\\* and MGA-CLAP (reported) explained by the different training datasets?**  \nYes, this gap is due to dataset differences. MGA-CLAP was trained on large-scale web data, including YouTube, which aligns with DESED and AudioCaps. MGA-CLAP* was retrained on our proprietary dataset, explaining the lower performance.\n\n**There seems to be a trade-off between frame representation and global representation [...]. Could this difference be due to model drift when training FLAM with the final loss? And if so, could a KL-penalty term with respect to the predictions of FLAM - Global be helpful (similar to the KL penalty used in DPO [1])?**  \nExcellent observation! Indeed, your observation reflects a trade-off between local and global representation alignment. To investigate further, we conducted additional experiments training FLAM without the global loss. Removing global loss results in a slightly better SED performance but a significant drop in retrieval and zero-shot performance. Please refer to the results in the link above. We mitigate this trade-off via joint global contrastive optimization. As the reviewer suggested, incorporating a KL penalty to align frame-level outputs with the global model is a promising direction. We’ve noted this in the discussion for future work.\n\n**The impact and intuition about logit scale**  \nIntuitively, a smaller logit scale increases the cosine distance between negative frame and text embeddings for the same loss effect. This helps the model capture finer distinctions in cosine similarity. Due to space constraints, please refer to the link above for more discussion.\n\n---\n\n### Other Comments\n\n**Missing impact statement**  \nMany thanks for pointing this out. We have added an impact statement at the end of the paper. See the link above for the updated version.\n\n**Typos**  \nWe thank the reviewer for highlighting the typos—these have been corrected. Due to space constraints, full details are in our response to reviewer **GGdQ**.", "claims_and_evidence": "The paper makes the following claims:\n\n**C1. FLAM produces frame-level representations for open-vocabulary event detection**\n\nThe theoretical analysis and the results (particularly Table 1) both support this claim.\n\n**C2. FLAM effectively handles label imbalance in sound-event detection training**\n\nThis is supported by the overall results (which would not be as good as they are if FLAM could not  handle dataset imbalances) and specifically by Figure 3. \n\n**C3. A pipeline for captioned frame-level audio generation**\n\nThe augmentation pipeline is well described in the paper, but key factors are omitted: \n\n * How large is the dataset?\n * Is every one of the 1M samples 10 seconds long? \n * How many types of events are covered in the captions?\n * How were the proprietary source audio samples licensed? Were they licensed at all??\n * How effective are the dataset augmentation techniques?\n\n**C4. State-of-the-art performance in sound-event detection, strong performance on retrieval and zero-shot classification**\n\nThis claim is supported by the results presented on the text.\n\n**Post-rebuttal update**\n\nThe authors answered all the questions regarding the training dataset, and committed to releasing the validation dataset ASFX-SED.", "methods_and_evaluation_criteria": "The methods and evaluations make sense for the problem at hand.", "theoretical_claims": "I did not review the theoretical claims in the papers as those were in the appendix (which I did not have time to review).\n\nI did go through all the derivations in the main text and they seem reasonable. However, as a nitpick, the paper does spend a lot of time justifying the use of a learned per-caption bias term ($\\beta^t$), and then figure 3 shows what makes the highest difference is a learned per-caption weight scale ($\\alpha^t$) whose use is not justified nor explained in the text (beyond \"we experimentally found that it is beneficial\").", "experimental_designs_or_analyses": "The experimental analyses that are present seem reasonable.", "relation_to_broader_scientific_literature": "FLAM builds on previous Audio-Language Models by introducing a _frame-level representation_, which (as the paper proves) is really important for event detection.", "essential_references_not_discussed": "None that I could find.", "other_strengths_and_weaknesses": "**Other weaknesses**:\n\n* Missing impact statement.", "other_comments_or_suggestions": "* Line 244 in page 5 (left column) repeats \"the loss $\\mathcal{L}_p =$\"\n* Lines 273-274 in page 5 (left column) are corrupted.\n* Line 360, page 7, left column, \"ustic events\" should be \"Acoustic events\"", "questions_for_authors": "Beyond the questions under **C3** (which I urge authors to address, since they are weighting heavily on my review), I have the following questions:\n\n* **Q1**: Will the training dataset for FLAM be made available? And if so under what terms?\n* **Q2**: How large is FLAM? How long does it take it to train? The text states in line 400 (page 8) that LAION-CLAP is larger scale, but it remain unclear by how much.\n* **Q3**: What is the effect of varying $L$, the number of frames? Does increasing/decreasing $L$ affect SED/Retrieval/Zero-shot classification performance?\n* **Q4**: In table 1, is the gap between `MGA-CLAP*` and `MGA-CLAP (reported)` explained by the different training datasets?\n* **Q5**: There seems to be a trade-off between frame representation and global representation (see differences in retrieval scores between `FLAM - Global` and `FLAM` in Table 2). Could this difference be due to model drift when training FLAM with the final loss? And if so, could a KL-penalty term with respect to the predictions of `FLAM - Global` be helpful (similar to the KL penalty used in DPO [1])?\n\n\n[1] Rafailov et al. (2023) Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.\n\n\n**Post-rebuttal update**\n\nAll these questions were addressed in the rebuttal. Specifically the global loss was confirmed to have an important effect on retrieval (and not much on sound event classification). FLAM with L=128 which performed worse on retrieval/zero-shot accuracy, but similar to FLAM L=32 for sound event classification.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r8o23eVzh7", "reviewer": "Reviewer_d9oS", "summary": "The paper introduces FLAM, a Frame-Wise Language-Audio Model for open-vocabulary sound event detection. FLAM enhances traditional audio-language models by incorporating frame-level contrastive learning and logit adjustment to handle label imbalance. It leverages a large-scale dataset synthesized from text-labeled audio events, enabling precise event localization.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer d9oS for reviewing our manuscript and recognizing the novelty of our frame-wise contrastive learning, data augmentation, and FLAM’s strong open-vocabulary SED performance. We respectfully clarify and address the concerns you raised. Notably, these concerns were not shared by the other reviewers. For example:\n\n- **Reviewer Rbvm** commented that “the experiments are fair, with no clear advantages given to the proposed method in comparison to the baseline,” and further stated that “the supplementary material... [is] complete and satisfactory.”\n\n- **Reviewer GGdQ** noted that “FLAM mostly outperforms baselines in some audio-text tasks” and that “the datasets and metrics used in the experiments are standard in this field.”\n\n- **Reviewer bqpP** emphasized that “FLAM builds on previous audio-language models by introducing a frame-level representation, which (as the paper proves) is really important for event detection,” and remarked that “the theoretical analysis and the results... both support [the main claims].”\n\nWe hope this provides context for why we believe the raised concerns do not substantially weaken the validity of our contributions. We now respond point-by-point:\n\n>  The paper lacks supplementary material, which could provide additional details on experiments, datasets, and implementation. The paper does not present any formal theoretical proof.\n\nWe’d like to clarify that our submission does include detailed supplementary material from page 11 to page 20, covering:\n- A full derivation of the FLAM frame-wise contrastive objective  \n- Sound event detection results on open-vocabulary tasks  \n- Detailed descriptions of our training procedure, dataset construction, and architectural hyperparameters  \n\nIn addition, we provide a demo website at [https://flam-model.github.io/](https://flam-model.github.io/), which showcases FLAM's localization results on unseen real-world samples.\n\nBoth **Reviewer Rbvm** and **Reviewer GGdQ** explicitly confirmed they reviewed the supplementary materials and found them complete and satisfactory. We invite Reviewer d9oS to revisit this part of the submission.\n\n---\n\n### Regarding baseline comparison:\n\n> The baseline comparisons seem weak, as the primary baseline (MGA-CLAP) is re-trained on the authors' dataset, potentially limiting its effectiveness.  \n> Why did you choose the specific baselines for comparison, and how do you justify the selection given the availability of other state-of-the-art models in the field?\n\nWe chose MGA-CLAP as the primary comparison for open-vocabulary SED because it is the most recent and relevant baseline in this domain. Specifically:\n- **MGA-CLAP** was accepted as an **oral paper at ACM Multimedia 2024** (top 3.97% of submissions), indicating strong peer recognition and technical merit.  \n- Its training objective and model design also aim at audio-text alignment, making it a natural benchmark for FLAM.  \n- Since our training set comprises licensed proprietary sound effects, we **retrained MGA-CLAP on the same dataset** to ensure a fair comparison. This avoids domain shift effects that would unfairly disadvantage the baseline.\n\nTo help readers interpret both in-domain and cross-domain generalization, we also include the original MGA-CLAP performance on public datasets. Additionally, on retrieval and classification tasks, we compare to other baselines like **CompA** and **LAION-CLAP**, where FLAM remains competitive.\n\nAs **Reviewer Rbvm** stated: “The experiments are fair, with no clear advantages given to the proposed method.” **Reviewer GGdQ** similarly noted that “FLAM mostly outperforms baselines in some audio-text tasks,” reinforcing that our baseline selection and evaluation strategy are reasonable.\n\n---\n\n> The paper could benefit from a clearer discussion of the limitations and potential future work.\n\nThank you for this helpful suggestion. We added a paragraph at the end of Section 7 discussing limitations and future work:\n\n“FLAM represents an initial step toward large-scale open-vocabulary sound event detection, but several aspects remain to be improved. The current training corpus, while diverse, is still limited in scale; future work could explore larger and more diverse corpora, potentially by synthesizing additional labeled mixtures or leveraging web-scale audio. The lightweight model could benefit from scaling or more expressive architectures. Additionally, FLAM uses a fixed 10-second audio input and a coarse frame resolution, which constrains its ability to handle longer or more temporally nuanced recordings. Future efforts could focus on supporting variable-length audio and adopting encoders with finer temporal granularity. Beyond architectural and data improvements, future work could explore the use of real-world frame-level annotations, better evaluation protocols, KL penalty to align frame-level outputs with global model, and generative augmentation strategies to further enhance open-vocabulary localization.”", "claims_and_evidence": "The claims in the submission are not fully supported by clear and convincing evidence. Key issues include: 1) Lack of supplementary material, which limits reproducibility and detailed analysis. 2) Weak baseline comparisons, as MGA-CLAP is the only baseline re-trained on the same dataset, making it difficult to assess FLAM's true improvements. 3) Limited discussion on the effectiveness of the proposed logit adjustment and data augmentation techniques. These shortcomings reduce the persuasiveness of the claims.", "methods_and_evaluation_criteria": "The proposed methods, including frame-level contrastive learning with logit adjustment and scalable data augmentation, are well-suited for open-vocabulary sound event detection. The evaluation criteria, using both synthetic and traditional SED datasets, effectively assess FLAM's ability to localize events and generalize. However, the baseline comparison could be stronger to better highlight FLAM's advancements.", "theoretical_claims": "The paper does not present any formal theoretical proof. It focuses on empirical evaluations and methodological contributions, such as the frame-level contrastive objective and logit adjustment techniques. Therefore, there are no proofs to verify.", "experimental_designs_or_analyses": "The paper lacks supplementary material, making it difficult to verify the experimental designs and analyses. The baseline comparisons seem weak, as the primary baseline (MGA-CLAP) is re-trained on the authors' dataset, potentially limiting its effectiveness. More robust baselines and detailed experimental setups would enhance the paper's validity.", "relation_to_broader_scientific_literature": "No connection to the broader scientific literature.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: FLAM introduces a novel frame-wise contrastive objective and logit adjustment for open-vocabulary SED, achieving state-of-the-art performance. The scalable data augmentation pipeline synthesizes a large-scale dataset, enhancing generalization.\n\nWeaknesses: The paper lacks supplementary material, limiting reproducibility. The baseline comparisons are weak, potentially overstating FLAM's advantages.", "other_comments_or_suggestions": "1. The paper lacks supplementary material, which could provide additional details on experiments, datasets, and implementation.\n2. The baseline comparisons seem weak; including more state-of-the-art models would strengthen the paper's claims.\n3. The paper could benefit from a clearer discussion of the limitations and potential future work.", "questions_for_authors": "Why did you choose the specific baselines for comparison, and how do you justify the selection given the availability of other state-of-the-art models in the field?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hmoHWQeKh2", "reviewer": "Reviewer_GGdQ", "summary": "The paper introduces an open vocabulary SED model, FLAM, trained with sigmoid loss. It outperforms the baseline MGA-CLAP on open vocabulary SED datasets and most closed-set SED datasets. The model is also tested in other tasks such as retreival and classification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer **GGdQ** for their thoughtful and constructive feedback, and for recognizing several key strengths and contributions of our work. We are encouraged by your positive assessment of our model and contributions—for instance, your remark that \"FLAM mostly outperforms baselines in some audio-text tasks\", your recognition that FLAM is the \"first SED system trained with sigmoid loss in a contrastive learning setting\", and your appreciation of our \"proprietary high-quality sound dataset\". We also value your comment that \"the proposed architecture and training scheme make sense\", and your confirmation that \"the datasets and metrics used in the experiments are standard in this field\". We found your comments particularly insightful and have revised the manuscript to incorporate your suggestions, which we address point-by-point below.\n\n---\n\n> It would be insightful to see a comparison in closed-set SED tasks between other SED methods not designed for open vocabulary SED.\n\nWe appreciate your suggestion to compare FLAM against strong closed-set SED models. In response, we conducted additional evaluations of FLAM on closed-set SED benchmarks alongside state-of-the-art closed-set systems. Due to space constraints, we summarize these results here: https://flam-model.github.io/response_reviewer_ggdq.html\n\nWhile FLAM performs competitively, especially considering its generalization capabilities, there remains a performance gap compared to highly specialized closed-set models. This result is expected, as closed-set SED methods are explicitly trained on a fixed, predefined label set and can thus heavily optimize for those specific categories. However, this design inherently limits their applicability to new or unseen sound events, since they cannot recognize or adapt to events outside their training vocabulary. In contrast, FLAM supports open-vocabulary detection by leveraging natural language descriptions, enabling broader generalization and better alignment with real-world, evolving sound taxonomies.\n\nFrom a downstream application perspective, this flexibility makes FLAM more broadly applicable, particularly in dynamic or user-driven scenarios where defining a fixed vocabulary in advance is infeasible. We believe these findings motivate further research into bridging the gap between open and closed-set SED and enhancing generalization across domains.\n\n---\n\n> Ablation studies to confirm the strength of sigmoid loss, its low computational cost and its insensitivity to batch size, should be performed under the SED scenario\n\nThank you for encouraging a deeper investigation into the role of sigmoid loss. We conducted the following ablation studies to evaluate its effectiveness under the SED setting, with results in https://flam-model.github.io/response_reviewer_ggdq.htm\n\n- We trained a version of FLAM without sigmoid loss (please see the exact object in link).\n  This version fails to converge and results in substantially degraded SED performance, highlighting the importance of the sigmoid loss in stabilizing training and enabling convergence in the frame-level contrastive setup.\n\n- We trained FLAM using two smaller batch sizes: 256 and 128 (original batch size = 512). As shown in our results, FLAM remains robust across batch sizes, with only marginal drops in performance at smaller sizes. We attribute this to fewer negative examples being sampled during training.\n\nThese results support our claim that sigmoid loss is both effective and computationally efficient for frame-level SED. While it is not entirely insensitive to batch size, its formulation allows us to scale to larger batches under the same memory constraints—unlike InfoNCE-based losses, which require global softmax computation and scale poorly with batch size. This makes sigmoid loss particularly well-suited for the resource-intensive frame-level SED setting.\n\n---\n\n**Typos:**  \nThank you and Reviewer **bqpP** for catching the remaining typos! We've corrected the following:\n\n- **Sec. 3.3**  \n  Original: _\"we allocate up t slots perup to five times the audio in a batchd pad any unused sloaudio or text ts with placeholders\"_  \n  **Fixed** (Lines 273–274, Page 5, left column):  \n  _“Since each audio clip may contain a varying number of events, we allocate text slots equal to five times the number of audio clips in a batch, padding any unused audio or text entries with placeholders.”_\n\n- **Sec. 5**  \n  Original: _\"How does FLAM fare on downstream tasks typically used to evaluate contrastive ALMs?\"_  \n  **Fixed**: _\"How does FLAM perform on downstream tasks typically used to evaluate contrastive ALMs?\"_\n\n- **Sec. 5.2**  \n  Original: _\"ustic\"_  \n  **Fixed** (Line 360, Page 7, left column): _\"Acoustic events\"_\n\n- **Table 2 (A2T experiment on Clotho):**  \n  We fixed the bolding of the correct model.\n\n- **Line 244 (Page 5, left column):**  \n  We removed the duplicated phrase: _“the loss $\\mathcal{L}_p = $”_", "claims_and_evidence": "Exisiting contrastively trained multimodal models are not accurate in providing frame-level results. FLAM extends CLAP to frame-level granuarity. To mitigate increased computational cost and biased class variations in available audio-text datasets, sigmoid loss, proposed in SigLIP, is introduced.", "methods_and_evaluation_criteria": "The proposed architecture and training scheme make sense. The datasts and metrics used in the experiments are standard in this field", "theoretical_claims": "Sigmoid loss can reduce computational and memory costs because it does not need to compute the global view necessary in InfoNCE loss. It is also evident from the equations of InfoNCE and sigmoid loss that the latter is not affected by batch size", "experimental_designs_or_analyses": "It would be insightful to see a comparison in closed-set SED tasks between other SED methods not designed for open vocabulary SED. Ablation studies to confirm the strength of sigmoid loss, its low computational cost and its insensitivity to batch size, should be performed under the SED scenario", "relation_to_broader_scientific_literature": "Different from othre audio fields such as sound separation, open vocabulary SED is relatively unexplored field", "essential_references_not_discussed": "The reference section covers a descent amount of past litterature", "other_strengths_and_weaknesses": "[Strength]\n- FLAM mostly outperforms baselines in some audio-text tasks\n- First SED system trained with sigmoid loss in a contrastive learning setting\n- Proprietary high-quality sound dataset \n\n[Weakness]\n- The strength of sigmoid loss in SED tasks is not verified in the experiments nor discussed in the paper \n- FLAM sometimes underperforms MGA-CLAP in audio-text retreival tasks\n- Some typos remain", "other_comments_or_suggestions": "- In table 2, for the A2T experiment on Clotho, should MGA-CLAP be made in bold?\n- Some typos?\n\nSec. 3.3 \"we allocate up t slots perup to five times the audio in a batchd pad any unused sloaudio or text ts with placeholders\"\n\nSec. 5 \"How does FLAM fare on downstream tasks typically used to evaluate contrastive ALMs?\"\n\nSec. 5.2 \"ustic\"", "questions_for_authors": "Please see \"Experimental Designs Or Analyses\" and let me know your thoughts", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oubKxOjTn2", "reviewer": "Reviewer_Rbvm", "summary": "The paper develops a contrastive audio-language model that is capable of doing frame-level sound event detection. The paper's focus is on using contrastive techniques and off-the-shelf encoders, as well as the correction of bias caused by the imbalance in event labels. The contrastive training, logit adjustment for bias correction and the SED task are all trained for simultaneously, and show good performance compared to recent models in terms of SED, while showing extremely accurate frame alignment of events. The main contribution of the paper is the focus on frame-level SED and the model is trained on a combination of synthetically constructed data (by using sound effect and general audio).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Rbvm for their thoughtful and constructive review. We are especially grateful for your recognition of our theoretical and mathematical contributions. As you noted, “the theoretical claims around bias correction… [are] broadly correct,” and your review confirmed that you “reviewed the appendix and broadly found the theoretical claims to be correct.” \n\nWe also appreciate your positive assessment of our methodological choices—“the method is simple and elegant”—as well as your acknowledgment that our bias correction and synthetic dataset construction are “well-justified.” Your remark that “the focus on bias correction and robustness is admirable and should be standard in the contrastive literature” particularly resonates with our goals. We are hopeful that our logit-adjusted contrastive loss formulation will inspire future work in other domains such as vision and language.\n\nBelow, we address your primary concern and introduce a new alignment-specific evaluation metric, as requested.\n\n---\n> The improvement in alignment of the audio events, the whole point of doing frame level SED, is only shown through examples, and not robustly checked and justified. This is the paper's major weakness, because, without summary statistics and robust justification, it is not clear if the models improvements in accuracy come from better alignment or simply better classification accuracy. While it is probably that the improvements come from better alignment, I'd like to see this measured better than via examples and diagrams.  \n> Could the authors come up with a metric that focuses purely on alignment and compare the various models on it?\n\n\nWe fully agree with the reviewer that robust evaluation of alignment quality is essential to justifying the benefits of frame-level supervision. In our original submission, we report frame-level AUROC and PSDS, both of which reflect the model’s ability to distinguish when an event occurs, given a caption. AUROC is computed over all frame-caption pairs, including negative ones, and thus directly reflects alignment fidelity. PSDS (Polyphonic Sound Detection Score) is a DCASE challenge metric that integrates precision and recall across thresholds with a focus on time-sensitive detection.\n\nIn Table 1 of the main paper, we showed that FLAM significantly improves both AUROC and PSDS across open-vocabulary and closed-set datasets, which we argued reflects improved temporal alignment. However, we agree that these classification-based metrics only measure alignment indirectly.\n\nTo address this, we propose a new diagnostic metric specifically designed to directly evaluate alignment: **Spearman correlation between the model’s frame-text similarity scores and the ground truth label mask**. This measures how well the model’s similarity predictions track the actual occurrence of audio events over time, independent of decision thresholds or absolute similarity magnitude.\n\n**Definition**: For each captioned event, we compute the Spearman rank correlation between:\n- the model’s similarity scores across frames (either pre-sigmoid logits or dot products), and\n- the corresponding binary ground-truth label mask indicating event presence over time.\n\nA higher Spearman correlation indicates that the model more faithfully aligns audio with text on a frame level.\n\n---\n\n#### **Results: Alignment Correlation (Spearman ρ)**\n\n| Model        | ASFX ρ  | Synth ρ |\n|--------------|---------|---------|\n| **FLAM**         | **0.409** | **0.600** |\n| MGA-CLAP     | 0.256   | 0.352   |\n| FLAM-Global  | 0.197   | 0.262   |\n\nNote: for all Spearman ρ, the p-value is smaller than 0.01.\n\n---\n\nThese results confirm that **FLAM achieves substantially higher alignment correlation** than both MGA-CLAP and FLAM-Global on both ASFX-SED and synthetic datasets. This supports our hypothesis that FLAM’s improvements stem from better temporal alignment, not merely improved classification.\n\nWe will consider including this new metric, its formulation, and the results above in the revised manuscript. Thank you again for your insightful suggestion—it has helped strengthen the empirical justification for our frame-level approach.", "claims_and_evidence": "1. It is claimed that the negative examples in contrastive learning causes models to be biased - derivations and prior literature confirm this claim.\n2. The authors claim that frame-level SED is superior to instance-level SED - they somewhat demonstrate this by retraining MGA-CLIP on their data, as well as via FLAM-GLOBAL. This does not become generally true unless this method scales up and can outperform state-of-the-art models, but for the scale discussed in the paper, is sufficient. \n3. It is claimed that the method leads to highly accurate alignment of SED outputs - this is only demonstrated by example, not via summary statistics or dataset-level measures", "methods_and_evaluation_criteria": "The authors measure performance on three appropriate tasks - synthetic open vocab/closed SED, Text-to-audio and Audio-to-text retrieval, as well as zero-shot SED.\n\nThe methods used are standard and appropriate, and combining frame-level approaches with instance-level ones is a plausible thing to do in the audio domain.", "theoretical_claims": "I checked the theoretical claims around bias correction at a high level, and am broadly satisfied with them. I also reviewed the appendix and broadly found the theoretical claims to be correct.", "experimental_designs_or_analyses": "Yes, experiments are appropriately designed, with comparisons to baselines trained on the same data as well as different data. The authors conduct experiments both on held-out test sets of the training data and zero-shot classification studies, and as an added task, also study retrieval. The experiments are fair, with no clear advantages given to the proposed method in comparison to the baseline.", "relation_to_broader_scientific_literature": "The key contributions of the paper build off of the broader scientific literature in the world of sound event detection, with the authors using well-known models such as HTSAT [Chen et. al., 2022] and RoBERTa [Liu, 2019] to develop their methods. The authors also engage with the literature by comparing to standard baselines like MGA-CLAP [Li, 2024] on well-known datasets such as AudioSet", "essential_references_not_discussed": "No major concerns", "other_strengths_and_weaknesses": "Strengths:\n1. The method is simple and elegant, and seems to perform competitively to standard methods from the literature. \n2. The focus on bias correction and robustness is admirable and should be standard in the contrastive literature.\n3. The synthetic dataset construction is well-justified and the methods to create this data are appropriately explained. \n\nWeaknesses:\n1. The improvment in alignment of the audio events, the whole point of doing frame level SED, is only shown through examples, and not robustly checked and justified. This is the paper's major weakness, because, without summary statistics and robust justification, it is not clear if the models improvements in accuracy come from better alignment or simply better classification accuracy. While it is probably that the improvements come from better alignment, I'd like to see this measured better than via examples and diagrams.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Could the authors come up with a metric that focuses purely on alignment and compare the various models on it? If the frame level approach were strongly justified, I would increase my rating.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}]

Author rebuttal:

Meta-review: This paper introduces an audio language model ("FLAM") utilizing a frame-level sound-event localisation loss along with a contrastive learning objective to produce frame-level representations aligned with natural language. Authors develop open-vocabulary data augmentation (with localization) techniques with binary masks to indicate the presence of the captions in the sound at a given frame. FLAM also compensates for the label imbalance on the dataset (not all classes have the same duration!) by estimating a per-caption bias and logit-scale.

Experiments on text-to-audio and audio-to-text retrieval show FLAM being competitive with baseline audio-language models (eg. ~86% R@5 for both text-to-audio and audio-to-text retrieval on Clotho compared to LAION-CLAP), while showcasing very strong sound event detection on both open-vocabulary and closed-vocabulary sound event detection tasks. During the rebuttal, authors clarified several questions regarding the training process and data and committed to releasing the ASFX-SED validation dataset, which should be a useful resource to the community. Ablations confirmed that the global loss has an important effect on retrieval (and not much on sound event classification), leaving reviewers that engaged in the rebuttal process with a good sense of how the model works and an accept recommendation.
Decision: Accept (poster)