DeepASA: An Object-Oriented Multi-Purpose Network for Auditory Scene Analysis
Authors: Dongheon Lee
Younghoo Kwon
Jung-Woo Choi
Venue: NeurIPS 2025 poster ()
Keywords: audio signal processing
auditory scene analysis
universal sound separation
object-oriented processing
chain-of-inference
Primary area: applications
TL;DR: We propose DeepASA, an object-oriented one-for-all network that mimics human auditory scene analysis to perform source separation, sound event detection, and direction-of-arrival estimation, achieving state-of-the-art performance on downstream tasks.

Abstract: We propose DeepASA, a multi-purpose model for auditory scene analysis that performs multi-input multi-output (MIMO) source separation, dereverberation, sound event detection (SED), audio classification, and direction-of-arrival estimation (DoAE) within a unified framework. DeepASA is designed for complex auditory scenes where multiple, often similar, sound sources overlap in time and move dynamically in space. To achieve robust and consistent inference across tasks, we introduce an object-oriented processing (OOP) strategy. This approach encapsulates diverse auditory features into object-centric representations and refines them through a chain-of-inference (CoI) mechanism. The pipeline comprises a dynamic temporal kernel-based feature extractor, a transformer-based aggregator, and an object separator that yields per-object features. These features feed into multiple task-specific decoders. Our object-centric representations naturally resolve the parameter association ambiguity inherent in traditional track-wise processing. However, early-stage object separation can lead to failure in downstream ASA tasks. To address this, we implement temporal coherence matching (TCM) within the chain-of-inference, enabling multi-task fusion and iterative refinement of object features using estimated auditory parameters. We evaluate DeepASA on representative spatial audio benchmark datasets, including ASA2, MC-FUSS, and STARSS23. Experimental results show that our model achieves state-of-the-art performance across all evaluated tasks, demonstrating its effectiveness in both source separation and auditory parameter estimation under diverse spatial auditory scenes. The demo video, samples and code are available at https://huggingface.co/spaces/donghoney22/DeepASA.

DeepASA: An Object-Oriented Multi-Purpose Network for Auditory Scene Analysis
Dongheon Lee, Younghoo Kwon, Jung-Woo Choi
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon, Republic of Korea
{donghen0115, kOhoo, jwoo){kkiist.c .cc.stttcc... .c.
Abstract
We propose DeepASA, a multi-purpose model for auditory scene analysis that performs multi-input multi-output (MIMO) source separation, dereverberation, sound event detection (SED), audio classification, and direction-of-arrive esti- mation (DoAE) within unified framework. DeepASA designed for complex auditory scenes where multiple, often similar, sound sources overlap in time and move dynamically in space. To achieve robust and consistent inference across tasks, we introduce an object-orientee processing (OOP) strategy. This approach encapsulates diverse auditory features into object-centric representations and re- fines them through chain-of-inferene (Col) mechanism. The pipeline comprises dynamic temporal kernel-based feature extractor, a transformer- based aggrega- tor, and an object separator that yields per-object features. These features feed into multiple task-specific decoders. Our object-centric representations naturally resolve the parameter association ambiguity inherent in traditional track-wise processing. However, early-stage object separation can lead to failure down- stream ASA tasks. To address this, implement temporal coherence matching (TCM) within the chain-of--nfeee of-inference, enabling multi-task fusion and iterative re- finement of object features using estimated auditory parameters. We evaluate DeepASA on representative spatial audio benchmark datasets, including ASA2, MC-FUSS, and STARSS23. Experimental results show that model achieves state-of-the-ar performance across all evaluated tasks, demonstrating its effec- tiveness in both source separation and auditory parameter estimation under di- verse spatial auditory scenes. The demo video, samples and code are available at https://hugginngface. http:////.....poo//.////////////
Introduction
Auditory scene analysis (ASA) seeks extract information about individual sound sources from complex auditory environments [1] This includes identifying the class of each source, its onset and offset times, and its direction of arrival (DoA) [2]. In humans, ASA is facilitated by the brain's 's ability to organize sound into perceptual streams by integrating multiple auditory cues such as pitch, timing, and spatial location. in a complementary fashion 4 [
Inspired by this ability, computational auditory scene analysis (CASA) research 2 6 has aimed to replicate similar functionality using auditory cues like pitch, onset/offset, interaural level differences (ILDs), and interaural time differences (ITDs). With the rise of deep learning, a wide array of models has emerged to tackle specific ASA tasks, including audio tagging (AT) 8 9. sound event
detection (SED) [10 12), DoAE [1] [4 15], and blind source separation (BSS) [6] [7 [8
However, unlike the human auditory system, these task-specific models typically lack the capacity for relational reasoning across tasks and cues. As a result, they often fail when critical auditory cues
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

are missing or degraded [9]. Recent work highlights the benefit of combining multiple auditory
cues. For instance, target sound extraction (TSE), which uses class [20 21, activation [22] 23 and
spatial cues [24 25 outperforms universal sound separation (USS) in challenging scenarios [26 27
Similarly, joint SED and DoAE sound event localization and detection (SELD) has been shown to
enhance performance by leveraging interdependent auditory information 28 29
Building on this evidence, we propose general-purpon ASA model that emulates this cue integration process. Our model separates object-level auditory streams at an early stage and performs multiple downstream ASA tasks by exploiting the complementary relationships among estimated cues.
Contribution. We introduce DeepASA, a unified architecture for general auditory scene analysis, with two core contributions:
Object-Oriented Processing (OOP): We design an encoder-decoder framework that isolates features of individual sound objects. The encoder incorporates dynamic short-time Fourier transform (STFT) module and transformer-bases feature aggregator. These are followed by an object feature separator that disentangles per-object auditory representations. These object-centric features are then passed to multiple sub-decoders, each responsible for estimating a specific auditory parameter (e.g., class, DoA, activation) without requiring manual association or permutation invariant training between estimated auditory parameters.
2. Chain-of-Inferencee (Col) architecture: To emulate the human auditory system's ability to recover missing cues by reasoning from others, we propose a chain-of-inference architecture. This component progressively refines object representations by fusing the outputs of ASA sub-decoders using a temporal coherence-basee attention mechanism. enables the model to reinforce incomplete or ambiguous estimates via complementary information.
We validate DeepASA on multiple spatial audio benchmarks. Ablation studies the newly proposed ASA2 datase datase[Ishon show that task-specific sub-decoders mutually enhance one another when conditioned on shared object features, yielding strong results with an SI-SDRi of 11.2 dB and SELD score of 0.206. We also demonstrate that DeepASA pretrained by the ASA2 dataset can be generalized to various spatial audio benchmarks through fine-tuning with Col. The fine-tuned DeepASA achieves -tate-of-the-as (SOTA) results, including an SI-SDRi of 18.5 dB on MC-FUSS for multichannel universal sound separation
30 31, a SELD score of 0.253 on STARSS23
(a) Track-wise processing
(b) Object-oriented processing
Audio Decoder
Sound Detection (SED)
Separated audio UKM++N
relations
Deject Olgedl
Feature
modeling
Audio
Object separator
SED
Moltichonnel
extractor
CNNss
Confurner
Transfomte:
Decoder
SED (UKT"KC)
Mel-spoctrogram
GRU-LTM
Mamba
DoA Estimation
wavefform (MxN)
Object features (JDD
DoA Decoder
DoAE (3x7"x3)
(DoAE)
Figure 1: Comparison between (a) traditional track-wise processing and (b) proposed object-oriented
processing
2
Method
2.1
Overall framework: DeepASA
The multichannel audio mixture x E M where M is the number of microphones and V is the number of time samples of the waveform, can be modeled the summation of reverberant foreground signals and background noise v. The j-th foreground source can be further decomposed as the superposition of direct sound s, and reverberation h,.
inj
(sy[m)] hy [n]) + v/n].
(1)

---

Object
(LXMMM))
feature
Direct reverb audio
Time-varying
Object
Audio
(M*.V)
X
Iearnable
window
X
Lp-Conv P2 >
Feature aggregation blocks
Object Spiitter
Object
Decoder
SED
Background noise
UxTxC)
(M*N)
(2MXT*F)
> U hD
Object
DoA Decoder
Decoder
Sound
Noisc
(xTX*x3)
((--))xDxT*P)
DoA trajactory
Audio Mixture
Audio Encoder
Object Separator
Sub-decoders
Auditory Information
Figure 2: Basic architecture of DeepASA
We aim to estimate the auditory information of up foreground sources, including classes, activations of onsets and offsets, DoA trajectories, and multichannel waveforms of the direct and reverb audio signals, as well as one multichannel noise signal, from the multichannel audio mixture.
As illustrated in Figure the overall architecture of DeepASA comprises three components: audio encoder, object separator, and sub-decoders. The audio encoder extracts basic features from the M-channel input mixture. The object separator separates audio features extracted from the audio encoder into sound object. The sub-decoders then estimate the auditory information from each object feature. D,1" and C denoted channel dimension, time frames and number of classes.
object features, each of which encapsulates the auditory information of single
The proposed framework separates each sound object at a feature level, which is referred to as object- oriented processing (OOP) in this work. One advantage of OOP is that the object-wise permutation of the separated features is consistently inherited across all sub-decoders. For example, the j-th object estimated by the audio, SED, DoA decoders all include the waveforms, SED, DoA information from the same sound source. The characteristic the major difference to the conventional track-wise processing [28] 33 34 35
36 37), for which multiple source information can be aligned on the same track when they do not overlap in time. The OOP eliminates the requirement for pairing different auditory information and also enables the selective attention of estimated information across different objects and auditory information.
2.2
Audio encoder
Dynamic STFT with time-varying learnable window The audio encoder of DeepASA incorporates our proposed architecture, Dynamic STFT, for adaptive temporal focusing on the input waveform. Conventional STFT uses a fixed analysis window at all times, yet the locations and durations of salient information within time frames can vary across the waveform. Accordingly, we employ Iearnable Gaussian window whose mean (pe) and standard deviation (a,) are predicted at every frame, forming μ E RT and o IR. vectors for the total number of time frames T.. The frame-wise mean (He) aligns the center of the analysis window on the most informative region, while (ar) sculpts its taper: a large a, flattens the analysis window toward a rectangle, sharpening spectral focus with a thin main lobe, whereas small d contracts it toward an impulse, enhancing temporal focus.
The process extracting yl and or is depicted in Figure 3 For each mi- crophone channel, a parameter-share ID convolution with the same kernel size and stride as the window size K and hop size L used in STFT is ap- plied, producing outputs x' E DxT for each microphone. Two parallel linear layers then project x' to the mean and standard deviation, produc- ing μ1, a e R for every microphone channel.
Audio
Time-varying leanaalle window
Mixuure
Parameter Sharing
Parameter Sharing
Conv (1+0))
Linvar (D +
Linear (D
Conv1D (1
Linear (0 1)
Linear (D
ConvID <
Linear (2)
Linear (MxT)
(MKx))
(MxDx7)
(Mx7)
stridc:
Figure Time-varying learnable window
Processing STFT with the time-varying learnable window, the multichannel waveform is converted into complex spectrogram X E xxXX where F is the number of frequency bins.
These

---

complex spectrograms are then fed directly into an Up-Conv block composed of 2D convolution layer followed by layer nommalization, preserving the temporal and spectral dimensions. For training a learnable window, the parameters are frozen at the beginning of the training, and when the model has converged, they are unfrozen and trained together.
increase the channel dimension from 2M to D while
2.3 Object separator
Feature aggregation As the feature aggregation block of the proposed network, we modified the DeFT-Mamba (38) model designed for the USS task (Figure DeFT-Mamba is a comprehensive feature aggregation model utilizing Mamba and transformer layers to capture temporal, spectral, and interchannel relations in multidimensional data. To lighten the model, we adopt Mamba-FFN only for T-Hybrid Mamba and use a conventional feedforward network (FFN) for F-Hybrid Mamba. Then, we remove the unfolding process from the gated convolutional block (GCB) while keeping the gating module and the convolution kernel.
Object splitter The features arranged by the modified DeFT-Mamba are transformed into J + object features through the object splitting layer given by a 2D convolution kernel. The separated J object features corresponding to foreground sources pass through the direct and reverb decoders of the audio decoder, the SED decoder, and the DoA decoder. The last object among the separated objects is treated as a noise object, which is separately utilized by the noise decoder in the audio decoder to estimate background noise.
2.4 Sub-decoders
The sub-decoders decode each object feature into the direct, reverberant, and noise audio signals, class, activation, and DoA outputs. The detailed architectures of sub-decoders are depicted in Figure (b).
MIMO Audio decoder The audio decoder aims to estimate a maximum of foreground source signals and 1 background noise signal corresponding to individual sound objects. In the conventional multi-input single-output (MISO) model, spatial information is lost in the decoded single-chanen signal, and the audio decoder cannot assist DoAE in the DoA decoder. Therefore, the audio decoder in DeepASA is designed as a multi-input multi-output (MIMO) architecture, facilitating spatial information in the separated object features. To further assist the development of spatial information and dereverberation, the audio decoder is trained to separate direct sound s, and reverberant sound h, of each object across all microphone channels. In addition, for reducing the influence of background noise, the last object from the object separator is separately processed by the noise decoder trained to estimate the background noise When active foreground sources are fewer than J, inactive sources are trained to estimate zero target signals.
SED decoder The SED decoder decodes the object features of each sound source into the predicted class probability (1 x C), the binary object activation curve for event onset and offset (T' x 1), and the predicted SED map (T' x C). Here, T" and C denote the number of time frames and the number of classes, respectively. The predicted SED map is only used the training stage to guide
(a)
GCR
MHSA
Feediierward network (PFN)
(b)
Sacel presede
Audio Deceder
Carvdd
direct
Kilicared & rerah (ssMs
Caur2d
DeFI-Mamha black
SED Decoler
F-Hybril Mamba
T-Hybrid
Munths
ATST
FC
73
DxTxF
Multi-Head
DxTHF
L-NN
EGRU
FC
TriD (1 (
Gated Convolation Block
Self-Atrentonn
Mambu-Fuedlee Fwedforward Network (Mamba-FFN)
(GCB)
(MHSA)
ASP
1f (7+1)=
PENN
F-GRU
Mors FC
Prodicedd choc
LD Lo Foo
Curvil Cancht
DeA Deceder
Input
Flack Aitcetoon
Ri- Mandoal
T-CNN
T-GRU
Estirated (7* p
oxana
Clnvid Dilic
Matichanzel Backgnward take dids
Figure 4: Detailed architecture of (a) feature aggregation block, and (b) sub-decoders

---

2
Temperal Coherence Matching (TCM)
Chnin- Chatr-of-nlereece (Col)
Net
SED query branch
Sub Jabedeeseers
Gross
PC
Fratere Fusion (FF)
Sob- deceders 2
Attention
LCC +
Audio
Desoler
Deselkrr
(MKN]
Andia Encader
Festure fasrepation blacks
Object (Spliee
Drcoler
SED
DoA quers brnnch
Rr >
Ubject Spiiter
DoA
x (D*Ix)5
Cross
PC
Doseder
Atfention
Figure 5: Detailed architecture of Mhain-of--infrree
the joint detection of a sound event and class. In the inference stage, we use the SED map produced by multiplying the predicted class and activation curve, resulting in single SED map for each sound source. This is to exclude the possibility of more than one class being mapped to the SED map for each sound object. The SED decoder combines a pre-trained audio teacher-student transformer (ATST) [12] with two separate convolutional recurrent neural network (CRNN) [I]] [[s]eb branches summarizing time and frequency information. CRNN consists of multiple repetitions of Conv2d and Maxpool2d (denoted as CNN), followed by GRU and FC layers. Specifically, among the three red branches of the SED decoder, the middle path with seven Mo layers outputting (J x x D) is the T-CRNN path, and the bottom path with two MonxoMaxpool layers outputting (J x D) corresponds to the F-CRNN path. We employ the pre-trate pre-trained ATST to utilize various features learned from AudioSet, large-scale dataset containing a wide range of audio classes [39] The features from ATST are combined with auxiliary features developed from another Conv2d layer. The concatenated features are analyzed in time by Time (T)-GRU and summarized to predict the sound event map. For predicting a single-channel activation curve, adaptive statistical pooling (ASP) 40) and one fully connected layer applied along the class dimension. The class probability is predicted by superposing the logit derived from Frequency (F)-CRNN, which prioritizes the spectral information using Conv2d layers and F-GRU, with the logit obtained from ASP. For inactive sources, the class decoder is trained to predict an equal probability of 1/C' across all C' classes, while the sound event and activation decoders trained to produce zero values for all time frames. The detailed specifications of the SED decoder are provided in Appendix A
DoA decoder The DoA decoder has the CRNN structure and outputs stream of DoA vectors in Cartesian coordinates (x, y. 2). The DoA output estimates a DoA vector with a magnitude of I when source is present, and assigns all y. z values to O when no source is detected, allowing for the estimation of the activity and incident angle. DoA is estimated for each time frame, enabling the trajectory prediction for moving sources.
2.5
Chain-of-inference
The initial estimation through sub-decoders can include misalignment between auditory parameters. For example, the DoA vector stream can exhibit different onsets and offsets with the sound event map. These asynchronous estimations are refined from the object separation level through the fusion of initial estimations. This refinement step, chain-of-inferencee consists of two processing steps: temporal coherence matching (TCM) and feature fusion (FF) (Figure 5)
Temporal coherence matching In TCM, temporal coherence between SED and DoAE is evaluated through the multi-clue attention mechanism. One branch of multi-clue attention takes SED as queries and DoA as keys and values, and the attention output is added to SED to refine the SED information. Conversely, the other multi-clue attention branch uses DoA queries and SED keys and values. The SED decoder outputs three components, but ultimately, the SED map of T" x C is constructed for each object from the predicted class probability (1 x C) and object activation (T' 1). DoA decoder also generates T x' output for each object. The alignment between the SEDs and DoAs is accomplished TCM through the cross-attention by assigning the time dimension as a sequence dimension. The outputs of the two multi-clue attentions are passed through two individual linear layers to produce the D-channel clues and then superposed for fusion. The fused clue interpolated by transposed convolution layer to match its time dimension I" with that of object features (I).
5

---

Feature fusion In the feature fusion block, the fused clue is utilized to refine the object feature separation. TCM output processed by a feature-wise linear modulation (FiLM) layer 41, which outputs values of 3 and y for each channel and time frame. The output of the FiI.M layer is then combined with the output from the feature aggregation module (z) inject information required for object separation. The object features refined by the object splitter 2 are then plugged again into the sub-decoders 2 to improve ASA parameter estimation.
For efficient training of the Col architecture, we utilize multi-stage training. That is, Net I of Figure trained first, then Net 2 is trained with up the third DeFT-Mamnn block of Net 1 frozen.
3 Experimental Settings
3.1 Datasets
The proposed architecture is trained and evaluated on three spatial audio datasets: ASA2, MC- FUSS, and STARSS23. Auditory Scene Analysis dataset V2 (ASA2) provides ground truth signals and labels for all acoustic parameters addressed in this study, including comprehensive auditory information for moving source separation and scene analysis in noisy, reverberant environment. ASA2 based on the ASA dataset [38]. but we modified it extensively to enable the comprehensive tasks discussed in this study. ASA2 supports (1) the separation of direct and reverberation components (for dereverberation). (2) MIMO separation, as compared to MISO separation of the ASA dataset, and (3) an increased maximum number of foreground sound sources, from four to five. The balance in the number of audio clips has been changed be proportional to the number of sources constituting the scene. Multichannel-Free -reee Universal Sound Separation (MC-FUSS) 330 is a benchmark dataset for multichannel USS, including mixtures of 1-4 sound sources, with one being the background source. For fair comparison and with fine-tuning of the pre-trained model on the ASA dataset. Also, we compared the effect of pre-trained ATST in the MC-FUSS dataset. STARSS23 R2) is dataset for the SELD task, addressing various sound events with 3.8 hours of audio- video data collected from real environments. This dataset generally requires data augmentation, additional data augmentation of 80 hours was conducted using a spatial scaper [42). In this experiment, other SELD models were also trained using the large auxiliary datasets. The DCASE 2023 challenge allowed the use of external datasets and pre-trained models, so we applied this rule. All datasets were set to a 16 kHz sampling rate and a 4-second duration.
the MC-FUSS dataset, we trained DeepASA both from scratch
3.2 Training and model configuration
For the STFT, the Gaussian window of 40 ms length and hop size of 20 ms (50% overlap) was used. The initial weight for the mean and standard deviation of the window was set to 0 and 6.67 ms. ensuring 6 x covers the full window range. The number of feature aggregation modules was 6, and the channel dimension for the DeepASA was 64. The initial learning rate 10- was used for the Adam optimizer during pre-training, and 4 x 10 and 5 x 10 were used for fine-tuning to the MC-FUSS and STARSS23 datasets, respectively. The learning rate was halved the validation loss did not decrease for five consecutive epochs. The model was trained for 100 epochs, and the model from the epoch with the lowest validation loss was selected as the final model. The batch size was set to 2. The training was performed on eight GeForce RTX 3090 GPUs
The direct and reverb decoders in the audio decoder were trained with the source-aggregated signal- to-distortion ratio (SA-SDR) loss 4 with permutation invariant training (PIT) 44 We configured a reference channel among multichannel signals and applied different weights (1:0.1) the losses of the reference and the other channels. The noise decoder was trained by the scale-invariant signal- to-distortion ratio (SI-SDR) loss 45]. We employed one noise decoder, so the noise decoder was not trained with PIT. The joint loss was calculated by summing the losses with weights 1:1:0.01 for direct, reverb, and noise decoders. The losses for the SED decoder were cross-entron cross-entropy, binary cross-entropy and mean-square-erro (MSE), for the class decoder, activation decoder, and sound event decoder, respectively. The MSE loss was used for the DoA decoder, which was summed with all aforementioned losses with equal weights. In evaluation, scale-invariant -odis-ooo ratio improvement (SI-SDRi) (dB) and agnal-to-distortioo ratio improvement (SDRi) (dB) were used for USS performance evaluation, while SELD metrics (error rate (ER,%) Fl Fl-score (F1,%),

---

localization error (LE, degrees), localization recall (LR, %)) [28] were employed for SED and DoAE, We measured the model complexity by the parameter size and computational complexity.
4
Results
4.1
Ablation study of DeepASA blocks
We conducted ablation studies to assess the contribution of each module in DeepASA. Each row block of Table corresponds to the ablation of specific component. For sequential consistency, the final configuration from the previous block was used the baseline for the subsequent analysis.
Framework The first ablation study about incorporating the MIMO separation task, SED, and
DoA decoders. The baseline was the MISO USS model [38], and USS performance was measured based on the reconstruction of the reference channel. When MIMO USS is introduced, the separation performance of the reference channel decreases. This can be attributed to the features extracted for all channels rather than a single channel. Introducing the SELD task using the SELDNet (28) sub- decoder enhances the USS performance well, showing that auditory scene analysis can effectively guide the source separation task. The SELD performance is further improved when the CRNN architectures employed as SED and DoA decoders. Notably, USS performance metrics recovered to those of the baseline, indicating that a sophisticated SELD decoder can effectively improve SELD performance with minimal impact on the USS task.
Object separator Next, we examine the impact of downsizing on the baseline model (DeFT-Mamba- MIMO + SED, DoA decoder). The performance reduction by omitting unfold was negligible, showing only 1 dB difference SI-SDRi. Furthermore, when F-Mamba is removed, a performance degradation of 0.3 dB in SI-SDR and 0.002 in SELD score is observed. However, this comes with significant lightweighting in parameter size (0.9 M) and computational complexity (10.9 GMac/s), so adopted the conventional FFN in the F-HybridMamba block.
SED decoder The efficacy of more sophisticated sub-decoders tested in this section. Recent work has shown that combining pre-trained ATST and CRNN achieves high performance in sound event detection [46]. The SELD score is also significantly improved when the baseline, combined with the ATST + T-CRNN decoder, is fine-tuned to the ASA2 dataset. This stresses the importance of using the class decoder specialized to the classification task. Similarly, we combined the F-CRNN architecture capture and emphasize global frequency relations, improving the SELD score further up to 0.266. Since the improvement was less only with the F-CRNN decoder, we can conclude that capturing both local and global frequunyy relationships is critical for effective SED.
Audio decoder With the best SED decoder suggested above, we conducted the model analysis for the audio decoder. The first improvement was adding a noise decoder that estimates multichannel background noise. Both separation and SED performance are significantly improved when the noise decoder performing noise estimation is added. As analyzed in detail in Appendix B the noise decoder enhances the separation performance of domestic sounds that share similar time-frequency
Table 1: Ablation study on the proposed model using the ASA2 dataset. The models with (+) or (-) indicate the addition or removal of the corresponding blocks from the baseline. The (+) in the parameter size indicates the parameter size of the pre- trainrainedd ATST.
USS
SED
DoAE
Complexities
Model variation
LE 4 LR SELD
SI-SDRi f SDRi 1
ER L FL
Param.
MACKs
DcFT-Mamba- MISO
38
10.4
11.3
3.6 M
83,8 G
DeFT-Mamba-Ne MIMO
10.0
10.9
M
83.8 G
Framework
(+) SELDNet 2X
10.2
11.1
42.0
58.2
28.6
63.2
0.341
M.4
86.1 G
(+) SED, DoÅ decuder
10.4
11.4
40.0
60.3
22.9
65.7
0.317
7.2 M
88.4 G
Object
(-) Unfold
10.3
11.4
39.8
60.5
22.2
66.1
0.314
7.2M
88.0
separator
(-) Unfold, F-Mamba
10.0
11.1
39.9
60.4
22.0
65.9
0.315
6.3
77.1
(+) F-CRNN
10.3
11.2
38.0
61.1
21.9
68.8
0.301
8.1
90.7
SED
decoder
ATST T-CRNN 46
10.2
11.2
35.7
66.1
21.2
71.0
0.276
6.3 (496.8) M
96.5
ATST+ &F-CRNNS
10.3
11.2
34.1
66.6
21.3
72.8
0.266
8.1 (496.8) M
98.8 G
Noise decoder
11.0
11.7
30.3
69.8
21.2
76.0
0.241
8.1 (496.8) M
98.9
Audio
(+) Direethreverh,
decoder
noise decoder
10.8
11.5
30.0
69.6
19.1
76.2
0.237
8.1 (496.8) M
99.0
Dynamic STFT
(+) -iverinvariant window
10.7
11.4
30.2
69.8
19.3
76.0
0.238
8.966.8.
99.0
(+) Time-variant window
11.0
11.7
28.8
70.2
18.5
76.9
0.230
8.2 (+96.8)
99.1 G
Col
(+) Chain-of-inference
11.2
12.0
25.0
74.1
17.0
78.1
0.206
12.1 (496.8)
104.0 G
7

---

characteristics with background noise. Next, separate decoders for the direct and reverberant sounds were employed estimate s, and hj. respectively, instead of s; + h estimated by the baseline audio decoder. Here, the USS performance was evaluated by measuring SI-SDRí and SDRi on sj + h; for fair comparison to previous cases. While this change slightly reduces separation performance, DoAE performance (LE and LR) is markedly improved. This suggests that learning to separate direct audio positively impacts DoA accuracy by suppressing reverberant signals. A detailed analysis of the direct/reverb decoders is provided in Appendix C
Dynamic STFT Lastly, the effectiveness of dynamic STFT was validated. The first comparison set is the model using learnable but time- invariant window, which was designed by applying mean pooling over the temporal dimension before passing it through the linear layers of a time-varying learnable window. This model shows negligible performance improvement compared to the baseline STFT-based model. contrast, the time-varying learnable window increases all performance metrics, demonstrating the importance of dynamic temporal adjustment of the window position and length in feature extraction and separation. The detailed comparisons with various dynamic convolution kernels are presented in Appendix D
4.2 Ablation study of -ffffe-----nfeeeee (Col)
The results of the Col ablation study are presented in Table 2 The baseline model corresponds to the best-performing configuration without Col, identified in Table To isolate the effect of individual auditory cues, we ablated the Col module by including only a single query branch, either the SED or DoA branch in the Temporal Coherence Matching (TCM) module. Additionally, the audio decoder 2 was removed from Net 2 to eliminate the influence of two-stage training on the audio separation task. In these ablated configurations, USS performance evaluated using the output from audio decoder 1 only.
Table 2: Ablation study for the Sraii--ff-infrrnne architecture
USS
SED
DoAE
Complexitics
Model variation
SELD
SI-SDRi f SDRi
ER |
F1 f
LE
LR f
Param. MACKS
without chain-of-inferenee
11.0
11.7
28.8
70.2
18.5
76.9
0.230
8.2 8.2 (4-9.8) M 99.1
without (+) SED branch
11.0
11.8
26.6
71.8
18.2
76.5
0.221
10.3 (496.8) M
G
(+) DoA branch
11.0
11.8
28.2
70.6
17.6
76.0
0.228
10.3 (496.8) M 101.6
audio decoder (+) SED & DoA
11.0
11.7
26.5
73.0
17.3
77.2
0.214
12.1 (496.8) M 104,0 G
-Chain-of-nferrence
11.2.ao
12.0am:
25.0
74.1.m
17.0
78.1
0.206
12.1 (496.8) M 104.0 G
The results reveal that incorporating the SED branch improves sound event detection (as indicated by enhanced ER and F1-score), while incorporating the DoA branch improves DoA estimation (as shown by reduced localization error, LE). These findings suggest that cross-attention with SED or AoÅ queries selectively refines the corresponding auditory information. Notably, combining both branches (SED & DoA) leads to further improvements in both metrics, indicating that when one cue unreliable, the other can compensate to support more accurate inference. In the final configuration, the full Col architecture was restored, with Net 2 including audio decoder 2. In this setting, the USS output from audio decoder was used for evaluation. The complete Col architecture yields the best overall performance, achieving 12.0 dB SI-SDRi on the USS task and SELD score of 0.206. These results confirm that the proposed Col mechanism effectively enhances estimation across auditory domains by explicitly modeling interdependenccess and enabling complementary cue integration. We run five trials and report the average and standard deviation for the final model. A more detailed analysis of Col's contribution is provided in Appendix E
4.3 Generalization to spatial audio benchmarks & comparison with SOTA models
We evaluated the generalization capability of DeepASA on two spatial audio benchmarks: MC-FUSS for USS and STARSS23 for SELD, and compared it to SOTA models. DeepASA was pre-trained on the ASA2 dataset and fine-tuned using the ground-truth parameters available in each target dataset. Since no existing model can simultaneously perform USS, SED, and DoAE tasks, comparison models trained individually for the subset of tasks they support.
USS performance comparison on MC-FUSS dataset Table 3 presents a comparison between DeepASA (excluding the Col, SED, and DoA decoders) and existing SOTA models for universal sound separation (USS). Despite being unable to utilize the Col module during fine-tuning on MC- FUSS, DeepASA outperforms existing USS models, particularly in scenes with a larger number
8

---

of foreground sources. This performance gain is largely attributed to the noise decoder's 's explicit estimation of background noise and the benefits of fine-tuning from pre-trained weights.
Table 3: USS performance comparison on MC-FUSS dataset
Training
Model
J=2
J=3
1=4
Total
Param.
MACK
ByteDunce-uss
14.8
14.4
12.7
14.0
(48007) M
40.1G
MC-BSRNN
15.7
15.2
11.4
14.1
12.2 M
15.3
TF-GridNee 47
17.2
16.1
12.5
15.3
14.7 M
462 G
DeFTAN-II
48
17.6
16.3
12.8
15.6
4.1 M
66.1
From scratch
SpatialNet (49
17.8
16.5
13.1
15.8
7,3 M
7188
DeFT-Mamba [38
18.4
17.1
13.8
3.6 M
83.8
DeepASA (SEP, w/o ATST)
18.5
18.2
15.7
2.9 M
7
DeFT-Mamba 38
18.4
17.1
14.1
3.6 M
83.8
Finc-tuning (pre-trained on ASA2)
DeepASA (SEP, w/o ATST)
18.8
19.0
17.4
18.4
2.9 M
7
DeepASA (SEP)
18.9
19.1
17.6
18.5
2.9 M
77.3
SELD performance comparison STARSS23 dataset We evaluate two versions of DeepASA (with and without Col) against SOTA SELD models, including top-ranking submissions from the DCASE 2023 challenge. DeepASA achieves the highest overall performance the STARSS23 dataset, even without employing the audio decoder. Both versions of DeepASA surpass the per- formance of the top-ranked model (37. which relied on class deasndee separation networks and model ensembling. is worth noting that DeepASA was pre-trained on the ASAZ dataset prior to fine-tuning. To control for the effect of pre-training, able models that were also trained using ASA2. As detailed in Appendix [F] DeepASA consistently achieves SOTA performance across all evaluated tasks under the same training conditions. These results collectively demonstrate the strength of DeepASA across diverse ASA tasks and datasets, validating the effectiveness of combining class, activation, and DoA trajectory information through the object-oriented processing (OOP) and chain-of-inferene (Col) architecture.
further compared DeepASA publicly avail-
Table 4: SELD performance comparison on STARSS23 dataset
Model
ER [
FT1
L.
LR
SELD ]
CST-former
59.0
42.6
20.5
61.3
0.416
MFF-EINV2
50
54.0
42.5
18.7
62.6
0.398
CST-forme2
1
42.0
59.7
15.6
68.4
0.301
EINV-2 w/ data augmentation chain (2nd rank) 35
42.0
57.5
15.8
72.7
0.301
NERC-SLIP System (Ist rank) 17
(w/ Ensemble)
40.0
64.0
13.4
74,0
0.277
NERC-SLIP System (1st rank)
337
(w/ Ensemble)
38.0
66.0
12.8
75.0
0,260
DeepASA (SELD)
34.4
62.6
10.2
73.8
0.259
DeepASA (SELD) + chain-of-inferencc
33.7
63.1
74.6
0.253
Related Works
Universal Sound Separation (USS) USS addresses the challenge of separating various sound sources in an auditory scene where multiple sources overlap [18] 31). Humans perform source separation by leveraging various characteristics of individual sources, such class, activation, and DoA. However, existing USS methods rely on fime-frequency representations, which may limit their ability to exploit such source-specific attributes [26] 22)). To address this, the proposed method draws inspiration from human segregation processing and integrates multiple audio cues within unified framework, resulting in improved performance the USS task.
Target Sound Extraction (TSE) TSE refers to the separation specific sound sources from audio mixture, with a focus on the auditory information of the source 21 TSE plays a significant role in the source separation area, where spatial and temporal characteristics are leveraged to extract the target sound [22 26). A limitation of TSE is that humans must explicitly provide clues such class, activation, and DoA. However, the network itself needs to iteratively infer these clues and use them to refine the separation process. To overcome this limitation, the proposed method introduces a strategy in which the clues inferred by the network are integrated to improve separation performance.
Sound Event Localization and Detection (SELD) The SELD task involves performing both SED Nevertheless, most previous SELD models first analyze auditory cues using separate DNN branches for SED and DoA, and then perform track-wise
and DoAE within a single model (28
33
51
55) This approach can lead to incorrect pairing of SED and DoA outputs, and especially causes performance degradation when handling polyphonic
separation into auditory stream 52
53

---

audio. Additionally, models like [37 have improved performance by incorporating pre-trained target sound extraction model [7 but cannot reuse features learned from separation for SELD. Instead, the proposed architecture reutilizes multiple auditory information estimated from common per-object features refine the separation and SELD performance.
Conclusion
We proposed DeepASA for unified auditory scene analysis and separation, employing a multi- decoder architecture that utilizes relations between auditory information for both source separation and analysis tasks. The model leverages chain-of-inferene to refine performance in USS, SED, and DoAE tasks. Experimental results demonstrated the effectiveness of sub-decoders and multi-clue attention in improving feature extraction. Our results show significant performance improvements over existing methods in various downstream datasets.
This paper focuses on the framework that emulates how humans perform ASA by utilizing various types of auditory information to segregate sound objects solve the cocktail party problem. We believe that DeepASA offers unified framework that not only advances the state of the art in auditory scene analysis and sound separation but also opens new directions for research the intersection of spatial hearing, multi-task learning, and object-centric audio modeling.
Broader impacts
Limitations and future work The main limitation is the large parameter size of the ATST adopted for the SED decoder. In this work, the separated object features are reprocessed by the patch-based large set of audio classification datasets. For
ATST to keep the advantage of ATST pre-trained the aspect of the bias of the dataset, the reverberation time was set between 0.2 and 0.6 seconds, so the performance the model may degrade environments with reverberation times longer than 0.6 seconds. Additionally, because the background noise was simulated with an SNR range of 6 to 30 dB, the performance the model is also expected to decline when the background noise has an SNR below () dB and is louder than the foreground source. Next, regarding the potential misuse, since this model can extract speech of individual speakers and analyze their types and directions, there is concern that it could be exploited for eavesdropping.
Future research could explore pre-training the novel classification layer designed to be compatible with object features, using a large audio dataset like ATST.
Acknowledgements and Disclosure of Funding
This work was supported by the National Research Foundation of Korea (NRF) grant (No. RS-2024- 00337945) and STEAM research grant (No. RS-2024-004664669) funded by the Ministry of Science and ICT of Korea government (MSIT), the BK21 FOUR program through the NRF grant funded by the Ministry of Education of Korea government (MOE), and the Center for Applied Research in Artificial Intelligence (CARAI) funded by DAPA and ADD (UD230017TD).
References
[1] A. S. Bregman. Auditory scene analysis. In Proceedings of the 7th International Conference Pattern Recognition. Citeseer, 1984.
[2] G. J. Brown and M. Cooke. Computational auditory scene analysis. Computer Speech & Language, 1994.
[3] E. C. Cherry. Some experiments the recognition of speech, with one and with two cars. Journal of the acoustical society of America, 1953.
[4] A. S. Bregman. Auditory scene analysis: The perceptual organizationn of sound. MIT press, 1994.
[5] Shihab A. S., Mounya E., and Christophe M. Temporal coherence and attention in auditory scene analysis. Trends in Neurosciences, 2011.
[6] D-I.. Wang and G. J. Brown. Computational auditory scene analysis: Principles, algorithms, and applications. WiLE press, 2006.
10

---

[7] Q. Kong, Y. Xu, W. Wang, and M. D Plumbley. A joint detection caatiionncassiiicatiinn model for audio tagging of weakly labelled data. In IEEE International Conference on Acousties, Speech and Signal Processing, 2017.
[8] Y. Gong, Y-A. Chung, and J. Glass. PSLA: Improving audio tagging with pretraining, sampling, labeling, and aggregation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
[9] Schmid, K. Koutini, and G. Widmer. Efficient large-scale audio tagging via transformer-to- cnn knowledge distillation. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2023.
[10] A. Mesaros, Heittola, T. Virtanen, and M. D. Plumbley. Sound event detection: tutorial. IEEE Signal Processing Magazine, 2021.
[11] E. Cakır, G. Parascandolo. T. Heittola, H. Huttunen, and T. Virtanen. Convolutional recurrent neural networks for polyphonic sound event detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2017.
[12] N. Shao, X. Li, and X. Li. Fine-tune the pretrained ATST model for sound event detection. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2024.
[13] P-A. Grumiaux, Srdan Kitić, L. Girin, and A. Guérin. A survey of sound source localization with deep learning methods. The Journal of the Acoustical Society of America, 2022.
[14] D. Diaz-Gueray A. Miguel, and J. R. Beltran. Robust sound source tracking using SRP- PHAT and 3D convolutional neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020.
[15] Y. Wang, B. Yang, and X. Li. FN-SSL: Full-band and narrow- fusion for sound source localization.
[16] D-L. Wang and J. Chen. Supervised speech separation based deep learning: An overview. IEEE/ACM transactions on audio, speech, and language processing, 2018.
[17] Y. Luo and N. Mesgarani. Conv-TasNet: Surpassing ideal time- fine-freyyency magnitude masking for speech separation. JEEE/ACM transactions on audio, speech, and language processing, 2019.
[18] I. Kavalerov, S. Wisdom, H. Erdogan, B. Patton, K. Wilson, J. Le Roux, and J. R. Hershey. Universal sound separation. In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2019.
[19] W. He. Deep Learning Approaches for Auditory Perception in Robotics. PhD thesis, EPFL, 2021.
[20] B. Veluri, J. Chan, M. Itani, T. Chen, T. Yoshioka, and S. Gollakota. Real-time target sound extraction. In IEEE International Conference on Acoustics, Speech and Signal Processing. 2023.
[21] Mare Delcroix, Jorge Bennasar Vázquez, Tsubasa Ochiai, Keisuke Kinoshita, Yasunori Ohishi, and Shoko Araki. SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.
[22] H. Wang, D. Yang, C. Weng, J. Yu, and Y. Zou. Improving target sound extraction with timestamp information. 2022.
[23] M-S. Kim, D.and Back. Y. Kim, and J-H. Chang. Improving target sound extraction with timestamp knowledge distillation. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2024.
[24] T. Jenrungrot, Jayaram, S. Seitz, and Kenelmacher-Shiizemma. The cone of silence: Speech separation by localization. Advances in Neural Information Processing Systems, 2020.
11

---

[25] R. Gu and Y. Luo. ReZero: Region custonalust sound extraction. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
[26] Q. Kong. K. Chen, H. Liu, X. Du, T. Berg- Frng-KKikpatrick, S. Dubnov, and M. D. Plumbley. Universal source separation with weakly labelled data. arXiv preprini arXiv:2305.07447, 2023.
[27] K. Zmolikova, M. Delcroix, T. Ochiai, K. Kinoshita, J. Cernocký, and D. Yu. Neural target speech extraction: An overview. IEEE Signal Processing Magazine, 2023.
[28] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 2018.
[29] Y. He, N. Trigoni, and A. Markham. SoundDet: Polyphonic moving sound event detection and localization from raw waveform. In International Conference on Machine Learning, 2021.
[30] T. Aizawa, Y. Bando, K. Itoyama, K. Nishida, K. Nakadai, and M. Onishi. Unsupervised domain adaptation of universal source separation based neural full-rank spatial covariance analysis. In IEEE 33rd International Workshop on Machine Learning for Signal Processing, 2023.
[31] S. Wisdom, H. Erdogan, D. PW. Ellis, R. Serizel, N. Turpault, E. Fonseca, J. Salamon, Seetharaman, and R. Hershey. What's all the fuss about free universal sound separa- tion data? In IEEE International Conference on Acoustics, Speech and Signal Processing, 2021.
[32] K. Shimada, A. Politis, P. Sudarsanam, D. A. Krause, K. Uchida, S. Adavanne, A. Hakala, Y. Koyama, N. Takahashi, S. Takahashi, al. STARSS23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events. Advances in neural information processing systems, 2023.
[33] T. N. T. Nguyen, K. N. Watcharasupat, N. K. Nguyen, D. L. Jones, and W-S. Gan. SALSA: Spatial cug-augmented log-spectrogram features for polyphonic sound event localization and detection. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.
[34] Y. Shul and J-W. Choi. CST-former:
Transformer with channel- wwwsssppeeeepppoeee attention for sound event localization and detection. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2024.
[35] L. Xue, H. Liu, and Y. Zhou. Attention mechanism network and data augmentation for sound event localization and detection. Technical report, DCASE2023 Challenge, June 2023.
[36] J. Hu, Y. Cao, M. WU, F. Yang. W. Wang, M. D. Plumbley, and J. Yang. A data generation method for sound event localization and detection in real spatial sound scenes. Technical report, DCASE2023 Challenge, June 2023.
[37] Q. Wang, Jiang, S. Cheng, M. Hu, Z. Nian, P. Hu, Z. Liu, Y. Dong, M. Cai, Du, and C-H. Lee. The NERC-SLIP system for sound event localization and detection of DCASE2023 challenge. Technical report, DCASE2023 Challenge, June 2023.
[38] D. Lee and J-W. Choi. DcFT-Mamba: Universal multichannel sound separation and polyphonic audio classification. In IEEE International Conference Acoustics, Speech and Signal Processing, 2025.
[39] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2017.
[40] K. Okabe, T. Koshinaka, and K. Shinoda. Attentive statisties pooling for deep speaker embed- ding. arXiv preprint arXiv: 1803. 10963, 2018.
[41] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. FILM: Visual reasoning with general conditioning layer. Proceedings of the AAAI conference on artificial intelligence, 2018.
12

---

[42] I. R Roman, C. Ick, S. Ding, A. S. Roman, B. McFee, and J. Bello. Spatial scaper: library to simulate and augment soundscapes for sound event localization and detection in realistic rooms. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2024.
[43] T. von Neumann, K. Kinoshita, C. Boeddeker, M. Delcroix, and R. Haeb-Umbach. SA-SDR: A novel loss function for separation of meeting style data. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2022.
[44] D. Yu, M. Kolbæk, Z-H. Tan, and J. Jensen. Permutation invariant training of deep models for ppeaker-independentt multi-talker speech separation. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 241-245, 2017.
[45] Le Roux, S. Wisdom, H. Erdogan, and R. Hershey. SDR-half-baked or well done? In IEEE International Conference on Acoustics, Speech and Signal Processing, 2019.
[46] F. Schmid, P. Primus, T. Morocutti, J. Greif, and G. Widmer. Improving audio spectrogram trans- formers for sound event detection through multi-stage training. arXiv preprint -Xiv:2408.00791, 2024.
[47] Z-Q. Wang, S. Cornell, S. Choi, Y. Lee, B-Y. Kim, and S. Watanabe. TF-GridNet: Integrating full-and sub-band modeling for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[48] D. Lee and J-W. Choi. DeFTAN-II:
Efficient multichannel speech enhancement with subgroup processing. /EEF/ACM Transactions Audio, Speech, and Language Processing, 2024.
[49] C. Quan and X. Li. SpatialNet: Extensively learning spatial information for multichannel joint speech separation, denoising and dereverberation. IEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 2024.
[50] D. Mu, Z. Zhang, and H. Yue. MFF-EINV2: Multi-scale feature fusion across spectral-spatial- temporal domains for sound event localization and detection. arXiv preprint arXiv:2406.08771, 2024.
[51] Y. Shul, D. Choi, and J-W. Choi. CST-former: Multidimensional attention based transformer for sound event localization and detection in real scenes. arXiv preprint arXiv:2504.12870 2025.
[52] K. Shimada, N. Takahashi, Y. Koyama, S. Takahashi, E. Tsunoo, M. Takahashi, and Y. Mitsu- fuji. Ensemble of ACCDOA-and EINV2-based systems with D3Nets and impulse response simulation for sound event localization and detection. arXiv preprini orXiv:2/06 10806, 2021.
[53] Y. Cao, T. Iqbal, Q Kong, F. An, W. Wang, and M. D. Plumbley, An improved -ennt-nndependent network for polyphonic sound event localization and detection. In IEEE International Confer- ence on Acoustics, Speech and Signal Processing, 2021.
[54] Q. Vo and D. K. Han. Resnet-conformer network with shared weights and attention mecha- nism for sound event localization, detection, and distance estimation. Technical report, Technical report, DCASE2024 Challenge, 2024.
[55] D. Mu, Z. Zhang, H. Yue, Z. Wang, J. Tang, and Yin. leld-mamba: Selective state- sate-space model for sound event localization and detection with source distance estimation. arXiv preprint arXiv:2408.0 050557 2024.
[56] T. H Falk, C. Zheng, and W-Y. Chan. A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech. IEEE Transactions on Audio, Speech, and Language Processing, 2010.
[57] S-H. Kim, H. Nam, and Y-H. Park. Temporal dynamic convolutional neural network for text- independent speaker verification and phonemic analysis. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6742 6746. IEEE, 2022.
[58] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via wradient-based localization. In IEFE International Conference Computer Vision, pages
2017.
13

---

[59] S. Niu, J. Du, Q. Wang. L. Chai, H. Wu, Z. Nian, L. Sun, Y. Fang, J. Pan, and C-H. Lee. An experimental study on sound event localization and detection under realistic testing conditions. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2023.
14

---

Appendix / Technical Appendices and Supplementary Material
This appendix is organized as follows:
Appendix provides detailed specifications for the SED decoder.
Appendix provides an in-depth analysis of the impact of the noise decoder on classification.
Appendix presents experiments on the effects of the direct and reverb decoders on DoAE.
Appendix examines the experimental results using the time-varying learnable window.
Appendix presents an analysis of the chain-of-inference mechanism.
Appendix compares the proposed method with SOTA models on the ASA2 dataset and includes real-world demonstration.
Appendix presents the experimental results along with the number of sound sources.
A Detailed specifications of the SED decoder
In this section, we provide detailed specifications of the SED decoder. The SED decoder consists of three branches: ATST, T-CRNN, and F-CRNN. The specifications of the SED decoder are presented in Table 5 First, in the ATST branch, the object features are converted into patches of length 64. The sequence length corresponds to the number of these transformed patches, and the patches are reshaped along the channel dimension. Since the pre-trained ATST takes a single-channel mel spectrogram as input, linear layer is added before ATST to match the embedding dimension, followed by adaptive pooling to align the time frame to T" = 40. In the T-CRNN branch, seven convolution layers are used, as in [46), followed by a linear layer after concatenating the output of the ATST branch. The channel dimension and pooled frequency dimension are then reshaped into a single dimension, with the time dimension treated as the sequence dimension, and passed through a T-GRU. Finally, in the F-CRNN branch, pooling is focused on the time dimension, followed by two convolution layers, after which the channel dimension and pooled time dimension are reshaped into single dimension, and the frequency dimension is treated as the sequence dimension before passing through an F-GRU. The combined output the ATST and T-CRNN branches then passes through fully connected layer with an output dimension of to produce the object activation curve, a fully connected layer with an output dimension of C = 13 (the number of classes) to produce the sound event map, and is subsequently combined with the output of the F-CRNN after passing through attentive statistics pooling (ASP) [40] and a fully connected layer, producing the final class output.
Tp 4,
Table 5: Detailed specifications of the SED decoder
SED decoder
ATST branch
T-CRNN branch
F-CRNN branch
(3,3) LN, Mish
patchifiication TF T,7, x Dr,FF
Pool 2x3
(3,3)@64, LN, Mish
(3,1)664, LN, Mish
Pool 2x1
(3,3)0664 LN, Mish
Pool 4x5
FC 768
Pool 2x1 (3,3)064, LN, Mish
Pre-trained ATST
Pool (3.3)0/64, LN, Mish
(3,3)@64, LN, Mish
Pool 2x]
Adop + 40)
(3,3)@64, LN, Mish
Pool 4x4
FC320
Pool 2x1
FC320
GRU 320
GRU 320
FC512, Dropout FC13
Global pooling
FC64, Tanh, FC13
FC512, Dropout, FCI
softmax
soltmaa
FC512, Dropout, FC13
FCI3
Activation
SED
Class
B Impact of the noise decoder on classification
We conducted a detailed analysis to investigate the impact of the noise decoder on classification. Figure 6 presents a class-wise comparison of classification recall performance between the cases
15

---

with and without the use of the noise decoder. Without the noise decoder, the classification recall for domestic sounds is 49.4%, which is lower than other classes. This is due the similar feee-rreyy characteristics of domestic sounds and background noise (from the TAU-SNoise DE leading the model to misclassify domestic sounds as noise and remove them. However, after adding the noise decoder, the classification recall for domestic sounds increased significantly to 64.9%, showing an improvement of 15.5% points. Furthermore, the recall performance for almost classes improved. These results indicate that although the noise decoder estimates the noise waveform, it significantly influences classification performance.
In Figure 7 the t-SNE analysis further confirmed that without the noise decoder, domestic sounds (represented purple) have unclear boundaries with other classes, but when the noise decoder is employed, the boundaries between the domestic sound and other classes are more clearly delineated. This suggests that the noise decoder explicitly estimates the noise, allowing the model better distinguish between foreground sources and noise. In summary, employing noise decoder addresses the issue in previous approaches where the undenoised background noise degraded classification performance.
Per-class classification performance comparison
100
K-o Scne decede
Recall (%))
domestic
male
musical instrument
clapping
music
laughter
walk & footsteps
knock
water tap
door
female
hell
te telephooee average
sound
speech
speech
Figure 6: Per-class classification performance with and without noise decoder
w/o Noisc decader
(b) w/ Noise decoder
Dotnestl sound
T-SNE- tiremsion
Dameelic sound
T-SNE dimersson
T-SNE dimension
Figure T-SNE comparison with and without noise decoder
C Effect of direct and reverb decoder on DoAE
To investigate the effect of estimating direct and reverberant foreground signals separately on DoAE, we conducted detailed analysis. Estimating direct and reverberant signals separately introduces the dereverberation task in addition to the denoising and separation tasks, which can lead to a decrease in overall separation performance. However, as shown Figure g the LE histogram for DoAE reveals an increase in the number of samples with LE within 10 degrees, while the number of samples with LE greater than 10 degrees decreases. This indicates that even though the direct and reverb decoders are part of the audio decoder, they positively influence DoAE performance.
To understand how the dereverberation task affects DoAE, we performed a cosine similarity analysis between the convolution kernel weights. Figure compares the cosine similarity between the first convolution kernel weights of the DoA decoder and those of the direct and reverb decoders. The results show that the similarity between the DoA decoder and the direct decoder is higher than that between the DoA decoder and the reverb decoder. This suggests that the DoA decoder relies more on the direct components than the reverb components for DoAE. In reverberant environments, DoAE
16

---

Estimating the direct reverb signal
(b) Estimating the direct signal roverb signal separately
namber of sumples
Humber of samples
000000000000 0...000....0.................... w.w...........0000020.00........
0000000.000000 2...00.......................... 233.229922.9 Localization Error (LE, degree)
Localization Error (LE, degree)
Figure 8: Comparison of LE histograms when estimating (a) direct + reverb signals together and (b) direct and reverb separately.
(a) Cosine similarity between first convolution kemnel of DoA decoder and that of direct decoder
(b) Cosine similarity between first convolution komol of decoder
decoder
chanzel fist convelttoo: bernel of Dok decoder
culput charnd covoolation komel decoder
copper dhatnel or ordrect
0.4
CuRUr dhatnel iovel of deculer
0.4
0.3
01
0.0
0.11
Figure 9: Comparison of cosine similarity between the weight of the first convolution kernel of the DoA decoder and the convolution kernel weights of (a) direct decoder and (b) reverb decoder
more challenging compared to anechoic environments. This observation implies that learning to separate direct and reverb components within the object features can improve DoAE by ensuring that the direct components are properly embedded in the features.
Furthermore, to evaluate the performance of dereverberation, we compared the results of reverberant, direct (dereverberation, SI-SDR,) and reverb source separation (SI-SDR,,), as well as the speech reverberation modulation ratio (SRMR) [56]. SRMR is only calculated for classes corresponding to male and female speech. The experimental results demonstrate that the proposed model performs well in dereverberation, achieving performance comparable to full source separation. Additionally, with an SRMR of 6.5, the model indicates strong dereverberation effectiveness.
Table 6: Dereverberation performance of proposed model SI-SDR
SI-SDR,
SI-SDR,
SRMR 6.5
10.8
10.8
10.1
D Detailed analysis on time-varying learnable window
In this section, we present experiments to investigate the role and funetionality of the time-varying learnable window. In the audio encoder, the short-time Fourier transform (STFT) is widely used to convert waveform into a complex spectrogram. Unlike speech separation, which considers only the speech class, USS involves various classes, necessitating dynamic adjustments the configuration of the window. Although many conventional methods rely on fixed windows (28 12 38), this reliance
limits their performance [57]. To address this limitation, we propose time-varying learnable window that can dynamically adjust the window configuration for both USS and ASA.
We conducted a detailed ablation study on the effect of the time-varying learnable window. The results are presented in Table 7 Initially, compared encoding methods using complex STFT and learnable 1D convolution kernels. The performance of complex STFT generally exceeded that of the ID convolution kernel, suggesting that while the 1D convolution kernel extracts learnable features. complex STFT is more effective at explicitly capturing frequency information in the feature representation. Additionally, the time-invariant learnable window learns a fixed window across time, resulting in window length that is similar to that of the STFT. The time-invariant learnable window can be obtained by performing mean pooling the temporal dimension before being passed through the linear layers of the varying learnable window. Next, examined TDY-CNN [57], which
17

---

Table 7: Ablation study results of
learnable window
USS
SED
DoAE
Complexities
Variation
SELD 4
S1-SDRi
SDRi
ER
F1
LE
LR
Parum.
MAC/s
STFT
10.8
11.5
30.0
69.8
19.3
76.0
0.237
(+96.8 M 99.0 G
ID Conv
7.8
8.8
40.2
59.0
27.1
61.2
0.338
9.0 (496.8) M 99.2 G
Time-invariant
learnable window
10.7
11.4
31.3
68.1
19.5
75.6
0.246
8.2 (496.8) M 99.1 G
TDY-CNN
10.5
11.3
29.7
70.0
18.9
76.3
0.235
9.0 (+96.8 M 99.4 G
Time-variant
learnable window
11.0
11.7
28.8
70.2
18.5
76.9
0.230
8.2 (+96.8) 99.1
conventional STFT
time-varying learaalee window
40
30
20
16.4
10
104
11:0
000
-10
-10
-20
COUSOO RETOISAO
luughtes
&
knock
door
fenale
hell
tolphene
Concesji
specss
ciepping
music
laughter
ktookk
tup
door
spaich
hell
telephoree
Class
Class
Figure 10: Comparison of source separation performance for each class: (a) conventional STFT, (b)
vary-ng learnable window
learns attention weights for each time frame to control how much information is passed through, capturing phoneme variation. However, this approach has the limitation being unable to adjust the frequency band information. In contrast, the proposed method allows for control over the frequency band by adjusting the width the main lobe.
Figure presents comparison of separation performance for each class. The results show signifi- cant improvement in the separation performance of sporadic sounds, such as door and knock events. Despite being transient, the time-varying learnable window effectively captures the characteristics of these sound objects, allowing the model to improve separation performance.
Figure T1 shows a comparison of the sample between conventional STFT and the proposed time- varying learnable window. Figure a] shows the time-domain waveform of the audio mixture, and Figure libb plots its frame-wise standard deviation over time. Figure [c), (d), and (e) present the SED result of conventional STFT, that of time-varying learnable window, and ground truth, respectively, with binary values between 0 and (black indicates O, white indicates 1). The results show that with conventional STFT. object was not captured, and object 3 failed in both classification
(a) audio mixture
conventional
time-varying leurnable window
(e) ground truth
Object
STFT
Class
(b) standard deviation along time
Object
140
Voll Awots
Clase
110
Time
3bject
Index
Time (4)
Time (5)
Time (5)
Figure 11: (a) Time-domain waveform of the audio mixture, (b) window length along the time frame,
SED results for (c) conventional STFT, (d) vareing learnable window, (e) ground truth.
18

---

(3) Mixture
(b) conventional STFT
learnable window
(c) time-varying
Froguemi (M14
Frequency (M
Female speech
Female speech
19
1.0
0.4
0.4
Time (4)
Priquency (abl))
Male speech
Male speech
05
0.5
0.2
Freqpetcy (ele)
Telephone
Freueere (ile))
Telephonc
0.
05
(s)
Time
Figure 12: (a) Spectrogram of the audio mixture, GradCAM results for (b) conventional STFT, (c)
time-varying learnable window.
and activation estimation. However, using the proposed method, the model can detect object 1, which improves the sound event detection performance of object 3.
Finally, Figure 12 shows GradCAM [58] analysis of the separated object features immediately after the object splitter. to examine how well the object features encapsulate the properties of the corresponding sound objects. The model with conventional STFT struggled with object separation. as regions of speech are still embedded within telephone regions. In contrast, the proposed method more effectively isolates the telephone regions by suppressing most of the speech-related components. This demonstrates that the proposed approach effectively separates sound objects based on auditory information.
E Detailed study on Chain-of-Inference
In this section, we conduct case study to understand the underlying mechanism of the Col. It was observed that the model struggled with SED before applying the Col, whereas the SED performance improved after its application. We then analyzed the fundamental reasons for this improvement. Figure 13 a). (b), and (c) present the SED output for the model without Col, the model with Col, and the ground truth, respectively. By comparing the performance before and after applying Col, an improvement is observed the SED estimation for object 1 and object 3. To investigate the cause of this difference, we examined the DoAE results for object and object 3 before applying Col in Figure d) and (c), and the multi-clue attention maps with SED as the query and DoA as the query in Figure 13 () and (g), respectively. In the DoAE results, for object 1, DoA vectors below the onset threshold (0.5) suggest the potential presence of an object, while for object 3, the estimated DoA closely matches the ground truth. This indicates that the classification of object 3 was refined through Col by leveraging the nearly accurate DoAE. Furthermore, examining the attention maps, the attention map with SED as the query identified four objects, whereas the one with DoA as the query identified five. This demonstrates the successful correction of object 1, which was mistakenly classified silence, by utilizing DoA information.
Table 8: Performance comparison by repeating chain-of-inference
Stage
USS
SED
FI 1 LE J LR + SELD |
DoAE
Complexities
SI-SDRi 1 SDRi 1
ER L
Param.
MAC/s
without Col
11.0
11.7
28.8
70.2
18.5 76.9
0.230
8.2 (+96.8)) M 99.1
Col (Ist stage)
11.2
12.0
25.0
74.1
17.0
78.1 0,206
12.1 (+96.8) M 104.0 G
Col (2nd stage)
11.1
11.9
26.8
72.3
17.2
77.5
0.216
16.0 (+96.8 M) 118.9 G
Additionally, the Col mechanism utilizes the high-level information (DoA, SED) to refine the object feature separation based on the temporal coherence matching (TCM), so using the same mechanism multiple times can bias the results towards the direction of prioritizing TCM. We present the results of performing the chain-of-inferrnee multiple times as Table As a result, using Col only once resulted in the highest performance.
19

---

(a) Col
Object
Object
Object
Object
Object
Class index
(b)
Class index
(c) ground Iruth
Class index
Time
Time
Time (s)
Time (8)
Object 1
Multiclue map (query:: DoA)
(e)
(query: map
DoA ouiput
18-
005
Time (5)
Time (s)
(d) DoA estimation result w/o CO1
(leff: prediction, right: ground truth)
DoA
Object
20
30 SED
40
40
04
DoA ourpot
50-
62
10
00
Time (5)
Time (8)
DoA
(c) Do estimation result wio COT (left: prediction, right: ground truth)
Figure 13: SED result (a) without Col, (b) with Col, (c) ground truth, and (d) DoAE result of object without Col model, (e) DoAE of object 3 without Col, multiclue attention map for (f) DoA query, (g) SED query
F Comparison with SOTA models for ASA2 dataset and real-world demonstration
Comparison USS and SELD performance ASA2 dataset We compared the performance the proposed DeepASA model on the ASA2 dataset with that of other SOTA models in Table 9) First, the proposed model achieved superior USS performance compared other models 26
even though it operates with MIMO setup and performs dereverberation task. Despite having lightweight feature aggregation module compared to DeFT-Mamba, the model still achieved SOTA performance. This improvement is likely due the ability of the model to utilize various auditory information, such as class, activation. DoA, and noise, enabling better separation foreground sources. Next, in terms of SELD performance, the proposed model significantly outperformed existing models. This is because ASA2 consists entirely of polyphonic audio, and carlier approaches that estimate the outputs directly from mixtures unsuitable for such overlapped sources. Our model avoids this limitation by separating the objects.
Real-world demonstration To evaluate the applicability of the proposed method and dataset in real-world scenarios, we experimented using a pre-trained model without fine-tuning. The experiment was carried out in a real office environment, where three sound objects (male speech, music, and telephone) were placed. Among these, the male speech was a moving source. The miniDSP ambiMIK- I was utilized for capturing the sound, which is different from the microphone configuration for the pre-training dataset. The results demonstrate that the pre-trained DeepASA model can be applied to USS, SED, and DoAE tasks in real-world conditions, confirming its feasibility. A demo video of the
20

---

Table 9: Comparison with SOTA models ASA2 dataset
USS
SED
DoAE
Complexities
model
SELD
SI-SDRi
SDRi 1 FR.
Fl
LE
IRT
Param.
MAC/s
ByleDance-uss
5.2
8.3
28 (+80.7) M
40.1 G
TF-GridNet 47
8.7
10.7
14.7 M
462 G
SpatialNet 44
9.6
10.2
7.3 M
71.8
DeFT-Mamba 38
10.4
11.3
M
83.8
EINV2
48.5
27.1
51.5
0.431
6.7.
ResNet Conformer
45.7
41.0
26.6
53.7
0.414
13.6
7.6
SELD-Mamba [55
43.5
42.7
25.5
56.7
0.396
75.1 M
G.3
MFF-EINV2 [50]
42.1
43.2
25.8
60.7
0.381
54.8 M
14.3
DeepASA (wlo ATST)
11.0
11.7
30.1
69.5
18.5
74.8
0.240
8.2
91.0G
DeepASA
11.7
28.8
70.2
18.5
76.9
0,230
(+96.8) M
99.1
(+) Chain-of-inference
11.2.m
12.0am:
25.0.m
74.1us
17.0wa
78.1au 0.206
(496.8)
104.0
real-world experiment and example results from the ASA2 dataset are available on the page linked below
Additionally, we conducted an experiment by fine-tuning the pretrained DeepASA over datasets generated from real-world room impulse responses (RIRs). The fine-tuned model was then evaluated by unseen real-world RiRs. From Table TO, the experimental results show a slight decrease in USS performance, as well in most of SELD performance. This performance degradation can be attributed to the discrepancy in the hidden representation of simulation and real RIRs. Ideally, training with large of RIRs, including both simulated and measured RIRs, would improve the generalizability of models.
Table 10: Comparison between datasets with simulated and real-world RIRs
USS
DoAE
RIR
ER. FI LF... 1.R SELD
SI-SDRi
SDRi
simulated RIR
11.0
11.7
28.8
70.2
18,5
76.9
0.230
real RIR
10.6
11.4
32.7
66.4
13.4 72.3 0.254
G Experimental results along with the number of sound sources
We performed inference when the number of sound sources within the maximum estimable range [2, 5]. and when it exceeded the maximum estimable range [6. 7]. The experimental results are presented in Table II When the number of foreground sources is 6 or 7, resulting in sharp decline in performance. The experimental results show that when there are 6 sources, two sources are not separated, leading very low performance only for the two sources. When there are sources, three sources are mixed and mapped to one track in most cases, resulting in very low performance for three of the sources.
Table 11: Experimental results along with the number of sound sources
USS
SED
DoAE
Number sources
SELD
S1-SDRi
SDRi
ER
F1
LE
LR
17.7
17.1
22.1
82,4
10.5
83.5
0.143
3
14.6
14.5
22.4
80.8
13.1
86.3
0.160
8.5
10.6
33.8
65.6
20.9
72.5
0.269
7.9
10.3
34.9
0.280
Total
2.4
5.9
47.7
48.5
54.5
0.418
Top 4 average
7.0
9.6
38.2
61.2
25.6
66.7
0.311
Bottom 2 average
-6.8
-1.5
69.1
23.1 69.7
30.1
0.636
Total
0.2
4.5
59.0
36.6
51.7
42.4
0.521
Top 4 average
6.6
9.1
38.9
60.6
29.2
64.5
0.325
Bottom average
-8.3
-1.6
85.8
81.7
12.9
0.784
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
These are the provided in the lines 1-9 in the Abstract and the lines 42-58 in the Introduction. We propose novel approach that not only performs multi-task learning as a foundation model but also emulates human auditory scene analysis by reasoning over multiple auditory cues.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
This is provided in the lines 357-361 in the Conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The Method section and the Appendix section contain the details of our proposed method for reproducing the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
23

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
To support reproducibility, we provide access to the ASA2 dataset used in our experiments. The code will be released publicly if the paper is accepted.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppbllcgggudeeeeeeeeeeee Coaee//ddeeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This is provided in the lines 207 226 in the Experimental settings sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The performance of the final proposed model was evaluated by conducting statistical significance tests five runs with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resource in the Experimental Settings section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Authors carefully read the NeurIPS Code of Ethics and preserved anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This is provided in the lines 352-356 in the Conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve IRB approvals nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
27

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neure wwwparripp...coooeeeeeeeeeea cc/Conferences//2022 for what should or should be described.
28


Reviews:
[{"review_id": "wFpBj0HPP6", "reviewer": "Reviewer_gK5M", "summary": "This paper proposes DeepASA, which trains a single deep neural network (DNN) to do source separation, dereverb, sound event detection, audio tagging, and direction-of-arrival (DOA) estimation.\n\nThis is realized by first implicitly separating the mixtures, creating hidden representations for each sound object, and then using various decoders on the hidden representations to estimate auditory parameters such as sound event and DOA.\n\nEvaluation results on their own customized training and test dataset (i.e., ASA2) and two public datasets (MC-FUSS and STARSS23) show that the proposed method can produce strong performance for sound separation and auditory parameter estimation.", "questions": "1.\tIn Section 2.2, how is the synthesis window in inverse STFT computed?\n\n2.\tLine 146, what is “a single-channel object activation (onset and offset) curve”. From Figure 4(b), its dimension is JxT’ (without dimension C, so it is a label for all the sources combined). Is it a binary vector for each source, denoting when each source is active? It is really difficult to understand and this makes the reviewer unable to fully follow the “SED decoder” part, please explain and improve.\n\n3.\tLine 150, “During evaluation, only the predicted class probability and object activation curve are used to calculate the SELD scores.” Can the authors explain how? The predicted class prob has a dimension of JxC, and the object activation curve has a dimension of JxT’, according to Figure 4(b).\n\n4.\tLine 152, “The SED decoder combines a pre-trained audio teacher-student transformer (ATST) [12] with two separate convolutional recurrent neural network (CRNN) [11]-based branches summarizing time and frequency information.”, how is CRNN reflected in Figure 4(b)? The reviewer only notices AIST, but not CRNN. Are all the compared benchmark systems using the AIST model, which is trained on AudioSet, a much larger scale dataset than ASA2, MC-FUSS and STARSS23? If not, the comparisons in the experiments may not be fair at all.\n\n5.\tLine 114, “from 2M to C” should be “from 2M to D”? as C is defined as the number of sound types the model can handle.\n\n6.\tFigure 5, what is symbol “S”? Why the embedding after SED Decoder 1 is SC and why the embedding after DoA Decoder 1 is 3S? Are there any motivations of choosing the embedding dimension?\n\n7.\tFigure 5, in SED query branch, the attention output should be 3SxT’? Then why is it followed by a linear layer SC->D, where the dimension does not match.\n\n8.\tFigure 4(b), “SED decoder” contains multiple outputs including estimated activations, predicted classes and predicted SED. In Figure 5, “SED Decoder 1” outputs a SCxT’ tensor. How is this performed? \n\n9.\tLine 197, is ASA2 built upon the dataset proposed in [38]. From Section 3.1, this is not clear.\n\n10.\tLine 211, “The initial learning rate 4 ×10−4 was used for the Adam optimizer during pre-training, and 4 ×10−4 and 5 ×10−5 were used for fine-tuning to the MC-FUSS and STARSS23 datasets, respectively.” How is pre-training performed? So far, the authors talk nothing about pre-training. Do the authors first train DeepASA on ASA2, and then fine-tune the trained model on MC-FUSS and STARSS23? If this is the case, are all the compared benchmarks using similar ways and trained on the same training data?\n\n11.\tLine 221, what is “timestamp decoder”? this is a new thing that has not been described. It is not shown in Figure 4(b).\n\n12.\tLine 221, any motivations on using MSE loss for sound event detector?\n\n13.\tIn Figure 4(b), what loss function is used for the audio decoder part? PIT? There is no description, and this is an important detail.\n\n14.\tSection 2.5, SED decoder has 3 outputs, which themselves may not be aligned with each other. Would this be possible and if so, how is this solved? and, in TCM, how the authors align DOA results with each of the 3 outputs in the SED decoder?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nDeepASA can simultaneously deal with multiple tasks, and obtains reasonable improvement over existing methods.\n\nWeaknesses:\n\n1.\tTitle, “one-for-all” is not accurate, as auditory scene analysis contains many more tasks, while DeepASA only addresses a subset of them. In addition, the considered combination of tasks is not so novel, as compared with existing SELD tasks (which already deal with SED, audio tagging and DOA estimation simultaneously), DeepASA only introduces separation.\n\n2.\tThe evaluation is only performed on simulated datasets, and lacks convincing evaluation results on real-recorded datasets for sound separation. On real-recorded data, sound separation is usually difficult, as real data often mismatches simulated data. In this case, would the method still perform well, considering that the method relies so much on obtaining good hidden representations for each sound object (i.e., if the hidden representations for each sound object is not good on, e.g., real-recorded data, then auditory parameter estimation would be also not good)?\n\n3.\tSection 2.2, better to use “analysis window” to replace the word “window”, as “window” is overloaded with many meanings in signal processing.\n\n4.\t“SED decoder” in Section 2.4 and Figure 4(b) feel messy and are difficult to follow. The writing needs improvement.\n\n5.\tMany details are missing regarding the model. See my detailed questions to the authors.\n\n6.\tThe comparison with existing models in Table 3 and 4 are not so fair, as they are trained on different sets of data, and DeepASA uses the ATST model (which is pre-trained on a very large dataset). It might not be surprising that DeepASA can obtain better performance. See also my detailed questions.", "was_revised": true, "final_justification": "I'd like to thank the authors for addressing my comments and providing additional experiments. Many of my concens are addressed and I'd like to increase my rating as boardline accept.", "rebuttal": "**Weakness**\n\n**A1.** Thank you for the reviewer’s comment. We will change the title of the manuscript to “DeepASA: An object-oriented multi-purpose network for auditory scene analysis.”  Additionally, the ‘separation’ task of our work represents ‘multichannel-to-multichannel separation’ and ‘dereverberation’, which involves separating the direct and reverberant waveforms in a multichannel format, unlike conventional separation models that produce single-channel output.  \n\n**A2.** The STARSS23 dev-test dataset is a real-world dataset captured in a real environment. So the SELD performance has been validated on a real dataset. For the USS task, we conducted an additional experiment by fine-tuning the pretrained DeepASA over datasets generated from real-world room impulse responses (RIRs). The fine-tuned model was then evaluated by unseen real-world RIRs.\n\nComparison between datasets with simulated and real-world RIRs\n||SI-SDRi↑|SDRi↑|ER↓ |F1↑ |LE↓|LR↑|SELD↓|\n|-|-|-|-|-|-|-|-|\n|simulated RIR|11.0|11.7|28.8|70.2|18.5|76.9|0.230|\n|real RIR|10.6|11.4|32.7|66.4|13.4|72.3|0.254|\n\nThe experimental results show a slight decrease in USS performance, as well as in most of SELD performance. As the reviewer mentioned, this performance degradation can be attributed to the discrepancy in the hidden representation of simulation and real RIRs. Ideally, training with a large set of RIRs, including both simulated and measured RIRs, would improve the generalizability of models. \n\n**A3.** We will use “analysis window” instead of “window.”\n\n**A4.** According to the reviewers’ comments, we will amend the explanation of “SED decoder” in Section 2.4 as follows. \n“The SED decoder decodes the object features of each sound source into the predicted class probability (1xC), the binary object activation curve for event onset and offset (T’x1), and the predicted SED map (T’xC). Here, T’ and C denote the number of time frames and the number of classes, respectively. The predicted SED map is only used in the training stage to guide the joint detection of a sound event and class. In the inference stage, we use the SED map produced by multiplying the predicted class and activation curve, resulting in a single SED map for each sound source. This is to exclude the possibility of more than one class being mapped to the SED map for each sound object.”\n\n**A5.** We will update the text and figures (see below).\n\n**A6.** The reviewer’s comments can be summarized as (1) Use the same training dataset for comparison, and (2) The large size of the pretraining dataset for the ATST (SELD task).\nAs for the comment (1), the objective of the comparisons is to demonstrate the “generalizability” of the proposed model pretrained on the multi-task dataset. The “fair comparison using the same dataset” is separately studied in the table below, where the same dataset (ASA2 dataset) was used and compared to the SOTA USS model (DeFT-Mamba-MISO). For the case when using only MC-FUSS, we conducted additional experiments, demonstrating that the proposed model still outperforms DeFT-Mamba even when trained from scratch without pretraining.\n\nResults on the MC-FUSS dataset (SI-SDRi)\n||| S=2  | S=3  | S=4  | Total |\n|-|-|-|-|-|-|\n|pre-trained on ASA2|DeFT-Mamba| 18.4 | 17.2 | 14.1 | 16.6  |\n||DeepASA (w/o ATST)| 18.8 | 19.0 | 17.4 | 18.4  |\n||DeepASA (w/ ATST)|18.9|19.1|17.6|18.5|\n|from scratch|DeFT-Mamba|18.4|17.1|13.8|16.4|\n||DeepASA (w/o ATST)|18.5|18.2|15.7|17.5|\n\nFor comment (2), the SELD models in Table 4 were also trained using the large auxiliary datasets. The DCASE 2023 challenge allowed the use of external datasets and pre-trained models, so ‘NERC-SLIP’ and ‘EINV2 with data augmentation chain’ were also trained by large auxiliary datasets like AudioSet, FSD50K. \n\n**Questions**\n\n**A1.** We used a fixed Gaussian window with a zero mean and standard deviation of 6.67 ms for the synthesis window in inverse STFT.\n\n**A2.** Thank you for pointing out this ambiguity in the figure. SED decoders for “J” different objects share the same architecture and weights (parameter-shared), and this is expressed as the batch dim. Accordingly, for each object, the object activation curve is a single-channel binary vector across time (1xT'). To avoid confusion, we will revise Fig. 4(b) with a clear presentation of this parameter-shared architecture.\n\n**A3.** For each object in the batch(J), a T'xC-sized SED map is produced from the estimated activation curve and the class probability vector (please refer to W-A4). The SED maps for J objects are concatenated to form the output of size JxT'xC. We will reflect this detailed explanation in the revised manuscript along with the modified Fig. 4(b).\n\n**A4.** “CRNN” denotes multiple repetitions of Conv2d and Maxpool2d, followed by GRU and FC layers in Fig. 4(b). Specifically, among the three red branches of the SED decoder, the middle path with seven Conv+Maxpool layers outputting (JxT'xD) is the T-CRNN path, and the bottom path with two Conv+Maxpool layers outputting (JxF'xD) corresponds to the F-CRNN path. We will revise Fig. 4(b) to enhance its clarity.\n Again, Tables 3 and 4 are for demonstrating the generalization capability, and the architectural performance comparison under the same training dataset should be referred to Table 1. \n- The other models compared in Table 4 do not use the pre-trained ATST, but top-rank models (NERC-SLIP, EINV2 with data augmentation chain) also use large datasets (AudioSet, FSD50K) or pre-trained models. This is because the STARSS23 dataset is somewhat small to achieve robust classification across diverse classes of data. \n- The other models in Table 3 also do not utilize the ATST, but the use of ATST is less relevant for the USS task because the SED decoder is removed in fine-tuning for the USS task. \n-  For showing the architecture-originated difference only, we will provide the results of DeepASA trained solely on each dataset without using the pre-trained ATST or any pre-training on ASA2, in which DeepASA still outperforms existing SOTA models. \n\nASA2 dataset result comparison between DeepASA with and without ATST\n|DeepASA|SI-SDR↑|SDRi↑|ER↓|F1↑|LE↓|LR↑|SELD↓|Param.|MAC/s|\n|-|-|-|-|-|-|-|-|-|-|\n|w/ pre-trained ATST|11.0|11.7|28.8|70.2|18.5|76.9|0.230|105.5M|99.1G|\n|w/o ATST|11.0|11.7|30.1|69.5|18.5|74.8|0.240|8.2M|91.0G|\n\nMC-FUSS dataset result comparison with DeepASA without ATST (trained from scratch)\n||S=2|S=3|S=4|Total|Param.|MAC/s|Runtime|\n|-|-|-|-|-|-|-|-|\n|DeFT-Mamba|18.4|17.1|13.8|16.4|3.6 M|83.8 G|39.2 min|\n|DeepASA (w/o ATST)|18.5|18.2|15.7|17.5|2.9 M|77.3 G| 26.8 min|\n|DeepASA (w/ ATST)|18.9|19.1|17.6|18.5|2.9 M|77.3 G|26.8 min|\n\n**A5.** Thank you for pointing out the typo. We will change “from 2M to C” into “from 2M to D.”\n\n**A6** S is equivalent to J (maximum number of sound sources). We will change S to J in Fig. 5 and will use S to denote the ‘actual’ number of sources in the mixture.\n- For each sound object, the SED decoder and DoA decoder output T'xC and T'x3 (3: X, Y, Z coordinates) size tensors, respectively. These are concatenated for J sound objects and then transposed to JCxT' and 3JxT' tensors. \n- In TCM, the embedding dim. is T' and the sequence dim. is JC or 3J. This is for emphasizing channels showing similar temporal activities across SED and DoAE through the cross-attention. After the cross-attention, the output is transposed again to make T' the sequence dim.\n\n**A7.** In the SED query branch, the query is the SED output (JCxT'), and the key and value are the DoA output (3JxT'). The output size of the attention follows that of the query, followed by transpose; it becomes T'xJC, and then the channel dim. is changed from JC to D through a linear layer (size: T'xJC → T’xD).\n\n**A8.** The SED decoder outputs three outputs, but only the SED map is used for training Net 2. As answered in replies W-A4 and Q-A3, J SED maps with a size of T'xC are produced and processed to form a T'xJC tensor in Fig. 5. \n\n**A9.** ASA2 is based on the ASA dataset, but we modified it extensively to enable the comprehensive tasks discussed in this study. ASA2 supports (1) the separation of direct and reverberation components (for dereverberation), (2) MIMO separation, as compared to MISO separation of the ASA dataset, and (3) an increased maximum number of foreground sound sources, from four to five. We will add these details to the appendix.\n\n**A10.** DeepASA's pre-training was conducted on the ASA2 dataset, followed by fine-tuning on the datasets for each downstream task. We will add a detailed explanation in the manuscript.\nModels in the MC-FUSS dataset benchmark did not involve pre-training, whereas some models for the STARSS23 included pre-training and data augmentation on large datasets (AudioSet, FSD50K). To show the influence of pretraining a USS model on a large dataset, we will add the experimental results presented in replies W-A6 & Q-A4 to the manuscript. \n\n**A11.** It is another consistency fault. We will unify the terminology to \"object activation curve estimation\" and also revise other parts with that expression (Line 163 and Line 221).\n\n**A12.** We used MSE because it is a typical choice in previous works for SED. (“CSTFormer” (Y. Shul, et. al), “Ensemble of ACCDOA and EINV2” (K. Shimida, et. al,))\n\n**A13.** The SA-SDR was used as a loss for the audio decoder, which is mentioned in line 216. Also, the PIT technique was used, and we will mention this in the revised manuscript.\n\n**A14.** As mentioned in the previous reply, the SED decoder outputs three components, but ultimately, the SED map of T'xC is constructed for each object from the predicted class probability (1xC) and object activation (T'x1). DoA decoder also generates T'x3 output for each object. The alignment between the SEDs and DoAs is accomplished in TCM through the cross-attention by assigning the time dim. as a sequence dim. We will add this explanation regarding this part in the manuscript and revise Fig. 4(b) and 5 accordingly."}, {"review_id": "mj92NpjZwX", "reviewer": "Reviewer_FCLK", "summary": "Authors present DeepASA as a unified framework for acoustic scene analysis, adopting an Object-Oriented Processing approach that separates object-centric features at an early stage to avoid permutation ambiguity, and the Chain-of-Inference mechanism that iteratively refines predictions using temporal coherence matching between different auditory cues", "questions": "Could authors provide evidence for true object correspondence? .. some validation that object features actually correspond to individual sound sources rather than just being abstract encoded representations.\n\nIs it possible to show that each of the J=5 object features consistently tracks the same sound source across time? When sounding objects enter and exit a scene, are object features able to maintain semantic coherence?", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nThis paper addresses a fundamental challenge of comprehensive scene analysis, maintaining consistent object-wise associations across all sub-decoders task heads, as opposed to prior track0wise processing systems.\n\nThe motivation for object separator, CoI, as well as dynamic stft is clearly articulated and supported by ablation evidence. Temporal coherence matching is also shown to provide measurable improvements.\n\nWeakness:\nThe overall model, even with pre-trained AST is notably bulky compared to task-specific approaches and is a concern for practical deployment. Additionally, given the multistage training, can authors provide an analysis of training convergence properties and sensitivity to initialization. Are there convergence guarantees for the chain-of-inference mechanism? I believe the 97M AST limits the novelty for SED component + is a computational bottleneck.\n\nWith regards to ASA2 dataset, the balanced distribution proportional to source count and increased maximum sources from 4 → 5 compared to prior versions suggests substantial modifications that could favor the proposed approach. Overall paper can benefit from highlighting some potential biases in ASA2\n\nForeground sources are fixed to J=5, however, some analysis regarding possible performance degradation when this assumption is violated would be helpful. Some analysis varying the degree of polyphonicity is expected. What is the computational and memory scaling behaviour?\n\nMinor weakness:\nThe relationship between different loss components and their relative weighting could be better motivated in the main paper than in the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "**Weak-Q1.** The overall model, even with pre-trained AST is notably bulky compared to task-specific approaches and is a concern for practical deployment. Additionally, given the multistage training, can authors provide an analysis of training convergence properties and sensitivity to initialization. Are there convergence guarantees for the chain-of-inference mechanism? I believe the 97M AST limits the novelty for SED component + is a computational bottleneck.\n\n**A.** As for the concern on the 97M ATST, we conducted an extra experiment using the same training dataset (ASA2) without using pre-trained ATST. The experimental results indicate that the USS performance remains consistent, and the SELD score is slightly lowered by 0.01. When compared to other state-of-the-art models in Table 8 of Appendix F, the performance does not differ significantly. The ATST was used to achieve the best SELD performance possible, but it can be omitted when the space complexity matters. In addition, the size of ATST (97M) is much larger than DeepASA without ATST (8.2M) but similar to those of SELD-Mamba (75.1 M) and MFF-EINV2 (54.8M).\n\nComparison between DeepASA models with and without ATST\n|DeepASA|SI-SDR↑|SDR↑|ER↓|F1↑|LE↓|LR↑|SELD↓|Param.|MAC/s|\n|-|-|-|-|-|-|-|-|-|-|\n|w/ pre-trained ATST|**11.0**|**11.7**|**28.8**|**70.2**|**18.5**|**76.9**|**0.230**|105.0 M|99.1 G|\n|w/o ATST|**11.0**|**11.7**|30.1|69.5|**18.5**|74.8|0.240|**8.2 M**|**91.0 G**|\n\nFor the initialization issue, we conducted experiments 5 times with different initializations and included the mean and standard deviation in the last row of Table 2. The influence of different initializations was not significant, only showing std 0.001 in SELD score and 0.1 dB in SI-SDR.\n\nThe CoI mechanism utilizes the high-level information (DoA, SED) to refine the object feature separation based on the temporal coherence matching (TCM), so using the same mechanism multiple times can bias the results towards the direction of prioritizing TCM. We present the results of performing the chain-of-inference multiple times as follows.\n\nComparison of multi-stage COIs\n|DeepASA|SI-SDR↑|SDR↑|ER↓|F1↑|LE↓|LR↑|SELD↓|Param.|MAC/s|\n|-|-|-|-|-|-|-|-|-|-|\n|w/o CoI|11.0|11.7|28.8|70.2|18.5|76.9|0.230|**105.0 M**|**99.1 G**|\n|CoI (1 stage)|**11.2**|**12.0**|**25.0**|**74.1**|**17.0**|**78.1**|**0.206**|108.9 M|104.0 G|\n|CoI (2 stages)|11.1|11.9|26.8|72.3|17.2|77.5|0.216|112.8 M|118.9 G|\n\n**Weak-Q2.** With regards to ASA2 dataset, the balanced distribution proportional to source count and increased maximum sources from 4 → 5 compared to prior versions suggests substantial modifications that could favor the proposed approach. Overall paper can benefit from highlighting some potential biases in ASA2\n\n**A.** In Table 8, all models were trained using the same ASA2 dataset, so the composition of this dataset does not particularly favor the proposed approach. However, since the pre-trained ATST on AudioSet might have an impact, we will add the performance of DeepASA without using the pre-trained ATST to the table (refer to the previous reply).\n\n As the reviewer mentioned, the pre-trained DeepASA could be advantageously applied for fine-tuning on the MC-FUSS dataset. To ensure transparency in performance evaluation, we present the performance of DeepASA trained from scratch on the MC-FUSS dataset, also without using a pre-trained model (ATST), similar to other models in Table 3.  DeepASA without pretraining and ATST achieves performance surpassing the previous best, DeFT-Mamba, with lower parameter size, computational complexity, and runtime (training time per epoch).\n\nMC-FUSS dataset result comparison with DeepASA trained from scratch (w/o ATST)\n(S denotes the number of sources in the mixture). \n||S=2|S=3|S=4|Total|Param.|MAC/s|Runtime|\n|-|-|-|-|-|-|-|-|\n|DeFT-Mamba|18.4|17.1|13.8|16.4|3.6 M|83.8 G| 39.2 min|\n|DeepASA (w/o ATST)|**18.5**|**18.2**|**15.7**|**17.5**|**2.9 M**|**77.3 G**|**26.8 min**|\n\nAdditionally, the benchmark for the STARSS23 dataset in Table 4 allows the use of external datasets and pre-trained models, following the criteria of the previous DCASE challenge. The models that achieved the best performance before DeepASA (NERC-SLIP, EINV2 with data augmentation chain) similarly performed data augmentation using large datasets like AudioSet and FSD50K or pre-trained their models for fine-tuning on the STARSS23 dataset. Consequently, the comparison in Table 4 can be considered fair, and the reason for increasing the number of foreground sources in ASA2 from 4 to 5 is specifically for fine-tuning on STARSS23.\n\n**Weak-Q3.** Foreground sources are fixed to J=5, however, some analysis regarding possible performance degradation when this assumption is violated would be helpful. Some analysis varying the degree of polyphonicity is expected. What is the computational and memory scaling behaviour?\n\n**A.** We conducted extra experiments with a larger number of sources than that assumed in the model (five). There is inherent performance degradation due to an increased number of sources, but the unseparated source signals are mostly merged into a single track without severely affecting the other four tracks. We will add this important discussion to the revised manuscript.\n\n(1) When the number of sources is increased to 6 and 7, there is a significant decrease in USS and SELD performance. This is obvious because (A) separating a larger number of sources is inherently difficult, (2) the model is designed to separate five sources, so unseparated objects are merged in some of the separated signals.\n\nResults for changing the number of sources (S)\n||SI-SDRi↑|SDRi↑|ER↓|F1↑|LE↓|LR↑|SELD↓|\n|-|-|-|-|-|-|-|-|\n|S<=5|**11.0**|**11.7**|**28.8**|**70.2**|**18.5**|**76.9**|**0.230**|\n|S=6|2.4|5.9|27.7|48.5|40.3|54.5|0.418|\n|S=7|0.2|4.5|59.0|36.6|51.7|42.4|0.521|\n\n(2) However, this does not mean the total collapse of the model. When we evaluate the performance of individual sound objects, there is a big performance gap between the top-4 performance objects and the rest. This result stresses that four object signals are relatively well separated, whereas the rest objects are mapped to one remaining track.\n\nDetailed results for S=6 and 7\n|||SI-SDRi↑|SDRi↑|ER↓|F1↑|LE↓|LR↑|SELD↓|\n|-|-|-|-|-|-|-|-|-|\n|S=6|Top 4 average|**7.0**|**9.6**|**38.2**|**61.2**|**25.6**|**66.7**|**0.311**|\n||Bottom 2 average|-6.8|-1.5|69.1|23.1|69.7|30.1|0.636|\n|S=7|Top 4 average|**6.6**|**9.1**|**38.9**|**60.6**|**29.2**|**64.5**|**0.325**|\n||Bottom 2 average|-8.3|-1.6|85.8|4.6|81.7|12.9|0.784|\n\n(3) Computational and memory scaling behaviour\n- To train the model separating a larger number of sources, the output dimension of the object splitter must be increased. In contrast, the parallel decoders are parameter-shared, so the parameter size remains the same and the computational complexity only increases linearly. \n\n- The increased computational load and memory we measured are as follows. In the main paper, the training was performed with a batch size of 2. However, due to the memory limit of our GPU, the batch size was changed to 1 to measure memory size and runtime.\n\nComplexity comparisons for changing the number of sources\n||Param.|MAC/s|GRU memory|runtime (/epoch)|\n|-|-|-|-|-|\n|S<=5|**105.0 M**|**99.1 G**|**12.3 GB**|**60.2 min**|\n|S=6|105.1 M|104.4 G|12.6 GB|62.3 min|\n|S=7|105.1 M|109.9 G|12.9 GB|64.1 min|\n\n**Minor weak-Q1.** The relationship between different loss components and their relative weighting could be better motivated in the main paper than in the appendix.\n\n**A.** In the main paper, lines 216-223, we have explained the weights of the loss functions and their relationships. We will also include additional information regarding permutation invariant training (PIT).\n\n**Question-Q1.** Could authors provide evidence for true object correspondence? .. some validation that object features actually correspond to individual sound sources rather than just being abstract encoded representations.\n\n**A.** The decoders processing different object features are identical parameter-shared decoders. If object features of individual sound objects include similar abstract representations of multiple objects, the outputs from the decoders should be the same. \n- In detail, the object splitter splits features for J objects and one noise. At this point, the size of the object feature map is (B, JxD, T, F), and to process each object in parallel, the object dimension J is reshaped into the batch dimension, resulting in (JxB, D, T, F). In other words, because a parameter-shared decoder is applied to the batch data, the decoder cannot be structurally tailored to a particular object. Our unclear illustration in Figure 4 might have caused this confusion, so we will amend the figure to clarify this mechanism.\n\n**Question-Q2.** Is it possible to show that each of the J=5 object features consistently tracks the same sound source across time? When sounding objects enter and exit a scene, are object features able to maintain semantic coherence?\n\n**A.** To verify this, we conducted a performance analysis on samples with an offset of more than 1 second during the occurrence of audio sources. The experimental results show that, although the performance of samples with an offset of more than 1 second decreases, the difference is not significant. This indicates that the proposed DeepASA can detect the same sound object through spatial and frequency information, even when there is an offset between the occurrences of the sources.\n\nInfluence of the offset inserted between source onsets/offsets\n||SI-SDRi↑|SDRi↑|ER↓|F1↑|LE↓|LR↑|SELD↓|\n|-|-|-|-|-|-|-|-|\n|Total|**11.0**|**11.7**|**28.8**|**70.2**|**18.5**|**76.9**|**0.230**|\n|with offset (> 1 s)|9.3|10.5|31.7|68.3|19.7|74.5|0.250|"}, {"review_id": "4ZvCDLw668", "reviewer": "Reviewer_Qqmd", "summary": "This paper presents DeepASA, a unified framework for auditory scene analysis that performs multichannel source separation, dereverberation, sound event detection, audio classification, and direction-of-arrival estimation in one model. The core contributions are: (1) an Object-Oriented Processing (OOP) strategy, which isolates per-object audio features early in the pipeline via a transformer-based aggregator and object separator, and (2) a Chain-of-Inference (CoI) mechanism that iteratively refines object features by fusing sub-decoder outputs through temporal-coherence attention. Evaluated on three benchmarks (ASA2, MC-FUSS, STARSS23), DeepASA achieves state-of-the-art performance across all tasks.", "questions": "See Weaknesses.", "limitations": "The authors discuss the ATST size but could further elaborate on dataset biases (e.g., reverberation conditions) and potential misuses (e.g., surveillance).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Ablations isolate each module’s contribution; end-to-end performance on multiple datasets is compelling.\n2. Real-world examples on the demo webpage lend practical credibility.\n3. Well-designed multi-decoder architecture eliminates permutation issues and enables complementary cue integration.\n\nWeaknesses:\n1. Figures 2 & 5 (model diagram & CoI) are dense and lack intuitive legends. Readers may struggle to follow data flow. I think a step-by-step schematic or callouts would help.\n2. Dynamic STFT (Sec. 2.2) is novel but under-explained: it would be clearer with visualizations of learned windows over time and ablations on σₜ initialization.\n3. The choice of maximum object count J is not justified: how sensitive is performance if fewer/more sources appear?\n4. Computational cost of the ATST-based SED decoder is high. It would be great to discuss the runtime and efficiency trade-offs.\n5. Related Work (Sec. 5) omits object-centric learning: key works like \"Slot Attention\" (Locatello et al., 2020), \"Sound of Pixels\" (Zhao et al., 2018), \"Objects That Sound\" (Arandjelović et al., 2017), \"SSV2A\" (Guo et al., 2024), \"AVObject\" (Li et al., 2025), are not discussed. The distinction between those and DeepASA’s OOP should be clarified.", "was_revised": true, "final_justification": "The paper contributes a unified framework for auditory scene analysis. The experiments demonstrate its effectiveness. After reading other reviewers' comments, I would like to recommend Borderline Accept.", "rebuttal": "**Weak-Q1.** Figures 2 & 5 (model diagram & CoI) are dense and lack intuitive legends. Readers may struggle to follow data flow. I think a step-by-step schematic or callouts would help.\n\n**A.** Thank you for pointing this out. We will revise the figures by creating a step-by-step schematic and adding detailed explanations in the captions so that the model structure can be understood just by looking at the figures. Specifically, in Figure 2, we aim to highlight how each object is processed in parallel on a batch basis through the object splitter. In Figure 5, we will clarify the signal flow from the SED and DoA query branches to TCM in the chain-of-inference. \n\n\n**Weak-Q2.** Dynamic STFT (Sec. 2.2) is novel but under-explained: it would be clearer with visualizations of learned windows over time and ablations on σₜ initialization.\n\n**A.** Thank you for mentioning the dynamic STFT. We briefly introduced its ablation study and the visualization of the learned window (changes in standard deviation over time) in Appendix D, but as per the reviewer's suggestion, we will provide a more detailed visualization in the revised paper. First, we will add visualizations of the learned window (mean and standard deviation) from various samples. To summarize the results here, the mean slightly alters around zero for most samples, but the standard deviation (width of window) dynamically changes, indicating that the dynamic STFT is utilized for adjusting the temporal and spectral resolution.\n- The ablation study on initialization of $\\sigma_t$ shows that proper initialization is important. If $\\sigma_t$ is not initialized with a size comparable to the width of the STFT analysis window, e.g., zero initialization, the features after the encoder do not sufficiently contain the information of the audio mixture, and the subsequent parameters cannot be trained for each task. Therefore, initialization is an essential element for learning, and as mentioned in the main text, $\\sigma_t$ was set to 6.67 ms to ensure that $6\\sigma_t$ can cover one frame.\n\n\n**Weak-Q3.** The choice of maximum object count J is not justified: how sensitive is performance if fewer/more sources appear?\n\n**A.** We set J=5 because the minimum maximum number of sources that allows the pre-trained model to be fine-tuned on the STARSS23 dataset is 5.\n We performed inference when the number of sound sources is within the maximum estimable range (S=2–5<=J) and when it exceeded the maximum estimable range (S=6–7>J). When the number of foreground sources is 6 or 7, resulting in a sharp decline in performance. The experimental results show that when there are 6 sources, two sources are not separated, leading to very low performance only for the two sources. When there are 7 sources, three sources are mixed and mapped to one track in most cases, resulting in very low performance for three of the sources.\n\n\nASA2 dataset result for changing the number of sources (S)\n||SI-SDRi↑|SDRi↑|ER|F1↑|LE↓|LR↑|SELD↓|\n|-|-|-|-|-|-|-|-|\n|S=2|**17.7**|**17.1**|**22.1**|**82.4**|**10.5**|**88.5**|**0.143**|\n|S=3|14.6|14.5|22.4|80.8|13.1|86.3|0.160|\n|S=4|8.5|10.6|33.8|65.6|20.9|72.5|0.269|\n|S=5|7.9|10.3|34.9|64.5|22.1|71.3|0.280|\n|S=6|2.4|5.9|47.7|48.5|40.3|54.5|0.418|\n|S=7|0.2|4.5|59.0|36.6|51.7|42.4|0.521|\n\nDetailed results for S=6 and 7\n|||SI-SDRi↑|SDRi↑|ER↓|F1↑|LE↓|LR↑|SELD↓|\n|-|-|-|-|-|-|-|-|-|\n|S=6|Top 4 average|**7.0**|**9.6**|**38.2**|**61.2**|**25.6**|**66.7**|**0.311**|\n||Bottom 2 average|-6.8|-1.5|69.1|23.1|69.7|30.1|0.636|\n|S=7|Top 4 average|**6.6**|**9.1**|**38.9**|**60.6**|**29.2**|**64.5**|**0.325**|\n||Bottom 2 average|-8.3|-1.6|85.8|4.6|81.7|12.9|0.784|\n\n\n**Weak-Q4.** Computational cost of the ATST-based SED decoder is high. It would be great to discuss the runtime and efficiency trade-offs.\n\n**A.** We additionally compared the performance of the final DeepASA model without the pre-trained ATST on the ASA2 dataset and the MC-FUSS dataset. The experimental results showed that while the parameter size significantly decreased, the model's performance did not change significantly, and the reduced performance still surpassed other existing state-of-the-art models. Specifically, on the MC-FUSS dataset, DeepASA achieved higher performance than DeFT-Mamba, even without the pre-trained ATST and pre-training on the ASA2 dataset, despite its smaller parameter size, computational complexity, and runtime (training time per epoch). We will add these experimental results to the main text.\n\nASA2 comparison between DeepASA with and without ATST\n|DeepASA|SI-SDRi|SDRi|ER|F1|LE|LR|SELD|Param.|Runtime|MAC/s|\n|-|-|-|-|-|-|-|-|-|-|-|\n|w/ ATST|**11.0**|**11.7**|**28.8**|**70.2**|**18.5**|**76.9**|**0.230**|105.0 M|30.1 min|99.1 G|\n|w/o ATST|**11.0**|**11.7**|30.1|69.5|**18.5**|74.8|0.240|**8.2 M**|**27.6 min**|**91.0 G**|\n\nMC-FUSS comparison between DeepASA with and without ATST (SI-SDRi)\n(S denotes the number of sources in the mixture).\n|Model|S=2|S=3|S=4|Total|Param.|MAC/s|Runtime|\n|-|-|-|-|-|-|-|-|\n|DeFT-Mamba|18.4|17.1|13.8|16.4|3.6 M|83.8 G|39.2 min|\n|DeepASA (w/o ATST, w/o pre-train)|18.5|18.2|15.7|17.5|**2.9 M**|**77.3 G**|**26.8 min**|\n|DeepASA (w/ ATST, w/ pre-train)|**18.9**|**19.1**|**17.6**|**18.5**|**2.9 M**|**77.3 G**|**26.8 min**|\n\n\n**Weak-Q5.** Related Work (Sec. 5) omits object-centric learning: key works like \"Slot Attention\" (Locatello et al., 2020), \"Sound of Pixels\" (Zhao et al., 2018), \"Objects That Sound\" (Arandjelović et al., 2017), \"SSV2A\" (Guo et al., 2024), \"AVObject\" (Li et al., 2025), are not discussed. The distinction between those and DeepASA’s OOP should be clarified.\n\n**A.** [Slot Attention] The key difference is that Slot Attention sets the slots first and then trains the mapping of objects to these slots, whereas OOP learns the spatial, frequency, and temporal relationships of the mixture audio first and then separates the objects through an object splitter. \n\n The main difference between these works and the proposed OOP is that, unlike OOP’s core characteristic, these works do not process the separated features in parallel by permuting them along the batch dimension for each task (localization, separation, retrieval, and audio generation). In OOP, features are separated by the object splitter and processed independently in the batch dimension, fundamentally preventing interference from other object features.\n\n [Sound of Pixels] is a network that separates audio occurring at specific pixels on a pixel-level basis. It extracts video features through a video analysis network and audio features through an audio analysis network and then performs source separation in an audio synthesis network. This approach is similar to conventional audio source separation networks, which extract information for source separation and ultimately perform the separation.\n\n [Object that Sounds] proposes a network that fuses video and audio features for cross-modal retrieval, enabling semantic object localization. Similar to the previous Sound of Pixels, it utilizes object-centric learning. If OOP were applied to this work, features could be extracted through AVEnet, and the object splitter would separate each audio feature, allowing parallel retrieval of each sound source at the batch level.\n\n [SSV2A, AVObject] also employ object-centric learning in the multi-modal domain, similar to the previous two papers. In conclusion, the key difference between object-centric learning and OOP lies in whether objects are explicitly separated and then processed in parallel through a parameter-shared decoder by passing them as batches.\n\n\n**Limitation-Q1.** The authors discuss the ATST size but could further elaborate on dataset biases (e.g., reverberation conditions) and potential misuses (e.g., surveillance).\n\n**A.** Thank you for the suggestion. We will add the following concerns in the Limitation & Future Work section: The reverberation time was set between 0.2 and 0.6 seconds, so the model’s performance may degrade in environments with reverberation times longer than 0.6 seconds. Additionally, because the background noise was simulated with an SNR range of 6 to 30 dB, the model’s performance is also expected to decline when the background noise has an SNR below 0 dB and is louder than the foreground source.\n\nNext, regarding the potential misuse, since this model can extract individual speakers’ speech and analyze their types and directions, there is a concern that it could be exploited for eavesdropping."}, {"review_id": "BGeeqXOgWC", "reviewer": "Reviewer_Mddv", "summary": "The paper proposes DeepASA, a new model for auditory scene analysis, taking in a multichannel audio mixture capable of performing source separation, sount event detection, audio classification and direction-of-arrival estimation. The DeepASA model itself is based on the DeFT-Mamba model.\nThe two core contributions are the chain-of-inference architecture and object oriented processing.", "questions": "1. In lines 307-308 you write \n>This performance gain is largely attributed to the noise decoder’s explicit estimation of background noise and the benefits of fine-tuning from pre-trained weights.\n\nThis is something I've wondered about when I saw Figure 2; why exactly is background noise estimated and how would it contribute to an improvement in estimating the foreground sources?\n\n2. Figure 2 shows how the channel dimension is increased from $2M$ to $D$ but the text (line 114) claims it increases from $2M$ to $C$. Is that just a mistake, given that $C$ is later (line 162) defined as the number of classes?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- DeepASA is capable of performing universal sound separation, sound event detection and direction-of-arrival estimation all within one model \n- DeepASA outperforms previous models in most metrics on the three datasets it was tested on\n- The proposed model is significantly more accurate with more than two foreground sources compared to the previous work on the MC-FUSS dataset\n- Detailed ablation study was done for all the proposed changes\n\n\nWeaknesses\n- DeepASA has significantly higher computational cost than DeFT-Mamba, the model it is based on despite often only achieving modest accuracy gains.\n- Chain-of-inference is defined as one of the two major contributions of the paper. But the results do not seem to warrant this focus. In fact, the biggest improvements over previous models can be seen in the MC-FUSS dataset where chain-of-inference is not even used at all.\n- Other than the MC-FUSS, the accuracy improvements of the DeepASA over previous work are more modest, a combination of many minor improvements adding up.", "was_revised": true, "final_justification": "The provided updated experiments, particularly regarding the computational cost of the proposed architecture made me increase the rating to borderline accept.", "rebuttal": "**Weak-Q1.** DeepASA has significantly higher computational cost than DeFT-Mamba, the model it is based on despite often only achieving modest accuracy gains.\n\n**A.** (1) Compared to DeFT-Mamba, DeepASA covers a wider range of tasks. It can perform separation, denoising, dereverberation, sound event detection, and DoA estimation for up to 5 sources. In the ASA2 dataset, it showed an improvement of 0.8 dB in SI-SDR and 0.7 dB in SDR compared to DeFT-Mamba.\n\n(2) When only performing USS, DeepASA has a lower computational load but higher performance than DeFT-Mamba. Initially, we incorrectly reported the computational cost on the MC-FUSS dataset. In the MC-FUSS dataset, only the audio decoder is used, and the other decoders are not utilized, which reduces the parameter size and computational complexity of DeepASA. \nTo ensure a fair comparison with DeFT-Mamba, we compared the performance of DeepASA trained from scratch on the MC-FUSS dataset without using the pre-trained ATST with the performance in Table 3. The experimental results are as follows: DeepASA without using a pre-trained ATST has a lower parameter size and computational complexity than DeFT-Mamba, but its performace improved for all source counts, even when it is trained from scratch. This is attributed to the exclusion of F-Mamba, which improves sound separation performance but has a high computational load, and the introduction of the proposed dynamic STFT and noise decoder. We will add these experimental results to the main paper.\n\nASA2 comparison between DeepASA with and without ATST\n|Model|SI-SDRi|SDRi|ER|F1|LE|RL|SELD|Param.|MAC/s|\n|-|-|-|-|-|-|-|-|-|-|\n|DeFT-Mamba|10.4|11.3||||||**3.6 M**|**83.8 G**|\n|DeepASA (w/o ATST)|11.0|11.7|30.1|69.5|18.5|74.8|0.240|8.2 M|91.0 G|\n|DeepASA (w/ ATST)|11.0|11.7|28.8|70.2|18.5|76.9|0.230|105.0 M|99.1 G|\n|DeepASA (w/ ATST, w/ CoI)|**11.2**|**12.0**|**25.0**|**74.1**|**17.0**|**78.1**|**0.206**|108.9 M|104.0 G|\n\nMC-FUSS comparison between DeepASA with and without ATST (SI-SDRi)\n(S denotes the number of sources in the mixture).\n|Model|S=2|S=3|S=4|Total|Param.|MAC/s|Runtime|\n|-|-|-|-|-|-|-|-|\n|DeFT-Mamba|18.4|17.1|13.8|16.4|3.6 M|83.8 G|39.2 min|\n|DeepASA (w/o ATST, w/o pre-train)|18.5|18.2|15.7|17.5|**2.9 M**|**77.3 G**|**26.8 min**|\n|DeepASA (w/ ATST, w/ pre-train)|**18.9**|**19.1**|**17.6**|**18.5**|**2.9 M**|**77.3 G**|**26.8 min**|\n\n**Weak-Q2.** Chain-of-inference is defined as one of the two major contributions of the paper. But the results do not seem to warrant this focus. In fact, the biggest improvements over previous models can be seen in the MC-FUSS dataset where chain-of-inference is not even used at all.\n\n**A.** The improvement on the MC-FUSS dataset is due to the comprehensive object separation and analysis introduced in deepASA. These include dereverberation, multichannel-to-multichannel (M2M) separation, dynamic STFT process, and noise estimation. These modifications brought the biggest improvements when only the USS task was performed. However, the improvement on the USS task does not mean the inefficacy of the chain-of-inference (CoI). In our ablation study with and without CoI, the model with CoI demonstrates +0.3 dB in the SDRi and -0.024 in a SELD score, compared to the model without CoI in the ASA2 dataset. Additionally, the model is able to surpass the existing models by using CoI in the STARSS23 dataset.\n\nASA2 dataset results compared with and without chain-of-inference and state-of-the-art models\n||SI-SDRi↑|SDRi↑|ER↓|F1↑|LE↓|LR↑|SELD↓|\n|-|-|-|-|-|-|-|-|\n|DeFT-Mamba|10.4|11.3||||||\n|MFF-EINV2|||42.1|43.2|25.8|60.7|0.381|\n|DeepASA(w/oCoI)|11.0|11.7|28.8|70.2|18.5|76.9|0.230|\n|DeepASA(w/CoI)|**11.2**|**12.0**|**25.0**|**74.1**|**17.0**|**78.1**|**0.206**|\n\nAdditionally, we applied the CoI to existing SELD models and observed performance improvements. Existing models do not employ object-oriented processing, so only the cross-attention from temporal coherence matching was applied to the existing models. The experimental results are shown in the table below, and since both models exhibited similar enhancements, this demonstrates the generalizability of CoI.\n\nInfluence of employing CoI to existing SELD models\n||Col|ER↓|F1↑|LE↓|LR↑|SELD↓|\n|-|-|-|-|-|-|-|\n|MFF-EINV2|X|42.1|43.1|25.8|60.7|0.381|\n||O|**38.7**|**47.2**|**22.1**|**63.5**|**0.351**|\n|SELD-Mamba|X|43.5|42.7|25.5|56.7|0.396|\n||O|**40.1**|**46.6**|**22.7**|**61.1**|**0.363**|\n\n**Weak-Q3.** Other than the MC-FUSS, the accuracy improvements of the DeepASA over previous work are more modest, a combination of many minor improvements adding up.\n\n**A.**\n- In terms of performance, the benchmark for the STARSS23 dataset in Table 4 of the DCASE challenge permitted the use of pre-trained models and external large datasets, as well as data augmentation, resulting in each system using different training data and pre-trained models. Consequently, in Table 8 of the Appendix, the performance of SOTA models for the sound event localization and detection (SELD) task without the use of external data or data augmentation is listed. The comparison with the performance of DeepASA without using the pre-trained ATST can be found in the reply to the previous comment (Weak-Q1-A). Ultimately, DeepASA achieved a 0.6 dB improvement over the best performance of previous works in terms of SI-SDRi and a 0.141 improvement in SELD score compared to the best performance of previous works in a fair comparison. This can be considered a sufficiently significant gap.\n- Also, the main contribution of DeepASA is not its exceptionally high performance on a specific task but rather its ability to enhance performance across various tasks by learning multiple auditory information (class, timestamp, and DoA) through several sub-decoders to separate object features. In other words, DeepASA contributes not only in terms of model architecture but also in its training methodology. From a model structure perspective, it improves efficiency by excluding the computationally intensive F-Mamba and allows for the learning of time resolution and frequency lobes over time through dynamic STFT. Additionally, it enhances the estimation performance of individual auditory information through cross-referencing multiple auditory information sources, similar to human processing. The DeepASA and chain-of-inference structure is a novel architecture not previously proposed.\n\n**Question-Q1.** In lines 307-308 you write \"This performance gain is largely attributed to the noise decoder’s explicit estimation of background noise and the benefits of fine-tuning from pre-trained weights.\"\n\n**A.** Estimating background noise is essential because, when only the foreground source is separated, noise often mixes into the inactive source, which should be estimated as silence. To prevent this, noise separation is explicitly learned. A similar approach based on loss functions has existed in speech enhancement [**]. Specifically, by estimating noise, the sum of the foreground source and noise is constrained to be the audio mixture, which indirectly constrains the phase of the estimated foreground sources. This helps reduce artifacts that occur when using loss in the STFT domain. This approach also contributes to improving classification performance in this paper.\n\n[**] A. Pandey, D.L. Wang, \"Dense CNN with self-attention for time-domain speech enhancement.\" IEEE/ACM Trans. Audio, Speech, Lang. Process., 29, pp. 1270 – 1279, 2021.\n\n**Question-Q2.** Figure 2 shows how the channel dimension is increased from 2M to D but the text (line 114) claims it increases from 2M to C. Is that just a mistake, given that C is later (line 162) defined as the number of classes?\n\n**A.** Thank you for the reviewer’s comment. We defined C as the number of classes, and it needs to be D. We will change “from 2M to C” into “from 2M to D.”\n\n**Paper Formatting Concerns.** This is a minor nitpick, but many metrics are only named by their abbreviations (SI-SDRi, SDRi, SA-SDR) and should be defined the first time they are named, e.g. scale-invariant signal-to-distortion ratio improvement (SI-SDRi), etc. The same is true for D, T' and C Figure 2. It is not immediately obvious what they are supposed to be from the Figure alone. The description of Table 3 should also mention it is the SI-SDRi in dB which is reported.\n\n**A.** Thank you for pointing that out. We will define the first time the evaluation metrics are named. (scale-invariant signal-to-distortion ratio improvement (SI-SDRi), signal-to-distortion ratio (SDR), source-aggregated signal-to-distortion ratio (SA-SDR). As the reviewer commented, we will define D, T’, and C in the caption of Figure 2. We will mention that SI-SDRi is reported on the dB scale."}]

Author rebuttal:

Meta-review: This paper proposes DeepASA, which trains a single deep network to perform multiple audio scene analysis tasks, including source separation, dereverb, sound event detection, audio tagging, and direction-of-arrival estimation. This is realized by first separating the mixtures (Object-Oriented Processing strategy), creating hidden representations for each sound object, and then using various decoders on the hidden representations to estimate auditory parameters, with further refinement achieved by a Chain-of-Inference mechanism. Through extensive discussions the reviewers are happy with the method and results presented in this paper.
Decision: Accept (poster)