Title: DeepASA: An Object-Oriented Multi-Purpose Network for Auditory Scene Analysis
Venue: NeurIPS 2025
Accepted: yes
Keywords: audio signal processing

auditory scene analysis

universal sound separation

object-oriented processing

chain-of-inference
Abstract: We propose DeepASA, a multi-purpose model for auditory scene analysis that performs multi-input multi-output (MIMO) source separation, dereverberation, sound event detection (SED), audio classification, and direction-of-arrival estimation (DoAE) within a unified framework. DeepASA is designed for complex auditory scenes where multiple, often similar, sound sources overlap in time and move dynamically in space. To achieve robust and consistent inference across tasks, we introduce an object-oriented processing (OOP) strategy. This approach encapsulates diverse auditory features into object-centric representations and refines them through a chain-of-inference (CoI) mechanism. The pipeline comprises a dynamic temporal kernel-based feature extractor, a transformer-based aggregator, and an object separator that yields per-object features. These features feed into multiple task-specific decoders. Our object-centric representations naturally resolve the parameter association ambiguity inherent in traditional track-wise processing. However, early-stage object separation can lead to failure in downstream ASA tasks. To address this, we implement temporal coherence matching (TCM) within the chain-of-inference, enabling multi-task fusion and iterative refinement of object features using estimated auditory parameters. We evaluate DeepASA on representative spatial audio benchmark datasets, including ASA2, MC-FUSS, and STARSS23. Experimental results show that our model achieves state-of-the-art performance across all evaluated tasks, demonstrating its effectiveness in both source separation and auditory parameter estimation under diverse spatial auditory scenes. The demo video, samples and code are available at https://huggingface.co/spaces/donghoney22/DeepASA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DeepASA, a new model for auditory scene analysis, taking in a multichannel audio mixture capable of performing source separation, sount event detection, audio classification and direction-of-arrival estimation. The DeepASA model itself is based on the DeFT-Mamba model.\nThe two core contributions are the chain-of-inference architecture and object oriented processing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- DeepASA is capable of performing universal sound separation, sound event detection and direction-of-arrival estimation all within one model \n- DeepASA outperforms previous models in most metrics on the three datasets it was tested on\n- The proposed model is significantly more accurate with more than two foreground sources compared to the previous work on the MC-FUSS dataset\n- Detailed ablation study was done for all the proposed changes\n\n\nWeaknesses\n- DeepASA has significantly higher computational cost than DeFT-Mamba, the model it is based on despite often only achieving modest accuracy gains.\n- Chain-of-inference is defined as one of the two major contributions of the paper. But the results do not seem to warrant this focus. In fact, the biggest improvements over previous models can be seen in the MC-FUSS dataset where chain-of-inference is not even used at all.\n- Other than the MC-FUSS, the accuracy improvements of the DeepASA over previous work are more modest, a combination of many minor improvements adding up.\n\nQUESTIONS:\n1. In lines 307-308 you write \n>This performance gain is largely attributed to the noise decoder’s explicit estimation of background noise and the benefits of fine-tuning from pre-trained weights.\n\nThis is something I've wondered about when I saw Figure 2; why exactly is background noise estimated and how would it contribute to an improvement in estimating the foreground sources?\n\n2. Figure 2 shows how the channel dimension is increased from $2M$ to $D$ but the text (line 114) claims it increases from $2M$ to $C$. Is that just a mistake, given that $C$ is later (line 162) defined as the number of classes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DeepASA, a unified framework for auditory scene analysis that performs multichannel source separation, dereverberation, sound event detection, audio classification, and direction-of-arrival estimation in one model. The core contributions are: (1) an Object-Oriented Processing (OOP) strategy, which isolates per-object audio features early in the pipeline via a transformer-based aggregator and object separator, and (2) a Chain-of-Inference (CoI) mechanism that iteratively refines object features by fusing sub-decoder outputs through temporal-coherence attention. Evaluated on three benchmarks (ASA2, MC-FUSS, STARSS23), DeepASA achieves state-of-the-art performance across all tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Ablations isolate each module’s contribution; end-to-end performance on multiple datasets is compelling.\n2. Real-world examples on the demo webpage lend practical credibility.\n3. Well-designed multi-decoder architecture eliminates permutation issues and enables complementary cue integration.\n\nWeaknesses:\n1. Figures 2 & 5 (model diagram & CoI) are dense and lack intuitive legends. Readers may struggle to follow data flow. I think a step-by-step schematic or callouts would help.\n2. Dynamic STFT (Sec. 2.2) is novel but under-explained: it would be clearer with visualizations of learned windows over time and ablations on σₜ initialization.\n3. The choice of maximum object count J is not justified: how sensitive is performance if fewer/more sources appear?\n4. Computational cost of the ATST-based SED decoder is high. It would be great to discuss the runtime and efficiency trade-offs.\n5. Related Work (Sec. 5) omits object-centric learning: key works like \"Slot Attention\" (Locatello et al., 2020), \"Sound of Pixels\" (Zhao et al., 2018), \"Objects That Sound\" (Arandjelović et al., 2017), \"SSV2A\" (Guo et al., 2024), \"AVObject\" (Li et al., 2025), are not discussed. The distinction between those and DeepASA’s OOP should be clarified.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors discuss the ATST size but could further elaborate on dataset biases (e.g., reverberation conditions) and potential misuses (e.g., surveillance).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors present DeepASA as a unified framework for acoustic scene analysis, adopting an Object-Oriented Processing approach that separates object-centric features at an early stage to avoid permutation ambiguity, and the Chain-of-Inference mechanism that iteratively refines predictions using temporal coherence matching between different auditory cues\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThis paper addresses a fundamental challenge of comprehensive scene analysis, maintaining consistent object-wise associations across all sub-decoders task heads, as opposed to prior track0wise processing systems.\n\nThe motivation for object separator, CoI, as well as dynamic stft is clearly articulated and supported by ablation evidence. Temporal coherence matching is also shown to provide measurable improvements.\n\nWeakness:\nThe overall model, even with pre-trained AST is notably bulky compared to task-specific approaches and is a concern for practical deployment. Additionally, given the multistage training, can authors provide an analysis of training convergence properties and sensitivity to initialization. Are there convergence guarantees for the chain-of-inference mechanism? I believe the 97M AST limits the novelty for SED component + is a computational bottleneck.\n\nWith regards to ASA2 dataset, the balanced distribution proportional to source count and increased maximum sources from 4 → 5 compared to prior versions suggests substantial modifications that could favor the proposed approach. Overall paper can benefit from highlighting some potential biases in ASA2\n\nForeground sources are fixed to J=5, however, some analysis regarding possible performance degradation when this assumption is violated would be helpful. Some analysis varying the degree of polyphonicity is expected. What is the computational and memory scaling behaviour?\n\nMinor weakness:\nThe relationship between different loss components and their relative weighting could be better motivated in the main paper than in the appendix.\n\nQUESTIONS:\nCould authors provide evidence for true object correspondence? .. some validation that object features actually correspond to individual sound sources rather than just being abstract encoded representations.\n\nIs it possible to show that each of the J=5 object features consistently tracks the same sound source across time? When sounding objects enter and exit a scene, are object features able to maintain semantic coherence?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DeepASA, which trains a single deep neural network (DNN) to do source separation, dereverb, sound event detection, audio tagging, and direction-of-arrival (DOA) estimation.\n\nThis is realized by first implicitly separating the mixtures, creating hidden representations for each sound object, and then using various decoders on the hidden representations to estimate auditory parameters such as sound event and DOA.\n\nEvaluation results on their own customized training and test dataset (i.e., ASA2) and two public datasets (MC-FUSS and STARSS23) show that the proposed method can produce strong performance for sound separation and auditory parameter estimation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nDeepASA can simultaneously deal with multiple tasks, and obtains reasonable improvement over existing methods.\n\nWeaknesses:\n\n1.\tTitle, “one-for-all” is not accurate, as auditory scene analysis contains many more tasks, while DeepASA only addresses a subset of them. In addition, the considered combination of tasks is not so novel, as compared with existing SELD tasks (which already deal with SED, audio tagging and DOA estimation simultaneously), DeepASA only introduces separation.\n\n2.\tThe evaluation is only performed on simulated datasets, and lacks convincing evaluation results on real-recorded datasets for sound separation. On real-recorded data, sound separation is usually difficult, as real data often mismatches simulated data. In this case, would the method still perform well, considering that the method relies so much on obtaining good hidden representations for each sound object (i.e., if the hidden representations for each sound object is not good on, e.g., real-recorded data, then auditory parameter estimation would be also not good)?\n\n3.\tSection 2.2, better to use “analysis window” to replace the word “window”, as “window” is overloaded with many meanings in signal processing.\n\n4.\t“SED decoder” in Section 2.4 and Figure 4(b) feel messy and are difficult to follow. The writing needs improvement.\n\n5.\tMany details are missing regarding the model. See my detailed questions to the authors.\n\n6.\tThe comparison with existing models in Table 3 and 4 are not so fair, as they are trained on different sets of data, and DeepASA uses the ATST model (which is pre-trained on a very large dataset). It might not be surprising that DeepASA can obtain better performance. See also my detailed questions.\n\nQUESTIONS:\n1.\tIn Section 2.2, how is the synthesis window in inverse STFT computed?\n\n2.\tLine 146, what is “a single-channel object activation (onset and offset) curve”. From Figure 4(b), its dimension is JxT’ (without dimension C, so it is a label for all the sources combined). Is it a binary vector for each source, denoting when each source is active? It is really difficult to understand and this makes the reviewer unable to fully follow the “SED decoder” part, please explain and improve.\n\n3.\tLine 150, “During evaluation, only the predicted class probability and object activation curve are used to calculate the SELD scores.” Can the authors explain how? The predicted class prob has a dimension of JxC, and the object activation curve has a dimension of JxT’, according to Figure 4(b).\n\n4.\tLine 152, “The SED decoder combines a pre-trained audio teacher-student transformer (ATST) [12] with two separate convolutional recurrent neural network (CRNN) [11]-based branches summarizing time and frequency information.”, how is CRNN reflected in Figure 4(b)? The reviewer only notices AIST, but not CRNN. Are all the compared benchmark systems using the AIST model, which is trained on AudioSet, a much larger scale dataset than ASA2, MC-FUSS and STARSS23? If not, the comparisons in the experiments may not be fair at all.\n\n5.\tLine 114, “from 2M to C” should be “from 2M to D”? as C is defined as the number of sound types the model can handle.\n\n6.\tFigure 5, what is symbol “S”? Why the embedding after SED Decoder 1 is SC and why the embedding after DoA Decoder 1 is 3S? Are there any motivations of choosing the embedding dimension?\n\n7.\tFigure 5, in SED query branch, the attention output should be 3SxT’? Then why is it followed by a linear layer SC->D, where the dimension does not match.\n\n8.\tFigure 4(b), “SED decoder” contains multiple outputs including estimated activations, predicted classes and predicted SED. In Figure 5, “SED Decoder 1” outputs a SCxT’ tensor. How is this performed? \n\n9.\tLine 197, is ASA2 built upon the dataset proposed in [38]. From Section 3.1, this is not clear.\n\n10.\tLine 211, “The initial learning rate 4 ×10−4 was used for the Adam optimizer during pre-training, and 4 ×10−4 and 5 ×10−5 were used for fine-tuning to the MC-FUSS and STARSS23 datasets, respectively.” How is pre-training performed? So far, the authors talk nothing about pre-training. Do the authors first train DeepASA on ASA2, and then fine-tune the trained model on MC-FUSS and STARSS23? If this is the case, are all the compared benchmarks using similar ways and trained on the same training data?\n\n11.\tLine 221, what is “timestamp decoder”? this is a new thing that has not been described. It is not shown in Figure 4(b).\n\n12.\tLine 221, any motivations on using MSE loss for sound event detector?\n\n13.\tIn Figure 4(b), what loss function is used for the audio decoder part? PIT? There is no description, and this is an important detail.\n\n14.\tSection 2.5, SED decoder has 3 outputs, which themselves may not be aligned with each other. Would this be possible and if so, how is this solved? and, in TCM, how the authors align DOA results with each of the 3 outputs in the SED decoder?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-09"}