SelM: Selective Mechanism based Audio-Visual Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 06:32:30

Abstract: Audio-Visual Segmentation (AVS) aims to segment sound-producing objects in videos according to associated audio cues, where both modalities are affected by noise to different extents, such as the blending of background noises in audio or the presence of distracted objects in video.Most existing methods focus on learning interactions between modalities at high semantic levels but is incapable of filtering low-level noise or achieving fine-grained representational interactions during the early feature extraction phase. Consequently, they struggle with illusion issues, where nonexistent audio cues are erroneously linked to visual objects.In this paper, we present SelM, a novel architecture that leverages selective mechanisms to counteract these illusions. SelM employs State Space model for noise reduction and robust feature selection. By imposing additional bidirectional constraints on audio and visual embeddings, it is able to precisely identify crutial features corresponding to sound-emitting targets.To fill the existing gap in early fusion within AVS, SelM introduces a dual alignment mechanism specifically engineered to facilitate intricate spatio-temporal interactions between audio and visual streams, achieving more fine-grained representations. Moreover, we develop a cross-level decoder for layered reasoning, significantly enhancing segmentation precision by exploring the complex relationships between audio and visual information.SelM achieves state-of-the-art performance in AVS tasks, especially in the challenging Audio-Visual Semantic Segmentation setting.Source code will be made publicly available.

Main paper: 
 introduction: 
 Audio-Visual Segmentation (AVS) is an emerging multimedia technology with wide applications, such as video editing, industrial maintenance, and surveillance. It aims at leveraging audio cues to enable models to locate sound-producing objects and generate pixel-level segmentation masks. Introduced by AVSBench [56, 57] , this task encompasses three settings: Semi-supervised Single Sound Source Segmentation (S4), Fully-supervised Multiple Sound Source Segmentation (MS3), and Audio-Visual Semantic Segmentation (AVSS), covering a spectrum of segmentation complexities.In the AVS task, video provides explicit spatial information, while audio reveals changes in sound over time. For instance, if the model detects human vocals suddenly emerging within a piano accompaniment through the audio, then the model should accordingly segment both the singer and the piano. Similarly, upon observing guitar strings being plucked in the video, it should accordingly segment the guitar. Therefore, the comprehensive integration of both modalities is crucial for accurately inferring the location of entities. Current methods [8, 29, 32, 38, 57] predominantly focus on developing more effective auditory-visual information fusion methods. For example, AVSBench [57] integrates the TPVAI module to facilitate interaction between Audio-Vidual features, while CATR [29] introduces a decoupled Audio-Visual Transformer for integration across both spatial and temporal dimensions. From another perspective, some researches [18, 39, 57] explores the diversity of feature representation. AVSBench employs regularization to reduce the distance between modal features, narrowing the domain gap. One representative study is ECMVAE [39] , which utilizes multiple regularizations to decompose features into shared and specific components, achieving more robust feature representations.Despite significant advancements in quantitative evaluation metrics achieved by current methods, extensive visualizations unveil 'hallucination' as depicted in Figure 1 . Despite the performers do not sing, current methods are influenced to different degrees and erroneously segment the silent objects. We term this phenomenon the Auditory Illusions issue, highlighting that in existing methods, audio fails to play a decisive role in entity selection. We identify two reasons: (1) Both audio and video contain considerable noise, and when multiple sound sources appear in the same scene, key information overlaps, challenging the model's ability to decouple different sound-emitting entities, thereby misleading its segmentation. (2) Although current methods employ feature fusion or regularization, they predominantly utilize intermediate or late fusion techniques, as depicted in Figure 2 . Given the substantial disparities between the modalities, previous approaches result in a deficiency of nuanced fusion and fine-grained representations.To address the first issue, we consider leveraging the selective mechanism of Mamba [11] to suppress noise. Mamba, renowned for its powerful long-sequence modeling capability and computational efficiency, garners widespread attention within the community. Its reliable modeling prowess stems from the selective mechanism that compresses redundant representations and noise, isolating key and robust features. This capability is particularly suited to the AVS task, prompting our use of Mamba coupled with the design of a bidirectional conditional constraint for further cross-modal cue selection. For the second issue, beyond facilitating high-level information exchange, we also propose the dual alignment approach between the two encoders to achieve early fusion. The differences between the previous method and ours are demonstrated in Figure 2 (a) and (b). Numerous multimodal studies [7, 10, 42, 54] highlight the benefits of early fusion, such as the capability to overcome domain gaps and generate more detailed representations. Thus, for the first time in the context of AVS tasks, we introduce a dual alignment module as an early fusion strategy to enhance the information diversity and uniformity of the feature distributions.Specifically, similar to AVSBench [57] , we utilize two encoders to extract features from video and audio. Between these encoders, we design a Dual Alignment Module (DAM) for early fusion, facilitating fine-grained interaction. The encoded features are then directed to a Bidirectional Conditioned Selective Mechanism Module (BCSM) for the selection of relevant information and the imposition of bidirectional constraints. To ensure a comprehensive understanding during the decoding phase, we devise a Cross-LEVEl Reasoning (CLEVER) decoder that alternately performs high-level fusion and decoding of Audio-visual information, which yields the segmentation mask. Lastly, the auxiliary loss is applied to the cross-attention map of the decoder to further refine segmentation quality.In summary, our contributions are threefold:• We employ a selective mechanism to filter out noise from features, using bidirectional constraints to select spatiotemporal information relevant to the sound-emitting entities, addressing the issue of Auditory Illusions.we propose an elegant alignment module dedicated to early fusion, cultivating more nuanced and enriched representations. This innovation fills a crucial gap in the current fusion methodologies for AVS tasks. • We introduce an cross-level reasoning decoder that alternately merges and analyzes information from both modalities, yielding precise segmentation results. 
related work 2.1 audio-visual segmentation: 
 The purpose of Sound Source Localization (SSL) is to identify objects within a video that produce sound. Early efforts [1, 2, 5, 23, 24, 24, 33, 44, 44, 45, 47] in this domain remain at the patch level due to the lack of fine-grained annotations. Recent research [57] introduces the Audio-Visual Segmentation benchmark with pixel-level annotations, capturing considerable attention from communities. A considerable portion of the latest studies [18, 31, 39] focuses on effectively integrating audio-visual information. For example, AVS-Bench [57] proposes the TPAVI module for multimodal information integration, incorporating regularization terms into optimization objectives to bridge the gap between audio and visual features. AVS-BG [18] utilizes a bidirectional generation manner to reinforce the audio-visual relationship through maintaining cycle consistency. ECMVAE [39] divides the features of the two modalities into shared and specific knowledge to enhance representation diversity. CATR [29] introduces a decoupled audio-visual transformer, taking audio features and learnable embeddings as queries. AVSC [31] employs a two-stage paradigm consisting of instance segmentation followed by sound source classification to alleviate ambiguity. Our research aims to address the problem of datasets bias-induced Auditory Illusions, effectively leveraging audio information and preventing model degradation.The State Space models, originating from classical control theory [27] , is introduced into the field of deep learning to model dependencies in long sequences. Recent advancements in the State Space model are driven by a variety of works [12-15, 17, 19, 35, 40, 48] that focus on refining its computational efficiency and analytical robustness. Notably, there is a particular focus on Mamba [11] , which introduces a selective mechanism to enhance the model's information compression capabilities, while leveraging hardware-aware algorithms to accelerate computation. Due to its input-dependent parameters, Mamba demonstrates formidable capabilities in modeling long sequences and possesses linear scalability properties. A wide array of research [16, 25, 34, 43, 58] harnesses Mamba for the extraction of hierarchical features in images, addressing directional sensitivity by augmenting the model with varied scanning directions. Mamba even shows promising performance in point After alignment, a selective mechanism is applied for noise suppression and robust feature representation, with bidirectional constraints yielding information pertinent to the sound-emitting objects. (c) In the decoding phase, learnable queries alternately interrogate both modalities, culminating in the predicted segmentation results.cloud analysis [30] and generative tasks [4, 25] . Audio-Visual Segmentation (AVS) depends crucially on the careful selection and highlighting of visual segments that match specific audio cues, ensuring segmentation targets only relevant data from both modalities. Consequently, delving into Mamba's selective mechanism for AVS tasks presents an intriguing idea. To our knowledge, this marks the first attempt to refine Mamba's selection mechanism for AVS tasks, indicating a new direction in this field of study.Integrating features and aligning latent representations across modalities are key for multimodal tasks [3, 7, 10, 26, 42, 50, 52, 54] . Early fusion, the process of combining modalities before processing, leverages all available data to effectively capture intermodal correlations. Building on this foundation, intermediate fusion takes place at a model's mid-stage, where it merges features to enhance the understanding of complex data relationships. Finally, late fusion, applied at the decision level, ensures modality independence by integrating insights only after individual analyses, thereby maintaining the distinctiveness of each data source. In the AVS field, both intermediate and late fusion techniques are widely used. TPAVI [57] leverages intermediate fusion to enhance audio-visual interactions, while AVSegFormer [8] applies late fusion, integrating audio-visual tokens in its decision stage to capitalize on the strengths of each modality. This research builds upon existing concepts while innovatively integrating early fusion to bridge the domain gap between auditory and visual pre-training. By doing so, it aims to achieve comprehensive multimodal integration and understanding at every stage. 
preliminaries: selective mechanism: 
 State Space Models (SSMs) [19, 35, 40, 48] are usually employed as linear time-invariant systems to transform the input 𝒙 (𝑡) ∈ R 𝑀 to the output 𝒚(𝑡) ∈ R 𝑀 via a hidden state 𝒉(𝑡) ∈ R 𝑁 . Mathematically, this system can be formulated as a linear ordinary differential equation:EQUATIONwhere 𝑨 ∈ R 𝑁 ×𝑁 is the state transition matrix, 𝑩 ∈ R 𝑁 ×1 and 𝑪 ∈ R 1×𝑁 are projection matrices, and 𝐷 ∈ R 1 denotes the skip connection. Considering that the continuous-time models cannot be directly applied to deep learning algorithms, the structured state space sequence models and Mamba [11] employ the principle of zero-order hold (ZOH) for discretization, resulting in the following formula:EQUATIONwhere the continuous parameters 𝑨 and 𝑩 are discretized into 𝑨 and 𝑩 through a time scale parameter △. After discretization, Eq. ( 2) can be written as follows: The final output 𝒚(𝑡) can be computed through a convolution, directly establishing a relationship with the input 𝒙 (𝑡). The formula is as follows:EQUATIONEQUATIONwhere 𝑲 ∈ R 𝑁 is a structured convolutional kernel. Based on this, Mamba [11] further introduces an input-dependent modeling approach, and the selective mechanism can be expressed as follows:EQUATIONwhere 𝑠 𝐵 (•), 𝑠 𝐶 (•), and 𝑠 𝐴 (•) are linear projections. 𝜏 𝐴 (•) denotes the Softplus activation function. 
method: 
 This section starts by introducing the overall architecture of the proposed Selective Mechanism-based AVS method (SelM), followed by elaborating its core components.The proposed SelM is built upon the Selective Mechanism to select and align the representative features of visual and audio inputs. It mainly consists of two feature encoders, a Dual Alignment Module (DAM), a Bidirectional Conditioned Selective Mechanism Module (BCSM), and an Audio-visual Cross-LEVEl Reasoning decoder (CLEVER). As shown in Figure 3 , given the video frames and corresponding audio clip, they are first fed into the visual encoder and audio encoder, respectively, to obtain the multi-stage features. Meanwhile, the DAM is applied to align and fuse the audio and visual features gradually for all the stages. Then, BCSM is designed to model the spatial and temporal information, and filter out the feature noise from audio and visual in a bidirectional manner. Finally, the filtered multi-modal features are fed into CLEVER to obtain the final segmentation result. To make a fair comparsion, we follow the prior methods [22, 57] by employing ResNet50 [20] or Pyramid Vision Transformer (PVT-v2) [51] as the visual encoder and VGGish [22] as the audio encoder. In the following, we will describe the core components in detail.Given the multi-stage audio and visual features extracted by the encoders, the Dual Alignment Module for Early Fusion (DAM) is designed for cross-modal feature alignment and integration. Different from the conventional works that only employ late fusion for audio and vision features at the high semantic level, we propose a dual alignment manner and apply it at multiple feature stages to achieve sufficient integration between the two modalities. While the two modalities of data have different distribution characteristics, they exhibit strong semantic correlations and complementarity. Therefore, DAM is designed to comprise two symmetric subblocks named V2ABlock and A2VBlock, to enhance the relevant information from audio and vision features upon each other, respectively.To simplify the description, we will focus on providing a detailed description of the A2VBlock. The structure of the V2ABlock is identical to the A2VBlock, with the only difference being the interchange of data modalities between audio and video. As shown in Figure 3 (a), given the audio tokens 𝑻 and the visual feature 𝑭 from the same stage of encoders, A2VBlock aims to enhance the representative information from the visual features according to the audio tokens. Therefore, we design to learn an alignment weight map 𝑾 upon the audio-visual features. Specifically, both the audio tokens and visual features are fed into convolution branches, respectively, and their outputs are element-wise multiplicated to produce the weight map, which is further utilized to re-weight the visual feature and output the enhanced version 𝑭 𝒂 . In our designs, Conv1D is adopted as the convolutional layer, followed by a GELU [21] activation function and a dropout [49] operation to introduce non-linearity and prevent overfitting.To ensure the model achieves thorough understanding and alignment across all levels, DAM is applied at all the feature stages, allowing a seamless harmonization of features between the two branches. Compared with the existing AVS works, our method adopting the proposed DAM enjoys at least two benefits. First, it utilizes both low-level fine-grained and high-level semantic features, offering comprehensive knowledge about the segmentation target. Second, with the light-weight DAM, the visual and audio information are bilaterally and gradually enhanced and aligned, bringing more reliable representations of the target features.To investigate whether source signals significantly impact model decisions, we visualize the predicated segmentation maps of the existing methods, which show a Auditory Illusions issue. As demonstrated in Figure 1 , models erroneously segment out the non-sounding performers. This indicates that models tend to take shortcuts by memorizing the segmented objects associated with specific scenes rather than fully utilizing audio information. The issue is challenging to settle. First, the input data from two modalities are complex and contain noise, making it challenging to decouple the key information, especially in multi-source scenarios where critical cues overlap. Second, video frames and audio signals change over time, and they struggle to perform mutual selection effectively. Based on the above analyses, we introduce the Bidirectional Conditioned Selective Mechanism Module (BCSM). Details can be found in Figure 3(b) . Specifically, we employ the Mamba [11] to suppress the irrelevant information of the given input. The design of Mamba incorporates a selection mechanism that emphasizes robust feature representation while filtering out noise and redundant information, thereby making it highly suitable for the AVS task. In addition, we design a bidirectional constraint that discerns relevant information across both modalities.As shown in Figure 3 (b), the BCSM consists of two symmetric branches. Taking the upper branch as an example, the aligned video feature 𝑭 𝒂 is transformed into sequences through average pooling, then processed through V2A-SeM which stacks three consecutive Mamba blocks for filtering out irrelevant information. This produces high-level weight maps which contain appearance cues about sound-emitting objects for selecting audio tokens by element-wise multiplication. The process mirrors in the lower branch, with the note that audio tokens bypass pooling operation. For a Mamba block, it comprises a vanilla Mamba equipped with state space model principles (recommended in Section 3 and Mamba [11] ), supplemented by residual connections and normalization operations. Overall, A2V-SeM and V2A-Sem are utilized for denoising within each modality, further refined by bidirectional constraints for feature selection. Ultimately, after processing through BCSM, we obtain the selected 𝑭 𝒔 and 𝑻 𝒔 .Existing approaches [31] utilize the FPN [28] decoder, a single-level decoding technique. They usually combine audio tokens and video features via a TPAVI [57] module or other feature fusion module to create a mixed audio-visual representation. The combined tensors are directly used as inputs to the decoder without any discrimination.To facilitate a comprehensive understanding and interaction of audio-visual information at various levels, we design the Cross-LEVEl Reasoning (CLEVER) decoder, depicted in Figure 3 (c) . CLEVER is structured around three identical Audio-Visual (AV) Transformer layers, where each layer sequentially processes through audio crossattention followed by video cross-attention mechanisms. This layered operation marks our primary distinction from previous singlelevel decoders. While single-level decoders may utilize multi-scale features, they do not engage in alternate reasoning between different modalities.Specifically, for an AV Transformer layer, the input consists of two parts: the selected audio tokens 𝑻 𝒔 𝒊 and the video features 𝑭 𝒔 𝒊 . The outputs of the first two layers are concatenated with the subsequent 𝑭 𝒔 𝒊+1 , forming mixed features that are fed into the next AV Transformer layer. Each AV transformer layer comprises two sequentially arranged cross-attention modules. For audio crossattention, selected tokens 𝑻 𝒔 serve as keys and values, while for video cross-attention, the selected features 𝑭 𝒔 assumes these roles. In the same AV Transformer layer, learnable tokens are shared between two cross-attention modules as queries. Through alternate reasoning and comprehension, the queries are imbued with meaning related to the sound-producing entities. In the third layer, we perform a dot product between the learnable tokens of outputs and mixed video features to produce the final estimated masks.For the S4 and MS3 settings, we employ a synergistic blend of DICE [41] and BCE (Binary Cross Entropy) loss functions, further augmented by an auxiliary loss component. This integrated approach is formalized as follows:EQUATIONIn this formulation, the DICE [41] and standard BCE losses are leveraged to quantify the similarity between the estimated masks produced by the model and the ground truth. The auxiliary loss introduces an additional constraint on the intermediate layers of the CLEVER decoder. Specifically, the auxiliary loss is defined by the following equation:EQUATIONwhere ℓ 𝑖 represents the DICE loss between the upsampled heat map of the 𝑖 𝑡ℎ layer and the ground truth. The term 𝛼 𝑖 denotes the associated weight for the 𝑖 𝑡ℎ layer. For our implementation, we empirically set the weights 𝜶 1 , 𝜶 2 , and 𝜶 3 to 0.001, 0.01, and 0.1, respectively.For the AVSS setting, we forego the application of DICE loss, opting instead to utilize the second and third components delineated in Equation 6. 
experiments 5.1 implementation details: 
 5.1.1 Datasets. We validate our ideas across three settings: singlesound and multi-sound under AVSBench-object [57] , and AVSBench-Semantic [56] . AVSBench-Object targets audio-visual segmentation with two subjects -Semi-supervised Single-sound Source Segmentation (S4) with 4932 videos, and Fully-supervised Multisound Source Segmentation (MS3) with 424 videos -each offering pixel-level segmentation across 23 categories. The 2023 expansion, AVSBench-Semantic (AVSS), enhances AVSBench-Object with semantic labels and 10-second clips, broadening the dataset for audiovisual semantic segmentation research.. We train our model on an NVIDIA Tesla A100 using PyTorch, employing the AdamW optimizer with a initial learning rate of 2 × 10 -5 . The batch size is set at 4, with training durations of 40 epochs for the S4 setting, 100 epochs for the MS3 setting, and 30 epochs for the AVSS setting. Unless specifically mentioned, video frames are uniformly resized to 224 × 224. For the audio and visual encoders, we adopt the same configuration as used in previous works [29, 57] .We employ the Jaccard index 𝑀 𝐽 and F-score 𝑀 𝐹 as our primary evaluation metrics. The 𝑀 𝐽 metric evaluates the segmentation accuracy by measuring the overlap between predicted and ground truth areas. Meanwhile, 𝑀 𝐹 provides a combined measure of precision and recall, which is formulated as follows:𝑀 𝐹 = 1 + 𝛽 2 × 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑟𝑒𝑐𝑎𝑙𝑙 𝛽 2 × 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑟𝑒𝑐𝑎𝑙𝑙 (8)where 𝛽 2 is set to 0.3 in our experiments, as in [57] . 5.2 Comparison with SOTA Methods.To ensure a fair comparison, consistent with AVSBench [57] , we employ ImageNet-1K [46] pretrained ResNet-50 [20] or PVT-v2 [51] as the backbone for extracting visual features, alongside AudioSet [9] pre-trained VGGish [22] for audio feature extraction. We compare AVS-related research and SOTA methods within three settings: Semi-supervised Single Sound Source Segmentation (S4), Fully-supervised Multiple Sound Source Segmentation (MS3), and Audio-Visual Semantic Segmentation (AVSS). The related body of work includes sound source localization (LVS [5] and MSSL [44] ), video object segmentation (3DC [36] and SST [6] ), and salient object detection (iGAN [37] ), each sharing some degree of relevance with AVS tasks. Table 1 showcases our performance on the AVSS setting. Notably, SelM outperforms related works (3DC [36] and AOT [53] ) and existing methods (AVSBench [57] , CATR [29] ), comprehensively surpassing the current best method, AVSegFormer [8] , regardless of whether ResNet-50 or PVT-v2 is used for video feature extraction. For both 𝑀 𝐽 and 𝑀 𝐹 metrics, our approach achieves approximately an 11% performance increase over the baseline model AVSBench [56] . Table 2 presents the performance of SelM on the S4 and MS3 settings, where we slightly outperform current methods (AVS-BG [18] , AUTR [32] , AVSC [31] , CATR [29] , DiffusionAVS [38] , ECMVAE [39] , AVSegFormer [8] , etc.) using PVT-v2 as the feature extractor, and we alse realize comparable or slightly superior performance with ResNet-50. It is particularly noteworthy that the AVSS task, with its more complex and varied sound sources and greater number of categories, is the most challenging. This underscores the capability of SelM to discern sound source information and achieve superior segmentation results.To showcase SelM's effectiveness in mitigating Auditory Illusions, we display comparative visualization results from AVSBench [57] , AVSegFormer [8] , and SelM. Figure 4 presents examples from the MS3 and AVSS settings, which involve multiple sound sources. For additional visualizations related to the S4 setting, please refer to the supplementary materials. It can be observed that previous methods struggle with Auditory IllusionsMethod Backbone S4 MS3 𝑀 𝐽 ↑ 𝑀 𝐹 ↑ 𝑀 𝐽 ↑ 𝑀 𝐹 ↑ LVS [5]ResNet-18 37.9 51.0. 29.5 33.0 MSSL [44] ResNet-18 44.9 66.3 26.1 36.3 3DC [36] ResNet-152 57.1 75.9 36.9 50.3 SST [6] ResNet-101 66. issues, whereas our approach demonstrates stronger sound source localization capabilities and avoids segmenting irrelevant objects. Moreover, our segmentation is more precise and refined. This is due to the implementation of information interaction measures at various stages, especially via BCSM, which effectively suppresses noise and isolates robust features, thereby improving the precision of localization.In this part, we detail the contributions of each module, the efficacy of symmetric design, and the impact of loss functions. To achieve a representative analysis, we conduct ablation studies under single-source (S4) and multi-source (MS3) audio settings, with all experiments utilizing the PVT-v2 as the video encoder. For additional subsidiary ablation studies, such as the effects of different stages in early fusion, please refer to the supplementary materials.To dissect the impact of each component on the effectiveness of SelM, we conduct ablation studies by sequentially removing the three modules from the complete SelM architecture. For these experiments, the S4 and MS3 settings are chosen for representative analysis. Regarding DAM and BCSM, these modules are simply removed, while for the condition 'without CLEVER', the decoder from AVSegFormer [8] is used as a substitute because it similarly utilizes the query-based transformer structure and operates on a single level. that eliminating any module reduces the performance of SelM, with removal of BCSM having the most substantial effect under MS3 setting. This emphasizes the effectiveness of our design in enhancing the selection and interaction of information in complex scenes. Furthermore, the incorporation of DAM and CLEVER plays distinct roles in bolstering performance. Dual Alignment Module. Additionally, to validate the necessity of introducing the Dual Alignment Module for early fusion, we compare concatenation or addition of the features 𝑭 and tokens 𝑻 output from the intermediate layers of both encoders. To align the dimensions, we apply average pooling to the 𝑭 and a straightforward repeat operation to the 𝑻 . The results, displayed in Table 4 , indicate that neither of these elementary approaches results in performance gains, and simple concatenation leads to significant decline in capability. This indicates that a thoughtful design of early fusion is necessary as merely interacting the modalities does not automatically enhance performance.Bidirectional Conditioned Selective Mechanism Module. For BCSM, we employ a qualitative analysis approach, visualizing the video feature maps with and without BCSM processing. Comparative observations in Figure 5 reveal that without processing by the BCSM module, interference object such as the dog exhibits notably high responses, leading to segmentation failure. In contrast, maps tend to ignore the interfering, with attention directed towards sound-emitting objects. This indicates that selective mechanism is effective, as stacking Mamba blocks suppresses noise in both modalities, and through bidirectional constraints, the model enhances its focus on information relevant to sound-producing objects.Cross-LEVEl Reasoning Decoder. To further analyze the benefits of CLEVER for the AVS task, we perform ablation studies by replacing the cross-level reasoning with single modality reasoning. Specifically, we maintain the same number of cross-attention operations and queries but restrict the focus to purely video features or audio tokens. As shown in Table 5 , resorting to any single modality results in performance degradation. Particularly, excluding audio interactions proves to be significantly detrimental in the MS3 setting. This is attributed to MS3 being a multi-source setting where a single video may contain multiple sound-emitting objects, rendering audio information crucial. This also demonstrates the capacity of CLEVER to adequately incorporate information from both modalities.To delve into the interplay between audio and visual modalities, as well as to quantify the benefits brought by symmetric design, we conduct ablation studies on the DAM and BCSM modules. In these experiments, "A2V" denotes the alignment or selection of video features based on audio information, producing video features as output, whereas "V2A" represents the reverse, yielding audio tokens as output. Table 6 illustrates the relationship between unilateral modeling capabilities and symmetric design during the early fusion with DAM. We Table 8 : Ablation study on loss function find that V2A alignment is superior to A2V, and a symmetric alignment method exceeds any unilateral approach. Similarly, Table 7 reveals that selecting audio tokens using video features is more beneficial, and implementing bidirectional constraints yields the best performance.We conduct ablation studies on the loss functions within the S4 and MS3 settings to thoroughly understand the contributions of each component. Table 8 demonstrates that incorporating DICE loss and auxiliary losses both contribute to enhance the final predictions. We observe that adding auxiliary losses leads to greater improvements in the MS3 setting, likely because the MS3 setting has less training data, and auxiliary losses help the model converge more effectively. 
conclusion: 
 In this paper, we introduce the SelM model to address the Auditory Illusions issue in Audio-Visual Segmentation (AVS) tasks. We pioneer the application of early fusion in AVS by designing a Dual Alignment Module that enables fine-grained feature interactions and aligns the distributions of both modalities, filling the existing gap in early fusion approaches. We then incorporate the selective mechanism of Mamba to suppress noise and model robust representations, while bidirectional constrains further select relevant sound-emitting information. Finally, we develop a cross-level decoder that uses learnable queries to alternately interrogate audio and video cues, facilitating comprehensive interactions between the modalities and achieving effective segmentation results. Our approach sets new state-of-the-art performance across all three AVS settings and is thoroughly validated by extensive quantitative experiments. Qualitative visualizations confirm that SelM effectively mitigates Auditory Illusions issues. Future work: The effectiveness of selective mechanism in AVS tasks is validated in this paper. We will fully utilize mamba as a feature extractor for both modalities to achieve more robust feature representations in the future.  
 
 Appendix:  


Decision: The authors address selective mechanism-based audio-visual segmentation. The proposed method is intriguing; however, the introduction of several new components could result in increased complexity and computational costs. It would be beneficial to provide more discussion on computational efficiency and potential trade-offs. While the method demonstrates strong performance on the evaluated datasets, additional experiments on more diverse and challenging datasets would better assess its generalizability. Moreover, analyzing the individual contributions of each module and their effects could be beneficial.

Reviews: review: paper_summary: paper_summary: The paper introduces SelM, a novel architecture for Audio-Visual Segmentation (AVS) tasks that addresses the limitations of current methods in dealing with auditory illusions. SelM employs a selective mechanism for noise reduction and robust feature selection, coupled with a dual alignment mechanism for early fusion of audio and visual features. It also includes a cross-level reasoning decoder to enhance segmentation precision. The method achieves state-of-the-art performance on various AVS benchmarks.
strength_weakness: strength_weakness: 1. Novelty: The introduction of a selective mechanism for noise reduction and the use of dual alignment for early fusion in AVS tasks are innovative contributions.
2. Technical Correctness: The paper presents a well-structured methodology with sound theoretical underpinnings and a detailed explanation of the architecture.
3. Evaluation: The proposed method is thoroughly evaluated on multiple datasets and settings, demonstrating significant improvements over existing state-of-the-art methods.
4. Clarity: The paper is well-organized and clearly written, with comprehensive explanations of the proposed components and their integration.
5. Applications: The method has broad applicability in multimedia tasks, such as video editing, industrial maintenance, and surveillance, making it highly relevant to the field.
limitations: limitations: 1. Complexity: The architecture introduces several new components which may increase the complexity and computational cost. Further discussion on computational efficiency and potential trade-offs would be beneficial.
2. Generalization: While the method shows strong performance on the evaluated datasets, additional experiments on more diverse and challenging datasets could further validate the robustness and generalizability of the approach.
3. Ablation Studies: Although the paper includes ablation studies, more detailed analyses of the individual contributions of each module and their interactions could provide deeper insights into the architecture’s performance.
paper_summary: paper_summary: The paper introduces the SelM model for Audio-Visual Segmentation (AVS), utilizing a novel early fusion strategy and a Dual Alignment Module to address Auditory Illusions. It incorporates noise suppression mechanisms and a cross-level decoder to enhance interaction between audio and video cues, significantly advancing AVS performance.
strength_weakness: strength_weakness: This paper focuses on noise reduction and robust feature selection by applying state space models to audio-visual segmentation (AVS), which is a novel approach. Additionally, the paper is clear and well organized.
limitations: limitations: I have some minor questions regarding the paper.

[1] Could you please provide the input and output dimensions for each layer in the CLEVER module?

[2] In Table 7, the performance appears lower when using A2V-SeM or V2A-SeM compared to not using BCSM at all in Table 3. Could you explain the reason for this observation?
paper_summary: paper_summary: The paper introduces SelM, a novel architecture for Audio-Visual Segmentation (AVS). SelM addresses the limitations of existing methods that often fail to filter low-level noise or achieve fine-grained representational interactions during the early feature extraction phase. It employs a State Space model for noise reduction and robust feature selection, and a dual alignment mechanism for early fusion of audio and visual features. Additionally, a cross-level decoder is developed for layered reasoning, enhancing segmentation precision. The method is evaluated on the AVSBench dataset and demonstrates state-of-the-art performance in AVS tasks.
strength_weakness: strength_weakness: 1. Novel Approach: The introduction of the selective mechanism and dual alignment module addresses the critical issue of auditory illusions and improves the precision of AVS tasks. This dual alignment strategy facilitates early fusion, which is a significant departure from traditional methods that primarily focus on high-level semantic interactions.
2. Technical Correctness: The use of a State Space model for noise reduction is technically sound and innovative. The paper effectively demonstrates how this model can suppress noise and isolate key features, which is crucial for accurate AVS.
3. Comprehensive Evaluation: The paper provides extensive experimental results, including ablation studies and comparisons with state-of-the-art methods. The results show significant improvements in segmentation accuracy and robustness across multiple settings.
4. Clarity and Presentation: The methodology is well-explained, and the use of figures and tables effectively illustrates the model architecture and experimental results. The qualitative comparisons further highlight the superiority of SelM over existing methods.
5. Applications: The proposed method has wide-ranging applications in multimedia, video editing, industrial maintenance, and surveillance, where precise AVS is critical.
limitations: limitations: 1. Complexity: The introduction of multiple novel modules, including the dual alignment module and the cross-level reasoning decoder, adds to the complexity of the model. This might make it challenging to implement and optimize in practical scenarios.
2. Generalization: The evaluation is primarily conducted on the AVSBench dataset. Additional experiments on other datasets and real-world scenarios would strengthen the claims about the generalization capabilities of SelM.
3. Sensitivity to Parameters: The performance of the model is sensitive to the choice of certain hyperparameters, such as the weights in the loss function. This might require careful tuning and could impact the ease of use in different applications.
4. Fine-Grained Feature Extraction: While the selective mechanism helps in filtering out noise, there may still be challenges in distinguishing between very similar sound sources or visual objects, as noted in some qualitative analyses.