Unveiling and Mitigating Bias in Audio Visual Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-03 06:10:28

Abstract: Community researchers have developed a range of advanced audio-visual segmentation models aimed at improving the quality of sounding objects' masks. While masks created by these models may initially appear plausible, they occasionally exhibit anomalies with incorrect grounding logic. We attribute this to real-world inherent preferences and distributions as a simpler signal for learning than the complex audio-visual grounding, which leads to the disregard of important modality information. Generally, the anomalous phenomena are often complex and cannot be directly observed systematically. In this study, we made a pioneering effort with the proper synthetic data to categorize and analyze phenomena as two types “audio priming bias'' and “visual prior'' according to the source of anomalies. For audio priming bias, to enhance audio sensitivity to different intensities and semantics, a perception module specifically for audio perceives the latent semantic information and incorporates information into a limited set of queries, namely active queries. Moreover, the interaction mechanism related to such active queries in the transformer decoder is customized to adapt to the need for interaction regulating among audio semantics. For visual prior, multiple contrastive training strategies are explored to optimize the model by incorporating a biased branch, without even changing the structure of the model. During experiments, observation demonstrates the presence and the impact that has been produced by the biases of the existing model.
Finally, through experimental evaluation of AVS benchmarks, we demonstrate the effectiveness of our methods in handling both types of biases, achieving competitive performance across all three subsets.

Main paper: 
 related works 2.1 audio-visual segmentation: 
 Before the emergence of AVS, traditional AVL tasks [17, 32, 36] relied on unsupervised learning that used bounding boxes or coarse heat maps for predicting the positions of sounding objects. However, for applications such as autonomous driving [37, 48] , and embodied intelligence [13, 31] , a more precisely multimodal grounding task to localize sounding objects at a pixel level is required. This is where AVS comes into the stage. The existing AVS approaches can be broadly categorized into fusion-based [18, 22, 25, 28, 43, 46] and prompt-based [10, 29, 38, 41] . The pioneering fusion-based work [46] employs a multi-stage strategy to integrate audio with multi-scale visual features. Building upon this, CATR [22] further proposes a segmentation paradigm that combines both temporal and fusion information. Taking the thought of bidirectional generation, Hao el al. [14] achieve a further boosting on the performance. In contrast, prompt-based methods like AVSegFormer [10] and GAVS [38] directly decode fused features using projected audio queries. Notably, GAVS leverages rich visual information to achieve a generalizable model in zero-shot and few-shot scenarios. Moreover, taking a holistic perspective, Yan et al. [41] present a general framework for AVS and Ref-VOS [39] (Referring Video Object Segmentation), which both are cross-model guided video segmentation. However, previous AVS studies have primarily focused on achieving better multimodal fusion and plausible masks from the framework perspective, without delving into the challenges underlying the establishment of reasonable grounding.Due to the widespread interest in the Visual Question Answering (VQA) task [3, 4] , researchers have tackled two types of bias, namely "visual priming bias" and "language prior", as discussed in previous works [33, 42] . "Visual priming bias" was initially identified by Antol et al. [4] , where models tend to prioritize visually salient content while disregarding the other context [12, 42] . Subsequently, the issue of "language prior" has gained attention from researchers [2, 33, 42] , referring to the tendency of models to generate statistically common answers (e.g. the problem of green and yellow banana). The approaches for addressing the aforementioned bias can be broadly categorized into non-extra-based methods [5, 8, 33] , and extra-based methods, involving additional human supervision [35, 40] . Different from extra-based methods, non-extra-based methods introduce a branch of the question-only model and conduct joint training with the target VQA model by specific learning strategies. Researchers have proposed several customized learning strategies including sharing the same question encoder [33] , blocking the backpropagation of the question-only model [5] , pretraining question-only model [8] . Moreover, it has been discovered recently that there is also a certain similarity between bias in VQA and hallucination [21, 47] in the popular Multimodal Large Language Model (MLLM). However, the bias in AVS has never been explored by researchers before. Here, we hold the holistic opinion that VQA and AVS, as cross-model guided tasks, share certain similarities. VQA involves obtaining answers to questions based on image cues, while AVS involves obtaining masks from images based on audio cues. Therefore, in simple terms, VQA can be understood as a symmetrical problem to AVS in terms of modalities. So, the anomalous phenomena observed in AVS can potentially be explained by the bias theory present in VQA. 
problem and analysis: 
 Based on the same thought of bias categories in VQA [42] , we similarly classify the phenomena of anomalies into two types, "audio priming bias", and "visual prior" in Fig. 1 . The segmentation failure cases observed in the previous model often tend to be a combination of both biases. The illustration of audio priming bias. We have discovered that higher audio intensity results in stronger guiding capabilities in the green block. Also, the audio with distinct semantic attributes is easier to learn and possesses stronger guiding capabilities in the grey block. For instance, music has a greater guiding capability compared to human sound. Consequently, the diverse guiding capabilities cause the phenomenon of dominance in the red block.The phenomenon that the model tends to focus on audio salient content but not whole content is called "audio priming bias". Firstly, through observation experiments involving manual intervention in audio (detailed in the appendix), we have observed the phenomenon of audio priming bias as illustrated in Fig. 2 . Hence, the following observations can be discerned. 1) Audio with different intensities demonstrates varying guiding capability, as shown in Fig. 2 green block. 2) When controlling other variables including volume, we can observe a clear variance of the guiding capability by different semantics through the box plot. 3) In cases where multiple audios are simultaneously present, the overlaying of audio does not always lead to separate related masks being superimposed. Instead, the dominant of the audio appears as shown in Fig. 2 red block. Therefore, we contend that the primary determinants of audio guiding abilities are the intensity and semantic attributes of the audio. Consequently, the diverse guiding capabilities give rise to the phenomenon of dominance. Therefore, a well-designed mechanism is demanded to alleviate audio priming bias.The phenomenon that the model may directly segment the commonsounding objects is called "visual prior". Figure 4 reveals that, regardless of the audio provided, the AVS model consistently segments the partial or entire region of the particular object. Since the piano generally appears with a high sounding probability in the training data, the model tends to segment once sees the piano. Statistically, the occurrence frequency in [45] and the sounding probability in Fig. 4 between different semantics are imbalanced in the dataset. According to previous works [21, 23, 33] , such preference and distribution provide strong prior information and make the model inclined to obtain statistically plausible results, rather than achieving the desired challenging grounding behavior. Moreover, due to the imbalance of inherent preference and distribution in the real world, addressing visual prior becomes more significant. 
methods: 
 The overall pipeline for the AVS task is normally constructed by the encoder-decoder structure in Fig. 3 involving an image encoder, a pixel decoder, and a transformer decoder. Generally, the multimodal information is fused and perceived in the transformer decoder [7, 10, 19, 22] . Therefore, the mitigation of audio priming bias requires an enhancing mechanism in the transformer decoder, while the mitigation of visual prior requires distribution reorganization after acquiring the logits. Methodologically, on one hand, to enhance the audio sensitivity and cooperation of different intensity and semantic attributes, we introduce the semantic-aware active queries utilizing a perception module and interaction enhancement mechanism. On the other hand, multiple training strategies are explored to contrastively optimize the debias model and reorganize the logits without modifying the structure.Given an audio-visual video sequence, we first divide it into 𝑡 nonoverlapping audio and visual segment pairs with a length of 1𝑠.Visual. After the division, we extract visual representations 𝐹 𝑉 ∈ R 𝑡 ×𝑑 𝑉 ×𝐻 ×𝑊 using a pre-trained Swin-base model [9] . The visual encoder is frozen and only tuned with two-layer MLP adapters.Audio. Similarly, audio representations 𝐹 𝐴 ∈ R 𝑡 ×𝑑 𝐴 is encoded from VGGish [11, 15] , where 𝑡 represents the duration of the audio in seconds and matches the number of video frames. The audio representations are extracted in advance by the frozen encoder.Task. Given a dataset D = {𝐹 𝑖 𝑉 , 𝐹 However, in an ideal AVS model with a similar structure, the learnable queries need to not only generate regions but also perceive and regulate the interaction between audio semantics. So, this mechanism is designed to enhance the understanding of latent semantics and improve the interaction to only occur between semantic-aware active queries.In transformer decoder of conventional semantic segmentation, each learnable query (∈ R 1×𝑑 𝐴 among initialized learnable queries 𝑞 ∈ R 𝑡 ×𝑁 ×𝑑 𝐴 , where 𝑁 is the number of learnable queries in 1𝑠 interval) functions like a region proposal network [34] and has the ability to generate mask proposals. These queries are simply inherited by repeatedly adding audio features 𝐹 𝐴 , in the AVS model [10, 22, 38] as shown in the upper block in Fig. 2 . However, in an ideal AVS model, the learnable queries need to not only generate regions but also perceive and regulate the interactions between audio semantics adaptively.To achieve the basic perception of audio semantics, we first introduce the following perception module aiming to enhance sensitivity to audio of various intensities and semantics. The logical and straightforward approach is to employ a separately trained projection Φ(•) module with an optimization objective for audio alone to perform a coarse perception of latent audio semantics and to finally project to a latent semantic mark Φ(𝐹 𝐴 ) ∈ R 𝑡 ×𝑁 with padding of 0.EQUATIONEQUATIONHere, we use K-means, Gaussian Mixture Models (GMMs), or BEATs [6] to perform projection in both unsupervised and supervised ways. So each audio segment will be assigned with 1 ≤ C ≤ 8 classes or C = 1 clusters. The overall goal of a latent semantic mark is a coarse filter to have a coarse and basic perception of the audio semantics. The items encoded by 1 in Φ(𝐹 𝐴 ) are termed active marks, whereas those encoded by 0 in Φ(𝐹 𝐴 ) are termed dormant marks. The active or dormant states are relative and vary from samples. Afterward, each audio segment will receive the latent semantic marks Φ(𝐹 𝐴 ) ∈ R 𝑡 ×𝑁 based on the projection mentioned above. Then the semantic-enhanced audio features are obtained by element-wise product ⊙ with broadcastEQUATIONwhere 𝐹 ′ 𝐴 ∈ R 𝑡 ×𝑁 ×𝑑 𝐴 is semantic-enhanced audio features. Furthermore, we consistently encode the 0 𝑡ℎ mark out of 𝑁 as an active mark shown in Eq. 1 to capture global audio semantic information.Building upon the semantic-enhanced audio features, we carry out self-attention SA(•) with the residual of initialized learnable queries 𝑞 and obtain the semantic-aware queries 𝑞 ′ asEQUATIONwhere 𝑞 ′ ∈ R 𝑡 ×𝑁 ×𝑑 𝐴 is the semantic-aware queries. Here, to enhance conciseness, the equations in this paper for attention have excluded projection and scaling factors. The semantic-aware queries obtained are utilized as the learnable query inputs for the transformer decoder layers. Similarly, the queries in the same position corresponding to latent semantic active marks are active queries 𝑞 ′ 𝑎 , while those corresponding to dormant marks are dormant queries 𝑞 ′ 𝑑 , where {𝑞 ′ } = {𝑞 ′ 𝑎 } ∪ {𝑞 ′ 𝑑 }. Moreover, the adequate number of active queries 𝑞 ′ 𝑎 per segment is ensured in multiple ways and please refer to the appendix for more details.After introducing active and dormant queries, we observed that regions proposed by dormant queries are sometimes unnecessarily optimized and backpropagated. It is clearly detrimental since dormant queries do not contain adequate audio semantic information.The widely used transformer decoder layer in per-mask segmentation consists of three modules to process learnable query features 𝑞 ∈ R 𝑁 ×𝑡 ×𝑑 𝐴 in the following order: a cross-attention CA(•), a selfattention module SA(•), and a feed-forward network 𝐹 𝐹 𝑁 (•). For the original transformer decoder, the interactions occur between all learnable query features, and the same query can propose regions of different semantics. Original cross-attention and self-attention in 𝑙 𝑡ℎ layer can be denoted asEQUATIONEQUATIONwhere 𝑙 is the layer index, 𝑞 (𝑙 ) refers to the query features corresponding to the learnable queries 𝑞. 𝑞 𝑐𝑎 (𝑙 ) and 𝑞 𝑠𝑎 (𝑙 ) is the temporary However, we argue that audio interactions should only occur between active queries related to perceived latent semantics. This ensures that a specific set of active queries should only propose the region of the same semantics.EQUATIONEQUATIONwhere only active query features 𝑞 ′ 𝑎 (𝑙 -1) are updated to 𝑞 ′ 𝑎 (𝑙 ) by 𝐹 𝐹 𝑁 (𝑞 𝑠𝑎 𝑎 (𝑙 ) ), and dormant query features 𝑞 ′ 𝑑 (𝑙 ) , are masked during backpropagation. Finally, based on the 𝑞 ′ 𝑎 (𝑙 ) in each layer, the mask logits ∈ R 𝑇 ×𝑁 ×𝐻 ×𝑊 and class logits ∈ R 𝑇 ×𝑁 × K are obtained, where K is the number classes of semantics (K=1 for binary).Following each transformer decoder layer, query matching aims to determine which of the predicted regions fits the referred objects. Therefore, we match the predicted regions from the active queries and the ground truth regions by minimizing the matching costEQUATIONwhere M 𝑞 ′ 𝑎 are the regions proposed by 𝑞 ′ 𝑎 , and M 𝑔𝑡 is the ground truth of original image. M 𝑞 ′ 𝑎 is calculated by multiplying mask logits and class logits, so the cost 𝐶 match evaluates the predicted region in both mask and semantic level. In practice, we separately compute the costs of mask and class logits and jointly optimize the summation as 𝐶 match . Then, the losses L are calculated in every intermediate transformer decoder layer, and only the one in the final layer is given a higher weight and used for inference.Based on the method above, interactions related to active queries are retained and enhanced, while interactions of dormant queries are suppressed with a reduction of the computational workload. Although this method slightly reduces the number of proposed valid regions M 𝑞 ′ 𝑎 , this trade-off is proven advantageous for better cooperation between audio semantics in further experiments.To recognize bias and then avoid the biased region appearing along with the referred region, it is a natural idea to introduce a new biased branch and conduct joint training to the current AVS model without modifying their structures. We first briefly bring up two simple and direct methods and carry out analysis and experiment for comparison. Subsequently, a well-designed uncertainty-based strategy is proposed with competitive performance and versatility. To maintain brevity, operations on 𝑙𝑜𝑔𝑖𝑡𝑠 in Sec. 4.4 are applied to both mask logits and class logits separately.To begin with, the simplest approach to reorganize the distribution is the direct logits ensemble strategy of separately trained biased model G 𝑏 (𝐹 𝑉 ) → M 𝑔𝑡 and vanilla AVS model G(𝐹 𝑉 , 𝐹 𝐴 ) → M 𝑔𝑡 , and subsequently merge their output logits using simple operations such as weighted subtraction and multiplication. As the second strategy, the bias-only strategy requires a new objective of training by involving a bias-only branch with the mapping of G 𝑏 (𝐹 𝑉 ) → M 𝑔𝑡 and jointly optimize the overall model with a bias-only branch. To obtain the debias logits, we simply compute an element-wise product ⊙ of the logits of G and G 𝑏 𝑙𝑜𝑔𝑖𝑡𝑠 ( Ḡ(𝐹 𝑉 , 𝐹 𝐴 )) = 𝑙𝑜𝑔𝑖𝑡𝑠 (G(𝐹 𝑉 , 𝐹 𝐴 )) ⊙𝜎 (𝑙𝑜𝑔𝑖𝑡𝑠 (G 𝑏 (𝐹 𝑉 ))), (10) where Ḡ is the debias projection of the triplets D and 𝜎 is the sigmoid function. The final optimization goal can be calculated from the logits asEQUATIONBased on the updated logits, the region of the man in Fig. 4 will be given fewer logits and yield a greater loss than the original.Although plausibly reorganizing the logits distribution, these two methods demonstrate fluctuations in experiments and do not consistently improve performance in popular methods. These issues can be attributed to the hard and direct introduction of the biased branch, which forces the network to learn from the large distribution gap, resulting in fluctuations in gradient direction and magnitude. To address this, we propose a soft and gradual strategy by introducing audio uncertainty through Gaussian noise.Our proposed uncertainty-based strategy provides a gradual and soft way to estimate the biased output distribution. It does not require multitask-like loss but involves contrast G 𝑛 output distributions derived from image and distorted audio inputs. We first follow the idea of the forward diffusion process 𝑟 (𝐹 𝐴(𝑇 ) | 𝐹 𝐴(0) ) in image generation [16] to construct the distorted audioEQUATIONwhere 𝐹 𝐴(0) denotes the original audio features without any noise and 𝐼 refers to an identity matrix. We incrementally add a small amount of Gaussian noise for 𝑇 steps. As step 𝜏 goes larger, the amount of noise added in each step is controlled by 𝛽. Finally, if 𝑇 → ∞, 𝐹 𝐴(𝑇 ) will be completely distorted. Then, the final logits are updated asEQUATIONwhere larger 𝛼 values indicate a stronger amplification of differences between the two distributions (𝛼 = 0 reduces to regular logits).During inference, 𝑇 → ∞, so 𝐹 𝐴(𝑇 ) only contains Gaussian noise for the biased branch. Based on such logits, the overall lossEQUATIONis calculated by updated logits. Finally, the loss and mask result will be calculated from the debias 𝑙𝑜𝑔𝑖𝑡𝑠 ( Ḡ(𝐹 𝑉 , 𝐹 𝐴 )). In summary, replacing the hard bias-only branch with a gradually distorted audio branch provides a soft and learnable transition for the model to reorganize the distribution caused by visual prior.Considering both masks and semantics, the overall loss L mentioned in Sec.4.4 is obtained within every intermediate transformer decoder layer by mask loss L 𝑚𝑎𝑠𝑘 [24] and class loss L 𝑐𝑙𝑠 based on the updated mask logits and class logits by debias strategy asEQUATIONEQUATIONwhere L 𝑚𝑎𝑠𝑘 is implemented by a combination of the dice loss L 𝑑𝑖𝑐𝑒 [29] and the focal loss L 𝑓 𝑜𝑐𝑎𝑙 functions, and L 𝑐𝑙𝑠 is implemented by the cross-entropy loss.After adopting the active queries, both losses L 𝑚𝑎𝑠𝑘 , L 𝑐𝑙𝑠 are calculated and backpropagated within the matched pairs of masks M from active queries and ground truth M 𝑔𝑡 . During inference, the overall logits ∈ R ( K+1) ×𝐻 ×𝑊 are obtained by the multiplication of mask logits and class logits in the last layer.To address the shortcut effect resulting from easily learned regions proposed by specific queries, we drop the relatively welllearned pairs (M, M 𝑔𝑡 ) under the probability of 𝑝 in the calculation of L and empirical results show slight improvements. 
experiments 5.1 implementation details: 
 Dataset. Our proposed method is evaluated on the AVS Benchmarks [46] , which contains three subsets. Firstly, the single-source subset (V1S) contains 4932 videos over 23 categories. In this subset, only the first sampled frame is annotated. Secondly, the multisource subset (V1M) contains 424 videos that include two or more sound sources and all sounding objects are visible in the frames. Finally, the AVSBench-semantic (AVSS) subset, an extension of V1S and V1M, contains 12,356 videos of 10𝑠, differing from the 5𝑠 videos in V1S and V1M. Additionally, to validate the cooperation performance in complex audio scenarios, a small multi-source test set of AVS called Co-AVS is created, containing 450 videos of 5𝑠. Original audio clips are replaced with synthesized audio clips of 6 different levels intensities of each semantics, like the red block in Fig. 2 . Setting. We conduct training and evaluation using the VGGish backbone pretrained on Youtube-8M [1] and Swin-base Transformer backbone pretrained on semantic-ADE20K [44] by Mask2Former [7] . The parameters 𝜆 𝑓 𝑜𝑐𝑎𝑙 , 𝜆 𝑑𝑖𝑐𝑒 , 𝜆 𝑐𝑙𝑠 in the loss are set to 5, 5, 2, 1. The AdamW optimizer is adopted with a learning rate of 1e-4 for the visual encoder adapters and 1e-3 for other learnable parameters, Table 1 : Comparison of performance on all three subsets of AVS-Benchmarks. The bold score is utilized to represent optimal results. "-" refers to the methods that do not support the corresponding subsets. Except for specific references, the "strategy" mentioned in the comparison refers to the uncertainty-based strategy that achieves the best outcomes.Audio-backbone Visual-backbone V1S V1M AVSS mIoU(%) F-score mIoU(%) F-score mIoU(%) F-score AVSBench [46] VGGish Metrics. To conduct a comprehensive evaluation of our model, we carry out tests using mean Intersection over Union (mIoU) and F-score as the performance metrics [10, 38, 46] .When compared to methods that do not incorporate the designed debias method, our method, which leverages methods for both biases, demonstrates strong competitiveness. As illustrated in Tab. 1, our method achieves comparable results across all V1S, V1M, and AVSS subsets, showcasing performance improvements of up to 0.63%, 3.22%, 7.76% in mIoU, respectively. Even when considering slight variations in the backbones used by different methods, our approach consistently outperforms others. The limited improvement observed in V1S can primarily be attributed to the constrained manifestation of biases stemming from the singular and simplistic nature of the data. However, we still believe that this does not undermine our ability to demonstrate the effectiveness of our bias handling techniques. Furthermore, both quantitative performance in Fig. 5 and qualitative analysis in Fig. 6 indicate that our model successfully alleviates both biases mentioned before and our predicted masks exhibit superior quality. Furthermore, due to the increased complexity of synthesized audio scenarios in Co-AVS subset, achieving a lower performance drop on synthesized audio necessitates better semantic collaboration. As shown in Fig. 5 , our demonstrates stronger robustness on complex audio scenarios.Overall ablation. We conduct an ablation experiment on the module and the strategy designed for biases. In this experiment, we progressively remove the modifications of each module in the set of V1M. Through this process, we aimed to assess the impact of these modifications on the segmentation results. As demonstrated in Tab. 2, the experiments indicate that the methods we designed for audio priming bias and visual prior respectively achieved an increase of 3.96% and 1.10% in mIoU. Different modules have all achieved an increase in both mIoU and F-score, reaching the design objectives. The ablation study further indicates that the inclusion of active queries alone does not directly result in a significant increase in performance. It necessitates the implementation of enhancement of interaction, ultimately leading to a combined improvement of 3.76%. This finding emphasizes the importance of enhancing interaction in conjunction with active queries, as they aid the transformer encoder in establishing audio-visual grounding.Ablation on projection of Equation 2. The objective of projection is to map each audio clip to several classes or a cluster with latent semantic information. Therefore, we naturally conducted experiments using both supervised and unsupervised approaches. In the supervised experiments, we employed Beats [6] for supervised multi-class classification and used threshold 𝜃 or top𝐾 strategies to roughly filter the samples with abstract semantic information in Tab. 3. The results showed that the different settings in the multi-classification do give an improvement of mIoU on V1M up to 67.22%. In the unsupervised experiments, we utilized the K-means and Gaussian Mixture Model (GMM) with Expectation Maximization for clustering, with the same number of clusters. Moreover, an adequate number of active queries per sample is ensured here for fairness. The results demonstrated that, regardless of supervised or unsupervised approaches, introducing latent semantic information along with active queries consistently improved performance, with a maximum improvement of 3.63% in Tab. 3.Mask of single active queries. Besides the quantitive results in Tab. 2, it is qualitatively found that the masks generated by each active query are consistent with their respective semantics, which further validates the effectiveness of the semantic-aware active queries in audio perception. Detailed qualitative results of cases are provided in the appendix.As general strategies, all three contrastive debias strategies are able to reorganize the distribution of logits. To validate the versatility, we not only applied the debias strategies in our framework but also explored its application in other popular models, as shown in Tab. 4. The performance of the logits ensemble and bias-only strategy fluctuates, achieving performance improvements only in certain methods. When considering all four methods, the uncertainty-based approach emerges as a superior solution due to its general improvement. Within our framework, it attained mIoU of 67.22% and exhibited commendable training stability. 
conclusion: 
 This study presents a novel and systematic exploration of the bias issues existing in the previous AVS model. By defining and analyzing these biases as "audio priming bias" and "visual prior", we propose targeted solutions for each bias. For audio priming bias, we introduce semantic-aware active queries that enhance the audio sensitivity of intensity and semantics with a designed perception module. Then further interaction of active queries is forced to collaborate on semantic understanding. For the visual prior, we employ a contrastive debias strategy that improves the overall performance of the model without modifying its structure. The meticulously designed methods and comprehensive experiment across all three subsets, demonstrate the superiority and versatility of our approach. In summary, our work introduces a novel perspective within the prevailing AVS network, opening up new possibilities for the future development of cross-model guided segmentation. 
: 
 In training data, the piano generally sounds. During testing, regardless of the sound presented, the model still tends to prioritize the piano. These two phenomena of the illogical anomalies can be categorized into "audio priming bias", and "visual prior", which typically are simultaneously observed as a general impediment in the AVS model. embodied intelligence [13, 31] , there is an urgent to establish finer grounding beyond bounding boxes and heat maps between audio and visual elements. Therefore, a new task called Audio-Visual Segmentation (AVS) has emerged. It introduces a pixel-level, finegrained scene understanding, bringing novel challenges to the field of audio-visual understanding. Furthermore, the performance of AVS showcases the current ability of machines to understand and integrate modalities in complex scenarios.The existing works on AVS can be broadly categorized into fusion-based [18, 22, 25, 28, 46] and prompt-based methods [10, 29, 30, 38] according to Wang et al. [38] . The former primarily integrates audio and visual information at different stages to localize sounding objects, while the latter focuses on generating effective audio prompts and further finetuning the visual foundation model pre-trained on large segmentation datasets [20, 44] . Currently, prompt-based methods are the cutting-edge models in this field, demonstrating exceptional performance. However, researchers have observed illogical anomalies in these approaches [38, 43] . As illustrated in Fig. 1 , even when the mask of the object seems plausible, two anomalous but interesting phenomena can be observed. 1) The segmentation results after overlaying audio do not align with the superimposition of masks guided by separate audios, even when each audio yields a satisfactory mask. Instead, dominance occurs. 2) Regardless of whether the piano produces sound, the model usually tends to segment the piano. Phenomena like Fig. 1 have a wide impact on the current model, which makes analyzing and addressing these phenomena an urgent work to improve the grounding behavior.Generally, the phenomena depicted on the left and right sides of Fig. 1 are often coupled and cannot be directly observed individually. We manage to control the influencing factor with artificial data, including volume, etc., and reasonably categorize phenomena into two types of biases: "audio priming bias" and "visual prior". Specifically, "audio priming bias" refers to the phenomenon that the model tends to focus on audio salient content but not whole content. This bias is characterized by the model's insensitivity to audio of specific intensity or semantics and consequently the dominance of certain semantic audio in multi-source scenarios. As for "visual prior", it refers to the phenomenon that the model may directly segment the common-sounding objects. The most common observed form of such bias is that regardless of the audio guidance employed, the model consistently segments the whole or part of certain objects. It is evident that both of these biases significantly impact the behavior of the current AVS models.To provide the community with a comprehensive solution to address these biases, we adopt the divide-and-conquer thought. For audio priming bias, we first introduce semantic-aware active queries. These active queries contain rich latent semantic information gathered by the perception module, enhancing the sensitivity to audio cues. Furthermore, we employ a customized interaction mechanism in the transformer decoder specially designed to enhance the active queries and suppress the dormant ones for better collaboration of specific audio semantics. As for visual priors, we explore three types of contrastive debias strategies without changing the structure of the model. Our findings indicate that the soft and gradual debias strategy based on uncertainty yields superior results across various AVS methods including ours.Finally, through experimental evaluation of AVS benchmarks, we demonstrate the effectiveness of our methods in handling both types of biases, achieving highly competitive performance across all three subsets. In summary, our contributions are threefold:• We make a pioneering attempt to categorize the complex phenomena into two types of bias: "audio priming bias" and "visual prior" with necessary observation and analysis. • For audio priming bias, we propose semantic-aware active queries and customized interaction mechanisms to improve the sensitivity and cooperation of complex audio scenarios. Then, for visual prior, the novel debias strategies are utilized to contrastively reorganize the distribution of logits. • Experimental results reveal that the proposed method and strategy effectively mitigate biases while achieving versatility and comparable performance. 
 
 Appendix:  


Decision: The paper is well written. The reviewers acknowledge the novelty and technical correctness of this work and are satisfied with the response. I am delighted to recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper analyzes the issue of bias in audio-visual segmentation models which may lead to inaccurate segmentation results. The authors categorize two main types of biases, "audio priming bias" and "visual prior". For "audio priming bias", they propose semantic-aware active queries to enhance audio sensitivity and customized interaction mechanisms in the transformer decoder for better audio-semantic collaboration. For  "visual prior", they explore contrastive debias strategies that optimize the model without structural changes. They conduct experiments on AVS benchmarks and achieve competitive performance compared with prior methods.
strength_weakness: strength_weakness: - **Novelty**: The paper introduces an approach to identifying and mitigating biases in the task of audio-visual segmentation, which has not been explored in prior works.
- **Technical Correctness**: The technical solutions proposed in the paper appear to be correct and well-implemented. The authors have detailed the methodology for enhancing audio sensitivity and addressing visual prior biases with clarity. The use of contrastive debias strategies, especially the uncertainty-based approach, is technically innovative.
limitations: limitations: - **Motivation**: The two types of bias come from the VQA task. The authors replace "Visual priming bias" and "language prior" in the VQA task with "audio priming bias" and "visual prior" in the AVS task. The issues observed from the AVS task itself are more appreciated.
- **Datasets**: Do V1S and V1M subsets represent S4 and MS3 subsets in the AVS benchmarks? If not, why not compare with previous methods on these two subsets?
- **Ablation studies**: Table 2 did not list the result of the vanilla model without all proposed modules and the comparison with it.
paper_summary: paper_summary: The paper presents a new exploration on bias issues prevalent in the exiting audio-visual models. To this end, the authors discuss two critical aspects by formulating audio priming bias and visual prior bias. The problem is very relevant to the community and is underexplored. The paper presents an intuitive approach to tackle these challenges and experimental results show that the presented method achieves considerable improvements over the prior baselines.
strength_weakness: strength_weakness: 1. The paper presents two very important anomalies commonly observed in audio-visual space : "audio priming bias" and "visual prior" and explores possible ways to mitigate these issues in existing approaches. These problems are pressing yet underexplored in the community this exploration can serve as the ground work for future investigations in this direction. 
2. The authors present intuitive ways to tackle these challenges and through experimental analysis the paper demonstrates the efficiency of the proposed solutions.
3. The paper is generally well written, easy to follow, although I would recommend the authors to consider re-writing some parts of the papers (especially introduction and problem motivation).
limitations: limitations: 1. Although the problem is very relevant to the community and needs attention, the paper in patches lack motivating the problem statement clearly and can do a better job there. I would recommend the authors to consider laying a strong foundation by adding more practical and real world examples and connecting that with the failure models in existing works and how their proposed solution is better equipped at tackling these issues.
2. The paper doesn't discuss some important literatures in this space:
[a] Self-supervised Audio-Visual Co-segmentation - Rouditchenko et al.
[b] Weakly-Supervised Audio-Visual Segmentation - Mo et al. 
[c] Listen to the pixels - Chowdhury et al.  
These works were initial attempts in the audio-visual segmentation area. 
3. I might be missing this, but why does the authors assign each audio segment into 8 classes? Is there a theoretical or empirical baking behind this?
4. Have the authors tried training their model with hard negatives? This can serve as a baseline for the visual prior task at least. This analysis can clearly underline the drawbacks in existing approaches and provide us with an estimate of the performance gain
5. The contrastive debias strategy is interesting, I wonder what physical aspects aspects do their approach consider while learning this debiasing? How much of a role does the datasets play and what are some good ways the authors think can be applied reuse the existing datasets with limited data distillation?
6. In table 4 there are quite a few cases where  employing the contrastive debiased strategy results in performance drop. Can the authors explain this behaviour or at least add some intuition what might be the reasons behind this observation?
7. The could have considered adding some physical examples through a video in the supplementary for better visualization.
8. The experimental section looks a bit weak. More extensive experiments on the choice of encoder, training strategy (effect of number of input tokens), comparisons in setting with >2 sounding objects are missing

Minor:
1. grammatical mistakes 'there is an urgent.. ' line 88 
2. Can the authors explain in brief their reasoning behind employing swin backbone?
paper_summary: paper_summary: This paper addresses the critical issue of biases in Audio-Visual Segmentation (AVS) models, specifically focusing on two types of biases: "audio priming bias" and "visual prior." The authors introduce novel methods to mitigate these biases, including semantic-aware active queries for enhancing audio sensitivity and a contrastive debias strategy for addressing visual prior. The paper presents extensive experimental evaluations, demonstrating the effectiveness of the proposed methods across multiple AVS benchmarks, achieving competitive performance and improving the quality of sounding objects' masks.
strength_weakness: strength_weakness: 1. Novelty and Theoretical Approach: The paper introduces innovative concepts such as "audio priming bias" and "visual prior," which are well-defined and systematically analyzed. The proposed solutions, including semantic-aware active queries and contrastive debias strategies, represent significant advancements in the field of AVS.

2. Technical Correctness: The methods are technically sound, leveraging advanced techniques like transformer decoders, semantic-aware queries, and contrastive learning. The detailed mathematical formulations and logical flow of the proposed methods indicate a high level of technical rigor.

3. Comprehensive Evaluation: The authors conduct extensive experiments on multiple AVS benchmarks, including V1S, V1M, and AVSS subsets, as well as a newly created Co-AVS subset. The results are presented in a clear and detailed manner, showing significant improvements in performance metrics such as mIoU and F-score.

4. Clarity and Organization: The paper is well-structured and clearly written, with each section logically building upon the previous one. The figures and tables are effectively used to illustrate the key points and results.

5. Applications and Impact: The proposed methods have broad implications for improving AVS models, which are crucial for various applications in multimedia processing, autonomous driving, and embodied intelligence. The systematic approach to addressing biases enhances the robustness and accuracy of AVS models in real-world scenarios.
limitations: limitations: 1. Evaluation Scope: While the paper presents a comprehensive evaluation on multiple benchmarks, it would benefit from additional comparisons with more recent and diverse AVS methods to further validate the generalizability of the proposed approaches.

2. Complexity and Implementation: The proposed methods involve sophisticated techniques that may be challenging to implement and require substantial computational resources. A discussion on the computational complexity and potential optimizations would be valuable.

3. Synthetic Data: The use of synthetic data for certain experiments, while useful for controlled studies, may not fully capture the complexity of real-world audio-visual scenarios. More experiments with real-world data would strengthen the conclusions.
paper_summary: paper_summary: This study identifies and addresses "audio priming bias" and "visual prior" in previous Audio-Visual Segmentation (AVS) models. To mitigate these biases, the authors introduce semantic-aware active queries for enhanced audio sensitivity and a contrastive debias strategy to improve visual performance without altering the model structure. Comprehensive experiments demonstrate the effectiveness of these methods, offering new insights for future development in cross-modal guided segmentation.
strength_weakness: strength_weakness: The strength of this paper lies in its novel definition of "audio priming bias" and "visual prior" as the sources of bias in Audio-Visual Segmentation (AVS). Additionally, the proposed solutions, including semantic-aware active queries and a contrastive debias strategy, are well-structured and effectively address these biases, enhancing the overall performance and robustness of the AVS model.
limitations: limitations: While the paper presents a novel approach and effective solutions, there are some areas that could benefit from further explanation:

[1] Line 387-388: “These queries are simply inherited by repeatedly adding audio features.” Can you explain more about how the queries are generated?

[2] Could you elaborate on the relationship between the number of learnable queries and the visual size of HxW?

[3] Can you provide more details about the dimensions related to Equations (1) and (2)? Specifically, does Equation (1) imply there are txN elements? If so, are the elements defined in Equation (2) also defined for txN elements?

[4] In Equation (1), what is the intuition behind designating the first element as 1 for capturing global audio semantic information?

[5] Line 507: Could you clarify the dimensions of mask logits and class logits and explain why N is needed? Additionally, Could you describe the dimensions from the input to the logits step by step? Plus, it would be more consistent to use 't' instead of 'T' in Line 507.

[6] Line 389: Figure reference should be corrected from Fig. 2 to Fig. 3.

[7] Equation 9: The notation 𝑀_{𝑞’_𝑎} should be corrected to 𝑀.