Title: Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal

representation alignment

neuroscience

human behavior

sound localization

auditory processing
Abstract: Imagine hearing a dog bark and instinctively turning toward the sound—only to find a parked car, while a silent dog sits nearby. Such moments of sensory conflict challenge perception, yet humans flexibly resolve these discrepancies, prioritizing auditory cues over misleading visuals to accurately localize sounds.  Despite the rapid advancement of multimodal AI models that integrate vision and sound, little is known about how these systems handle cross-modal conflicts or whether they favor one modality over another. Here, we systematically and quantitatively examine modality bias and conflict resolution in AI models for Sound Source Localization (SSL). We evaluate a wide range of state-of-the-art multimodal models and compare them against human performance in psychophysics experiments spanning six audiovisual conditions, including congruent, conflicting, and absent visual and audio cues. Our results reveal that humans consistently outperform AI in SSL and exhibit greater robustness to conflicting or absent visual information by effectively prioritizing auditory signals. In contrast, AI shows a pronounced bias toward vision, often failing to suppress irrelevant or conflicting visual input, leading to chance-level performance. To bridge this gap, we present EchoPin, a neuroscience-inspired multimodal model for SSL that emulates human auditory perception. The model is trained on our carefully curated AudioCOCO dataset, in which stereo audio signals are first rendered using a physics-based 3D simulator, then filtered with Head-Related Transfer Functions (HRTFs) to capture pinnae, head, and torso effects, and finally transformed into cochleagram representations that mimic cochlear processing. To eliminate existing biases in standard benchmark datasets, we carefully controlled the vocal object sizes, semantics, and spatial locations in the corresponding images of AudioCOCO. EchoPin outperforms existing models trained on standard audio-visual datasets. Remarkably, consistent with neuroscience findings, it exhibits a human-like localization bias, favoring horizontal (left–right) precision over vertical (up–down) precision. 
This asymmetry likely arises from HRTF-shaped and cochlear-modulated stereo audio and the lateral placement of human ears, highlighting how sensory input quality and physical structure jointly shape precision of multimodal representations. All code, data, and models are available \href{https://github.com/CuriseJia/SSHS}{here}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors created a visual-sound source localization dataset (AudioCOCO) and benchmarked ten AI models and human. The \"Random\" model serves as the low bound and performance from human psychophysics experiment serves as the high bound. This AudioCOCO dataset includes six conditions. Among those conditions, especially the conflicting visual cue (ConflictVcue), has almost never been tested by others.\n\nSTRENGTHS AND WEAKNESSES:\nI really like the idea of comparing the task performance and bias of AI models against humans or non-humans. Figures 4 and 5 and Table 1 clearly shown how the AI models fail under challenging conditions. This finding will benefit people from both AI and neuroscience field.\n\nWith that being said, the benefit I take from this paper is very limited. This model behaves very differently from human. In the L20-22 of Abstract, quote \"Remarkably, consistent with neuroscience findings, our model exhibits a human-like horizontal localization bias—favoring left-right over up-down precision.\" The sentence refers to Figure 6 where the human has 52.1% at horizontal plane and 31.5% at vertical plane, so the accuracy dropped around 40%. In contrast, the SOTA AI model, IS3-FT-Stereo, has less than 10% of drop (the authors do not provide values). I do not think this single, very small, and likely insignificant change could be highlighted as finding the correspondance between AI and humans. \n\nMore importantly, the same Figure 3 also shows that the accuracy of both AI models on the vertical plane is much better than human. This is not suprsing to me at all: human relys on monaural spectral cues but AI models can still use ITD/ILD cues for localization sound on the vertical plane. \n\nIf the authors really want to claim they found human-like sound localization behaviors, they should check their third referred paper (A Francl, JH McDermott - Nature human behaviour, 2022) and show us compariable evidence as shown in that paper. Since the primary area of this paper is neuroscience instead of datasets or machine/deep learning, the strong connection with neuroscience is necessary.\n\nIn the first sentence of Introduction (L27-28), quote \"Sound source localization (SSL) is the task of identifying the spatial origin of a sound within a visual scene.\" This is simply wrong. By this definitation, SSL will fail if sound occur in the darkness or from the human or animal's back. The third sentence (L29-30), quote \"SSL requires integrating information across auditory and visual modalities\". Multisensory integrations is beneficial but not required for SSL. I suggest the authors to revise many sentences like this which imply that SSL needs visual inputs.\n\nQUESTIONS:\n1. In the Abstract (L24) and Introduction (L91), the authors mentioned \"physical structure of sensory systems\". Does this sentence indicate the spectral cues created by human or animal's pinna? Unlike the AI model from Francl and McDermott which has two human-like ears and HRTF, I did not find any similarity between this paper's model and human auditory systems.\n\n2. L72-73: Stereo audio, widely studied in neuroscience for its role in spatial hearing through inter-channel differences, remains underexplored in SSL benchmarks in AI. Do you mean all the SSL beachmarks in AI only use mono audio?\n\n3. L84-86: Neuroscience research [3] shows that humans exhibit greater precision in horizontal (left-right) sound localization than vertical (up-down). Could you point out which sentence or figure in your reference support this? You referred a pure modeling paper without any human psychophysics experiments. There are lots of papers in SSL to support this, for example, JC Makous, JC Middlebrooks - The journal of the Acoustical Society of America, 1990.\n\n4. In the your Figure 8, the performance of IS3 model is so bad even under the audio-visual congruent condition. I don't understand this since IS3 was published on 2023 and achieved SOTA during that time.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates modality bias and conflict resolution in multimodal AI systems for sound source localization (SSL). Through a carefully designed psychophysics-inspired framework, the authors benchmark human and AI performance across six audio-visual conditions (e.g., congruent, conflicting, absent cues). They introduce *AudioCOCO*, a large-scale dataset synthesized via 3D simulation, containing stereo audio rendered from static images. Their experiments reveal that current AI systems are heavily biased toward visual cues and struggle under conflicting or missing modality conditions. By fine-tuning a state-of-the-art model on their new dataset with stereo input, they achieve significant performance gains and observe human-like spatial biases, such as better horizontal than vertical localization.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper is clearly written with good motivation, explanations, and visualizations.\n- Few existing works on sound source localization explore the modality bias, object bias, and the stereo audio. The proposed dataset with stereo audio simulation is novel, scalable, and would beneficial for the SSL community.\n- Based on the proposed dataset, this paper also provided some insightful conclusions which may inspire future works.\n\n## Weaknesses\n\n\n\n\n\n- **Unclear evaluation metrics.** 1) The authors introduce the evaluation metrics of 'A-Acc' and 'V-Acc' at Lines 235-242. As shown in Table S1, only one metric is reported for some SSL cases. This raises questions on the application and suitable scenarios of the metrics. Could the authors provide more explanations on this? 2) Prior SSL works usually adopt the CIoU to evaluate the model's performance. Can this metric be used in this paper? If not, could the authors provide some discussion on this?\n- **Finetuning and Inference time cost**. The proposed dataset is large-scale. I wonder about the time cost of fine-tuning the IS3 model and the inference.\n- **Questions on experiments** 1) According to Table 1 (1), the CAVP has better performance than IS3, so can the CAVP also be adopted for Finetuning using the proposed dataset? This would make the comparison more comprehensive. 2) The authors choose to fine-tune the prior model IS3. Could the IS3 model be directly trained from scratch? \n- **Minor issues on Figure presentation (do not affect my score).** 1) In Figure 1(a), the texts of 'left' and 'right' seem to be hard to correspond to the figure. 2) In Figure 3, there are no 'blue arrows' mentioned in the caption.\n\nQUESTIONS:\nPlease refer to the 'weakness' part.\n\nLIMITATIONS:\nYes, the authors have clearly articulated limitations, including dataset realism and the restriction to static images. They also discuss potential directions to incorporate video and better audio realism in future work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a detailed and systematic analysis of modality bias and conflict in the task of sound source localization (SSL). The authors propose six possible audiovisual scenarios: Congruent, ConflictVcue, AbsVcue, Aonly, Vonly, and MultiInstLoc. They synthesize an audiovisual dataset by pairing semantically similar images and sounds using the MSCOCO image dataset and the VGGSound audio dataset. Depth information is inferred from the images using the DepthAnything model, and depth-aware stereo audio rendering is performed in Unity.\n\nThe authors fine-tune the state-of-the-art SSL model IS3 on their newly constructed AudioCOCO dataset, resulting in IS3-FT. In the experiments, both human psychophysics studies and quantitative evaluations are conducted. The results show that object size is positively correlated with sound localization accuracy; audiovisual scenarios involving modality conflict harm localization accuracy. Existing AI models struggle with sound source localization in audio-only settings, while achieving relatively good V-ACC in vision-only conditions—indicating a preference for visual information. Human performance surpasses that of AI models across all audiovisual scenarios, particularly under audio-only conditions. Both humans and AI models perform poorly on multi-instance SSL tasks. The fine-tuned model IS3-FT outperforms the original IS3 on AudioCOCO. Moreover, the stereo-audio-aware IS3-FT-stereo model outperforms its mono counterpart IS3-FT-mono, and displays human-like localization biases—favoring horizontal over vertical precision.Strengths And Weaknesses\n\nSTRENGTHS AND WEAKNESSES:\na) Strengths\n\n1.\tThe paper introduces a new and challenging benchmark, UniAV, for the SSL task. The authors conduct a detailed analysis of various possible audiovisual scenarios in SSL, with a particular focus on modality conflict—an interesting and meaningful case that addresses issues not previously considered in the literature and has practical relevance.\n\n2.\tA large-scale, high-quality audio-visual dataset is constructed, simulating realistic, spatialized stereo audio that obeys the physical laws of sound propagation. By fine-tuning existing models on this dataset, the authors achieve performance that surpasses the current state-of-the-art on the UniAV benchmark.\n\n3.\tThe paper includes both psychophysics and quantitative experiments, analyzing the performance differences between existing AI models and humans in SSL tasks.\n\nb) Weaknesses\n\n1.\tThe synthesized dataset is based on static images rather than videos, which leads to a natural temporal inconsistency between the visual and audio information. This creates a gap between the benchmark and real-world sound source localization scenarios.\n\n2.\tThe evaluation is primarily conducted on the authors' proposed dataset (AudioCOCO) and benchmark (UniAV), lacking assessments on commonly used benchmarks in the field. For example, the paper does not demonstrate whether IS3-FT outperforms IS3 on VGG-SS [1] and SoundNet-Flickr-Test [2]. Therefore, the claimed improvement over the state-of-the-art may result from overfitting to the AudioCOCO dataset.\n\n3.\tThe paper lacks a review of related work.\n\n[1] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.\n\n[2] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.\n\nQUESTIONS:\n1. It is unclear whether the fine-tuning of the IS3 model was conducted solely on congruent data, or if modality conflict data was also used. If the latter, the introduction of ambiguous or conflicting audiovisual signals could potentially harm the model’s original performance. The paper should clarify whether modality conflict data was included during fine-tuning and analyze its effect on model robustness.\n\n2. Does IS3-FT outperform IS3 on standard SSL test datasets such as VGG-SS and SoundNet-Flickr-Test? The paper currently lacks evaluations on these widely used benchmarks, making it difficult to assess the generalizability of the improvements reported on AudioCOCO.\n\n3. The paper analyzes the impact of object size on SSL accuracy. However, based on the evaluation metric used for SSL accuracy—where a prediction is considered positive if the location of the maximum activation on the heatmap lies within the segmentation mask of the ground truth sounding object—it is important to note that larger objects naturally have a higher probability of being matched by chance. That is, even a randomly selected pixel from the image is more likely to fall within the object mask if the object is large (as supported by the authors' own experiment involving random sampling). Therefore, the higher SSL accuracy observed for larger objects may largely result from the limitations of the evaluation metric, rather than indicating a true model sensitivity to object size. While the conclusion that “Object size matters for humans and AI” may subjectively make sense, the current experimental design and analysis are insufficient to convincingly support this claim.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on audiovisual sound source localization. It is a study focussing on the interactions between the different modalities. Particularly interesting are cases where the two modalities have conflicting cues or if either is missing or noisy. They construct a new dataset from images in COCO and audio from VGGSound and smartly pair them. They use depth estimation models to construct stereo sounds. They compare results of AI models with a human study in similar conditions and derive interesting observations from them.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is well written and clear, with sufficient detail for the dataset construction. I liked the overall problem itself, and believe it can have a good impact on audiovisual research.\n2. The experiments are well designed and thorough. \n3. The dataset itself is a solid contribution and can also be useful for other audiovisual tasks beyond source localization.\n\n**Weaknesses**\n\n[Metric]\n\n4. I am skeptical about the metric. I may be underestimating performance in many cases. If a human click is just a few pixels outside the segmentation map, I would argue this is still a correct (almost) prediction, but the metric would mark it as 0. Similarly, if the peak of predicted probabilities is even slightly off, it is marked as completely wrong. \n5. The metric may also be overestimating performance, for example, in the ConflictVCue case of Fig. 8 where a prediction over the entire horse would be marked correct, even though its truly the horse’s head that should be marked correct.\n6. The trend observed in Fig. 4 across different sizes may actually be because of the inherent bias in the metric itself. Bigger objects have more pixels that can be predicted to score correctly. The increase in the performance of the random predictor seems to be very similar to that of the models and human performance.\n\n[Model]\n\n7. Based on the heatmaps in Fig. 8, I think that IS3 results may not be a good datapoint to compare against. I would recommend the authors to revisit claims based on IS3 and maybe constrain their analysis to IS3-FT. \n8. I am not familiar with the fixation cross. Can the authors include a brief description for why its needed.\n\n\n[Results]\n\n9. In Fig. 6, I could not see the dashed legend of vertical bias. May be a rendering issue at my end. Are the left ones for vertical and right ones for horizontal?\n10. Was IS3 originally trained with stereo audio? Based on Table 1 (b) it seems like the gains from switching to stereo is somewhat consistent for IS3 and IS3+FT. If IS3 is not trained on stereo, I would find this trend curious and wonder if the performance difference between IS3 and IS3+FT is primarily due to smaller domain gap and not better utilization of stereo audio.\n\nQUESTIONS:\nI would gladly increase my rating if the authors can defend the choice of metric and address my concerns about its possible effects on the paper's key takeaways.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}