Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
Authors: Yanhao Jia
Ji Xie
S Jivaganesh
Li Hao
Xu Wu
Mengmi Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: multimodal
representation alignment
neuroscience
human behavior
sound localization
auditory processing
Primary area: neuroscience_and_cognitive_science
TL;DR: We present a unified audio-visual framework to uncover how humans and AI respond to modality conflicts and bias.

Abstract: Imagine hearing a dog bark and instinctively turning toward the sound—only to find a parked car, while a silent dog sits nearby. Such moments of sensory conflict challenge perception, yet humans flexibly resolve these discrepancies, prioritizing auditory cues over misleading visuals to accurately localize sounds.  Despite the rapid advancement of multimodal AI models that integrate vision and sound, little is known about how these systems handle cross-modal conflicts or whether they favor one modality over another. Here, we systematically and quantitatively examine modality bias and conflict resolution in AI models for Sound Source Localization (SSL). We evaluate a wide range of state-of-the-art multimodal models and compare them against human performance in psychophysics experiments spanning six audiovisual conditions, including congruent, conflicting, and absent visual and audio cues. Our results reveal that humans consistently outperform AI in SSL and exhibit greater robustness to conflicting or absent visual information by effectively prioritizing auditory signals. In contrast, AI shows a pronounced bias toward vision, often failing to suppress irrelevant or conflicting visual input, leading to chance-level performance. To bridge this gap, we present EchoPin, a neuroscience-inspired multimodal model for SSL that emulates human auditory perception. The model is trained on our carefully curated AudioCOCO dataset, in which stereo audio signals are first rendered using a physics-based 3D simulator, then filtered with Head-Related Transfer Functions (HRTFs) to capture pinnae, head, and torso effects, and finally transformed into cochleagram representations that mimic cochlear processing. To eliminate existing biases in standard benchmark datasets, we carefully controlled the vocal object sizes, semantics, and spatial locations in the corresponding images of AudioCOCO. EchoPin outperforms existing models trained on standard audio-visual datasets. Remarkably, consistent with neuroscience findings, it exhibits a human-like localization bias, favoring horizontal (left–right) precision over vertical (up–down) precision. 
This asymmetry likely arises from HRTF-shaped and cochlear-modulated stereo audio and the lateral placement of human ears, highlighting how sensory input quality and physical structure jointly shape precision of multimodal representations. All code, data, and models are available \href{https://github.com/CuriseJia/SSHS}{here}.

Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
Yanhao Jia', Ji Xie', S Jivaganesh', Hao Li", Xu Wu'd, Mengmi Zhang
Nanyang Technological University, Singapore
Peking University, China Shenzhen University, China
i Corresponding author; address correspondence to mengmi zhangontu. edu.
Abstract
Imagine hearing a dog bark and instinctively turning toward the sound-only to find parked car, while silent dog sits nearby. Such moments of sensory conflict challenge perception, yet humans flexibly resolve these discrepancies, prioritizing auditory cues over misleading visuals accurately localize sounds. Despite the rapid advancement of multimodal Al models that integrate vision and sound, little is known about how these systems handle cross-modal conflicts or whether they favor one modality over another. Here, we systematically and quantitatively examine modality bias and conflict resolution in Al models for Sound Source Localization (SSL). We evaluate a wide range of state-of-the-et multimodal models and compare them against human performance in psychophysics experiments spanning six audiovisual conditions, including congruent, confficting. and absent visual and audio cues. Our results reveal that humans consistently outperform AI in SSL and exhibit greater robustness to conflicting absent visual information by effectively prioritizing auditory signals. In contrast, Al shows a pronounced bias toward vision, often failing suppress irrelevant conflicting visual input, leading to chance-level performance. To bridge this gap, we present EchoPin, a neuroscience-inspiee multimodal model for SSL that emulates human auditory perception. The model is trained on our carefully curated AudioCOCO dataset, which stereo audio signals are first rendered using a physics-based 3D simulator, then filtered with Head-Relatee Transfer Functions (HRTFs) to capture pinnae, head, and torso effects, and finally transformed into cochleagram representations that mimic cochlear processing. To eliminate existing biases in standard benchmark datasets, carefully controlled the vocal object sizes, semantics, and spatial locations in the corresponding images of AudioCOCO. EchoPin outperforms existing models trained on standard audio-visual datasets. Remarkably, consistent with neuroscience findings, it exhibits human-like localization bias, favoring horizontal (left-right) precision over vertical (up-down) precision. This asymmetry likely arises from HRTF-shaped and cochlear-modulatee stereo audio and the lateral placement of human ears, highlighting how sensory input quality and physical structure jointly shape precision of multimodal representations. All code, data, and models are available here.
1 Introduction
Sound source localization (SSL) is the task of identifying the spatial origin of a sound within a visual scene. It plays a fundamental role in both biological perception [1] and artificial intelligence (AI) [2], enabling systems to connect what they hear with what they see. In natural environments, visual inputs interact with auditory cues, often dominating recalibrating sound perception phenomenon exemplified by classic cross-modal illusions [3]. In practice, accurate SSL is critical for
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ap 4p.d))
Audia Library
Depth image
Audio Renderer
Maff Wiff
Computer screen
Interaural
Left
Multi-source
Single source
distance
0. 7.7mees
Right
0.17meters
x Figure 1: (a) Sound source localization challenge in naturalistic images. In the multi-source
Human
(a) Sound source localization challenge
(b) Depth- stereo audio synthesis
scenario (orange panel), multiple objects in the scene-such as pedestrians, birds, cars, and trucks (highlighted by yellow boxes)-emit sounds, whereas in the single-source scenario (red panel), only one object, telephone (red box), produces sound. In both settings, the task is localize all sounding objects based on two-channel stereo audio. To allow systematic and controllable benchmarking of human and AI performance, we focus on the single-source localization task (red panel), which remains challenging due to scene clutter, occlusions, and ambiguous visual cues. (b) Depth-aware stereo audio synthesis. In the 3D simulator, a human listener (interaural distance: 0.1 7m) is placed at the origin, facing the RGB image on the screen. This image and its depth image are aligned in the same direction. Using spatial audio renderer and audio from our library, stereo audio of the target car (red box) in the RGB image can be synthesized (see See 2.1).
real-world systems such as autonomous vehicles anticipating hazards [4], assistive technologies for visually impaired users [5], and robots operating in buman-centered environments [6, 7, 8, 9, 10]. These applications demand robust auditory inference under noisy, cluttered, and ambiguous sensory conditions. Consider real-life example in Fig. 1a depicting a busy traffic intersection: cars honk, people converse, birds chirp, and an ambulance siren approaches from behind building. Humans must rapidly detect and localize the siren despite competing sounds, partial visual occlusion, and environmental noise. Such everyday scenes highlight the challenges of SSL: resolving ambiguous visual or auditory cues, handling conflicting signals across modalities, and focusing attention on the most relevant source amidst distractions. Building AI systems capable of performing SSL tasks in these conditions remains an open problem.
Recent AI research has developed multimodal models for SSL [11, 12, 13, 14, 15, 2, 16). Methods such as contrastive learning [17, 18, 19] aim align audio-visual embeddings by maximizing the similarity between matching pairs and minimizing it between mismatched ones. Other approaches leverage cross-modal attention [20, 21, 22] in transformer architectures [23, 24] to allow sound features to dynamically query relevant visual regions. Some works [25, 26, 27] also incorporate object priors, leveraging knowledge of what typically makes sound to guide localization. However, despite these advances [28], little is known about how such models behave when the modalities conflict or when biases emerge-stie as favoring visual cues over auditory ones.
To address this gap, we systematically evaluate model behavior under six controlled audio-visual conditions: (1) Congruent audio and visual cues align in both semantics and location; (2) conflicting visual cues, where the visual scene misleads localization; (3) absent visual cues, where the sounding object is completely occluded in the visual scene; and (4) vision-only and (5) audio-only conditions, where either vision or audio is entirely omitted. Moreover, similar to the cocktail party problem [29, 30], we further extend the single-source stereo audio to (6) the multi-instance SSL [31], where multiple instances of the same semantic category are present, potentially distracting the model from correctly localizing the target instance. These manipulations allow us to probe how models resolve cross-modal ambiguity and characterize their reliance on each modality. To benchmark these model behaviors, we additionally conduct human behavioral studies under the same experimental conditions. Results show a clear performance gap: humans consistently outperform Al models in handling both congruent and incongruent conditions.
While numerous SSL datasets [32. 33, 34] exist, they often suffer from limitations that hinder robust multimodal alignment in AI models. Typically, these datasets [35, 36, 37, 38, 39, 40, 41J consist of scenes with a single, large, centrally placed sounding object, making them vulnerable to visual shorteut learning, where models perform well without truly integrating audio information [42, 43]. Others [44, 45, 46] are constructed by pairing unrelated images and sounds from independent

---

datasets [47, 48, 49, 50, 51, 52, 53, 54, 55], leading to weak cross-modal entanglement. More recent efforts [56, 57, 58] synthesize audio based on ahysics-informed rules [59, 60] or generative Al models [61, 62], but they still frequently rely on mono audio, neglecting the richer spatial cues provided by stereo audio signals.
Inspired by neuroscience findings highlighting the role of inter-channel differences in spatial hearing [63], we propose method that leverages 3D simulation engines to generate stereo audio from images by integrating separate image and sound datasets. Unlike neuroscience approaches that require physically setting up microphones in real-world spaces, which is both time-conee consuming and costly, our simulation based method offers scalable and efficient alternative for producing spatialized audio paired with complex visual scenes, With this method, we contribute a large-scale AudioCOCo dataset, comprising 28,224 image-audio pairs with ground truth annotations. By simulating stereo audio that adheres to physical principles of sound propagation and spatial cues, AudioCOCO provides realistic and diverse audio-visual scenes.
Human sound localization relies on cascade of acoustic transformations shaped by the ear, head, and torso. Direction-dependerr spectral notches introduced by the pinna help resolve elevation and front-back ambiguities [1, 64, 65]. These cues are formalized in the Head-Related Transfer Function (HRTF), which encodes interaural time (ITD) and level differences (ILD) alongside fine-grained spectral features. At the cochlea, sounds are further decomposed into frequency frequency-seleetive selective channels that preserve ITD/IDD while transforming
modulations into neural representations.
To model these biological mechanisms, we introduce EchoPin, neuroscience-inspiree model for SSL.. EchoPin pre-processes stereo audio using Head-Related Transfer Function (HRTF)--baed filtering and cochleagram representations derived from Equivalent Rectangular Bandwidth (ERB) filters. These designs capture the tonotopic organization and temporal dynamics of the auditory periphery [66, 67] more faithfully than conventional mel-spectrograms [68, 69, 70]. EchoPin then employs dual encoders to jointly process audio and visual inputs, trained with contrastive learning on our AudioCOCO dataset. Experimental results suggest that EchoPin outperforms existing models. Notably, without any human behavioral supervision, EchoPin reproduces a human-like horizontal localization bias [71], an emergent property that was not prominent in previous AI systems. We attribute this to EchoPin's 3D stereo audio pipeline, which integrates interaural spacing, HRTF filtering, and ERB-based cochlear processing to jointly enhance sensory fidelity and multimodal alignment. We highlight our key contributions below:
1. We introduce unified framework to systematically benchmark audio-visual localization models under modality conflicts, absence, and misalignment. We propose a scalable pipeline that synthesizes two-channel stereo audio for static images via 3D simulation, and construct large-scale, naturalistic, and spatially grounded audio-visual dataset, named as AudioCOCO.
2. We design and conduct psychophysics experiments to assess human strategies in resolving audio-visual conflicts, providing a strong baseline for Al-human comparison. We provide a detailed analysis of modality confficts and biases in existing SSL models and humans, highlighting key performance and behavioral differences under challenging multimodal conditions.
3. We introduce EchoPin, neuroscience-instee features a dual-encoder architecture that processes stereo audio-visual pairs. The stereo audio is rre-processed using HRTF-based spatial filtering and cochlear-inspired frequency decomposition. Experimental results show that precise audio-visual alignment emerges from high-fidelity sensory inputs and biologically grounded ear-structure priors. EchoPin not only achieves superior localization accuracy but also exhibits human-like localization biases, favoring horizontal over vertical precision.
model trained on our curated AudioCOCO dataset.
2 Experiments
2.1
AudioCOCO dataset
Image selection. We use the MSCOCO [47] dataset for its broad coverage of everyday objects and apply our selection criteria below to all images from the standard training and test sets. From its dataset annotations, we manually select 12 audible object categories encompassing humans, animals, vehicles, and electronic devices. Recognizing that larger objects may be easier to detect and localize, we further categorize sounding objects by their relative size in the image. Object size defined as the ratio of the object's segmentation mask area to the total image area, independent of their
3

---

a. Congruent
b. Conflicting Vcue
c, Absent Vcue
h.
Click the mouse to locate the sound within 20 secs
VIr
Center fixation for 500ms
d. Audio Only
e. Vision Only
f. Multi-instance
Q
Time
Figure 2: Overview of congruent and manipulated vision-audio conditions in our UniAV framework and task schematic. An example of the vision-audio congruent condition (a) is shown, where dog sound is played (black box) with matching visual source. Visual and auditory modifications for the other five experimental conditions (b-f) are also displayed. See See 2.2 for more details. (h) Each trial began with fixation cross (500 ms), followed by the presentation of an image-audio pair from either of the six conditions (a-f). Participants were instructed to use the computer mouse to click the perceived location of the sound source within 20 seconds.
real-world scale. We define three size bins: Sizel (0-5%), Size2 (5-15%), and Size3 (15-30%). Objects occupying more than 30% of the image are excluded, as they make the localization task trivially easy. To maintain class balance, we limit the number of images per object size per category to 150 within each training or test set. For experimental control, each image contains only one target sounding object, without any other instances of the same category. For the multi- instti-inttance SSL (Sec 2.2), we select 150 images per object size per category, each containing 2-5 instances of the same semantic category, with exactly one designated as the sounding target. See Supp Fig. S1, Supp Fig. S2, and Supp Fig. S3 for the distribution of image counts by category and their target spatial locations in both the training and test sets. After applying the selection criteria, we randomly sample 4,953 qualified images from the MSCOCO training set for training, and 5,500 images from the official test set set-comprising 2,840 single-object and 2,660 multi-object scenes-for testing.
Audio selection. Prior work often pairs MSCOCO images with external audio datasets like VGGSound [49] or FSDnoisy 18k [50]. but these audio sources often lack spatial and temporal consistency. For example, moving sound sources (e.g., dog running across the video frames) cause spatial shifts that make them unsuitable for generating realistic stereo audio on static images. Additionally, recordings may include background noise or mixed sounds from multiple objects, reducing semantic clarity and overall quality. To address these issues, we apply three filtering steps to the VGGSound videos, retaining 3,533 clips from the standard training set and 727 clips from the standard test set that contain high-quality audio, spatially and semantically aligned with the visual content. See Supp Sec. S1, Supp Fig. S4, and Supp Fig. S5 for detailed filtering steps and results.
Next, we randomly pair the selected MSCOCO images with high-quality audios from VGGSound to construct the AudioCOCO dataset. The dataset comprises 9. 9,360 audio-image pairs the training set and 18,864 pairs in the test set, which are further divided into six experimental conditions (Sec. 2.2).
Depth-aware stereo audio synthesis. To generate spatialized stereo audio for their corresponding sounding objects on static images, we employ Unity [72] as our 3D simulation engine, which allows us control precise object locations and simulate realistic stereo sound based on the spatial layout of visual scenes. As illustrated in Fig. 1b, we define a Cartesian coordinate system within Unity. The listener is positioned at the origin (0. O, 0). The computer screen displays the image to listener, is placed parallel to the x-z plane and aligned along the positive y-axis, with a fixed physical distance of 0.76 meters from the listener in the human psychophysics experiment (Sec 2.3). To estimate the depth of objects within the 2D image, we utilize the DepthAnything model [73]. which outputs a relative depth map with values ranging from to 10. However, without access to the original camera intrinsics of MSCOCO images, determining absolute scene scale in Unity is nontrivial. To address this, we normalize the depth values dp of the image as de dp.mux - dp where dp.max are the maximum depth values in the image. Sound loudness decreases logarithmically with distance from the listener. To maintain sound amplitudes within a comfortable and perceptible range, we rescale the normalized depth by 0.5. The final y-coordinate in Unity ya for the sounding object is computed as y, = d /2 + 0.76. For the x and z coordinates of the sounding object in Unity, we map the object's is pixel location from the 2D image to the physical dimensions of the monitor. Given that the display has resolution of 90 pixels per inch, we compute xu = 'p,90 and on = 2p/ /90, where Tp and Zp are the pixel coordinates of the target object's center in the image.

---

Once the sounding object's 3D position (Iu, B/. (Iu, audio source this location. Using an interaural distance of 0.17 meters, reflecting the typical distance between human ears, Unity simulates realistic two-channel stereo sound based on the spatial relationship between the listener and the sound source. This procedure allows us to synthesize spatially grounded, depth-aware stereo audio for each image-sound pair.
vu. zu) is established in Unity, we place static
2.2 Experimental conditions in the AudioCOCO test set
As shown in Fig. 2, the AudioCOCO test set includes six experimental conditions, totaling 18,864 image-audio pairs systematically probe modality biases and conflicts in humans and Al models. Each condition contains 2,900 pairs, except MultilnstLocc, which includes 4,364. All models are trained only on congruent conditions to evaluate generalization, while these six conditions are used exclusively for testing.
Audio-visual Congruent (Congruent) represents the ideal scenario where both the audio semantics and localization align perfectly with the corresponding visual target's 's semantics and location. This should serve as the upper bound for performance in both humans and Al models. For instance, as shown in Fig. 2(a), a dog sound is played at the same location as the dog in the image.
Confficting Visual Cue (ConffictVcuns cuee) examines the scenario where the semantics of both the visual and audio cues belong to the correct category but are spatially misaligned. In Fig. 2(b), a dog sound is played at the location of a cat, while the silent dog is visually present at different location. Among all the image-audio pairs in Congruent condition, we randomly choose an object from non-target category the sound source. We do not limit the distance between the distractor and the target.
Absent Visual Cue (Abs explores the case when a target sound is present but the visual scene contains non-relevant objects, and visual cue matches the sound. For instance, in Fig. 2(c), a dog sound might be played on the cat, but no dog is visually present. From the image-audio pairs in the congruent condition,
randomly select target sound play at randomly selected sounding object in the scene where no relevant objects aligning with the semantics of the sound source exists. This condition is more stringent than Conflicting Visual Cue, as it lacks any visual cues altogether.
Audio Only (AOnly) represents the extreme case where no meaningful visual information is provided, and the sound source is randomly placed anywhere within the image. The image could be blank gray image with pixel values set to 128 (Fig. 2(d)) or Gaussian noise image with a mean of 0 and a standard deviation of 1. For example, dog sound could be randomly played on the left side of pure gray background.
Vision Only (VOnly) exploits multi-modal biases or priors. In this condition, only visual scenes are provided to the AI models, and they must localize the sounding object despite the absence of any meaningful sound. The audio could either be completely silent or filled with random Gaussian noise (mean O, standard deviation 1). For example, the same visual stimulus as in the congruent condition is presented, but the dog sound is replaced with silence noise (Fig. 2(c)).
Multi- Localization (MultilnstLoc follows the same motivation as the cocktail party problem [29] and features several objects of the same category, but the audio corresponds to only one specific instance, testing localization accuracy illustrated in Fig. 2(f), a dog sound is played at the location of the left dog, while both dogs are visually present in the scene. This condition follows the same setup as the Congruent condition, but is more challenging due to the presence of multiple visually relevant objects. We selected images from the test set of the MSCOCO dataset, where the number of object instances within given image is restricted to between 2 and 5 for the target category.
multi-instance scenario. For example, as
2.3 Human psychophysies experiment
We conducted in-lab psychophysics experiments on the AudioCOCO test set with 14 participants, collecting total of 2,0 trials. All the experiments are conducted with the subjects' informed consent and according to protocols approved by the Institutional Review Board of our institution. Every experiment lasted approximately 40 minutes. The experimental setup is schematically illustrated in Fig. 2(h). Each trial began with a fixation cross displayed for 500 milliseconds, followed by the presentation of an image and audio pair drawn from one of the six experimental conditions (see Sec 2.2). The 6-second audio clip paired with the image stimulus continuously loops until
5

---

Image-Steret Pair
Localization
Augmentation
Vision
Compute Accuracy
Encoder
Cochleagram
Contrastive
Audio
Pinna
Cochlear
Learning
Encoder
Ground Truth
Figure 3: Overview of our neuroscience-insprred EchoPin model. EchoPin takes as input a static image paired with two-channel stereo audio signal. The stereo waveforms are first filtered using the llead-Related Transfer Function (HRTF) to simulate sound filtering by the pinnae, and then converted into cochleagrams to mimic auditory processing in the human cochlea. The audio and visual streams are independently processed through dedicated encoders. During training, semantic alignment between the two modalities is enforced using contrastive loss applied to paired audio and visual embeddings, while localization alignment is achieved by regressing the predicted sound source location (indicated by red triangle) from the multimodal feature similarity map the ground-truth location (red bounding box).
the trial concludes. Participants location of the sound source within a time limit of 20 seconds, while wearing stereo headphones throughout the experiment. All trials were randomly sampled, and their presentation sequences were shuffled to minimize order effects. If participants failed to respond within the allotted 20 seconds, the trial automatically ended, and the next trial commenced. Instead of relying image-audio pairs, we conducted an audio calibration procedure at the start of the experiment to account for individual differences in auditory perception. To achieve this, we implemented real-time stereo audio synthesis in Unity. Following calibration, an audio validation task was conducted to ensure successful calibration and adequate spatial hearing accuracy from the participants. See Supp Sec. S2 and Supp Fig.S6 for additional details on the human psychophysics experiments.
instructed
computer mouse to click the perceived
the pre-rendered
2.4 Our neuroscience-inspine EchoPin model
We introduce EchoPin, a ceuroscience-inspiree model for SSL (Fig. 3). The model takes image image-audio audio pairs as input and emulates the human auditory periphery decompose sounds in raw waveforms into frequency components. These representations are then aligned with visual features through a dual-encoder architecture for auditory and visual processing.
In human auditory neuroscience, incoming sound waveforms are directionally shaped by the pinna, head, and torso. These spectral transformations encode elevation-specifi notches and interaural differences, which are essential for resolving front-back and vertical ambiguities [1]. For implementation, cars, developed in Unity simulations and based on the extensive KEMAR dummy head dataset [74]. KEMAR is equipped with microphones in the ear canals to capture how sounds from different directions are filtered by the head and pinnae. The dataset includes left and right ear impulse responses recorded from a Realistic Optimus Pro 7 loudspeaker positioned 1.4 meters from KEMAR, covering 7 710 spatial positions with elevations from -40 to +900
pre-measured Head- Related Transfer Functions (HRTFs) from human
Next, the HRTF-filteed time-domain sound waveform is passed through a cochlear-inspired frequency decomposition, converting into a cochleagram using the PyCochleagram library [74]. The cochleagrams are constructed via Equivalent Rectangular Bandwidth (ERB) filterbanks, capturing the tonotopic and temporal resolution of sound across frequency channels. This representation retains key auditory features, including pitch, timbre, and spatial cues. The resulting 10-second stereo waveform, sampled at 16 kHz, is transformed into cochleagrams, yielding a tensor size 66 x 160,000 x 2, where the dimensions correspond to 66 ERB filters (after truncating 10 high-frequency channels for efficiency), 160k temporal samples, and two binaural channels. The binaural channels are first integrated using ID convolution kernels merge information across and then fed into the dual-encoder architecture of IS3 [45]. allowing separate audio and visual processing streams.
During training, we initialize all weights from the pre-trained IS3 model, except for the first ID-convolution layer in the audio encoder described above, and then optimize all parameters end-to-end using supervised learning. Two losses in the IS3 [45] model are employed: (i)

---

Triplet loss to enforce semantic alignment by pulling matched audio-visual embeddings closer than mismatched ones, and (ii) CloU loss to penalize spatial deviation between predicted and ground-truth sounding-object bounding boxes. This combination enables EchoPin to jointly capture what sounding and where it is located. See Supp Sec. S3 for extra implementation details.
Model variants of EchoPin. To study the effects of design components in EchoPin, we introduce two model variants: EchoPin-M (Mono) averages the two HRTF-filtered stereo channels into single time-domain waveform, allowing us to examine the impact of mono versus stereo audio on SSL tasks, EchoPin-S (Stereo) uses the HRTF-filtered stereo waveforms as imput, but processes them with standard mel-speetrograms instead of cochleagrams. Comparing these variants with the full EchoPin model allows us examine how stereo structure captured by the pinnae and seequency-speciffe cochlear modulation affect SSL performance. See Tab, 1(b) and Sec. 3 for results and discussion.
2.5 Baseline methods and evaluation metries
We benchmark EchoPin and the state-of-the-art multimodal models, including SSLTI [75], LVS [44], FNAC [43], CAVP [42], AV AVSegformer [23], IS3 [45], ImageBind [76], and LanguageBind [33], using the same stimuli in our human psychophysics experiments. While humans can leverage the pinna, head, and torso to encode elevation-specific auditory cues, models lack these physical structures. To ensure fair comparisons across models and with human participants, all AudioCOCO audios are HRTF-filtered to approximate human auditory processing, and these filtered sounds are used for all model evaluations. In the main text, we provide brief overviews of IS3 [45] and a random baseline, and report their performance alongside our proposed EchoPin model. Detailed descriptions of the other models and their extended experimental results are provided Supp. Sec. S3.
IS3 [45] is a dual-stream architecture with 2D CNNs, which processes visual and monaural auditory inputs separately using dedicated encoders before fusing the features for contrastive learning during training. IS3 also includes an Intersection-over-Uniie (IoU) loss and a semantic alignment loss to improve localization accuracy during supervised training. The model is trained the FlickrSoundNet [58] and VGG-Sound [49] datasets. Random is a chance model that randomly selects location on the image as the predicted sound source location. It serves lower bound for SSL without using any audio-visual information.
Predicting target sound locations. For IS3, EchoPin, and other 2D CNN -ased models, feature maps from the final layers of the visual and audio encoders are extracted, and cosine similarity is computed between them to generate similarity heatmap. The predicted sound location is taken as the point with the highest activation on this heatmap. See Supp. Fig. S7 for an illustration of how transformer-based baselines predict target sound source locations.
Evaluation metrics. To disentangle spatial localization from semantic alignment between visual and audio modalities, we define two metrics: Audio Accuracy (A-Acc) measures whether the model or human localizes the true sound source regardless of matching semantics. A-Acc 1 if the peak activation falls within the bounding box of the sounding object; 0 otherwise. Vision Accuracy (V-Acc) measures alignment with visual semantics. V-Aco = if the peak activation falls within any object of the correct category, even if it might not be the actual sound source, such in MultiInstLoc conditions. In VOnly condition. V-Ace = if the activation overlaps with any object from the 12 sound-emitting categories in AudioCOCO.
To robustly evaluate model performance, we consider three complementary factors that could influence results in Supp Sec. $5, First, A-Acc can be biased by object size, introduce chance-corrected A-Acc (Supp Tab. S5; Supp Tab. S1). Second, human clicks may fall near but not inside the target, so we treat clicks within thresholded radius correct in Supp. Tab. S2. None of these metric variants alter the conclusions. Finally, we evaluate the predicted target object bounding boxes by all the models using corrected Intersection over Union (cloU, [77]), where EchoPin continues to outperform other baselines (Supp Sec. S5 and Supp Tab. S3).
3 Results
We report results from both human participants and AI models across all experimental conditions and object sizes. For brevity, the main text focuses on comparisons between humans and the two
7

---

1.0
Random
0.8
Random
153
I53
0.8
EchoPin
EchoPin
Human
Human
0.6
A-Acc
0.6
A-Acc
0.4
0.2
0.2
0.0
0.0
VOnly(Noise) Size Size Size 3
ConflictVCue AbsVCue Congruent Figure 5: ConflietCCue and AbsVCue hurt SSL performance for both humans and Al models. The accuracy of humans and Al models under Conflic and AbsVeue conditions is shown for object size 2, with results from the congruent condition included for comparison. See Supp Tab. S4 for the results on object sizes and 3.
Figure 4: Object size matters for humans and models in the congruent condition. Accuracy increases with object sizes for both humans and AI models, with humans outperforming Al models, especially for small targets. Here and in subsequent figures, error bars represent Standard Error Mean (SEM).
Random
Vertical Horizontal
0.8
0.4
EchoPIn
Proportion
0.6-
V-Acc
0.3
0.4-
0.2
0.1
0.0
Random
153
EchoPIn
EchoPin-Ro
Human
0.0
VOnly(Silent)
Figure 6: EchoPin shows human-like borizontal-vertical asymmetry in SSL accuracy, while other models do not. We report the proportion of trials (object size 2) where the predicted sound location falls within 6 degrees of visual angle from the ground truth, separately along the horizontal (textured bars) and vertical (plain bars) directions.
Figure AI models show bias towards objects that emit sounds, even in the absence of sound or in the presence noisy sound. The V-Ace of Al models is presented in the VOnly condition, where either no sound or only noisy sound is present. Despite the lack of auditory cues, V-Ace of Al models on object size 2 remains higher than random (dark gray).
best-performing models, IS3 and EchoPin. Additional qualitative analyses for all other models are presented in Supp. Tab. $4.
Object size matters for humans and AI. As illustrated in Fig. 4, human A-Acc steadily increases with object size, suggesting that larger sounding objects easier for humans to localize. EchoPin exhibits a similar trend, with A-Ace rising substantially from 11.2% to 76.0% as object size grows. Across all object sizes, EchoPin consistently outperforms IS3 and the other models. This difference likely arises because IS3 is trained on datasets biased toward large, centered objects, whereas AudioCOCO provides wider range of object sizes and spatial locations. While both humans and AI models perform above chance for large objects, only humans and EchoPin maintain strong, consistent performance across all object sizes, including smaller ones perforaerfom that IS3 fails to achieve.
Conffict cues harm more than the lack of cues for AL As shown in Fig. 5, both the Conflict and VCse conditions impair SSL performance in humans compared the congruent conditions. While humans and EchoPin still perform significantly above chance, IS3 drops to near-chance levels under the ConflietVCue condition. This suggests that humans and EchoPin are more robust in SSL tasks involving conflicting or missing visual cues, whereas IS3 relies heavily on visual information. Notably, unlike humans, EchoPin shows greater performance decline when visual cues are conflicting but not when they are absent, indicating that it is more easily misled by incongruent visual information yet remains stable in the absence of such cues.
AT fails when there are only auditory eues but humans can. As shown in Supp. Fig. S8, humans achieve above-chance A-Acc even without visual information (c.g., gray Gaussian noise

---

(a) Multi- Localization
(b)
Overall Performance
Acc(
Rand
IS3
CAVP
AVSeg
EchoPin
Human
Acc(%)
Rand
IS3
EchoPin- MMS EchoPin
Sizel
1.6
4.8
2.9
2,5
4.5
25.7
Size1
1.6
3.0
3.6
Size2
9.1
7.9
7.5
7.3
24.1
36.4
Mono
Size2
9.4
13.9
15.8
Acc
Size3
21.3
22.4
20.4
20.2
47.1
38.6
Size3
19.8
28.7
31.4
Size1
8.4
11.9
10.5
11.2
37.5
60.9
Size1
1.6
5.3
9.7
Size2
17.8
24.1
23.0
23.7
53.8
82.8
Stereo
Size2
9.4
17.0
31.3
Acc
Size3
26.2
40.9
39.5
40.9
64.2
89.1
Size3
19.8
35.2
47.6
Table 1: Multi-instance SSL remains challenging task for both humans and AI models. The table (a) on the left summarizes the audio and visual localization accuracy of humans and AI models under the multi-instance condition across all object sizes. For AI models, both input data quality and the use of stereo audio substantially impact SSL performance. Table (b) on the right summarizes the average A-Acc across the Congruent, ConflietVcue, Abs and AOnly conditions for models trained with either mono or stereo audio. The second-to-last column shows the results of EchoPin-M (Rows 1-3) and EchoPin-S (Rows 4 4-6). See Sec. 2.4 for details these variants. (-) indicates that the results are not applicable due to the model configurations. In both tables, best is in bold and the second best is underlined.
backgrounds), confirming that they can perform SSL based solely on auditory cues. However, their performance remains lower than in the congruent condition, indicating that congruent visual information facilitates SSL. Similar trends are observed in IS3 and EchoPin, with EchoPin consistently outperforming IS3 under both AOnly conditions (gray or Gaussian background). Interestingly, both humans and EchoPin perform better with gray than Gaussian backgrounds, suggesting that incongruent visual noise can distract attention interfere with SSL, whereas neutral (gray) background minimizes interference.
EchoPin shows human-like asymmetry in auditory spatial precision. In neuroscience, auditory spatial precision is known to exhibit a horizontal-vertical asymmetry: humans localize sounds more accurately along the azimuth (horizontal) than the elevation (vertical) axis [71]. To quantify this effect, we measured the proportion of trials where predictions fell within six degrees of visual angle from ground ground-truth locations, separately for horizontal and vertical dimensions (Fig, 6). As expected, humans showed strong horizontal advantage, localizing targets in 86.1% of trials horizontally but only 66.7% vertically. Remarkably, EchoPin exhibited a similar asymmetry pattern despite being trained without any human behavioral data. In contrast, IS3, which relies solely on monaural also showed asymmetry but to a much lesser degree. Although both models have benefited from spatial filtering by the pinnae, the observed asymmetry in EchoPin arises from its biologically grounded auditory frequency decomposition in the cochlea and its stereo audio perception. To further validate this, we introduced an EchoPin variant, EchoPin-Ro, in which the interaural axis rotated by 90 degrees in Unity, effectively simulating vertically aligned ears. When audio was re- re-rendered under this configuration, the model's asymmetry was reversed, confirming the structural origin of this effect.
AI biases toward sound-emitting objects. Previous studies [42, 45] show that Al models often exploit visual shortcuts by favoring large or centered objects. To examine additional behavioral biases, we evaluate models under the VOnly condition. Without meaningful sound, any object could plausibly be the sound source. As shown in Fig. 7, models tend to localize sounds to vocal objects (e.g.. people, animals) rather than irrelevant ones (e.g., sky, trees), resulting in above-chance V-Ace. This indicates that models encode prior knowledge of which objects typically produce sound. Furthermore, for all the models, Gaussian noise audio input leads to lower V-Ace than the absence of audio, indicating that even noisy audio can reduce the models' over-reliance visual signals.
Multi-instance SSL remains challenging for humans and AI. As shown in Tab. la, V-Ace is high for both EchoPin and humans, indicating strong alignment between audio and visual semantics. This allows them to use audio cues to identify all visual objects with matching semanties. However, in scenes with multiple object instances, successful localization requires fine- -me-ggrieed SSL, beyond semantic matching. In these cases, A-Ace drops for both humans and EchoPin. Compared to the Congruent condition (with single sounding object), multi-instance SSL is notably harder, as it demands precise spatial disambiguation. Despite this, humans still perform above chance, especially for small objects. Similarly, EchoPin achieves high A-Ace object sizes 2 and 3. However, it still

---

Congruent
EchoPin
153
ConflictVCue
EchoPin
153
10
08
0.6
3 Mf
04
02
AbsVCue EchoPin
MultilnstLoc EchoPin
Figure 8: IS3 struggles to localize sound sources, whereas humans and EchoPin perform well across all four experimental conditions. The leftmost images (Columns and 4) show the correct localization results made by human participants. Red boxes mark the ground truth sound source locations, while green circles indicate mouse click responses from multiple participants. The middle columns and 5) and rightmost columns (3 and 6) display heatmaps predicted by EchoPin and IS3, respectively. Yellow triangles on the heatmaps denote the predicted sound source locations from each model. See the colorbar for the activation values of the heatmaps.
lags behind human performance, with the gap more pronounced in small object size. Despite this, both EchoPin and humans still perform above chance, especially for small objects. This demonstrates an ability to localize sounds at fine spatial resolution for both humans and EchoPin. Among all models, EchoPin performs best, even outperforms strong baselines like CAVP and AVSeg, which trained large-scale, standard audio-visual datasets.
Training data quality and stereo input are important for SSL. We report the average A-Acc across all object sizes for Random, IS3, and the EchoPin variants under four experimental conditions in Tab. lb. EchoPin consistently achieves the highest performance across all conditions. Notably, EchoPin-M surpasses IS3 despite using fewer fine-tuning examples, underscoring the importance of higl-quality training data. Moreover, EchoPin-S further improves over EchoPin-M, highlighting the advantage of incorporating human stereo configurations for spatial localization.
We further visualize the predicted sound source locations from human participants, IS3, and EchoPin in Fig. 8. EchoPin localizes sound sources more accurately and often aligns closely with human judgments. contrast, IS3 struggles with small or peripheral targets- for instance, fails localize dog in the lep-left corner under the Congruent condition and frequently misattributes sounds to other vocal objects (e.g., person, motorbike). Although EchoPin markedly improves SSL robustness, is still inferior human performance in complex scenes. Failure cases and additional comparisons with other baselines, such as CAVP, are provided in Supp. Sec. S4, Supp. Fig. S9, and Supp. Fig. S10.
4 Discussion
We systematically and quantitatively examine modality biases and conflicts in SSL across humans and AI models. Our study covers six audiovisual conditions, including congruent cues, conflicting signals, and cases with missing audio or visual input natural scenes. Human listeners show strong robustness: although conflicting or absent visual cues reduce performance, they can still accurately localize even small sound sources under challenging or multi-instance conditions, and even in the absence of visual input. In contrast, current multimodal AI models rely heavily on vision misatrributing sounds large, centered, and salient objects, and suffering steep performance drops when visual cues are removed. Conflcttngg visuals further degrade their accuracy, with most models performing near chance for small or visually absent sound sources.
We identify two primary causes of these limitations: (1) low-quality, visually biased audiovisual datasets, and (2) monaural audio inputs lacking spatial fidelity. To overcome these issues, we curate AudioCOCO, a high-quality dataset built through rigorous filtering and 3D physical simulation. By integrating depth maps, HRTF-based filtering that mimics pinna effects, cochlear-inspired frequency decomposition and modulation, and physically grounded 3D sound propagation, AudioCOCo produces realistic, spatialized stereo audio aligned with human auditory processing. Building on this, we introduce EchoPin, neuroscience-insee inspired model trained on AudioCOCO. Despite fewer training examples, EchoPin surpasses state-of-the- models across all conditions and exhibits human-like localization biases, such as stronger precision along the horizontal plane. Ablation studies confirm the importance both high-quality datasets and stereo auditory input for capturing spatial cues. This work underscores the value of designing models and datasets that respect the physical constraints of
10

---

sensory systems. Future directions include scaling AudioCOCO to incorporate temporal dynamics from videos and improving realism in simulated sound rendering, such as the effect of refraction.
Acknowledgements
This research is supported by the National Research Foundation, Singapore under its NRFF award SEFF-NRFFI5-222--000 University, Singapore. We would also like to thank Qing Lin and Shuangpeng Han for their valuable advice and feedback on the project.
and Mengmi Zhang's Startup Grant from Nanyang Technological
References
[1] Kiki van Heijden, Josef Rauschecker, Beatrice de Gelder, and Elia Formisano. Cortical mechanisms of spatial hearing. Nature Reviews Neuroscience, 20(10):609 623, 2019. 1, 1,3,
[2] Simon Jenni, Alexander Black, and John Collomosse. Audio-visual contrastive learning with temporal self-supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7996 8004, 2023. 1,2
[3] Uta Noppeney. Perceptual inference, learning, and attention in multisensory world. Annual review of neuroscience.
473, 2021.
[4] Jianwu Fang, Jiahuan Qiao, Jie Bai, Hongkai Yu, and Jianru Xue. Traffic accident detection via self-supervised consistency learning in driving scenarios. IEEE Transactions Intelligent Transportation Systems, 23(7);9601-99))
2022.
[5] George Dimas, Eirini Cholopoulou, and Dimitris K Iakovidis. Self-supervised soft obstacle detection for safe navigation of visually impaired people. In 2021 IEEE International Conference Imaging Systems and Techniques (IST), pages 1-6. IEEE, 2021. 2
[6] Hongpeng Chen, Shufei Li, Junming Fan, Anging Duan, Chenguang Yang, David Navarro-Alarcon, and Pai Zheng. Human-in-thed toop- robot learning for smart manufacturing: A human-centric perspective. IEEE Transactions on Automation Science and Engineering, 2025. 2
[7] Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien. Multi-task self-supervised learning for human activity detection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(2):1300 2019. 2
[8] Daniel Dworakowski, Angus Fung, and Goldie Nejat. Robots understanding contextual information in human-centered environments using weakly supervised mask data distillation. International Journal of Computer Vision, 131(2)4007-330 2023.
[9] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Jong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEF/CVF conference on computer vision and puttern recognition, pages 19108-19118, 2022.
[10] Xizhou 7hu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, hetter results. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9308-9316, 2019.
[11] Mariana-luliana Georgescu, Eduardo Fonseca, Radu Tudor lonescu, Mario Lucie, Cordelia Schmid, and Anurag Arah. Audiovisual masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16144-16154, 2023.
[12] Po- Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang- Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, al. Mavil: Masked audio-video learners, Advances in Neural Information Processing Systems, 16:20371-20399 2023.
[13] Yanhao Jia, Xinyi Wu, Hao Li, Qinglin Zhang, Yuxiao Hu, Shuai Zhao, and Wenqi Fan. Uni-rctrieval: A multi-style retrieval framework for stem's education, 2025. 2
[14] Sizhe Li. Yapeng Tian, and Chenliang Xu. Space-time memory network for sounding object localization in videos. arXiv preprint arXiv:2/1.00555 2021.
[15] Hao Li, Yanhao Jia, Jin Peng. Zesen Cheng. Kehan Li, Jialu Sui, Chang Liu, and Li Yuan. Freestyleret: Retrieving images from stleeddiersified queries. In Computer Vision 2024, pages 258-274 274, Cham, 2025. Springer Nature Switzerland. 2

---

[16] Arda Senocak. Hyeonggon Ryu. Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Toward interactive sound source localization: Better align sight and sound! IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2025."
[17] Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the" chirp" from the" chat": Self-supervised visual grounding of sound and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13117-13127. 2024.
[18] Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. Advances in Neural Information Processing Systems, 35:37524- 37536, 2022. 2
[19] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XX 16, pages 292 308. Springer, 2020.
[20] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1290- 1290-1299, 1299,2022. 2
[21] Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang. Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6918 01888669,22228 2022. 2
[22] Yan- Bo Lin, Hung- Yu Tseng, Hsin- Hsin-Ying Lee, Yen- Yu Lin, and Ming-Hsuan Hsuan Yang. Unsupervised sound localization via iterative contrastive learning. Computer Vision and Image Understanding, 227:103602, 2023. 2
[23] Shengyi Gao. Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visual segmentation with transformer. In Proceedings of the AAAI conference on artificial intelligence. volume 38, pages 12163, 2024. 2, 5,
[24] Kexin Li, Zongxin Yang. Lei Chen, Yi Yang, and Jun Xiao. Catr: Combinatorialldeeeaddee audio audio--ueried transformer for vusiolvisual video segmentation. In Proceedings of the 31st ACM international conference multinedia, pages 1485-1494, 2023. 2
[25] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In European Conference on Computer Vision, pages 218 2344. Springer, 2022. 2
[26] Hanyu Xuan, Zhiliang Wu, Jian Yang, Yan Yan, and Xavier Alameda-Pinedaa A croposal-based paradigm for self-supervised sound source localization in vidcos. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1029-1038 2022. 2
[27] Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Less can be more: Sound source localization with a classification model. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3308-3317, 2022. 2
[28] Yuan Gong, Andrew Rouditchenko, Alexander Liu, David Harwath. Leonid Karlinsky, Hilde Kuehne. and James Glass. Contrastive audio-visual masked autoencoder. arXiv preprint 10.0022200.07839, 2022.
[29] Simon Haykin and Zhe Chen. The cocktail party problem. Neural computation, 17(9):1875-13 1902, 2005.
2.
[30] Peiwen Sun, Honggang Zhang, and Di Hu. Unveiling and mitigating bias in audio visual segmentation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages
2024. 2
[31] Yejiang Wang. Yuhai Zhao, Zhengkui Wang, and Meixia Wang. Robust self-supervised multi-instance learning with structure awareness. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 37, pages 10218-10225, 2023.
[32] Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song. Acav 100m: Automatic curation of large-scale datasets for audio-visual video representation learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10274-10284, 2021.
[33] Bin Zhu, Bin Lin, Munan Ning. Yang Yan, Jiaxi Cui, Wang HongFa, Yatian Pang. Wenhao Jiang. Junwu Zhang, Zongwei Li, Cai Wan Zhang, Zhifeng Li. Wei Liu, and Li Yuan. Languagebind: Extending vidco-language pretraining to n-modality by language- based semantic alignment, 2023. 2, 7,5
[34] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralb. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-32 2019. 2
12

---

[35] Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang. Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang. al. Audio-visual segmentation with semanties. International Journal of Computer Vision, pages 1-21, 2024. 2, 6
[36] Yaoting Wang. Peiwen Sun, Dongzhan Zhou, Guangyao Li, Honggang Zhang, and Di Hu. Ref-avs: Refer and segment objects in audio-visure scenes. IEEE European Conference on Computer Vision (ECCV),
2024. 2
[37] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang. Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. European Conference on Computer Vision, 2022. 2
[38] Lucas Goncalves, Prashant Mathur, Chandrashekhar Lavania, Metehan Cekic, Marcello Federico, and Kyu J. Han. Peavs: Perceptual evaluation of audio-visual synchrony grounded in viewers' opinion scores, 2024. 2
[39] Juan F Montesinos, Venkatesh Kadandale, and Gloria Haro. A cappella: Audio- singing voice separation. In 32nd British Machine Vision Conference, BMVC 2021, 2021. 2
[40] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting music source separation some Slakh: A dataset study the impact of training data quality and quantity. In Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2019. 2
[41] Justin Salamon, Duncan MacConnell, Mark Cartwright, Peter Li, and Juan Pablo Bello. Scaper: A library for soundscape synthesis and augmentation. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). pages 344-348. TEEE, 2017. 2
[42] Yuanhong Chen, Yuyuan I Liu, Hu Wang, Fengbei I Liu, Chong Wang. Helen Frazer, and Gustavo Carneiro. Unraveling instance associations: A closer look for audio-visual segmentation. In Proceedings of the IEEE/CVF Conference on Cumputer Vision and Pattern Recognition (CVPR), pages 26497-265077, June 2024. 2, 6,
[43] Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. Learning audio-visua. source localization via false negative aware contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6420 1000-6429,200000 2023. 2, 7.4
[44] Honglic Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In CVPR, 2021. 2, 7,
[45] Arda Senocak, Hyconggon Ryu, Junsik Kim, Hac-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Sound source localization is all about cross-modal alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7777-7787, 2023. 2, 6, 9,4
[46] Antonio Torralba and Alexei Efros. Unbiased look dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
[47] Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. 3,
[48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 -255. Ieee, 2009. 3
[49] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020. 3, 4, 7,
[50] Eduardo Fonseca, Manoj Plakal, Daniel PW Ellis, Frederic Font, Xavier Favory, and Xavier Serra. Learning sound event classifiers from web audio with noisy labels. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 21 25. IEEE, 2019. 3,4
[51] Nicolas Turpault, Romain Serizel, Ankit Parag Shah, and Justin Salamon. Sound event detection in domestic environments with weakly labeled data and soundscape synthesis. In Workshop on Detection and Classification of Acoustic Scenes and Events, 2019. 3
13

---

[52] Eric Guizzo, Christian Marinoni, Marco Pennese, Xinlei Ren. Xiguang Zheng, Chen Zhang, Bruno Masiero, Aurelio Uncini, and Danilo Comminiello. L3das22 challenge: Learning 3d audio sources in real office environment. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9186-998889 9990, IEEE, 2022. 3
[53] Yuan Gong, Jin Yu, and James Glass. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 155, 2022. 3
[54] Archontis Politis, Kazuki Shimada, Parthasaarathy Sudarsanam, Sharath Adavanne, Daniel Krause, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, Yuki Mitsufuji. and Tuomas Virtanen. Starss22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events. arXiv preprint ar 2ix:22206.01 0/948, 2022. 3
[55] Heinrich W Löllmann, Christine Evers, Alexander Schmidt, Heinrich Mellmann, Hendrik Barfuss, Patrick A Naylor, and Walter Kellermann. The locata challenge data corpus for acoustic source localization and tracking. In 2018 IEEE 10th sensor array and multichannel signal processing workshop (SAM), pages 410-414. IEEE, 2018. 3
[56] Valentina Sanguineti, Pietro Morerio, Alessio Del Bue, and Vittorio Murino. Audio-visual localization by synthetic acoustic image generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 2523-2531, 2021.
[57] Eduardo Fonseca, Manoj Plakal, Daniel PW Ellis, Frederic Font, Xavier Favory, and Xavier Serra. Learning sound event classifiers from web audio with noisy labels. In ICASSP 2019-2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 21-25, IEEE, 2019. 3
[58] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, 29, 2016. 3, 7.4
[59] Josh H McDermott and Eero P Simoncelli. Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis. Neuron, 71(5):22-9440, 2011. 3
[60] Junwoo Park, Youngwoo Cho, Gyuhyeon Sim, Hojoon Lee, and Jaegul Choo. Enemy spotted: In-game gun sound dataset for gunshot classification and localization. In 2022 IEEE Conference on Games (CoG), pages 56 63. IEEE, 2022. 3
[61] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 52-69. Springer. 2020. 3
[62] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. Proceedings of the IEEE international conference on computer vision, pages 609-617, 2017. 3
[63] Kevin JP Woods, Max H Siegel, James Traer, and Josh H McDermott. Headphone screening to facilitate web-based auditory experiments. Attention, Perception, & Psychophysics, 79:2064-2072. 2017. 3
[64] Muhammad SA Zilany, Ian C Bruce, Paul C Nelson, and Laurel H Carney. A phenomenological model of the synapse between the inner hair cell and auditory nerve: long-teny teretermmme adaptation with power-law dynamics. The Journal of the Acoustical Society of America, 126(5):239022) 241002222222220 2009. 3
[65] Richard McWalter and Josh H McDermott. Illusory sound texture reveals multi-sccond statistical completion in auditory scene analysis. Nature communications, 10(1):5096, 2019. 3
[66] Mark R Saddler, Ray Gonzalez, and Josh H McDermott. Deep neural network models reveal interplay of peripheral coding and stimulus statisties pitch perception. Nature communications, 12(1):7278, 2021.
[67] Mark R Saddler and Josh H McDermott. Models optimized for real-world tasks reveal the task-dependent necessity of precise temporal coding in hearing. Nature Communications, 15(1):10590, 2024. 3
[68] Kazuki Shimada, Archontis Politis, Iran R Roman, Parthasaarathy Sudarsanam, David Diaz-Guerra, Ruchi Panday, Kengo Uchida, Yuichiro Koyama, Naoya Takahashi, Takashi Shibuya, et al. Sterco sound event localization and detection with onscrecn/offsereen classification. arXiv preprint 250 2025.
[69] Hogeon Yu. A two-step learning framework for enhancing sound event localization and detection. arXiv preprint 2rXi:: 2025. 3
14

---

[70] Da Mu, Zhicheng Zhang, Haobo Yue, Zehao Wang, Jin Tang, and Jianqin Yin. Seld-mamba: Selective state-space model for sound event localization and detection with source distance estimation. arXiv preprint arXiv:2408, 05057, 2024.
[71] Andrew Francl and Josh H McDermott. Deep neural network models of sound localization reveal how perception is adapted to real-world environments. Nature human behaviour, 6(1): 1-133, 2022. 3, 9
[72] Philipp Bomatter, Mengmi Zhang. Dimitar Karev, Spandan Madan, Claire Tseng, and Gabriel Kreiman. When pigs fly: Contextual reasoning in synthetic and natural scenes, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 255-2644 265, 2021. 4
[73] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 4
[74] Bill Gardner, Keith Martin, et al. Hrft measurements of kemar dummy- microphone, 1994.
[75] Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Exploiting transformation invariance and equivariance for self-supervised sound localisation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3742-3753, 2022. 7, 4
[76] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023. 7.5
177] Zhaohui Zheng, Ping Wang, Dongwei Ren, Wei Liu, Rongguang Ye, Qinghua Hu, and Wangmeng Zuo. Enhancing geometric factors in model lcarning and inference for object detection and instance segmentation. IEEE transactions on cybernetics, 52(8):85748888 2021. 7
[78] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems. 2020. 2
[79] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[80] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational visual media, 8(3):415-244 2022.
[81] Shawn Hershey, Sourish Chaudhuri. Daniel PW Ellis, Jort Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif Saurous, Bryan Seybold, et al. Cun architectures for large-scale audio classification. In ICASSP 2017 2017 IERE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 131- TEFF, 2017. 5
[82] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo, Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34: 14777-21-0 2021. 5
[83] Yuan Gong, Yu-An Chung, and James Glass, AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 57 17155 2021.
[84] Alexey Dosovitskiy. An image is worth 16x words: Transformers for image recognition scale, arXiv preprint arXiv:2010. 1929, 2020. 5
[85] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale datasct for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 2022. 5
[86] Mehdi Cherti, Romain Beaumont, Ross Wightman. Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2818-2829, 2023.
[87] Raghav Goyal. Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitaee et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference computer vision, pages 5842-5850, 2017. 5
15

---

[88] Will Kay. Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vjjyanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:
2017.
[89] Diederik Kinga, Jimmy Ba Adam, al. A method for stochastic optimization. In International conference on learning representations (ICLR), volume Califoria:, 2015.
[90] Arda Senocak, Tae-Hyun Oh, Junsik Kim, sound source in visual scenes. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 4358 4366, 2018. 7
Yang, and In So Kweon. Learning localize
16

---

Supplementary Material
In the supplementary material, we provide additional information of our AudioCOCO dataset in S and human experiment setup in Sec.S2. Moreover, we include extra experimental results, an ablation study, and visualization results for predicted position bias, which are discussed in Sec.S4 and Sec.55.
S1 Details about the AudioCOCO dataset
More details on image selection. The MSCOCO 2014 dataset [47] offers annotations for 80 object categories, including both bounding boxes and segmentation masks. To ensure fair comparison- especially given that some model backbones are pre-trained on MSCOCO we utilized the MSCOCO 2014 validation set rather than the training set. From this, selected 12 common vocal categories frequently encountered in daily life: person, motorbike, train, boat, elephant, bird, cat, dog, horse, sheep, cow, and keyboard. A total of 29,737 images containing at least one vocal object were extracted to form the pool of image candidates for our dataset.
To investigate spatial distribution, we visualized the occurrence frequency of these categories across the 29,737 images Fig. S1b and Fig. Sla, grouping instances by three object area size bins defined in the main draft: object size1 (0-5%), size2 (5-15%), and size3 (15-30%). For smaller object sizes, category locations were relatively uniformly distributed across the image space. However, for larger objects, some categories- such as bus, train, and truck exhibited strong center bias. To correct for this, we filtered the dataset to ensure that no position in the final heatmap exceeded 50% in frequency.
Sputial Cutegories Object Single
Spatial Distribation All
arrllans
hird
best
Coarchaate
CCorrnaaaa eleahart
Condiinta herse
coew
dag
phont
Acaandaate
XCartinata
Cocriinse heyboard
moteroyde
(a) The spatial distribution of single objects by category. (b) The spatial distribution of multi-objects by category.
Figure S1: The visualization results of images' spatial distributions in the AudioCOCO test set presented for each object category, where the color bar indicates the frequency of object occurrences each spatial location within the images. Darker regions correspond to higher frequencies.
Multiple Object Distribution by Category
Single Object Distribution by Category
Bnd
1.40
1.20
Core
30
30
(a) The count distribution of multi oblti-o by category. (b) The count distribution of single objects by category.
Figure S2: The visualization results of the image count distributions in the AudioCOCO test set are presented, where blue, green, and red represent object sizel, size2, and size3, respectively. This visualization illustrates how instances are distributed across different object size categories, highlighting the dataset's 's balanced distribution in terms of object sizes.

---

Spotial Distribttion Categories Object Stigle
Single Object Distribution by Category
Coanchaate
Cortinate
KCaorehula sleshurt
Crediinte herse
120
cow
day
Objectt Caint
Ensrance Countt
XCortinat
(a) The spatial distribution of single objects by category (b) The count distribution of single objects by category for AudioCOCO's 's training set.
for AudioCOCO's 's training set.
Figure S3: The visualization results of the image count distribution and spatial distribution for the AudioCOCO training set. For the count distribution, object sizes are color-coded: blue for Size1, green for Size2, and red for Size3. This plot illustrates the number of object instances across different size categories, confirming that the dataset maintains a balanced distribution in terms of object size. For the spatial distribution, the heatmap reflects the frequency of object occurrences at each spatial location in the images, with the color bar indicating frequency. Darker regions correspond to areas of higher object density, highlighting positional biases or spread across the dataset.
Additionally, given that the image distribution over all object categories in COCO is not uniform, we performed random sampling to cap the number of instances to 150 per category per object size bin, promoting balanced representation within the AudioCOCO dataset. The final distributions after filtering are shown in Fig. S2a, Fig. S2b, and Fig. S3, demonstrating that AudioCOCO achieves a balanced dataset across object area sizes, object center positions, and categories.
More details audio selection. To ensure that only high-quality, semantically aligned clips are retained, we apply the following three filtering steps to the videos from VGGSound [49]. First, introduce Semantic Consistency (SeC) and select audios that are representative of the semantic object categories. Specifically, we take all audio files belonging to the same semantic category, extract audio features from the last layer of the Wav2vec [78] model, compute their pari-wise cosine similarities based on these features, and retain the top 80% audios with the highest cosine similarity scores. Second, ensure the audio is free from noise or interference from unrelated sources, we introduce Mel- Mel-Spectrogram Similarity (MSS) as filtering criterion within each category. For each audio clip, we compute its Mel spectrogram, average the frequency magnitudes over time, and apply logarithmic transformation to compress representation of the audio's 's overall spectral structure. We then calculate the cosine similarity between these representations and retain the top 65% of audio clips based on this MSS metric, following the SeC criteria. Third, to ensure the stereo audio corresponds to the sounding object being centered in the video frame, we introduce Spatial Consistency (SpC)) for each audio-image pair. We compute the Spearman correlation between the left and right audio channels; high correlation suggests the sound source is centrally located, producing similar waveforms in both channels. We retain the top 50% of audio clips based on this metric from the MSS-filtered set.
components -yieedii-ieldinggg a compact
For audio selection, we also enforce minimum threshold of 20 audio clips per category. If any filtering step results in category falling below this threshold, that specific step discarded. and the previous filtering output is retained as the final result to ensure sufficient data coverage across all categories. The final distributions after filtering are shown in Fig. S4.
S2 Details of human psychophysics experiments
All the experiments are conducted with the subjects' informed consent and according to protocols approved by the Institutional Review Board of our institution. Participants were instructed to use a computer mouse to click on the perceived location of the sound source within time limit of 20 seconds, while wearing stereo headphones (SENNHEISER MOMENTUM with active noise cancellation) throughout the experiment.
2

---

Spatial Conaistancy Before and After Fittring
Spectral Condstency Before and After Fitedin
Original
ebar filsing
Semantic Condictency Before and After Filternng
Nordo
(a) The three stages of audio filtering result for (b) The comparison between original audio counts and AudioCOCO test set.
filtering audio counts for AudioCOCO test set.
Figure S4: The visualization results of audio statistics in the AudioCOCO test set demonstrate that the audio quality across most categories has significantly improved following the filtering process. Additionally, we ensured a balanced count distribution among all categories. These outcomes highlight the effectiveness of our selection and refinement strategy in enhancing both the semantic consistency and acoustic quality of the dataset.
Spatial Conalstancy Before and After Fittering
Spectral Condistency Before and After Fitering
Original Files
Kbr Fitunng
Semantic Consictency Before and Filterng
(a) The three stages of audio filtering result for (b) The comparison between original audio counts and AudioCOCO training set.
filtering audio counts for AudioCOCO training set.
Figure S5: The visualization results of audio statistics in the AudioCOCO training set demonstrate that the audio quality across most categories has significantly improved following the filtering process. Additionally, we ensured a balanced count distribution among all categories. These outcomes highlight the effectiveness of our selection and refinement strategy in enhancing both the semantic consistency and acoustic quality of the dataset.
Audio calibration. Instead of relying on the pre-rendered image-audio pairs, we conducted an audio calibration procedure at the start of the experiment to account for individual differences in height and auditory perception. To achieve this, we implemented real-time stereo audio synthesis in Unity, which communicates with MATLAB (hosting the human behavioral experiment) via TCP connections. The measured delay for real-time stereo sound synthesis and presentation is within 500 milliseconds, ensuring seamless interaction between the two systems.
During calibration, participants were presented with a white dot at a random location on the screen, accompanied by an audio clip spatially rendered at the dot's position. A white cross at the center of the screen served as a spatial reference (see Supp Fig.S6). Using the keyboard's arrow keys, participants adjusted the perceived audio source position until the sound aligned with the white dot.
3

---

Move the speaker to match
with the position of the white circle
Click the quadrant to which the speaker belongs.
Time
Figure S6: The audio calibration and validation process. During calibration, the player is at random position close to the white dot. The red arrow represents a potential movement trajectory. During validation, the display is divided into equal quadrants, with the audio player appearing a random position three times within each quadrant.
Once satisfied, participants pressed the ESC key to confirm the alignment. This process was repeated six times with different white dot positions.
For each calibration step E 1, .... 6, we recorded the participants' adjustments of the sound source location in pixels (Az,, Az,). The calibration hyperparameters a and were computed the mean of Ax, and the sum of a respectively. These hyperparameters were then applied to scale the Unity coordinates x. and zu. correcting for individual perceptual biases in auditory localization during the main experiment. The rationale for using the mean adjustment for Az, (horizontal direction) and the sum for A2, (vertical direction) based on human spatial hearing characteristics listeners typically localize horizontal (azimuth) sound sources with higher precision than vertical ones. This design allows greater tolerance for variability the vertical dimension (altitude) during calibration, ensuring more robust alignment with participants' perceptual expectations.
....,,,,
successful calibration and adequate spatial hearing accuracy. Participants heard an audio clip played from one of two possible locations
Audio validation. Following calibration, audio validation task conducted
horizontally arranged 1x2 grid displayed the screen (see Supp Fig. S6). They were instructed to click on the half of the screen from which they perceived the sound. This validation procedure was repeated six times. If participant' spatial sound localization accuracy fell below 83%, the calibration and validation procedures were repeated to guarantee reliable data quality during the actual experiment.
Center fixation presentation before the visual stimulus onset. During the experiment, participants instructed to fixate central dot before each trial began. This pre-trial fixation is a standard element in human psychophysics and cognitive neuroscience, designed to recenter attention and minimize carry-over effects across trials. By requiring participants begin each trial from a common spatial and attentional baseline, this design ensures that any differences in response latency or eye movement patterns be attributed the experimental manipulation, rather than lingering attentional bias from the previous trial.
$3 More implementation details of AI models
SSLTI [75], LVS [44], FNAC [43], and IS3 [45] are SSL models based on dual-stream architectures with 2D Convolutional Neural Networks (CNNs). Each model processes visual and auditory inputs separately using dedicated encoders before fusing the features for contrastive learning during training. Both encoders are based on ResNet18 [79]. Beyond standard contrastive learning, IS3 introduces an Intersection-over- -overtttiioo-v---niiioo (IoU) loss and semantic alignment loss to improve localization accuracy and better alignment between audio and visual modalities during supervised training. All the models trained the FlickrSoundNet [58] and VGG-Sound [49] datasets.

---

Feature Map for CNN-based models
Heatmap
Image-Stereo Pair
Vision Encoder
Compute Similarity
Compute Accuracy
Ground
Truth
Audio
Encoder
Dot Product
Average Stereo
Heatmap
Token Sequence for Transformer-baset models
Figure S7: Overview of SOTA multi-modal models for sound source localization (SSL). All models receive paired images and mono audio inputs, where stereo signals are averaged into a single channel. Visual and auditory inputs are processed independently through separate encoders. For CNN-based models (indicated by blue arrows), feature maps from the final layers of each encoder extracted, compared via cosine similarity, and used to generate similarity heatmap. For transformer-basee models, output token sequences are obtained from both encoders. Dot products on their token embeddings are calculated, and a heatmap produced accordingly. During evaluation, SSL accuracy is determined by verifying whether the location of the maximum activation on the heatmaps (red triangles or red tokens) lies within the segmentation mask of the ground truth sounding object (red bounding box).
Figure S8: While humans benefit from visual cues, they remain capable without them in the AOnly condition, unlike AI models, which fail localize sound sources without congruent visual information. The accuracy under AOnly conditions with both grayscale and Gaussian noise backgrounds drops for both humans and models compared to the congruent condition on object size 2.
0.8
Random
EchoPin
0.6
Human
A-Acc
0.4
0.2
0.0
AOnly(Gray)
AOnly(Noise)
Congruent
CAVP [42] is sound source segmentation model that follows dual-stream CNN architecture. uses PVTV2-B5 [80] as the visual encoder and either VGGish [81] or ResNet18 as the audio encoder. CAVP is trained in fully supervised manner using cross-entropy loss for segmentation and contrastive learning to align audio-visual features. Training data includes AVSBench [35] and
VPO [42] datasets.
AVSegformer [23] is an audio-visual semantic segmentation model built on dual-stream transformer-baset architecture. It employs SegFormer [82] the visual backbone and Audio Spectrogram Transformer (AST) [83] as the audio encoder to capture fine-grained cross-modal representations. AVSegformer integrates both modality-specifft and fused token embeddings through a lightweight fusion decoder for pixel-level prediction. is trained in a fully supervised setting using a combination of cross-entrope loss for segmentation and audio-visua consistency loss. AVSegformer's training data includes three subsets of AVSBench.
ImageBind [76] and LanguageBind [33] are large-scale, multi-modal transformer models with billions of parameters. These models are trained with contrastive learning objectives across six modalities: image, audio, video, text, depth, thermal, and IMU, and embed these into a shared representation space. Both models use a Vision Transformert ViT) [84] for visual encoding and AST [83] for audio encoding. ImageBind is trained
large-scale datasets including LAION [85, 86], SSv2 [87], and K400 [88]. LanguageBind builds on the pre-trained encoders from ImageBind and further fine-tunes them the VIDAL-10M [33] dataset.
excelptexcept EchoPin-S and EchoPin-takke paired image and mono audio inputs, processing each modality independently through dedicated visual and audio encoders. For the mono audio setup, we follow the EchoPin-M design by averaging the stereo channels into single-channel input. As shown in Fig. S7, for CNN-based models, feature
Implementation details of models. All eleven
5

---

AV-Congruent
EchoPin
153
ConflictVCue
EchoPin
1S3
0.0
AbsVCue
EchoPin
IS3
Multilnstloc
EchoPin
153
0.0
Figure S9: The figure shows the failure examples of both IS3 and EchoPin compared to humans. The leftmost images (Columns and 4) show the correct localization results made by humans. Red boxes mark the ground truth sound source locations, while green circles indicate mouse click responses from multiple participants. The middle columns (2 and 5) and rightmost columns (3 and 6) display heatmaps predicted by EchoPin and IS3, respectively. Yellow triangles on the heatmaps denote the predicted sound source locations from each model. See the colorbar for the activation values of the heatmaps.
maps from the final layers of the encoders extracted, and cosine similarity is computed between them to produce similarity heatmap. For wransformer-basee models, token sequences from both encoders retrieved, and pairwise dot products of their token embeddings calculated generate the heatmap. The predicted sound source location is identified as the point with the highest activation on the heatmap.
evaluated all the current models using their publicly available, pre-trained weights and adhered to their original implementation details. Since AVSBench [35] shares some images with AudioCOCO test set, we exclude those overlapping image- pairs when evaluating CAVP [42] and AVSegformer [23]. Experiments for ImageBind and LanguageBind were conducted on 8 NVIDIA A100) GPUs, whereas all other models, including EchoPin and its variants, were trained and evaluated on 4 NVIDIA A6000 GPUs. We fine-tune the EchoPin models using the Adaptive Moment Estimation (Adam) optimizer [89] with a weight decay
10-4 for 10 epochs. The initial learning rate is set to 1 10-5, and the batch size is 16. Each fine-tuning session takes approximately 16 hours complete. To accelerate data loading, all audio waveforms are preprocessed and stored as cochleagram tensors in advance, with each .ny file occupying roughly 160 MB. All model evaluations are repeated three times with different random seeds to ensure statistical reliability.
$4 More qualitative results of humans and AI in SSL
As shown in Fig.S9 and Fig.S10, we further illustrate the limitations of EchoPin relative human performance and provide additional qualitative results for other baselines. Notably, EchoPin often fails to localize the correct sounding object when a visually salient distractor occupies large portion of the scene. This suggests that the model remains vulnerable to visual saliency bias, tending to prioritize large or central objects even when they are not the true auditory source-a. tendency that human listeners are better equipped to suppress.
Similar patterns are shown in CAVP, which exhibits modality conflict sensitivity akin IS3, Both models are frequently misled by conflicting audio-visual cues and demonstrate systematic bias toward the visual modality, highlighting a lack of robust auditory grounding under incongruent conditions.
S5 More quantitative results of AI models in SSL
The raw A-Acc may confounded by the area of the ground-truth mask-sines intuitively, smaller objects are more difficult to localize while larger ones are easier. To address this potential bias, we introduce chance-corrected gain metric, which quantifies the improvement human or model over random guess, normalized by the baseline accuracy of the random guess:

---

Congruent
EchoPin
CAVP
ConflictVCue
EchoPin
CAVP
0.0
AbsVCue
EchoPin
CAVP
MultilnstLoc
EchoPin
CAVP
0.0
Figure S10: The figure shows the failure examples of CAVP and EchoPin compared to humans. The rightmost columns (3 and 6) display heatmaps predicted by CAVP.
Accx
ACCrand
Gain
x 100
AcCrand
The AcCrand represents the accuracy achieved when predictions are sampled uniformly at random across the scene.
Normalized Percentage
Size
Size 2
Size 3
(EchoPin Random) / Random
5.2
4.6
2.9
(Human Random) Random
31.9
7.3
3.6
(Human FchoPin)/ Random
26.7
3.0
0.8
Table S1: We report the chance-corrected gain for humans and models across three object sizes under the congruent condition. While human performance increases significantly with larger object sizes, models show limited improvement. This discrepancy may stem from AudioCOCO uniform object size distribution, which limits the models' ability exploit size-dependent cues.
This normalization removes the influence of mask size and isolates the true localization capability of humans and models. As shown Ta even after correcting for chance, humans consistently outperform models- models-especially especially for small object sizes. For example, in the smallest size category, humans achieve a gain of 31.9% 31.9% compared to only 5.2% for models. These results suggest that object size modulates sound source localization performance in way that cannot be fully explained by ground-truth mask area alone, highlighting deeper perceptual and representational differences between human and model behavior.
apply the same evaluation criterion to both human participants and models to ensure fair comparisons: a prediction is considered correct if the peak activation (for models) or the human click falls within the bounding box of the sounding object. We also conducted an additional analysis by varying the pixel distance thresholds used to determine correctness. Specifically, prediction considered correct if it falls within ø pixels of the ground-truth bounding box. We report the function of pixel thresholds in Tab.S2 based on
resulting A-Acc (accuracy with spatial tolerance) congruent conditions for object size2. From these results, we observe that while larger thresholds naturally lead to higher A-Ace values, the relative performance trend between humans and models remains consistent. This further supports the validity our evaluation methodology.
Moreover, we conducted additional experiment by fine-tuning our EchoPin-S model using the VGG-SS and Flickr- SoundNet datasets, and report comparative results against IS3 and CAVP using the CloU metric under the default evaluation protocol provided by [90]. As localization-wwan metric, CloU accounts for overlap area, center distance, and aspect ratio aligument, offering a more comprehensive assessment of spatial prediction quality. From Tab. S3, observe that benefiting from the ine-ggrinedd spatial features provided by AudioCOCO, our EchoPin-S model achieves superior performance these standard SSL benchmarks, outperforming the baselines.
7

---

Threshold
Random
Human
EchoPin-S
EchoPin
(default)
9.5
79.3
17.3
50,9
10
9.6
79.9
18.5
52.0
25
9.9
80.4
19.8
52.4
Table S2: We compare performance across varying localization thresholds (0-25 pixels) under the congruent condition for object size 2. As shown, both human and model accuracy exhibit only marginal improvement with increasing thresholds, indicating that our evaluation metric stable and not overly sensitive to small shifts in the decision mundary-therelle validating its robustness.
Method
VGG-SS
Flickr-SoundNet
IS3
42.96
84.40
CAVP
43.58
85.03
EchoPin-S
43.61
85.25
EchoPin
45.02
85.87
Table S3: We conduct comparison experiments on the VGG-SS and the challenges posed by these large-scale, imbalanced public benchmarks, EchoPin consistently outperforms prior tfte-oofthee-a (SOTA) methods, highlighting the robustness of spatial cues and the effectiveness of our peerossceenceinspireee design.
datasets. Despite

---

perform performances. respectively, Since CAVP and AVSegforme are trained on MS COC0201 we evaluate them on a subset of AudioCOCo and denote their results with the symbol f.
Table S4: L'omparioon of Al models in our benchmarke across six conditions and three object sizes. Bold and underlined values indicate the best and second- best
LangiageBBind
Human
EchoPin
Inugefinnd
CAVP*
LVV
SSLITE
PNAC
IS3
Random
Acc(F)
Model
59.3
11.2
1.1
1.4
2.7
3.1
1.3
1.4
1.8
2.3
1.8
Size1
Congruent
79.3
50.9
4.6
5.8
15.5
15.2
9.7
9.9
12.5
16.5
9.5
Size2
90.7
76.00
12.7
14.5
39.5
38.7
23.6
23.7
26.9
31.1
19.6
Size3
40.0
16.6
1.1
1.2
3.6
388
2.0
2.1
1.9
1.7
1.8
Size
Conflicting
$7.9
21.9
5.4
5.7
7.1
7.0
6.5
6.8
7.2
12.3
9.5
Size2
A-Acc
67.9
26
10.7
11.3
12.8
13.2
12.1
12.4
13.3
24.8
19.6
Suz33
54.3
11.8
3.2
3.4
5.3
5.5
5.0
5.2
5.8
3.3
1.8
Sizel
Absant VCae
65.77
$4.0)
699
7.2
9.8
10.2
9.4
9.9
10.5
13.5
9.5
Size
80.0
77.2
10.6
12.8
17.2
18.4
16.0
16.5
18.6
29,8
19.6
Size3
9.3
10.0
0.2
0.3
0.9
0.9
0.7
0.7
1.1
2.3
1.8
Sizel
49.4
16.9
3.0
3.1
4.5
4.7
3.8
3.9
4.8
8.8
9.5
Size2
Cray
Vision
76.8
17.9
6.5
6.6
7.9
8.4
8.3
8.2
8.7
11.9
19.6
Size3
Andin Only
12.3
4.0
0.4
0.4
1.1
1.2
1.0
1.1
1.3
3.3
1.8
Sizel
45.1
2112
3.7
3.8
4.6
4.9
4.2
4.3
5.0
17.3
9.5
Size2
Notie
Visioo
80.3
40.8
7.6
7.5
9.4
9.8
8.8
8.9
9.6
39.7
19.6
Sizc3
26.3
5.6
5.8
8.0
8.4
6.2
6.1
6.2
21.5
1.8
Size
39,0
13.2
14.5
20.3
21.2
1644
165
16.5
35.2
9.5
Sinc2
Sileni
Audio
53.7
24.9
27.2
29,9
44.5
40.5
40.7
43.8
42.7
19.6
Size3
Vision Only
13.5
4.1
4.2
8.7
6.6
5.4
5.5
5.4
20.5
1.8
Sizel
17.6
13888
16.3
17.8
16.5
12.7
12.7
12.6
31.6
9.5
Sizc2
Noöse
Audio
21.9
24.8
24.0
32.6
36.3
12.2
12.8
11.9
38.9
19.6
Size3
25.7
4.5
0.4
0.5
2.5
2.9
0.5
0.6
0.7
4.8
1.6
Sizel
36.4
24.
2.1
2.7
7.3
7.5
3.9
2.5
3.9
7.9
9.1
Size3
A-Acc
38.6
47.1
13.0
15.3
20.2
20.4
25.3
24.0
26
22.4
21.3
Size3
Multi Instunce
60.9
37.5
6,2
6.5
11.2
10.5
12.5
13.2
13.4
11.9
8,4
Sizel
83.8
53
10.8
12.4
23.7
23.0
22.4
20.7
25
24.1
17.8
Size2
V-Acc
89.1
61.2
22.5
25.1
40.9
39.5
50.6
49.4
52.8
40,9
26.2
Size3

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction accurately summarize the paper's key contributions and align well with the scope and results presented throughout the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper explicitly discusses the limitations of the proposed method,
acknowledging its constraints and outlining areas for future improvement in section of Discussion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
10

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper provides sufficient details on the experimental setup, model architecture, and evaluation protocol to support reproducibility of the main results and states that the code, model, and data are publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
11

---

5. Open access to data and code
Question:
Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The paper indicates that the code and data made publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines futtps://nipe) cc/
public/ public/guides/ Stw.dddee//oeeSSmissssssse..s
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. cw/ccc/ppb........ coc/pullic/aae callic//giide gttgguuiss///////g Cese/oooduubbii ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
The paper specifies key training and testing details, including data selection pipeline, hyperparameters and optimizer settings, allowing readers to understand how the results were obtained both in the main body and the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper contains error bars and we test all models with runs and report its mean. For human experiments, we summarize all results, randomly select 60% data, calculate the accuracy with runs and report its mean.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
12

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computational resources used for model experiments and human experiments in the section of Experiment Details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerrpp..e/e cat cc/publice habbblcc//hhtt e/Ethi csGuidelines?
Answer: [Yes]
The research presented in the paper adheres the NeurIPS Code of Ethics, with no identified ethical concerns regarding the methods, data usage, or potential societal impact.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work shares the general ethical considerations common research,
and does not present any unique specific socictal impact that warrants separate discussion.
13

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For the human experiment data, all participants have approved to use their anonymous data for research activity and signed the consent form under the supervision of IRB.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
14

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode, thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
The paper uses existing assets, which the authors have properly acknowledged. It also contributes new assets, including novel models, datasets, and benchmarks. Details of these contributions are clearly described in the paper, and all associated code, data, and models are made publicly available.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Details of the human experiments are provided in both the main body and the Experiment Details section of the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
15

---

Justification: IRB has approved our human behavioral experiments. pose no risks to participants.
The human experiments
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy pettps://neuripee http://uriip..ccooeeeeeeeeeeeeee for what should or should not be described.
16


Reviews:
[{"review_id": "BXxAYp9feV", "reviewer": "Reviewer_f4Cu", "summary": "The paper focuses on audiovisual sound source localization. It is a study focussing on the interactions between the different modalities. Particularly interesting are cases where the two modalities have conflicting cues or if either is missing or noisy. They construct a new dataset from images in COCO and audio from VGGSound and smartly pair them. They use depth estimation models to construct stereo sounds. They compare results of AI models with a human study in similar conditions and derive interesting observations from them.", "questions": "I would gladly increase my rating if the authors can defend the choice of metric and address my concerns about its possible effects on the paper's key takeaways.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well written and clear, with sufficient detail for the dataset construction. I liked the overall problem itself, and believe it can have a good impact on audiovisual research.\n2. The experiments are well designed and thorough. \n3. The dataset itself is a solid contribution and can also be useful for other audiovisual tasks beyond source localization.\n\n**Weaknesses**\n\n[Metric]\n\n4. I am skeptical about the metric. I may be underestimating performance in many cases. If a human click is just a few pixels outside the segmentation map, I would argue this is still a correct (almost) prediction, but the metric would mark it as 0. Similarly, if the peak of predicted probabilities is even slightly off, it is marked as completely wrong. \n5. The metric may also be overestimating performance, for example, in the ConflictVCue case of Fig. 8 where a prediction over the entire horse would be marked correct, even though its truly the horse’s head that should be marked correct.\n6. The trend observed in Fig. 4 across different sizes may actually be because of the inherent bias in the metric itself. Bigger objects have more pixels that can be predicted to score correctly. The increase in the performance of the random predictor seems to be very similar to that of the models and human performance.\n\n[Model]\n\n7. Based on the heatmaps in Fig. 8, I think that IS3 results may not be a good datapoint to compare against. I would recommend the authors to revisit claims based on IS3 and maybe constrain their analysis to IS3-FT. \n8. I am not familiar with the fixation cross. Can the authors include a brief description for why its needed.\n\n\n[Results]\n\n9. In Fig. 6, I could not see the dashed legend of vertical bias. May be a rendering issue at my end. Are the left ones for vertical and right ones for horizontal?\n10. Was IS3 originally trained with stereo audio? Based on Table 1 (b) it seems like the gains from switching to stereo is somewhat consistent for IS3 and IS3+FT. If IS3 is not trained on stereo, I would find this trend curious and wonder if the performance difference between IS3 and IS3+FT is primarily due to smaller domain gap and not better utilization of stereo audio.", "was_revised": true, "final_justification": "The authors addressed most of my concerns, but I was still not quite convinced about the choice of metrics. \n\nOverall, I am in support of this paper being accepted, but I also feel there is room for improvement. Therefore, I maintain my original rating of Borderline Accept.", "rebuttal": "Thank you for the detailed review and thoughtful feedback on our work. Below, we address the concerns point by point and respond to the reviewer’s questions individually.\n\n**f4Cu.1 - Questions: I am skeptical about the metric. I may be underestimating performance in many cases. If a human click is just a few pixels outside the segmentation map, I would argue this is still a correct (almost) prediction, but the metric would mark it as 0. Similarly, if the peak of predicted probabilities is even slightly off, it is marked as completely wrong.**\n\nWe apply the same evaluation criterion to both human participants and AI models to ensure fair comparisons: a prediction is considered correct if the peak activation (for models) or the human click falls within the segmentation mask of the sounding object. In other words, all evaluations are conducted on equal footing.\nIn response to the reviewer’s suggestion, we conducted an additional analysis by varying the pixel distance thresholds used to determine correctness. Specifically, a prediction is considered correct if it falls within x pixels of the ground-truth segmentation mask. We report the resulting A-Acc (accuracy with spatial tolerance) as a function of pixel thresholds in Table R1 below.\nFrom these results, we observe that while larger thresholds naturally lead to higher A-Acc values, the relative performance trend between humans and models remains consistent. This further supports the validity of our evaluation methodology.\nWe will include these results and the corresponding discussion in the final version.\nTable R1. A-ACC as a function of thresholds under congruent conditions for object size2\n| Threshold      | Random | Human | IS3-FT-Stereo |\n|---------------:|:------:|:-----:|:-------------:|\n| 0 (default)    | 9.5    | 56.6  | 29.1          |\n| 10             | 9.6    | 57.4  | 29.9          |\n| 25             | 9.9    | 58.3  | 31.2          |\n\n**f4Cu.2 - Questions: The metric may also be overestimating performance, for example, in the ConflictVCue case of Fig. 8 where a prediction over the entire horse would be marked correct, even though its truly the horse’s head that should be marked correct.**\n\nA frog may emit vocalizations from its belly, a human from the throat, and a car from either the horn or the engine—highlighting that the exact sound-emitting region can vary across object categories. The reviewer raises an insightful point regarding fine-grained sound localization, which we agree is an important direction. We will include this consideration in the future work section.\nFor the current study, we use the ground-truth segmentation mask of the entire sounding object, rather than attempting to isolate fine-grained vocalization regions, which are often unavailable in existing datasets.\nAs mentioned in our earlier response to **f4Cu.1 - Questions**, we apply the same evaluation criterion to both human participants and AI models to ensure fair comparisons. While varying the evaluation criteria (e.g., localizing specific sound-emitting parts) could affect the absolute A-Acc values, we expect that the relative performance trends would remain consistent.\n\n**f4Cu.3 - Questions: The trend observed in Fig. 4 across different sizes may actually be because of the inherent bias in the metric itself. Bigger objects have more pixels that can be predicted to score correctly. The increase in the performance of the random predictor seems to be very similar to that of the models and human performance.**\nThe reviewer correctly points out that raw A-Acc may be confounded by the area of the ground-truth mask—since, intuitively, smaller objects are more difficult to localize while larger ones are easier. To address this potential bias, we introduce a chance-corrected gain metric, which quantifies the improvement of a human or model over a random guess, normalized by the baseline accuracy of the random guess:\n\n$$\n\\mathrm{Gain}(X) = \\frac{\\mathrm{Acc} _X - \\mathrm{Acc} _{\\text{rand}}}{\\mathrm{Acc} _{\\text{rand}}} \\times 100 \\%\n$$\n\n$\\displaystyle \\text{Acc}_{\\text{rand}}$ represents the accuracy achieved when predictions are sampled uniformly at random across the scene. \n\nThis normalization removes the influence of mask size and isolates the true localization capability of humans and models. As shown in **Table R2** below, even after correcting for chance, humans consistently outperform models—especially for small object sizes. For example, in the smallest size category, humans achieve a gain of 17.9%, compared to only 1.7% for models.\nThese results suggest that object size modulates sound source localization performance in a way that cannot be fully explained by ground-truth mask area alone, highlighting deeper perceptual and representational differences between human and model behavior.\nTable R2. Chance-corrected gain for humans and models across object sizes\n| Normalized Percentage  | Size 1 | Size 2 | Size 3 |\n|------------------------|:------:|:------:|:------:|\n| (model − random) / random | 1.7   | 0.4   | 0.4   |\n| (human − random) / random | 19.6  | 5.0   | 2.2   |\n| (human − model) / random  | 17.9  | 4.5   | 1.7   |\n\n**f4Cu.4 - Questions: Based on the heatmaps in Fig. 8, I think that IS3 results may not be a good datapoint to compare against. I would recommend the authors to revisit claims based on IS3 and maybe constrain their analysis to IS3-FT.**\n\nWe chose IS3 because it is a widely used baseline in the literature and achieves the best performance among all baselines under most of the conditions. Therefore, we use IS3 as the representative baseline for the main paper’s analyses. Additionally, we compare other baselines against humans and IS3-FT-Stereo, providing these results and analyses in the appendix due to space constraints.\nAs suggested by the reviewer, we will revise Figure 8 to include more visualization results from other competitive baselines such as CAVP, and we will reduce the amount of discussion focused solely on IS3 in the final version.\n\n**f4Cu.5 - Questions: I am not familiar with the fixation cross. Can the authors include a brief description for why its needed.**\n\nThe pre-trial fixation dot is a crucial experimental design element commonly employed in classical human psychophysical studies within neuroscience, cognitive science, and psychology. At the start of each trial, it recenters participants’ attention by requiring them to fixate at the center of the screen. This ensures that attention originates from a common spatial and attentional baseline for every trial, thereby eliminating potential carry-over effects from the preceding trial. As a result, any observed differences in response latency or eye movement trajectories can be confidently attributed to the experimental manipulation.\nIn our human psychophysics experiments, we use eye-tracking devices to monitor eye movements for attention control—an important detail not mentioned in the current version due to space constraints. We will include this information about our eye-tracking data quality control in the final version.\n\n**f4Cu.6 - Questions: In Fig. 6, I could not see the dashed legend of vertical bias. May be a rendering issue at my end. Are the left ones for vertical and right ones for horizontal?**\n\nThis issue is likely due to rendering differences caused by local machine configurations. In each group of the bar plots, the left bar, which features slanted line textures on its face, represents vertical localization performance, while the right bar, without textures, corresponds to horizontal localization precision. To avoid similar rendering problems on other systems, we will update the figure’s presentation style in the final version.\n\n\n**f4Cu.7 - Questions: Was IS3 originally trained with stereo audio? Based on Table 1 (b) it seems like the gains from switching to stereo is somewhat consistent for IS3 and IS3+FT. If IS3 is not trained on stereo, I would find this trend curious and wonder if the performance difference between IS3 and IS3+FT is primarily due to smaller domain gap and not better utilization of stereo audio.**\n\nWe introduce three IS3 variants in the paper:\nFirst, IS3: The original model trained on the Flickr-SoundNet [53] and VGG-Sound [46] datasets.\nSecond, IS3-FT-mono: The original model fine-tuned on our AudioCOCO dataset, using monaural audio as input.\nThird, IS3-FT-Stereo: The original model fine-tuned on our AudioCOCO dataset, using stereo audio channels as input.\nThe original IS3 model exhibits a bias toward large, centrally located objects. As discussed in the Introduction and illustrated in Supplementary Figures S1–S3, our proposed filtering method applied to the AudioCOCO dataset mitigates this bias, resulting in a higher-quality dataset drawn from the same source domain.\nThe observed performance improvements in IS3-FT-Stereo can thus be attributed to both the enhanced data quality and the inclusion of stereo audio. The performance gap between IS3-FT-Mono and the original IS3 model is 3.3% for Size1 (small objects), 7.7% for Size2, and 9.3% for Size3, clearly demonstrating the effect of improved data quality.\nFurthermore, the comparison between IS3-FT-Mono and IS3-FT-Stereo shows gains of 4.4% for Size1, 4.4% for Size2, and 1.4% for Size3, highlighting the contribution of spatial cues derived from stereo audio, rather than domain gap effects.\nTogether, these results validate the effectiveness of our dataset refinement and the utility of stereo input for improved spatial localization. We will clarify these IS3 model variants in the final version."}, {"review_id": "2PTYSh75ow", "reviewer": "Reviewer_BL6m", "summary": "This paper presents a detailed and systematic analysis of modality bias and conflict in the task of sound source localization (SSL). The authors propose six possible audiovisual scenarios: Congruent, ConflictVcue, AbsVcue, Aonly, Vonly, and MultiInstLoc. They synthesize an audiovisual dataset by pairing semantically similar images and sounds using the MSCOCO image dataset and the VGGSound audio dataset. Depth information is inferred from the images using the DepthAnything model, and depth-aware stereo audio rendering is performed in Unity.\n\nThe authors fine-tune the state-of-the-art SSL model IS3 on their newly constructed AudioCOCO dataset, resulting in IS3-FT. In the experiments, both human psychophysics studies and quantitative evaluations are conducted. The results show that object size is positively correlated with sound localization accuracy; audiovisual scenarios involving modality conflict harm localization accuracy. Existing AI models struggle with sound source localization in audio-only settings, while achieving relatively good V-ACC in vision-only conditions—indicating a preference for visual information. Human performance surpasses that of AI models across all audiovisual scenarios, particularly under audio-only conditions. Both humans and AI models perform poorly on multi-instance SSL tasks. The fine-tuned model IS3-FT outperforms the original IS3 on AudioCOCO. Moreover, the stereo-audio-aware IS3-FT-stereo model outperforms its mono counterpart IS3-FT-mono, and displays human-like localization biases—favoring horizontal over vertical precision.Strengths And Weaknesses", "questions": "1. It is unclear whether the fine-tuning of the IS3 model was conducted solely on congruent data, or if modality conflict data was also used. If the latter, the introduction of ambiguous or conflicting audiovisual signals could potentially harm the model’s original performance. The paper should clarify whether modality conflict data was included during fine-tuning and analyze its effect on model robustness.\n\n2. Does IS3-FT outperform IS3 on standard SSL test datasets such as VGG-SS and SoundNet-Flickr-Test? The paper currently lacks evaluations on these widely used benchmarks, making it difficult to assess the generalizability of the improvements reported on AudioCOCO.\n\n3. The paper analyzes the impact of object size on SSL accuracy. However, based on the evaluation metric used for SSL accuracy—where a prediction is considered positive if the location of the maximum activation on the heatmap lies within the segmentation mask of the ground truth sounding object—it is important to note that larger objects naturally have a higher probability of being matched by chance. That is, even a randomly selected pixel from the image is more likely to fall within the object mask if the object is large (as supported by the authors' own experiment involving random sampling). Therefore, the higher SSL accuracy observed for larger objects may largely result from the limitations of the evaluation metric, rather than indicating a true model sensitivity to object size. While the conclusion that “Object size matters for humans and AI” may subjectively make sense, the current experimental design and analysis are insufficient to convincingly support this claim.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "a) Strengths\n\n1.\tThe paper introduces a new and challenging benchmark, UniAV, for the SSL task. The authors conduct a detailed analysis of various possible audiovisual scenarios in SSL, with a particular focus on modality conflict—an interesting and meaningful case that addresses issues not previously considered in the literature and has practical relevance.\n\n2.\tA large-scale, high-quality audio-visual dataset is constructed, simulating realistic, spatialized stereo audio that obeys the physical laws of sound propagation. By fine-tuning existing models on this dataset, the authors achieve performance that surpasses the current state-of-the-art on the UniAV benchmark.\n\n3.\tThe paper includes both psychophysics and quantitative experiments, analyzing the performance differences between existing AI models and humans in SSL tasks.\n\nb) Weaknesses\n\n1.\tThe synthesized dataset is based on static images rather than videos, which leads to a natural temporal inconsistency between the visual and audio information. This creates a gap between the benchmark and real-world sound source localization scenarios.\n\n2.\tThe evaluation is primarily conducted on the authors' proposed dataset (AudioCOCO) and benchmark (UniAV), lacking assessments on commonly used benchmarks in the field. For example, the paper does not demonstrate whether IS3-FT outperforms IS3 on VGG-SS [1] and SoundNet-Flickr-Test [2]. Therefore, the claimed improvement over the state-of-the-art may result from overfitting to the AudioCOCO dataset.\n\n3.\tThe paper lacks a review of related work.\n\n[1] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.\n\n[2] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback on our work. Below, we address the concerns point by point and respond to the reviewer’s questions individually.\n\n**BL6m.1 - Weakness: The synthesized dataset is based on static images rather than videos, which leads to a natural temporal inconsistency between the visual and audio information. This creates a gap between the benchmark and real-world sound source localization scenarios.**\n\nWe agree with the reviewer that studying audio in videos is a natural direction, as both modalities inherently involve temporal information. We acknowledge this limitation in the Discussion section.\nOur decision to start with static images for studying sound source localization (SSL) was motivated by the relative scarcity of work focusing on modality conflict and bias. Static images provide better experimental control, allowing us to isolate key factors more precisely.\nIn future work, we plan to extend our approach to video settings and warmly welcome the community—including the reviewer—to join us in this endeavor.\n\n**BL6m.2 - Weakness: The evaluation is primarily conducted on the authors' proposed dataset (AudioCOCO) and benchmark (UniAV), lacking assessments on commonly used benchmarks in the field. For example, the paper does not demonstrate whether IS3-FT outperforms IS3 on VGG-SS [1] and SoundNet-Flickr-Test [2]. Therefore, the claimed improvement over the state-of-the-art may result from overfitting to the AudioCOCO dataset.**\n\nWe respectfully disagree with the reviewer’s suggestion that the performance improvement in IS3-FT-Stereo is due to overfitting on the AudioCOCO dataset. To clarify, we introduce three IS3 variants in the paper:\nFirst, IS3: The original model trained on the Flickr-SoundNet [53] and VGG-Sound [46] datasets.\nSecond, IS3-FT-mono: The original model fine-tuned on our AudioCOCO dataset, using monaural audio as input.\nThird, IS3-FT-Stereo: The original model fine-tuned on our AudioCOCO dataset, using stereo audio channels as input. \nThe performance gap between IS3-FT-mono and the original IS3 model is 3.3% for Size1 (small objects), 7.7% for Size2, and 9.3% for Size3, clearly demonstrating the positive effect of improved data quality. Furthermore, the comparison between IS3-FT-Mono and IS3-FT-Stereo shows gains of 4.4% for Size1, 4.4% for Size2, and 1.4% for Size3, highlighting the contribution of spatial cues derived from stereo audio, rather than the domain gap effect.\n We will clarify this point explicitly in the final version.\nAs suggested by the reviewer, we conducted an additional experiment by fine-tuning our IS3-FT-Stereo model using the VGG-SS and Flickr-SoundNet datasets, and report comparative results against IS3 and CAVP using the cIoU metric under the default evaluation protocol provided by [a]. From ** Table R3**, we observed that benefiting from the fine-grained spatial features provided by AudioCOCO, our IS3-FT-Stereo model achieves superior performance on these standard SSL benchmarks, outperforming the baselines. These results and discussions will be included in the final version of the paper.\n\n[a]. Arda Senocak et al. (2018) Learning to localize sound source in visual scenes.\nTable R3. Results in cIoU for models on VGG-SS and Flickr-SoundNet\n| Method        | VGG-SS | Flickr-SoundNet |\n|---------------|:------:|:---------------:|\n| IS3           | 42.96  | 84.40           |\n| CAVP          | 43.58  | 85.03           |\n| **IS3-FT-Stereo** | **43.79** | **85.65** |\n\n**BL6m.3 - Questions: It is unclear whether the fine-tuning of the IS3 model was conducted solely on congruent data, or if modality conflict data was also used. If the latter, the introduction of ambiguous or conflicting audiovisual signals could potentially harm the model’s original performance. The paper should clarify whether modality conflict data was included during fine-tuning and analyze its effect on model robustness.**\n\n\nWe fine-tuned IS3 (IS3-FT) using only congruent condition data, with the goal of evaluating the model’s out-of-distribution (OOD) performance under five other novel and more challenging conditions—such as audio-visual conflicts.\n\nTraining IS3-FT-Stereo on modality conflicts is not even valid. In such scenarios, there is often no definitive ground truth, as conflicting cues introduce inherent ambiguity. Localizing the car with a barking dog sound would confuse the models to learn correct visual semantic and audio semantic associations.  \n\nTherefore, rather than training on these conditions, our paradigm in benchmarking model behavior when confronted with such real-world inconsistencies becomes more valuable. \nConflict conditions are not pathological but common in natural environments (e.g., occlusions, off-screen sound sources, or mismatched audio and visual stimuli). Our long-term objective is to propose new methods capable of flexibly interpreting and resolving these complex scenes, building on the foundational contributions of this paper.\n\n\n\n**BL6m.4 - Questions: Does IS3-FT outperform IS3 on standard SSL test datasets such as VGG-SS and SoundNet-Flickr-Test? The paper currently lacks evaluations on these widely used benchmarks, making it difficult to assess the generalizability of the improvements reported on AudioCOCO.**\n\nSee the response to ** BL6m.2 - Questions **.\n\n**BL6m.5 - Questions: The paper analyzes the impact of object size on SSL accuracy. However, based on the evaluation metric used for SSL accuracy—where a prediction is considered positive if the location of the maximum activation on the heatmap lies within the segmentation mask of the ground truth sounding object—it is important to note that larger objects naturally have a higher probability of being matched by chance. That is, even a randomly selected pixel from the image is more likely to fall within the object mask if the object is large (as supported by the authors' own experiment involving random sampling). Therefore, the higher SSL accuracy observed for larger objects may largely result from the limitations of the evaluation metric, rather than indicating a true model sensitivity to object size. While the conclusion that “Object size matters for humans and AI” may subjectively make sense, the current experimental design and analysis are insufficient to convincingly support this claim.**\n\nThe reviewer correctly points out that raw A-Acc may be confounded by the area of the ground-truth mask—since, intuitively, smaller objects are more difficult to localize while larger ones are easier. To address this potential bias, we introduce a chance-corrected gain metric, which quantifies the improvement of a human or model over a random guess, normalized by the baseline accuracy of the random guess:\n$$\n\\mathrm{Gain}(X) = \\frac{\\mathrm{Acc} _X - \\mathrm{Acc} _{\\text{rand}}}{\\mathrm{Acc} _{\\text{rand}}} \\times 100 \\%\n$$\n\n$\\displaystyle \\text{Acc}_{\\text{rand}}$ represents the accuracy achieved when predictions are sampled uniformly at random across the scene. \n\nThis normalization removes the influence of mask size and isolates the true localization capability of humans and models. As shown in ** Table R2 ** above, even after correcting for chance, humans consistently outperform models—especially for small object sizes. For example, in the smallest size category, humans achieve a gain of 17.9%, compared to only 1.7% for models.\nThese results suggest that object size modulates sound source localization performance in a way that cannot be fully explained by ground-truth mask area alone, highlighting deeper perceptual and representational differences between human and model behavior.\n\n\n**BL6m.5 - Weakness:The paper lacks a review of related work.\n[1] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.\n[2] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.**\n\nThanks for recommending these two excellent works!\n\n[1] introduces the VGG-SS dataset, which extends VGG-Sound with fine-grained ground truth annotations to support audio-visual segmentation (AVS) tasks. It also proposes a method for mining hard samples and integrating them into contrastive learning pipelines to enhance model performance.\n[2] presents the Flickr-SoundNet dataset, aimed at enabling large-scale audio-visual learning and localization in unconstrained, real-world environments.\n\nWhile both datasets have significantly advanced benchmark development for sound source localization (SSL), they also face key limitations. Specifically, the original audio tracks are monaural, which limits the availability of spatial cues essential for accurate localization by AI models. In addition, neither dataset explicitly addresses location bias or category imbalance, both of which can hinder model generalization.\n\nMoreover, the hard sample mining strategy in [1] may be less effective—or even counterproductive—under multi-instance conflict scenarios, where overlapping or incongruent audiovisual cues are common. In such settings, contrastive learning objectives may be misled by ambiguous or conflicting signals.\n\nTo overcome these challenges, we introduce a filtering metric to curate high-quality, spatially balanced samples in our AudioCOCO dataset. We also render stereo audio using Unity, allowing us to create more realistic, spatially informative scenes. This enables AudioCOCO to support a wider range of multi-modal reasoning tasks under both congruent and conflicting conditions.\n\nWe will cite these important works and explicitly highlight the differences between them and ours in the final version of the paper."}, {"review_id": "0t56dfDaZu", "reviewer": "Reviewer_AmaU", "summary": "This paper investigates modality bias and conflict resolution in multimodal AI systems for sound source localization (SSL). Through a carefully designed psychophysics-inspired framework, the authors benchmark human and AI performance across six audio-visual conditions (e.g., congruent, conflicting, absent cues). They introduce *AudioCOCO*, a large-scale dataset synthesized via 3D simulation, containing stereo audio rendered from static images. Their experiments reveal that current AI systems are heavily biased toward visual cues and struggle under conflicting or missing modality conditions. By fine-tuning a state-of-the-art model on their new dataset with stereo input, they achieve significant performance gains and observe human-like spatial biases, such as better horizontal than vertical localization.", "questions": "Please refer to the 'weakness' part.", "limitations": "Yes, the authors have clearly articulated limitations, including dataset realism and the restriction to static images. They also discuss potential directions to incorporate video and better audio realism in future work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n- The paper is clearly written with good motivation, explanations, and visualizations.\n- Few existing works on sound source localization explore the modality bias, object bias, and the stereo audio. The proposed dataset with stereo audio simulation is novel, scalable, and would beneficial for the SSL community.\n- Based on the proposed dataset, this paper also provided some insightful conclusions which may inspire future works.\n\n## Weaknesses\n\n\n\n\n\n- **Unclear evaluation metrics.** 1) The authors introduce the evaluation metrics of 'A-Acc' and 'V-Acc' at Lines 235-242. As shown in Table S1, only one metric is reported for some SSL cases. This raises questions on the application and suitable scenarios of the metrics. Could the authors provide more explanations on this? 2) Prior SSL works usually adopt the CIoU to evaluate the model's performance. Can this metric be used in this paper? If not, could the authors provide some discussion on this?\n- **Finetuning and Inference time cost**. The proposed dataset is large-scale. I wonder about the time cost of fine-tuning the IS3 model and the inference.\n- **Questions on experiments** 1) According to Table 1 (1), the CAVP has better performance than IS3, so can the CAVP also be adopted for Finetuning using the proposed dataset? This would make the comparison more comprehensive. 2) The authors choose to fine-tune the prior model IS3. Could the IS3 model be directly trained from scratch? \n- **Minor issues on Figure presentation (do not affect my score).** 1) In Figure 1(a), the texts of 'left' and 'right' seem to be hard to correspond to the figure. 2) In Figure 3, there are no 'blue arrows' mentioned in the caption.", "was_revised": true, "final_justification": "The authors' rebuttal addressed my main concerns. Overall, I think the stereo audio-involved sound source localization is a novel and interesting problem. The proposed large-scale dataset is well-designed. Therefore, I would like to increase my final rating.", "rebuttal": "Thank you for the detailed review and thoughtful feedback on our work. Below, we address the concerns point by point and respond to the reviewer’s questions individually.\n\n**AmaU.1 - Questions: Unclear evaluation metrics. 1) The authors introduce the evaluation metrics of 'A-Acc' and 'V-Acc' at Lines 235-242. As shown in Table S1, only one metric is reported for some SSL cases. This raises questions on the application and suitable scenarios of the metrics. Could the authors provide more explanations on this? 2) Prior SSL works usually adopt the CIoU to evaluate the model's performance. Can this metric be used in this paper? If not, could the authors provide some discussion on this?**\n\nAudio Accuracy (A-Acc) measures whether the model or human correctly localizes the true sound source regardless of semantic matching between audio and vision. According to the definition of sound source localization (SSL), the primary goal is to localize the location of the sound. However, in the Vision-only condition, there is no audio input. Hence, there is no definitive ground truth for A-Acc evaluation, since all possible locations in the image could be considered correct or incorrect. To address this, we introduce another metric called V-Acc, which measures alignment with visual semantics. This metric is only applicable in special conditions where there is no sound or when multiple visual instances match the sound. We will clarify this distinction in the final version.\nThe reviewer also raises a good point regarding CIoU (Complete Intersection over Union), which is a common metric in computer vision. However, applying this metric in human psychophysics experiments is challenging because humans rarely draw bounding boxes around target objects during localization tasks. Moreover, it would be both difficult and unfair to require humans and models to draw bounding boxes on blank images under the audio-only condition, where no visual stimuli are present. We will clarify this point as well in the final version.\n\n**AmaU.2 - Questions: Finetuning and Inference time cost. The proposed dataset is large-scale. I wonder about the time cost of fine-tuning the IS3 model and the inference.**\n\nWe contributed a large-scale AudioCOCO dataset containing roughly one million audio-image pairs by combining selected images and audios using a stringent set of filtering criteria (Section 2.1 and 2.2). As the reviewer correctly suspects, we did not use all possible combinations of image-audio pairs to fine-tune the IS3 models. Instead, we randomly sampled a small subset of 9,356 image-audio pairs for fine-tuning IS3.\nWe used 4 A6000 GPUs to run the experiment over 10 epochs, with a total runtime of about 4 hours. Despite using only a small subset of the full training set, IS3-FT-Stereo has already demonstrated remarkable performance compared to other IS3 variants. This provides strong evidence emphasizing that data quality matters more than data quantity. We will clarify this point in the final version.\n\n**AmaU.3 - Questions: 1) According to Table 1 (1), the CAVP has better performance than IS3, so can the CAVP also be adopted for Finetuning using the proposed dataset? This would make the comparison more comprehensive. 2) The authors choose to fine-tune the prior model IS3. Could the IS3 model be directly trained from scratch?**\n\nWe chose IS3 because it is a widely used baseline in the literature and achieves the best performance among all baselines under most conditions. Therefore, we use IS3 as the representative baseline for the main paper’s analyses. Additionally, we compare other baselines against humans and IS3-FT-Stereo, providing these results and analyses in the appendix due to space constraints.\nAs the reviewer correctly noted, similar to IS3, CAVP and other audio-visual localization models can also be fine-tuned on our proposed AudioCOCO dataset. We have now fine-tuned CAVP on AudioCOCO and present a performance comparison in ** Table R4 **. Similar to the analysis on IS3, these results highlight consistent performance improvements attributable to both the high-quality data and the inclusion of stereo audio. We will include these findings in the final version of the paper.\nDespite these performance gains after fine-tuning on our dataset, it is important to note that AudioCOCO remains a synthetic dataset. While training models from scratch on AudioCOCO is certainly possible, we argue that leveraging pretrained weights—especially those obtained from large-scale, real-world datasets—offers substantial benefits. These pretrained weights encode rich semantic priors that enable models to generalize better.\nLoosely connected to the human learning process, our fine-tuning strategy reflects a similar approach: just as humans build upon years of real-world experience rather than learning each task in isolation, models can benefit from general pretraining followed by task-specific refinement. This fine-tuning helps to mitigate the domain gaps that often cause performance drops in AI models when compared with humans. Importantly, this does not imply that our analysis is confounded by domain gap issues. The comparison between IS3-FT-Mono and IS3-FT-Stereo shows gains of 4.4% for Size1, 4.4% for Size2, and 1.4% for Size3, highlighting the contribution of spatial cues derived from stereo audio. \nOverall, we advocate using foundation models pretrained on large-scale datasets that can be fine-tuned on domain-specific, high-quality datasets such as AudioCOCO to achieve optimal performance.\n\nTable 4. A-Acc on congruent conditions for CAVP and CAVP-FT-Stereo\n| Method            | Size 1 | Size 2 | Size 3 |\n|-------------------|:------:|:------:|:------:|\n| CAVP              | 5.1    | 15.2   | 38.7   |\n| CAVP-FT-Stereo    | 20.4 (+14.6 %) | 28.8 (+13.6 %) | 52.5 (+13.8 %) |\n\n\n**AmaU.4 - Questions: Minor issues on Figure presentation (do not affect my score). 1) In Figure 1(a), the texts of 'left' and 'right' seem to be hard to correspond to the figure. 2) In Figure 3, there are no 'blue arrows' mentioned in the caption.**\n\nThank you! We will fix all the figure presentation issues accordingly."}, {"review_id": "mnDx8mKVex", "reviewer": "Reviewer_fYXa", "summary": "The authors created a visual-sound source localization dataset (AudioCOCO) and benchmarked ten AI models and human. The \"Random\" model serves as the low bound and performance from human psychophysics experiment serves as the high bound. This AudioCOCO dataset includes six conditions. Among those conditions, especially the conflicting visual cue (ConflictVcue), has almost never been tested by others.", "questions": "1. In the Abstract (L24) and Introduction (L91), the authors mentioned \"physical structure of sensory systems\". Does this sentence indicate the spectral cues created by human or animal's pinna? Unlike the AI model from Francl and McDermott which has two human-like ears and HRTF, I did not find any similarity between this paper's model and human auditory systems.\n\n2. L72-73: Stereo audio, widely studied in neuroscience for its role in spatial hearing through inter-channel differences, remains underexplored in SSL benchmarks in AI. Do you mean all the SSL beachmarks in AI only use mono audio?\n\n3. L84-86: Neuroscience research [3] shows that humans exhibit greater precision in horizontal (left-right) sound localization than vertical (up-down). Could you point out which sentence or figure in your reference support this? You referred a pure modeling paper without any human psychophysics experiments. There are lots of papers in SSL to support this, for example, JC Makous, JC Middlebrooks - The journal of the Acoustical Society of America, 1990.\n\n4. In the your Figure 8, the performance of IS3 model is so bad even under the audio-visual congruent condition. I don't understand this since IS3 was published on 2023 and achieved SOTA during that time.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "I really like the idea of comparing the task performance and bias of AI models against humans or non-humans. Figures 4 and 5 and Table 1 clearly shown how the AI models fail under challenging conditions. This finding will benefit people from both AI and neuroscience field.\n\nWith that being said, the benefit I take from this paper is very limited. This model behaves very differently from human. In the L20-22 of Abstract, quote \"Remarkably, consistent with neuroscience findings, our model exhibits a human-like horizontal localization bias—favoring left-right over up-down precision.\" The sentence refers to Figure 6 where the human has 52.1% at horizontal plane and 31.5% at vertical plane, so the accuracy dropped around 40%. In contrast, the SOTA AI model, IS3-FT-Stereo, has less than 10% of drop (the authors do not provide values). I do not think this single, very small, and likely insignificant change could be highlighted as finding the correspondance between AI and humans. \n\nMore importantly, the same Figure 3 also shows that the accuracy of both AI models on the vertical plane is much better than human. This is not suprsing to me at all: human relys on monaural spectral cues but AI models can still use ITD/ILD cues for localization sound on the vertical plane. \n\nIf the authors really want to claim they found human-like sound localization behaviors, they should check their third referred paper (A Francl, JH McDermott - Nature human behaviour, 2022) and show us compariable evidence as shown in that paper. Since the primary area of this paper is neuroscience instead of datasets or machine/deep learning, the strong connection with neuroscience is necessary.\n\nIn the first sentence of Introduction (L27-28), quote \"Sound source localization (SSL) is the task of identifying the spatial origin of a sound within a visual scene.\" This is simply wrong. By this definitation, SSL will fail if sound occur in the darkness or from the human or animal's back. The third sentence (L29-30), quote \"SSL requires integrating information across auditory and visual modalities\". Multisensory integrations is beneficial but not required for SSL. I suggest the authors to revise many sentences like this which imply that SSL needs visual inputs.", "was_revised": true, "final_justification": "The authors have clarified my questions on the neuroscience part of this work.", "rebuttal": "Thank you for the detailed review and thoughtful feedback on our work. Below, we address the concerns and respond to the reviewer’s questions point by point.\n\n**Weakness1: few benefits; model diverges from humans, direction gap is trivial.**\n\nYes, the reviewer is correct in noting that the asymmetry bias between vertical and horizontal localization precision in IS3-FT-Stereo is relatively small—approximately a 4.6% difference (**Table R1**). To address the reviewer’s question more rigorously, we performed a two-tailed t-test comparing vertical and horizontal localization for IS3-FT-Stereo. The resulting p-value < 0.001, indicating a statistically significant difference between the two conditions.\n\nFurthermore, we conducted an ANOVA to compare vertical and horizontal localization biases across both human participants and IS3-FT-Stereo(p-value < 0.001). This analysis reveals that the asymmetry bias is significantly more pronounced in humans than in IS3-FT-Stereo.\n\nWe agree with the reviewers that IS3-FT-Stereo represents a relatively simple extension of the IS3 model and is only loosely inspired by neuroscience at the high level. It lacks key features of the human auditory system, such as the use of monaural spectral cues, the complex structure of the pinna, and the anatomical organization of the cochlea for processing auditory signals. Despite these limitations, it is nonetheless noteworthy that IS3-FT-Stereo exhibits some degree of behavioral alignment with human localization performance—even if the effect is weaker than in more biologically plausible models, such as [3] in the paper. \nWe will include these analyses and clarifications on neuroscience plausibility in the final version. We will also tone down the claim on asymmetry in audio perception in IS3-FT-Stereo.\n\n**Weakness2: AI uses ITD/ILD**\n\nAnother excellent point! We agree with the reviewer that IS3-FT-Stereo performs considerably better than humans in the vertical plane, likely due to the model's advantage in utilizing ITD/ILD cues. Following the reviewer’s suggestion, we integrated human-like pinna and HRTF into IS3-FT-Stereo. As a result, the model's advantage in vertical localization is reduced! Please see **Weakness3** for details.\n\n**Weakness3: Reproducing Francl's evidence and bolstering neuroscience links.**\n\nAs suggested by the reviewer and following the implementation in Francl(2022), we converted each stereo track in AudioCOCO into a 64-band time-frequency cochleagram using the PyCochleagram[3]. These cochleagrams were then used to fine-tune the IS3-FT-Stereo model, resulting in a new variant we call IS3-FT-Cochleagram. Results are presented in **Table R5**. For ease of comparison, we also include results from IS3-FT-Stereo and human participants(copied from the main paper).\n\nFrom **Table R1**, we observe that IS3-FT-Cochleagram exhibits stronger behavioral alignment with humans than IS3-FT-Stereo. In particular, the asymmetry between vertical and horizontal localization biases becomes more pronounced, and vertical biases—reliant on monaural spectral cues from the pinna—are more prominent. We will include these additional results and discussions in the final version.\n\nTo further stress-test our core claim—that the physical structure of the sensory system shapes the multi-modal representations—we conducted an additional manipulation. We altered the input “diet” of IS3-FT-Stereo by rotating the human-like ears by 90 degrees. Specifically, we rotated the 3D coordinates in the Unity scene such that the physical left-right axis aligned with the image’s vertical axis, effectively simulating vertically arranged ears. We then re-rendered all stereo audio from this configuration. This variant is referred to as IS3-FT-Stereo-Rotated.\n\nInterestingly, in the IS3-FT-Stereo-Rotated condition, the asymmetry reverses, albeit with a relatively small effect size. This finding reinforces our central argument: the observed asymmetry in localization precision arises from the model's binaural receiver layout rather than any image-level artifact, supporting the idea that the physical structure of the sensory system directly shapes the multi-modal representations. Again, we acknowledge the absence of a pinna-like structure in IS3-FT-Stereo. To address this, we plan to modify the PyCochleagram to simulate a “90-degree rotated ear” configuration, and we will include the updated results in the final version of the paper.\n\n**Table R5. Vertical and horizontal localization bias for models and humans**\n|Method|Vertical|Horizontal|\n|-|:-:|:-:|\n|IS3-FT-Stereo|40.9|45.5|\n|IS3-FT-Stereo-Rotated|45.0|41.1|\n|IS3-FT-Cochleagram|34.7|51.8|\n|Human|31.5|52.1|\n\n**Weakness4:SSL definition**\n\nThank you—this is another excellent point. We will revise our broad definition of SSL to identify the spatial origin of a sound in a 3D environment. We agree with the reviewer that vision is not a necessary condition for SSL.\nIn this work, our focus is on studying multi-modal conflicts and biases, and therefore we specifically examine scenarios in which visual input is present. However, we acknowledge that conditions where auditory signals originate outside the visual field represent a compelling and important direction for future exploration. We will highlight this point in the future work section and revise all related statements about SSL throughout the manuscript for clarity and accuracy.\n\n**Question5: Clarify for model about the evident physiological analogue.**\nSee the response to **Weakness3**.\n\n**Question6: SSL benchmarks**\n\nAccording to the strict definition of SSL, audio-only localization has been the focus of many AI datasets—such as DCASE[a] and LOCATA[b]—and corresponding models[c-e] that use stereo audio for computational modeling and benchmarking. \nHowever, since our focus lies in multimodal representation learning, we primarily rely on multimodal benchmark datasets, including VGG-SS[f], MUSIC[g], and Flickr-SoundNet[h]. These datasets predominantly use monaural audio, where visual input often compensates for the lack of spatial cues in the auditory stream. Notably, most popular multimodal models—such as AVSegformer, CAVP, IS3, and others—also rely on mono audio as the primary input modality. This trend reinforces the relevance of our evaluation design and underscores the untapped potential of stereo audio for capturing richer spatial information in complex multimodal settings.\nMotivated by our interest in modeling audio-visual conflicts, we deliberately explored both stereo and mono configurations for the audio input and conducted comparative analyses against leading mono-audio visual localization models. These comparisons further highlight the importance and benefit of incorporating stereo audio when addressing more challenging and ecologically valid scenarios in multi-modal learning.\n\n**Questions7:More Ref**\n\nWe refer to Figures 4a and 4b in [3], which demonstrate that when listeners are forced to hear through altered ears, elevation localization degrades, while azimuthal localization remains largely unaffected. This finding suggests that azimuthal localization is robust due to its reliance on ITD and ILD, whereas elevation localization depends heavily on individual-specific spectral cues shaped by the pinna. Disrupting these cues leads to significantly reduced performance in estimating sound elevation.\nThanks for the reference! The amazing study by Makous et al. provides a more direct and stronger evidence of localization bias through human psychophysics experiments. They showed that localization accuracy is better in the horizontal than vertical dimension for stimuli near the frontal midline, but that the reverse is true for stimuli located further in the periphery. Since our work focuses on audiovisual story perception within the visual field, which primarily spans the frontal midline, the localization bias toward better horizontal accuracy still holds. We will cite this work and clarify this point in the final version of the paper.\n\n**Question8: IS3 performance**\n\nWe used the original implementation of IS3[i]. The primary reason for the poor performance of IS3 lies in the fact that many existing multi-modal methods inherit biased audio-visual representation alignments from their training datasets. As discussed in both the Introduction and Results sections, these models are heavily influenced by dataset bias, particularly the tendency for training data to center large, salient objects. Consequently, even state-of-the-art models struggle to accurately detect small or peripheral objects in a scene.\nThis limitation is further supported by prior work[j-k], which highlights that the spatial location of the target plays a critical role in localization performance. These findings align with our observations and underscore the importance of addressing spatial bias when developing more robust and generalizable multimodal models.\n\nRef:\n\n[a] Archontis et al.(2022) — STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events.\n\n[b] Löllmann et al.(2018) — The LOCATA Challenge Data Corpus for Acoustic Source Localization and Tracking\n\n[c] Hogeon Yu(2025) — A Two-Step Learning Framework for Enhancing Sound Event Localization and Detection\n\n[d] Kazuki et al.(2025) — Stereo sound event localization and detection with onscreen/offscreen classification\n\n[e] da et al.(2024) — SELD-Mamba: Selective State-Space Model for Sound Event Localization and Detection with Source Distance Estimation\n\n[f] Honglie et al.(2021) — Localizing Visual Sounds the Hard Way\n\n[g] Guangyao et al.(2022) — Learning to Answer Questions in Dynamic Audio-Visual Scenarios\n\n[h] Relja et al.(2017) — Look, listen and learn\n\n[i] Senocak et al.(2025) Toward Interactive Sound Source Localization: Better Align Sight and Sound\n\n[j] xizhou et al.(2018) Deformable ConvNets v2: More Deformable, Better Results\n\n[k] Antonio et al.(2011) Unbiased Look at Dataset Bias"}]

Author rebuttal:

Meta-review: (a, b) There are a number of strengths and impacts of this paper. The problem of understanding modeling and inductive/task bias between audio and visuals is not well understood in the multimodal learning literature. AV literature is very broad but some of the key points that authors raise -- human vs. non-human evaluation mismatches that drive model bias; lack of a controlled task and relevant data where various combinations of AV scenarios can be systematically tested for realism as well as representation-driven performance. It is interesting to see that localization task fit this quite well (this in itself is a surprising finding). The evaluations are done well; although there have been some questions about the metric, the rationale for using it makes sense. 

(c) More larger scale evaluations of the proposed algorithm/model especially on the traditional AV benchmarks could have boosted the paper. Bulk of the evaluations are centered on the dataset that the authors created. Bulk of the findings from mono vs. binaural/stereo are well known in psychoscoustics (e.g., ITD/ILD related) and so these are not surprising findings as such. 

(d) This is a well conducted study to understanding the representational power of audio vs. visual in joint audio-visual tasks. Separating the modeling bias from dataset bias by creating a novel task and an associated benchmark that allows interpreting the failure cases (eg., object sizes impact localization accuracy) and identifying psychocoustic outcomes (most of which are known, but atleast its retroactive evidence) is useful. Using human vs. non-human evals as a direct measure for performance is interesting. 

(e) Reviewers mostly agreed on the strengths and impact; and the rigor of execution in the paper. Bulk of their concerns were regarding miss interpretation / lack of clarity on metrics; potential limitations of dataset generality and missing details re: related work etc. All these are answered.
Decision: Accept (spotlight)