STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events
Authors: Kazuki Shimada
Archontis Politis
Parthasaarathy Sudarsanam
Daniel Aleksander Krause
Kengo Uchida
Sharath Adavanne
Aapo Hakala
Yuichiro Koyama
Naoya Takahashi
Shusuke Takahashi
Tuomas Virtanen
Yuki Mitsufuji
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: audio-visual dataset
sound event localization and detection
audio-visual sound source localization
direction of arrival estimation

Abstract: While direction of arrival (DOA) of sound events is generally estimated from multichannel audio data recorded in a microphone array, sound events usually derive from visually perceptible source objects, e.g., sounds of footsteps come from the feet of a walker. This paper proposes an audio-visual sound event localization and detection (SELD) task, which uses multichannel audio and video information to estimate the temporal activation and DOA of target sound events. Audio-visual SELD systems can detect and localize sound events using signals from a microphone array and audio-visual correspondence. We also introduce an audio-visual dataset, Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23), which consists of multichannel audio data recorded with a microphone array, video data, and spatiotemporal annotation of sound events. Sound scenes in STARSS23 are recorded with instructions, which guide recording participants to ensure adequate activity and occurrences of sound events. STARSS23 also serves human-annotated temporal activation labels and human-confirmed DOA labels, which are based on tracking results of a motion capture system. Our benchmark results demonstrate the benefits of using visual object positions in audio-visual SELD tasks. The data is available at https://zenodo.org/record/7880637.

STARSS23: An Audio-Visuaa Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events
Kazuki Shimada
Archontis Politis
Sony Al
Tampere University
kazuki. wwwsssassoonncooommmmo
com com
archontis ceccoonisspoliiiiii.o politis@tuni.ff
Parthasaarathy Sudarsanam
Daniel Krause Tampere University
Kengo Uchida
Tampere University
Sony
Sharath Adavanne
Aapo Hakala Tampere University
Yuichiro Koyama
Naoya Takahashi Sony Al
Tampere University
Sony Group Corporation
Shusuke Takahashi Sony Group Corporation
Tuomas Virtanen
Yuki Mitsufuji
Tampere University
Sony AI, Sony Group Corporation
Abstract
While direction of arrival (DOA) of sound events is generally estimated from multichannel audio data recorded in a microphone array, sound events usually derive from visually perceptible source objects, c.g., sounds of footsteps come from the feet of walker. This paper proposes an audio-visual sound event localization and detection (SELD) task, which uses multichannel audio and video information estimate the temporal activation and DOA of target sound events. Audio- visual SELD systems can detect and localize sound events using signals from a microphone array and audio-visual correspondence. We also introduce an audio- visual dataset, Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23), which consists multichannel audio data recorded with microphone array, video data, and spatiotemporal annotation of sound events. Sound scenes in STARSS23 recorded with instructions, which guide recording participants to ensure adequate activity and occurrences of sound events. STARSS23 also serves auman-annotated temporal activation labels and human-confirmed DOA labels, which are based on tracking results of motion capture system. Our benchmark results demonstrate the benefits of using visual object positions in audio-visual SELD tasks. The data is available htttps://zenodo org/record/7888637
Introduction
Given multichannel audio input from a microphone array, sound event localization and detec- tion (SELD) system [ 45 & 46 outputs a temporal activation track for each of the target sound classes along with around the microphone array, when the track indicates activity. Such a spatiotemporal characteri- zation of sound scenes be used in wide range of machine cognition tasks, such as inference on the type of environment, tracking of specific types of sound sources, acoustic monitoring, scene visualization systems, and smart-home applications. Recently neural network (NN)-based SELD
corresponding spatial trajectories, c.g., the direction of arrival (DOA)
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Multichannel audio input
Video input
Audio-Visual SELD System
Activity
DOA
Keeck
Knedk
frame
Tiee
Figure 1: An visual SELD system takes multichannel audio recording data and video data aligned with the audio data and estimates temporal activity and DOA of each class per frame. In a frame of a recording in STARSS23, a knocking sound comes from door while domestic sound derives from a vacuum cleaner behind a microphone array, as shown in an overlay of 360 video and spatial acoustic power map.
systems 45
46) show high localization and detection performance. These systems need data with activity and DOA labels of target sound events for training and evaluation. Because annotation
of DOA labels is challenging in real sound scene recordings, most SELD datasets 2 36 35
19 28
consist of synthetic audio data, which are made by convolution of monaural sound event signals and multichannel impulse response signals with DOA labels. The Sony-TAu Realistic Spatial Sound- scapes 2022 dataset (STARSS22) [39 tackles real sound scene recordings with DOA labels, which based on tracking results of a motion capture (mocap) system. Currently, STARSS22 is the only SELD dataset with real sound scenes, including overlapping sound events, moving source events, and natural distribution of temporal activation and DOA, e.g.. sounds of footsteps are relatively short and heard from a lower elevation. While this dataset is suitable for evaluating audio-only SELD systems in natural sound scenes, the dataset does not include other modality input, e.g., video data.
Sound events in real sound scenes originate from their sound source objects, e.g., speech comes from a person's 's mouth, sounds of footsteps are produced from the feet of a walker, and a knocking sound originates from a door. Such sound source object information is usually apparent in the visual modality. Video data aligned with audio recordings in SELD tasks have the potential mitigate difficulties and ambiguities of the spatiotemporal characterization of the sound scene as audio-visua
data improves source separation [59] 14 58 26 and speech recognition [2] 34 24). Visible people in the video data can provide candidate positions of human body-related sounds. When person walks the video, tapping sounds are easily recognized as footsteps. To investigate how a visual feature input affects SELD system, we propose an audio- visuul SELD task, which uses audio and video data to estimate spatiotemporal characterization of sound events. The left side of Figure [ shows audio-visual SELD system, which takes multichannel audio recordings and video aligned with the audio recordings and outputs activity and DOA target sound events in each frame. To tackle audio-visual SELD tasks, we need an audio-visual datasct consisting of multichannel audio, video, and activity and DOA labels of sound events per frame.
There is another interest in audio-visual sound source localization tasks 44 takes monaural audio and images and estimates where the audio is coming from the images. They focus on learning audio-visual semantic correspondence, not estimating physical DOA around microphone array. While the audio-visual sound source localization datasets [44 12 61 are adequate
12 61
33 which
to train NNs with audio-visual correspondence and evaluate localization performance in images, they are typically monophonic without spatial labels for the target sound events. Several datasets serve multichannel audio and video data with real sound scenes (27 53 54, 60 56 10
40 50), whereas only a few audio-visual datasets have multichannel audio data with DOA labels of speakers around microphone array 40 41 55). While these datasets help to evaluate audio-visual speaker DOA estimation (DOAE) tasks, the evaluation focuses only on speech, not sound events such as musical instruments footsteps.
To tackle audio-visual SELD tasks, we introduce an audio-visual dataset, the Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23)., consisting of multichannel audio, video, and spatiotemporal annotations of sound events, i.e., activity and DOA labels per each frame. The right part of Figure]] shows still in a frame of video in STARSS23, made from 360 vidco, spatial acoustic power

---

Table We compare STARSS23 to other real sound scene datasets from three points of view: whether audio data is multichannel, whether video data is included, and what annotation type is.
Multichannel audio
Dataset
Video
Annotation
STARSS22 39
Activity & DOA of sound events Bounding box Segmentation map
VGG-SS 12
AVSBench 61
YouTube-336 27
AVRI 41
Activity & DOA of speech
STARSS23 (Ours)
Activity, DOA & distance of sound events
map generated from microphone array, and sound event labels. The dataset contains over 7-hour recordings with 57 participants in 16 rooms with the spatiotemporal annotation as a development set. The participants are guided by generic instructions and suggested actívities in the recording to induce adequate occurrences of the sound events and diversity of content. There are 13 classes of target sound events, such as speech, musical instruments, and footsteps. To reveal whether the dataset has a natural distribution of sound events, we analyze the dataset regarding frame coverage, overlap, and DOA distributions per each sound event class. We develop and test an audio-visual SELD system with STARSS23. To investigate the effects of audio-visual input, we present overall localization and detection performance and per-class results.
Related Work
Sound event localization and detection SELDnet [ is the first SELD method, which uses convolutional recurrent neural network (CRNN) to output activity and DOA separately. An activity- coupled Cartesian DOA (ACCDOA) 455 vector, which embeds sound event activity information to the length of a Cartesian DOA vector, enables us to solve SELD tasks with a single output. While the two methods tackle overlaps from different classes, they cannot solve overlaps from the same class. Multi-ACCDOA [46] is an ACCDOA extension, allowing models to output overlaps from the same class. To handle that case effectively,
incorporates auxiliary duplicating permutation
invariant training (ADPIT) [46]. There are other SELD works about framework g 29 55
network
architecture audio feature 30 21,, and data augmentation [1 42).
To train or evaluate SELD methods, we need multichannel audio data with temporal activation and DOA labels. In synthetic multichannel audio datasets [2]
28 DOA labels can be easily annotated because the data are made from multichannel impulse response signals with DOA labels. tackled spatial recording with DOA labels, is limited to isolated single event recordings or combinations of two simultaneous events, ignoring spatiotemporal information linking events in natural scene. STARSS22 (39 tackled real spatial recording with temporal activation and DOA labels of each target class in natural scenes. Participants improvise natural scenes with mocap system, whose tracking results are used for DOA labels. However, the dataset does not release video data. Therefore, it cannot be used to evaluate audio-visual SELD systems.
36
35
19
While the SECL-UMons and AVECL-UMon datasets 6 5
Audio-visual sound source localization There is broad interest in audio-visual sound source localization tasks sources in video and evaluated the method on the VGG-SS dataset [12), which annotates bounding boxes of sound sources for sound and video pairs. The AVSBench dataset [6] serves pixel-level audio-visual segmentation maps for videos over 23 class categories. Because the datasets do not have multichannel audio recordings, they cannot be applied to evaluating SELD tasks.
33). Chen et al. have tackled unsupervised learning to localize sound
Audio-visual dataset with multichannel audio Several mudio-visual datasets include multichannel audio data 27
54
60
56
50
As many datasets are used for self-supervised
learning 27 54
53
or non-localization tasks 160
56 10], there are no DOA labels. The YouTube-
360 dataset 27 serves first-order ambisonics (FOA) signal and 360 video data without any labels
for self-supervised learning.
3

---

Room and prop
Sound Soene Recording
Data
wav: FOA and MIC
Participant Instruction
Blurred equirsciongslar video
Acsivity. DOA
Annotation
labels of sound events
deuree camera
microphone army
Class antotation
Activity annotation
Pesition annotation
Confirmation
Motion capture system
Spaens
Wireless microphooes
Figure 2: The pipeline data construction.
Qian et al. proposed an audio-visual DOAE system, which takes spectrograms and phase features from the audio input and face-bounding boxes from video input estimate the DOA of each speaker [4]]. The system was evaluated with the Audio Visual Robotic Interface (AVRI) dataset, recorded using Kinect and four-channel Respeaker array, along with activity and DOA labels. The audio-visual features are helpful for DOAE, and the dataset supports the evaluation of audio-visual speaker DOAE. However, the dataset only for speech, not various sound events such as clapping and knocks. We summarize the comparison of STARSS23 with other real sound scene datasets in Table [
few audio-visual datasets are collected for audio-visual DOAE tasks 40 50
3 STARSS23 Dataset
3.1 Overview
STARSS23 contains multichannel audio and video recordings of sound scenes in various rooms and environments, together with temporal and spatial annotations of sound events belonging to set of target classes. The dataset enables us to train and evaluate audio-visual SELD systems, which localize and detect sound events from multichannel audio and visual information. STARSS23 is available in public research data repository under the MIT license. There is also demo video
The contents are recorded with short instructions, guiding participants in improvising sound scenes. The recordings contain total of target sound event classes. The multichannel audio data are delivered as two 4-channel spatial formats: FOA and tetrahedral microphone array (MIC). The video data are anonymized 1920x960 equirectangular videos recorded by 360 camera. The annotations of STARSS23 consist of temporal activation, DOA, and source distance of the target classes. STARSS23 split into development set and an evaluation set. The development set totals about hours and 22 minutes, of which 168 clips were recorded with 57 participants in 16 rooms. The development set is further split into a training part (dev-set-trail) 90 clips) and a testing part (dev-set-test, 78 clips) to support the development process. In the evaluation set, publicly available annotations exist because the evaluation set prepared for a competition, which is described in Appendix [
One of the critical differ- ences is releasing video data aligned with multichannel audio data. While we maintain all the sessions of STARSS22, we add about 2.5 hours material to the development set. STARSS23 also serves source distance labels of sound events as additional annotations. We follow the recording procedure in STARSS22, where video data are used only to check labels internally. Adding descriptions about video data and distance annotation, show the data construction part an audio-visual dataset.
STARSS23 improves a multichannel audio dataset, i.c., STARSS22
3.2 Data Construction
As shown Figure 2 STARSS23 is constructed in three steps: sound scene recording, data conversion, and annotation. We explain each step follows.
attps://www. www.tyyuuubbb.oo http://wwww.uu..ccc/////cc/aaaaa

---

Sound scene recording STARSS23 was created in Tampere, Finland, and Tokyo, Japan. Recordings both sites shared the same process, organized in sessions corresponding to different rooms, sound- making props, and participants. In each session, various clips were recorded with combinations of that session's 's participants acting simple scenes and interacting among themselves and with the sound props. The scenes were based on generic instructions on the desired sound events. The instructions were a rough guide to ensure adequate event activity and occurrences of the target sound classes in a clip. The left photo of Figure shows that participants improvise following the instructions.
set of 13 target sound event classes are selected to be annotated, based on the sound events captured adequately in the recorded scenes. The class labels are chosen to conform to the AudioSet ontology [17]. They are: female speech, male speech, clapping, telephone, laughter, domestic sounds, footsteps, door, music, musical instrument, water tup, bell, knock. Music, c.g. background music or pop music, is played by a loudspeaker in the room. On the other hand, musical instruments are played by participants, including acoustic guitar, piano, and others. Domestic sounds consist of vacuum cleaners, mechanical fans, and boiling, which have strongly directional and loud sounds. They can be distinguishable from natural background noise in sound scenes. The scenes also contain directional interference sounds, such as computer keyboard or shuffling cards that are not labeled.
As shown in the left photos of Figure 2 each scene was captured with audio-visual sensors, i.e., a high nigh-resooluton 32-channel spherical microphone array (Eigenmike em3 with height set at 1.5 m, and 360 camera (Ricoh Theta v*) mounted 10 cm above the microphone array. For each recording session, a suitable position of the Eigenmike and Ricoh Theta V was determined cover the scene from central place. We also captured the scenes with two additional sensors for annotation: a mocap system of infrared cameras surrounding the scene, tracking reflective markers mounted on the participants and sound sources of interest (Optitrack Flex ..f). and wireless microphones mounted on the participants and sound sources, providing close-mic recordings of the main sound events (Røde Wireless Go 11. The origin of the mocap system was set at ground level on the exact position of the Eigenmike. In contrast, the mocap cameras were positioned at the room's 's corners.
Recording starts on all devices before the beginning of a scene and stops right after. A clapper sound initiated the acting, and it served as a reference signal for synchronization between the different types of recordings, including the mocap system that can record a monophonic audio side signal for synchronization. All types of recordings were manually synchronized based on the clapper sound and subsequently cropped and stored at the end of each recording session. The details of sound scene recording, e.g., generic instructions, sound events, and sensors, are summarized in Appendix A..]
Data conversion The original 32-channel recordings were converted to two 4-channel spatial formats: FOA and MIC. Conversion of the Eigenmike recordings to FOA following the SN3D normalization scheme (or ambiX) was performed with measurementt-ased filters (37.. Regarding the MIC format, channels 6, 10, 26, and 22 of the Eigenmike were selected, corresponding to a nearly tetrahedral arrangement. Analytical expressions of the directional responses of each format can be found in (36). Finally, the converted recordings were downsampled to 24kHz. The raw 360 video data were converted to equirectangular format with 3840x 1920 resolution at 29.97 frames per second, which was convenient to handle as planar video data. Based on the participant's is consent, the visible faces of all recordings were blurred. Finally, the video with face-blurring was converted to 1920x resolution.
Annotation Spatiotemporal annotations of the sound events were conducted manually by the authors and research assistants. As shown in the lower right part of Figure there were four steps: a) annotate the subset of the target classes that were active in each scene, b) annotate the temporal activity of such class instances, c) annotate the position of each such instance when active, moreover, d) confirm the annotations. Class annotations (a) were observed and logged during each scene recording. Activity labels (b) were manually annotated by listening to the wireless microphone recordings. Because each wireless microphone would capture prominent sounds produced by the participant source it was assigned to, onset, offsets, source, and class information of each event could be conveniently extracted. In scenes or instances where associating an event to a source was
5

---

Table 2: Frame coverage and max, mean, and the degree of polyphony globally and of each class separately on the dev-set-train of STARSS23. The mean polyphony is computed over active frames.
Global
Fem. speech
speech
Male
Clap
Phone
Laugh
Dom. sounds
Footsteps
Door
Music
Music.
instr.
Faneet
Bell
Knock
Prame awwnge
(56 total fromess
82.5
28.4
31.4
0.53
1,02
2.89
18.6
2.17
0.70
22.2
2.32
0.58
0.99
0.06
Max. polyphony
6 1.47
2
Mean polyphony
1,05
1.06
1,06
1.00
1.25
1.03
1.02
1.00
1.20
1.28
1.00
1.00
1.00
Polyphony
(%6 active frames)
62,878
95,44
94.35
93.74
78.67
91.27
97.87
100
79.99
71.66
100
100
100
Polyphony
28.413
1.94
Polyphuny
7,385
0.14
2.39
0.19
0
0
0
Polyphony
1.129
0.46
01
0
0
0
0
Polyphony
0.162
0
0
0
Polyphony
Asimuth
Asim.th
Devation
Devation
(a) Female speech
(b) Footsteps
Figure 3: The distribution of the azimuth and elevation. Each point in the scatter figure represents a frame. The top and right figures are stacked histograms of azimuth and elevation.
ambiguous purely by listening, annotators would consult the video recordings to establish the correct association. The temporal annotation resolution was set to 100 msec. After onset, offset, and class information of events was established for each source and participant in the scene, the positional annotations (c) were extracted for each such event by attaching tracking results to the temporal activity window of the event. Positional information was logged in Cartesian coordinates with respect to the mocap system's origin. The event positions were converted to spherical coordinates, i.e., azimuth, elevation, and distance, which are more convenient for SELD tasks. Then, the class, temporal, and spatial annotations were combined and converted the text format used in the previous dataset (39]. The annotation details are summarized in Appendix A.2 Confirmation of the annotations (d) was performed by listening to the Eigenmike recording while watching a synthetic video. The video was the equirectangular one overlapped with the event activities, which were visualized as labeled markers positioned at their respective azimuth and elevation on the video plane. If a clip was not passed the confirmation, the clip was annotated again.
3.3 Data Analysis
Having natural distribution of sound events is beneficial to evaluate audio-visusi SELD systems. We analyze the frame coverage, polyphony, and DOA per sound event classes on the dev-set-train of STARSS23. Table@]showe frame coverage and max, mean, and distribution of polyphony globally and of each class separately. Regular classes in frames are female and male speech, music, and domestic sounds. These classes also frequent in our daily lives. Musical instruments and laugh classes show a high mean polyphony of the same class, which are natural situations in jam sessions and conversations. Figure 3 shows the distribution of DOA with the axis of the azimuth and elevation. Regarding female speech in Figure 3a| the elevation distribution has a strong peak around -10 degrees, while that of azimuth seems uniformly distributed. Compared the speech class, footsteps appear in lower azimuth than -10 degrees. See Appendix B for further data analysis, e.g., duration.
4 Benchmark
In this section, we examine an audio-visual SELD task with STARSS23. For evaluation, we set the dev-set-train for training and hold out the dev-set-test for validation.

---

muttichaneel audio: FOAMIC
Feature extracio::
Object detection and featuee extracion: bounding encoded
amplitude spectogmune LPDs
alung szimath
layers
PCs
Object desection, e person
GRL layer
Muli-ACCIOOA
Visual encoded vectors Blong with azimalh and clevatios
Figure 4: An audio- visual SELD system uses a CRNN model with a multi-ACCDOA output. While multi-ACCDOA output can have several tracks, show that with single track for simplicity. We use an object detector to incorporate visual information, which outputs bounding boxes on target classes. The results encoded vectors along with azimuth and elevation. Then, the encoded vectors are embedded and concatenated before the GRU layer.
4.1 Audio-Visua SELD System
To build audio-visual SELD system, start with audio-only system based SELDnet
and multi-ACCDOA [46.. which widely used in audio-only SELD tasks (39 43 31. To extend the audio-only systems audio-visual, merge visual and audio information in the middle of the network, following the audio-visual speaker DOAE work [41]
First, we summarize the audio-only SELD system. Let X E RXPXX be input consisting of M-channel F-dimensional T-frame audio features, e.g., amplitude spectrograms. Convolutional layers are used audio embedding network, Fa,0, (-), where 9g is the network parameters:
E. Fa,. (Xa),
where E. E RKxx is audio embedding. Ka and 7 are the length of an embedding vector and the number of time frames, respectively. The embedding is followed by an audio-only decoder network with parameter Ono. Fao,9zo (+), which consists of gated recurrent unit (GRU) layer and fully connected layer (FC):
p= = Fao,0. (Ea);
(2)
where P NxCX""" is estimated output of N-track C-class T -fame multi-ACCDOA, Each class of the multi-ACCDO, output in track n and frame is represented by a three-dimensional vector with Cartesian coordinates æ. y. and as shown in the bottom of Figure The length of the vector shows activity, and the direction of the vector indicates the DOA around the microphone array. To train the SELD system, multi-ACCDOA outputs under the ADPIT scheme [46]. In inference, when the length of the vector is greater than threshold, the class is considered active.
mean squared error (MSE) between the estimated and target
Then, we prepare a visual embedding. As visual input, we use the corresponding video frame at the start of the audio features. With the corresponding image, object detection module, e.g.. YOLOX [15], outputs bounding boxes of potential objects on target classes, e.g., person class. Let us denote bounding box (us, tb. wa, wh,))) hy), where u, and Uú indicate the image position of the top-left corner, and w, and hú show the width and height of the bounding box. Each bounding box is encoded to two vectors Pazi (u) and Pate along the image's horizontal axis and vertical axis v. We follow the same formulation in 41 using a Gaussian distribution:
(u) exp
(3)
2

---

where the center Ms deviation on is proportional to wy. The vertical vector Pole has the same formula as Pauil (u) by replacing u with v. Pu with M4v = " + 2 h/b and 0n with dr in Equation 3 These vectors are concatenated into visual feature input X v E =R2xBxL, B is the maximum number of bounding boxes, and L is the length of the vectors. When no bounding box is detected, the visual feature imput is set to all-zero tensor. The right part of Figure @ deppcct the visual feature input. FCs are used as a visual embedding network, Fv,0, where B, is the network parameters:
2 wh is the horizontal center of the bounding box, and the standard
E, = Fv,o, Fx,
where Ey E IR is a Ky-dimensional visual embedding vector.
Using the visual embedding vector, we finally extend the audio-only system into audio-visual one. The visual embedding vector is repeated T" times to align the both embeddings in the time axis. Then the repeated visual embedding E,, E IR. K and the audio embedding are concatenated and fed into audio-visual decoder network with parameter Pav, Fav, Ow (.), to get multi-ACCIOO output:
p == Fov B.. (Ea, E,).
4.2 Evaluation Metric
We used four joint localization and detection metries [25 38), which are widely-used audio-only SELD tasks 46 39 31. Two metrics are referred to as location-awarc detection and are error rate (E R26)) ) and F-score (Fzu-) in one-sec non non-overlapping segments. We consider a prediction as a true positive if the prediction and reference class the same and the angle difference is below 20.. F20- is calculated from location-awee aware precision and recall, whereas E is the sum of insertion, deletion, and substitution errors, divided by the total number of the references. The other two metrics are referred to as class-aware localization and are localization error ( (I.EcD) ( in degrees and localization recall (LRcp) in one-sec con-overlapiing segments, where the subscript refers to glassfiictioon-deeedeen Unlike location-aware detection, we do not use any threshold but estimate the difference between the correct prediction and reference. LEOD expresses the average angular difference between the same class's 's predictions and references. LReD tells the true positive rate of how many of these localization estimates were detected in class out of the total number of class instances. We used the macro mode of computation while the mode does not apply E because it includes substitution errors between two classes. We first computed the metrics for each class and then averaged them for the other three metrics to obtain the final system performance. See Appendix C.T for further details.
4.3 Experimental Setting
As audio features, multichannel amplitude spectrograms and inter-channel phase differences (IPDs) used [46). Input features are segmented have a fixed length of 1.27 sec. To reduce the calculation cost of video, we use 360x 180 videos converted from the released 1920x960 videos. As visual input, extract the corresponding video frame at the start of the audio features. We use pretrained YOLOX object detection mode to get bounding boxes of person class. Other classes, e.g., cell phone or sink, are not stably detected in our preliminary experiments with STARSS23 videos. The bounding box results are encoded to vectors along azimuth and elevation as in Sec Sec|4.1 The maximum number of bounding boxes is B = 6. The vector size is L 37(= 36 + 1) cover 360 degrees of azimuth per 10 degrees. While we tested convolutional neural network (CNN)-based visual feature extraction methods 47 20), they performed worse than the object detection method.
To get audio embedding, we stack three convolutional layers with kernel size 3 x 3. We embed the visual encoded vectors with two FCs. The lengths of both embedding vectors are K. K, 64. The concatenated embeddings are processed with a bidirectional GRU layer with a hidden state size of 256. The number of target classes was C = 13. The number of tracks in the multi-ACCDOA format was fixed at N = 3 maximum simultancous sources. The threshold for activity was 0.3 to binarize predictions during inference.
The experiments are for the two formats: FOA and MIC. We compare the audio-visun SELD system with audio-only system based on the same data split and implementation. The difference is the
attps: cow. http////ww../a.aaaaa////////////

---

Table 3: SELD performance (C 13) in audio-visura and audio-only systems evaluated for the dev-set-test in STASS23.
Format
System
Rzu +
Fev 1
LECD +
LRCD f
FOA
Audio-Visual
1.03 + 0.03
13.2 + 0.6 %
51.6 5.6
34.2 + 1.0 %
Audio-Only
0.97 = 0.05
14.8 + 0.3
55.9 4.0
35.1 2.8
MIC
Audio-Visuin
1.08 = 0.04
10.0 + 0.8
60.0
27.6 1.7 %
Audio-Only
1.04 + 0.02
10.7 + 2.1 %
66.9 - 6.8
29.6 = 3.0 %
50
40-
2-0deg
20
10-
0-
Clep
Laugh
rootsteps
indtr
related related
Figure 5: Location-aware F-score per sound event class (C 13) audio-visual and audio-only
systems evaluated for the dev-set-test in STASS23.
presence or absence of video input. Because the visual input is bounding boxes of person class, especially focus on five classes related to the human body, i.c., female and male speeches, clapping, laughing, and footsteps. To further investigate the effect of the visual input on the body-related classes, we add experiments where we set only the body-related classes as the target classes of the SELD systems. The systems trained and evaluated with activities and DOAs of speeches, clap, laugh, and footsteps classes, i.e., C = 5. The other settings are the same as the 13-class SELD experiments. Details the experimental setting are in Appendix C.2 The code of training, inference, and evaluation is available in a GitHuu repository under the MIT license.
4.4 Experimental Results
Table summarizes the 13-class SELD performance of the audio-visual and audio-only SELD systems in both audio formats. Compared with both formats, SELD systems with FOA format show better SELD performance than those with MIC format. In FOA format, while the audio-visual SELD system shows a slightly worse location- aware F-score, the audio- visual system exhibited better localization error with comparable localization recall. There is a similar trend of lower localization error in MIC format.
We investigate the location-awanl F-score over 13 classes, considering both localization and detection aspects. Figure 5) shows the F-score per class in FOA format. We show the average score of the five body-related classes the left of the figure. The audio-visual system demonstrates higher location-aware F-score in the body-related. On the other hand, the audio-visual system performs worse in the average of the other classes, i.e., non-body-related. The results suggest that the visual input, i.e., bounding boxes of a person, contributes to localization and detection of body-related classes, whereas the visual input might limit the performance of non-body-related classes.
Table shows the body-related classes only SELD performance in audio-visual and audio-only systems. The audio-visual SELD system outperforms the audio-only system in all metrics. The visual location information of people enables the audio-visual system to localize and detect body- related classes in localization error. because a few classes are sometimes set 180 as they had no true positive output. Even if omit such cases, the audio-visual system still shows lower localization error.
accurately. Audio-only system in MIC format shows high standard deviation
https://githu/ http:///tttuuuoco/////e/aas httsssso///uuuu vieaaaliiiiiiieeeeee seld--eeeeddeee dcase2233

---

Table 4: Body-related classes only SELD performance (C 5) in audio-visual and audio-only systems evaluated for the dev-set-test in STASS23.
Format
System
22
F20 t
LECD +
LRCD 1
FOA
Audio- Visual
0.93 + 0,08
22.4 = 2,4 %
31.0 3.6 3
45,0 + 5.0 %
Audio-Only
0.93 0.02
18.0 1.9 %
40.9 + 5.5 %
MIC
Audio-Visual
0.95 0.01
20.9 = 1,9 %
28.9 + 1.5%
37.4 + 4.6
Audio-Only
0.97 = 0.10
16.0 = 3.7 %
58.4 26.8
30.9 = 10.9 %
Our experimental results show that visual location information of people improves SELD performance in human body-related classes. If visual location information of people is provided, should expect improvements in the human body-related classes, while we should not expect improvements in the rest. The results suggest that if visual information matches the target classes of an audio-visual SELD system, the audio-visual system performs better.
5 Discussion on Limitations and Future Work
Like any visual machine learning system, deploying audio-visual SELD systems might lead to privacy issues, e.g., capturing raw visual information, if the consent of the people being recorded not considered. We also recognize that improved SELD capability could empower surveillance technology without appropriate regulation.
As shown in T STARSS23 is imbalanced and long-tailed. When considering an evaluation set, the dataset would be appropriate as it can reflect real soundscape, which is usually imbalanced and long-tailed. On the other hand, we use difficult for model to learn about rare categories. We can use machine learning algorithms [22] 7 or [9] to tackle the data imbalance problem.
training set, the imbalance and long-tail make
data sampling strategies
The amount of STARSS23 is relatively small as training data. We can use a multichannel audio- visual simulation platform, e.g., SoundSpaces 2.0 [[]] for geometry- boased audio rendering in 3D environments. Also, we can use multichannel audio-only data such as TAU-NIGENS Spatial Sound train an audio branch separately. Unlabeled multichannel
Events (TNSSE) datasets 2 36 35)
audio-visual data, c.g., YouTube-360 (27] could help the pre-training for andio-visual SELD tasks.
There is room for developing visual features and model structure improve the audio-visual SELD performance. While the visual location information people improves the performance in the body-related classes, need to develop practical visual features for the non-body-relan more sophisticated model structure than the current concatenation could boost the performance. A recent work made the audio-visual SELD system better than the audio-only one in STARSS23, using decision-level audio-visual fusion with keypoints, e.g., feet or telephones (52)
classes. A
While we apply STARSS23 for audio-visual SELD tasks, STARSS23 would be applied to other tasks, c.g., audio-visual sound source localization 44
12
61
4 33 or oudio-visual cross-modal
retrieval tasks [57]. Also, when large multi multi-modal models 118
49 are extended to multichannel audio, STARSS23 is an appropriate audio-visual soundscape evaluation data for the models because STARSS23 has unique spatial annotations and multichannel audio data beyond stereo.
Conclusion
This paper attempts to broaden sound event localization and detection (SELD) to an audio-visual area by introducing an audio-visual SELD task. We present an audio-visual dataset, Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23), which consists of multichannel audio data, video data, and spatiotemporal annotation sound events natural sound scenes, Furthermore, we present quantitative evaluations for audio-visual SELD systems compared with audio-only systems and demonstrate the benefits of visual object positions. We still need to improve the SELD performance of various sound events using audio-visual data. Also, STARSS23 opens wide range of future research on spatial audio-visual tasks, taking advantage of the well-organized audio-visual recording and detailed labels about spatial sound events.
10

---

Acknowledgments and Disclosure of Funding
We thank Akira Takahashi for his helpful code review and thank Atsuo Hiroe, Kazuya Tateishi, Masato Hirano, Takashi Shibuya, Yuji Maeda, and Zhi Zhong for valuable discussions about the data construction process.
The data collection and annotation at Tampere University have been funded by Google. This work was carried out with the support of the Centre for Immersive Visual Technologies (CIVIT) research infrastructure at Tampere University, Finland.
References
[1] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE J. Selected Topics in Sig. Proc., 13(11)34-45) 2018.
[2] S. Adavanne, A. Politis, and T. Virtanen. A multi-toom reverberant dataset for sound event localization and detection. In Work. Detection and Classification of Acoustic Scenes and Events (DCASE), pages 10-14, October 2019.
[3] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman. Deep audio-visual speech recognition. IEEE transactions pattern analysis and machine intelligence, 41(1))8777-88777, 2018. [4] R. Arandjelovic and A. Zisserman. Look, listen and learn. In Proceedings of the IEEE international
conference on compuder vision, pages 609-617,2011 2017.
M. Brousmiche, Dupont, and Rouat. Avecl-umons database for audio-visual event classification and localization. arXiv preprint arXiv:2011.011 1.01018, 2020.
[6] M. Brousmiche, J. Rouat, and S. Dupont. Secl-umons database for sound event classification and localization. FEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP), pages 756-760, 2020. [7] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label-distribttion aware margin loss. Advances in neural information processing systems, 32, 2019.
[8] Y. Cao, T. Iqbal. Q. Kong. F. An, W. Wang. and M. D. Plumbley. An improved event-independen network for polyphonic sound event localization and detection. In IEÉE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP). pages 885-889, 2021.
[9] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[10] C. Chen, A. Richard, R. Shapovalov, V. Ithapu, N. Neverova, K. Grauman, and A. Vedaldi. Novel- view acoustic synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6409-6419) 2023.
[11] C. Chen, C. Schissler, S. Garg. P. Kobernik, A. Clegg, P. Calamia, D. Batra, P. Robinson, and Grauman.
Soundspaces 2.0: A simulation platform for visual-acoustic learning. Advances in Neural Information Processing Systems, 35:8896-89)
2022.
[12] H. Chen, W. Xie, T. Afouras, A. Nagrani, A. Vedaldi, and Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16867-16876, 2021.
[13] A. Dosovitskiy, I., Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprini wxXi:22010.1192) 2020.
[14] R. Gao, R. Feris, and K. Grauman. Learning to separate object sounds by watching unlaheled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35-53, 2018.
[15] 7. Ge, S. Liu, Wang, 7. Li, and J. Sun. YOLOX: Exceeding YOLO series in 2021. arXiv preprint 2021.
1xXiv:2107.08.. [16] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford. Datasheets for datasets. Communications the ACM, 64(12):86-92, 2021.
[17] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In IEFE Int. Conf. Acoustics, Speech and Sig. Proc. (ICASSP). pages 776-780, 2017.
[18] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1.51 11880015190 2023.
[19] E. Guizzo, R. F. Gramaccioni, S. Jamili, C. Marinoni, E. Massaro, C. Medaglia, G. Nachira, L. Nucciarelli,
I.. Paglialunga, M. Pennese, et al. L3DAS21 challenge: Machine learning for 3D audio signal processing. In IEFEE. Int. Work. Machine Learning for Sig. Proc. (ML.PP), pages 1-6, 2021.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
11

---

[21] Y. He. N. Trigoni, and A. Markham. Sounddet: Polyphonic moving sound event detection and localization from raw waveform. In International Conference on Machine Learning, pages 4160-4170. PMLR, 2021.
[22] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision. pages 2980-2988, 2017.
[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, Ramanan. P. Dollár, and C. Zitmick. Microsoft coco: Common objects in context. In Computer Vixion-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[24] P. Ma. S. Petridis, and M. Pantic. End-to-end audio-visual speech recognition with conformers. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7613-7617, 2021.
[25] A. Mesaros. S. Adavanne, A. Politis, T. Heittola, and T. Virtanen. Joint measurement of localization and detection of sound events. In JEEE Work. on Applications of Sig. Proc. to Audio and Acoustics ( (WASPAA),
pages 333-337, 2019. 2019.
[26] D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y. Xu, M. Yu, D. Yu, and J. Jensen. An overview of deep-lcarning- based audio-visual speech enhancement and separation. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 19613681,,, 2021.
[27] P. Morgado, Y. Li, and N. Nvasconcelos, Learning representations from audio-visual spatial alignment. Advances in Neural Information Processing Systems, 33:4733-4744, 2020.
[28] K. Nagatomo, M. Yasuda, K. Yatabe, S. Saito, and Y. Oikawa. Wearable seld dataset: Dataset for sound event localization and detection using wearable devices around head. In IEEE Int. Conf. Acousties, Speech and Sig. Proc. (ICASSP), pages 156-160, 2022.
[29] T. N. T. Nguyen, D. L. Jones, and W.-S. Gan. A sequence matching network for polyphonic sound event localization and detection. In IEEE Int. Conf. on Acoustics, Speech and Sig, Proc. (ICASSP), pages 71-75,
2020.
[30] T. N. T. Nguyen, D. L. Jones, K. N. Watcharasupat, H. Phan, and W.-S. Gan. Salsa-lite: A fast and effective feature for polyphonic sound event localization and detection with microphone arrays. In IEEE Int. Conf. Acoustics, Speech and Sig. Proc. (ICASSP), pages 116-720, 2022.
[31] S. Niu, J. Du, Q. Wang. L. Chai, H. Wu, Z. Nian, L. Sun, Y. Fang. J. Pan, and C.-H. Lec, An experimental study on sound event localization and detection under realistic testing conditions. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
[32] K. Noda, Y. Yamaguchi, K. Nakadai, H. G. Okuno, and T. Ogata. Audio-visual speech recognition using deep lcarning, Applied intelligence, 42:722-737, 2015.
[33] A. Owens and A. Efros. Audio-visual scene analysis with self-supervised multisensory fcatures. In Proceedings of the European Conference on Computer Vision (ECCV), pages 631-648, 2018.
[34] S. Petridis, T. Stafylakis, P. Ma, F. Cai, G. Tzimiropoulos, and M. Pantic. End-to-end audiovisual speech recognition. In IEEE international conference on acoustics, speech and signal processing (ICASSP), pages
6548-6552, 2018.
[35] A. Politis. S. Adavanne, D. Krause, A. Deleforge, P. Srivastava, and T. Virtanen. A dataset of dynamic reverberant sound scenes with directional interferers for sound event localization and detection. In Work. on Detection and Classification of Acoustic Scenes and Events (DCASE), pages 125-129, 2021.
[36] A. Politis, S. Adavanne, and T. Virtanen. A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection. In Work. on Detection and Classification of Acoustic Scenes and Events (DCASE), pages 165-169, 2020.
[37] A. Politis and H. Gamper. Comparing modeled and measuremenn--assed spherical harmonic encoding filters for spherical mierophone arryys In IEEE Work. Applications of Sig. Proc. to Audio and Acoustics
(WSPPAA). pages 224 228, 2017,
[38] A. Politis, A. Mcsaros, S. Adavanne, T. Heittola, and T. Virtanen. Overview and evaluation of sound event localization and detection in DCASE 2019. IEEE/ACM Trans. Audio, Speech, and Language Proc., 2020.
[39] A. Politis, K. Shimada, P. Sudarsanam, S. Adavanne, D. Krause, Y. Koyama, N. Takahashi, S. Takahashi, Y. Mitsufuji, and T. Virtanen. STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events. arXiv preprint arXiv:2206. 01948, 2022.
[40] x. Qian, M. Madhavi, Z. Pan, J. Wang, and H. Li. Multi target doa estimation with an audio-visie visual fusion mechanism. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4280-4284, 2021.
[41] X. Qian, Z. Wang. Wang, G. Guan, and H. Li. Audio-visual cross-attention network for robotic speaker tracking. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 31:550-562, 2022. [42] F. Ronchini, D. Arteaga, and A. Pérez-López. Sound event localization and detection based crnn using rectangular filters and channel rotation data augmentation. Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop (DCASE), pages 180-184, 184, Tokyo, Japan, November 2020.
[43] R. Scheibler, Komatsu, Fujita, and M. Hentschel. On sorting and padding multiple targets for sound event localization and detection with permutation invariant and location based training. In 2022 Asciffecific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, 2022.
12

---

[44] A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang. and I. S. Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4358-4366, 2018.
[45] K. Shimada, Y. Koyama, N. Takahashi. S. Takahashi. and Y. Mitsufuji. ACCDOA: Activity-cupled cartesian direction of arrival representation for sound event localization and detection. In IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP), pages 915-919, 2021.
[46] K. Shimada, Y. Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y. Mitsufuji. Multí-accdoa: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training. In IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP), pages 316-320, 2022.
[47] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv1409 1556, 2014.
[48] P. Sudarsanam, A. Politis, and K. Drossos. Assessment of self-attention on learned features for sound event localization and detection, arXiv preprint arXiv:2107.098) 2021.
[49] Z. Tang. Z. Yang. C. Zhu, Zeng, and M. Bansal. Any-to-any generation via composable diffusion. arViv preprint arXiv:2305. 1 1846, 2023,
[50] Q Wang, H. Chen, Y. Jiang. Z. Wang, Y. Wang. J. Du, and C.-H. Lee. Deep learning based audio-visual suuli--peakkrr doa estimation using permutation-frre loss function. arXiv preprint (tt::2200.14888. 2022. [51] Q. Wang, J. Du H.-X. Wu, J. Pan, F. Ma, and C.-H. Lee. A four-stage data augmentation approach to resnet-conformer based acoustic modeling for sound event localization and detection. arXiv preprint arXiv:2101.029/3 2021.
[52] Q. Wang, Y. Jiang. S. Cheng. M. Hu, Z. Nian, P. Hu, Z. Liu, Y. Dong. M. Cai. J. Du, and C.-H. Lee. The nere-slip system for sound event localization and detection of dcase2023 challenge. Technical report,
DCASE2023 Challenge, 2023.
[53] S. Wang. A. Politis, A. Mesaros, and T. Virtanen. Self-supervised learning of audio representations from audio-visual data using spatial alignment. IEEE Journal of Selected Topics in Signal Processing, 16(6):1467-144)
2022.
[54] K. Yang. B. Russell, and J. Salamon. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9932-9941, 2020.
[55] M. Yasuda, Y. Koizumi, S. Saito, H. Uematsu, and K. Imoto. Sound event localization based on sound intensity vector refined by DNN-based denoising and source scparation. In Proc. of IEEE ICASSP, pages 651-655, 2020.
[56] H. Yun, Y. Yu, W. Yang. K. Loe, and G. Kim. Pano-AVQA: Grounded audio-visual question answering on
360deg vidcos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2031-2041, 2021.
[57] D. Zeng, Y. Yu, and K. Oyama. Deep triplet neural networks with cluster-cea for audio-visual cross-modal retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 16(3):1-23, 2020.
[58] H. Zhao, C. Gan, W.-C. Ma, and A. Torralba. The sound of motions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1735-1744, 2019.
[59] H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018. [60] W. Zheng, R. Yoshihashi, R. Kawakami, I. Sato, and Kanezaki. Multi event localization by audio-visual fusion with ommidirectional camera and microphone array. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2565-2573, 2023.
[61] J. Zhou, J. Wang, J. Zhang, W. Sun, J. Zhang, S. Birchfield, D. Guo, L. Kong, M. Wang, and Zhong. Audio-visual segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 386-403, 2022.
13

---

Appendix
We show appendixes for data construction, data analysis, experiment, competition, social impact, and personal data handling. Finally, we answer the questions from Datasheets for Datasets [16].
A
Data Construction
A.1 Sound Scene Recording
The generie instructions for recording participants include information about duration, participants, props, active classes, and description of sound scenes. The recording duration can be shorter or longer as they are rough instructions. We set the basic structure of sound scenes, c.g., people gathering on sofa and discussing the weekend. Following the rough instructions, we leave the other details of sound scenes to the participants, e.g., they can walk wherever they want and talk about whatever they want without a fixed dialogue text. We show an example of the instructions:
Duration min
Participants people
Props Playing cards, mobile phone
Active classes Speech, laugh, mobile phone
Description
3 people gather on the sofa and talk about the weekend.
Propose to play a game of cards, shuffle, and distribute the cards.
Mobile phone rings while playing, attend the call, walk around and talk for a few seconds, and laugh during the call.
Come back, sit, and continue playing.
Sound scenes consist of target sound events, directional interference sounds, and background noise. Each target class contains diverse sounds, e.g., the speech classes include a few different languages, and the phone class has sounds from different mobile phones. In addition, several target classes correspond to super classes with some subclasses the AudioSet ontology [7]. e.g., the domestic sounds class contains the vacuum cleaner, mechanical fan, and boiling subclasses. We provide the subset of sounds encountered the recordings for the target classes in the form of more specific AudioSet-related labels. The subset information is summarized in Table which is a co-post of table in the STARSS22 paper [39]. Directional interference sounds are derived from computer key shuffling cards, dishes, pots, and pans; however, they not annotated. Natural background noise is mainly related to HVAC (heating, ventilation, and air-conditioning), ranging from low to high levels.
Spatial and temporal annotations relied on carefully mounting tracking markers and wireless mi- crophones (39] The tracking markers were mounted on independent sound sources such as mobile phones or hoovers. Head markers were also provided to the participants as headbands or hats. The head tracking results were reference points for human body-related sound classes. Mouth position for speech and laughter classes, feet stepping position for footstep class, and hand position for clap- ping class were each approximated with a fixed translation from the head tracking result. Regarding clapping, participants were instructed to clap about 20 cm in front of their faces, while footsteps were projected from the head coordinates to floor level. Hence, the mounting positions were considered in the annotation process translate the tracking data of each class into each sound source position. The wireless microphones were mounted to the lapel of each participant and to additional independent sound sources being far from the participants.
A.2 Annotation
At a certain time, the motion capture (mocap) system could not track the attached marker, e-g., when markers were out of the view of the mocap coverage, markers were moving too fast, or obstacles occluded markers. Whenever such misses were short, the tracking results were interpolated with If such misses were long and interpolating the results was not possible, azimuth and M 9
14

---

Table 5: Relation of target classes to specific AudioSet classes. Target classes not included in the table have a one-to-one one relationship with the similarly named AudioSet ones.
Target Class
Related AudioSet subclasses
Telephone
Telephone bell ringing, Ringione (no musical ringtones)
Domestic sounds
Vacuum cleaner, Mechanical fan, Boiling (produced by hoover, air circulator, water boiler)
Door, open or close
Combination of Door & Cuphoard, open or close
Music
Background music & Pop music, (played by a loudspeaker in the room)
Musical instrument Bell
Acoustic guitar, Marimba, Xylophone, Cowbell, Piano, Rattle (instrunent)
Combination of sounds from hotel bell and glass bell, closer to Bicycle bell & single Chine
elevation of sound sources were annotated based on the 360 video data. For example, direction of arrival (DOA) of the door class was often annotated from the video data because many doors were outside of the view of the mocap. To calculate source distance in such cases, we also used information on room dimensions and installation positions from our recording log in addition to the vidco data. Any interpolated spatial annotations were visually checked in the confirmation videos.
A.3 Data Format
We use WAV file format for the audio data and MP4 file format for the video data. The metadata are tabulated and served in CSV file format. The sound event classes, DOAs, and distances are provided the following format:
frame number, active class index, source number index, azimuth, elevation, distance
with all labels served as integers. Frame, class, and source enumeration begin at O. Frames correspond to temporal resolution of 100 msec. Azimuth and elevation angles are given in degrees, rounded to the closest integer value, with azimuth and elevation being zero at the front, azimuth e
[-100, 180%]. and elevation 0 E [-90%, 90% The azimuth angle increases counter-clockwise (Ø = 90% at the left). Distances are provided in centimeters, also rounded to the closest integer value. The source index is unique integer for each source in the scene. Note that each unique participant gets assigned one identifier, but not individual events produced by the same participant; e.g., clapping event and a laughter event produced by the same person have the same identifier. Independent sources that are not participants (e.g.. a loudspeaker playing music in the room) get O identifier. Note that the source index and the source distance only included as additional information that can be exploited during training. An example line could be as follows:
10, 1, 1, 50,, 30, 181
which describes that in frame 10, an event of class male speech (class 1) belonging to one participant (source 1) is active location (-509, 30% 181 cm).
B
Data Analysis
B.1 Duration
Figures Gjlllustrats the box plots of duration, i.e., how long a sound event lasts. The figure shows that there are classes with similar duration trends. Speech classes have wide range of plots, whereas laughing has a shorter duration than speech. While phone and bell have similar medians, the box of phone is longer phone calls repeat the recorded sound until it is answered. There several classes with longer duration, c.g., domestic sounds, music, musical instruments, and faucet, whereas collision or tapping sounds such as door and knock are relatively short.
B.2 DOA
Figure 7 shows the distribution of DOAs of the rest of the classes not depicted in Figure 3 While DOAS of human-produced classes such speech, laugh, clap, or footsteps are dispersed across the 360 plane, classes such as door, knock, or faucet result in specific discrete points in the plot due to their fixed position in the room. Music and bell classes also show similar trends to door class
15

---

10*
Fromes
107
301
107
90.
Clap
Phone
Lough
Dom,
Foctstops
Door
Music
Music
Raucet
Ball
K0ock
Figure 6: Box plots of the event durations across classes. The vertical axis indicates frame numbers in log scale. The center line of box shows the median. The top and bottom lines show the first and third quartile, while the whiskers extend from the box by 1.5 times the interquartile range. Each plot represents a data point of duration.
their sound sources rarely moved. Domestic sounds, phone, and musical instrument classes show similar trends to the speech classes as the respective sources are sometimes still and other times moving.
C Experiment
C.1 Evaluation Metric
We describe details on evaluation metrics [8) The metrics are based on true positives, T.P, and false positives, FP. determined not only by correct wrong detections but also based whether they are closer or further than a distance threshold (209 this paper) from the reference.
We associate Pa predicted events of class with R. reference events of class for each class e E [l...., C1 and each segment. count false negatives for misses: F FN. = max(0, 0, RS - Po). and count false positives for extraneous predictions: FP max(0, P. R.). Ke predictions are spatially associated with references based on Hungarian algorithm: Ke = min( Pc. R.). The number can also be considered as the unthresholded true positives: TPc Ke. We apply the spatial threshold to move Lo < Ke) predictions with a value greater than the threshold to false positives: FPe,2200 Le. and FP. FP c,
FPP. 2.200. We count the remaining matched estimates per class as true positives: T'P Ke F F..-FPeaeee PP.22000
Based on those, we form the location- F-score and error rate. We perform macro- anero-averaggnn of the location-aware F-score: F20e E. F2 /eeo Additionally, we evaluate localization accuracy through a class-aware localization error LE computed as the mean angular error of the matched true positives per class, and then macro-aveen -vero-averaged: LECD, = EE / TTP, for each segment with TP. > 0, and with 0j. being the angular error between the k-th matched prediction and reference, and after averaging across frames that have any true positives, LECD = Σ. F. E.CD.e/ c/ C. When there set 180 the localization
true positive outputs in a class, compute the macro average, error in the class. Complementary to the localization error, we compute a localization recall metric per class, also macro-averaged. LRCD, TP. PTTP FN c), and LRcD L.R.,. The localization error and recall are not thresholded to provide more varied complementary information to the Docation-depentee F-score, presenting localization accuracy outside the spatial threshold. For a more thorough analysis of the joint SELD metrics, please refer to 25 38
C.2 Experimental Setting
We add few details on experimental settings. The number of channels on audio features is M = consisting of 4ch amplitude spectrograms and 3ch IPDs. apply the short-term Fourier transform (STFT) to 1.27-sec audio signal with 20-ms frame length and 1 10-ms frame hop. The number frequency bins = 257, and the length of time frames is T 128. We keep the input feature length to 128 frames during inference and set the shift length to 120 frames.
16

---

For visual features, we use the pre-trained YOLOX object detection mode 1M which is trained with the COCO dataset [23]. While COCO has 80 object classes, we focus on person, cell phone, and sink classes because they are strongly related to the 13 sound events in STARSS23, Only person class is stably detected our preliminary experiments STARSS23 videos. Therefore, we use the model to get bounding boxes for the person class. We show an example of object detection results in Figure X While people are detected by the object detector, a sink is not detected by the detector.
Instead of the current object detection method, we also experimented with two convolutional neural network (CNN)-based visual feature extraction methods, i.e., VGG-16 47 and ResNet-18 [20]. While training losses of the CNN-based visual features are lower than the object detection method, validation results were worse than the object detection method. Bounding boxes of person class have simple but enough information on human positions. That would lead to generalization on human body-related classes. The CNN-based visual features retain richer visual information than bounding boxes, so they may require more training data to generalize. Visual features extracted from Transformer-bant based methods, e.g., ViT models 13, would face the same issues as CNN-based methods.
We use batch size of 16 and the Adam optimizer with a weight decay of 10-6 to train the audio- visual sound event localization and detection (SELD) system. The learning rate is set to 0.001. We validate and save model weights at every 1,000 iterations up to 20,000 iterations. We select a model that demonstrated the best aggregated SELD error, ESELD, calculated as
R2 + (1 F20-) ) + LEOD + (1 LRCD)
ESELD
180
(6)
4
We report the average scores and error bars of five experiments.
The model parameter size is 0,8 M. The model size is kept intentionally small for easier trials. A single GPU (e.g.. GeForce GTX 1080 Ti) is used for training. The training takes around six hours.
C.3 Experimental Results
In addition to the F-score per class in first-order ambisonics (FOA) format in Figure 5 we show the other SELD metries per class in both FOA and tetrahedral microphone array (MIC) formats. As shown in Figure 9c the F-score in MIC format shows a similar trend as in FOA format. While the audio-visual SELD system performs worse in the F-score of non-body-related classes, the audio- visual system demonstrates a higher F-score in body-related classes. Figure Paland [d show that the audio-visual system demonstrates lower localization error of body-related classes both formats. There is no significant trend of the localization recall between the two formats.
D Competition
STARSS23 has served as the development set and evaluation set for the SELD Task of the DCASE 2023 Challenge"" [T] which aims to accelerate audio-only and audio- visual SELD research. The task participants use the development audio/video recordings and labels to train and validate their SELD systems in the development process. The evaluation recordings without labels are used to produce system outputs for the challenge evaluation phase. researchers wish to compare their system against the submissions of the challenge, they will have directly comparable results if they use the evaluation data as their testing set. Also, the implementation the audio-visual SELD system described herein, trained and evaluated with STARSS23, has served as the baseline method for the audio-visual track of the challenge.
E Social Impact
The STARSS23 enables research audio-only and audio-visual SELD tasks, which form the backbone of various real-world applications on acoustic and audio-visual monitoring, intelligent home applications, audio-visual machine perception. The dataset, together with the associated
https: bttpe://github wwt..ccccccpccaeee /onn/open--aae http://wwwa.aaaaaccccaaa////////
8x8 _00_33cccoo.prrr coco..y
httpo://dcase http:///eeeseeeeceeeeeeeeeeeeeee commuitty/challleeee22233 sound event www.:teooaalaaattttiii.n www.lizataaiiioooooooooo and detection evalusted real
spatial- wal--oondd--a scenes
17

---

challenge, accelerates such research for research institutes and industry since it is the first of its kind based on real annotated recordings. Multiple research institutes, university laboratories, and industrial research and development groups have already shown interest in the dataset and its use, either as part of the DCASE challenge or outside of it in independent published studies. We expect the dataset to set the standard in the upcoming years in audio-visual SELD-related studies due to its unique spatial annotations from real tracked people and sound sources and its spatial audio content, which is becoming more and more relevant with multiple monitoring or smart home devices employing microphone arrays. The dataset also offers opportunities for cross-regional evaluation, with its recordings coming from two different sites geographically far apart. Of course, as with many strongly annotated datasets, the diversity of sound events is limited and cannot capture the conditions of many real-world application-specifs scenes. However, believe that it is valuable contribution to the development and maturation of such systems, at which point we expect more application-specifcc SELD datasets appear.
F
Personal Data Handling
STARSS23 data recorded with over 50 voluntary participants. Before the recording, explain our research purpose, how we record the sound scenes, and how we treat and release the recording data. Regarding the recording process, example of the generic instructions is in Appendix [ Our explanation is based on text format and verbal deseription. Participants can ask us questions related to recording and public release. We also explain potential risks, i.e., recording data containing personally identifiable information, and our Institutional Review Board (IRB) approvals. The personally identifiable information is raw speech and blurred faces. The participants not to reveal personal information during the recordings, and limit themselves to conversations of generic topics. After the explanation and Q&A, when participants understand the purpose and risk of recording and release, each participant signs the consent form.
also instructed
G Data Sheet
For dataset documentation, we take the questions from Datasheets for Datasets 16 and answer them.
G.1 Motivation
Q: For what purpose was the dataset created? Was there specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.
A: This dataset is created to tackle audio-visual sound event localization and detec- tion (SELD) tasks. While visual data about sound source objects can support SELD tasks, e.g., feet are a potential source of footsteps, the existing datasets did not contain the complete multichannel audio, video, and annotation set. STARSS23 serves real sound scene recording with multichannel audio, video data aligned with the audio, and spatiotemporal annotation of sound events. STARSS23 allows the incorporation of audio-visual correspondence into multichannel audio signal processing methods.
Q: Who created the dataset (c.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
A: Creative AI Laboratory (CAL) at Sony Group and Audio Research Group (ARG) at Tampere University.
Q: Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
A: The data collection and annotation at Tampere University has received funding by Google.
Q: Any other comments?
A: N/A.
18

---

G.2 Composition
Q: What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them: nodes and edges)? Please provide description.
A: Real sound scene recordings with multichannel audio data, video data aligned with the audio data, and annotations of temporal activation, the direction of arrival (DOA), and distance of sound events.
Q: How many instances are there in total (of each type, if appropriate)?
A: The development set totals about hours and 22 minutes, of which 168 clips were recorded with 57 participants in 16 rooms with annotation. The evaluation set totals about 3.5 hours and 79 clips without annotation.
Q: Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/ larger set, please describe why not (e.g.. to cover more diverse range of instances, because instances were withheld or unavailable).
If it is not representative of the
A: It contains all possible instances.
Q: What data does each instance consist of? "Raw" data unprocessd text images) or features? In either case, please provide a description.
A: Raw audio and video data, but we convert 32ch 48kHz audio recordings to 4ch 24kHz, and 3840x 1920 360" video recordings to x900 equirectangular. We also conduct face-blurring on video data to anonymize identical information.
Q: Is there label or target associated with each instance? If so, please provide description.
A: The dataset contains temporal activation, DOA, and source distance labels of 13 target sound event classes. The classes are female speech, male speech, clapping, telephone, laughter, domestic sounds, footsteps, door, music, musical instrument, water tap, bell, and knock.
Q: Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
The evaluation set has no annotation because the set is used in testing phase of the SELD task of the DCASE 2023 challenge. While almost all audio recordings in the development and evaluation set accompanied by synchronized video recordings, only 12 audio recordings in the development set are missing vidcos (from fold3 [dd3_room21__ mix001.wav to fold3 2ald3_room21 mix)02.wav)..
Q: Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
A: In each clip, we use the same tag, c.g., BoldXrroomi mixZ. We use the tag for audio (wwa)), video (.mp4), and labels (.csv).
Q: Are there recommended data splits (e.g., training, developpent/vvaiiattion, testing)? If so, please provide description of these splits, explaining the rationale behind them.
A: Yes. STARSS23 has the dev-set-train part and the dev-set-test part for the development process.
Q: Are there any errors, sources of noise, redundancies in the dataset? If so, please provide a description.
A: In addition to target sound events, the sound scene recordings contain directional inter- ference sounds and background noise. That makes the dataset a more realistic situation. We confirm all the labels by listening to the audio and watching the video. If there are any errors, they would be negligible.
19

---

Q: Is the dataset self-contained, or does it link to or otherwise rely on external resources (e-g,, websites, tweets, other datasets)? If it links to relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
A: Self-contained.
Q: Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-pune non-public communications)? If so, please provide description.
A: No.
Q: Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
A: No.
Q: Does the dataset identify any subpopulations (e.g., by age, gender)? so, please describe how these subpopulations are identified and provide description of their respective distributions within the dataset.
A: STARSS23 set female and male speech as two target sound event classes. The frame coverages of both classes almost equal: 28.4 % and 31.4 %, respectively.
Q: Is it possible to identify individuals (i.e., directly or indirectly (i.e., combination with other data) from the dataset? If so, please describe how.
more natural persons), either
A: As audio-visual sound scene recording dataset, STARSS23 contains raw speech data. However, faces in video data are blurred, and the talk contents have no personal topics. So, it is hard to identify individuals. We also explain to participants the potential risk of identical information before recording. After the explanation, participants sign a consent form for recording when they understand the potential risk.
Q: Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions union memberships, genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide description.
locations; financial or health data; biometric or
A: No.
Q: Any other comments?
A: N/A.
G.3
Collection Process
Q: How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If the data reported by subjects indirectly inferred/derived from other data, was the data validated/verrfied? If so, please describe how.
A: Each clip of a sound scene recorded in a room with participants and sound props. Participants improvise a scene following generic instruction about the sound scene and event.
Q: What mechanisms or procedures were used to collect the data (e-g., hardware apparatuses sensors, manual human curation, software programs, software APIs)? How were these mechanisms or procedures validated?
Multichannel audio and video data are recorded with a 32ch microphone array and 360 camera, respectively, A motion capture system and wireless microphones also recorded
20

---

for annotation. The specific recording equipment is Eigenmike em3 Ricoh Theta V Optitrack Flex 1 and Røde Wireless Go 1 respectively.
Q: If the dataset is sample from larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
A: N/A.
Q: Who was involved in the data collection process (e-g, students, crowdworkers, contractors) and how were they compensated (e.g,, how much were crowdworkers paid)?
Voluntary participants act in sound scene, and authors record the scene.
Q: Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (c.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
A: The first round of recordings was collected between September 2021 and April 2022. second round of recordings was collected between November 2022 and February 2023.
Q: Were any ethical review processes conducted (e-g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as link or other access point to any supporting documentation.
A: Yes. We get approvals from our Institutional Review Board (IRB). Following the discussion about personally identifiable information, we conduct face-blurring on video data and explain to participants about potential risks, i.e., recording data containing identifiable information.
Q: Did you collect the data from the individuals in question directly, or obtain via third parties or other sources (e.g,, websites)?
From the individuals.
Q: Were the individuals in question notified about the data collection? If please describe (or show with screenshots or other information) how notice was provided, and provide link other access point to, otherwise reproduce, the exact language of the notification itself.
A: Yes. Before the recording, we explain our research purpose, how we record sound scenes, and how we treat and release the recording data. Our explanation is based on text format and verbal description. Participants can ask us questions related to recording and release.
Q: Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
A: Yes. After our explanation and Q&A, when participants understand the purpose and risk of recording and release, each participant signs the consent form.
Q: If consent to revoke their consent in the future or for certain uses? If so, please provide description, as well as a link or other access point to the mechanism (if appropriate).
obtained, were the consenting individuals provided with mechanism
A: Yes. In such case, they can contact authors.
Q: Has an analysis of the potential impact the dataset and its use on data subjects (e.g., data protection impact analysis) been conducted? If so, please provide descrip- tion of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.
A: N/A.
Q: Any other comments?
A: N/A.
bttpe://baccoutiicc.comm http:///aaccaceeeee.ceee//carrar
21

---

G.4 Prggooesing/ClannnggLLbiiii
of the data done (e-g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a deseription. If not, you may skip the remaining questions in this section.
Q: Was any preprocesing/deeaninn/bbbe
A: We convert audio and video data to the appropriate size. We also annotate temporal activation and DOA of sound events. After annotating, we confirm the labels by listening the audio and watching the video.
Q: Was the "raw" data saved in addition to the preprocesea/ctaane/labeee data (e.g.- to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
We saved "raw" (large size) audio and video data for future use. If one is interested in them, one can contact the authors.
Q: Is the software that was used to preprocess/clean/label the data available? If so, please provide a link or other access point.
A: We use Python code for audio data conversion, and RICOH THETA 16 and FFmpeg for video data conversion. To annotate temporal activation, Audacity and REAPER are used. Tracking results are used in Motive 20
Q: Any other comments?
N/A.
G.S
Uses
Q: Has the dataset been used for any tasks already? If so, please provide description.
A: No, the dataset has not yet been used for any scientific papers. This paper is the first to use the dataset.
Q: Is there a repository that links to any or all papers or systems that use the dataset?
If so, please provide link or other access point.
A: Yes. We have the code repository of the SELD system: https://github. http:itthh..oo..eeeeeeemmeeeeeee /oon//
Q: What (other) tasks could the dataset be used for?
A: Apart from visual SELD tasks, could the dataset for audio-only SELD tasks, audio-visual speaker DOA estimation tasks, audio-visual sound source localization tasks, and audio-visual cross-modal retrieval tasks. Audio-visual sound source distance estimation could be another task.
Q: Is there anything about the composition of the dataset or the way was collected and preprocessed/cllanedllaaee anything that a dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals groups (e.g., stereotyping, quality of service issues) other risks or harms (e.g., legal risks, financial harms)? If so, please provide a description. Is there anything dataset consumer could do mitigate these risks harms?
that might impact future uses? For example, is there
A: No.
Q: Are there tasks for which the dataset should not be used? If so, please provide a description.
A: No.
Q: Any other comments? N/A.
bttee://beesa60.. http:////eeaaa....oooeeeeaaaaaae htt.a////..//..b//////////tt/ata bttpe://ffmpeg.. org/
bttpa://wonn http://wwwo.uuccttttoa.. porgs//tannonn////////
htpa://wue.raapar fn/
attps: / /ttp://///tttcckkkk http://ppwteeccc.cooeeeeaaaaa//a wowp//oowwwarre/ootvvvee
22

---

G.6
Distribution
Q: Will the dataset distributed third parties outside the entity (e.g,, company, institution, organization) on behalf of which the dataset was created? If so, please provide description.
A: Yes. STARSS23 is publicly available httt::::/eeaooooo///ereeer//////
Q: How will the dataset will be distributed (e.g., tarball website, API, GitHub)? Does the dataset have digital object identifier (DOI)?
STARSS23 is distributed via "10.5281 srrecnddo77799050
The DOI is
Q: When will the dataset be distributed?
A: STARSS23 already distributed.
Q: Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, well as any fees associated with these restrictions.
A: STARSS23 is licensed under MIT License.
Q: Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide link other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.
A: No.
Q: Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation. A: No.
Q: Any other comments?
N/A.
G.7 Maintenance
Q: Who will be supportigghotiig/nanttaiinng the dataset?
A: Sony Group and Tampere University.
Q: How can the owner/cuurator/manager of the dataset be contacted (e.g., email ad- dress)?
A: Please contact kazuki. shimada@sony......mooaa.aoaoooa. com and pecchontis.poittiiieio fi.
Q: Is there an erratum? If so, please provide link or other access point.
A: All changes to the dataset will be announced on ftttp:://zenodo.org//eeeee
7880637
Q: Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to dataset consumers (e.g., mailing list, GitHub)?
A: Yes, all the updates will be synced on the website.
Q: If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforeed. A: No.
Q: Will older versions of the dataset continue to be suppotec/nste/naitaindd?? If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers.
23

---

A: The older dataset versions remain in Zenodo if any changes are made.
Q: If others want to extend/angment/build on/contribute to the dataset, is there mechanism for them to do so? If so, please provide description. Will these contributions be validated/verified?? If so, please describe how. If not, why not? Is there a process for these contributions dataset consumers? If so, please provide
communicating/distrini a description.
A: Yes. Others can contact the authors of this paper to describe their proposed extension contribution. We would discuss their proposed contribution to confirm its validity, and if confirmed, we will release a new version of the dataset on Zenodo and announce accordingly.
Q: Any other comments?
A: N/A.
24

---

(a) Male speech
(b) Clap
Astmuth
Flusstion
(c) Phone
(d) Laugh
Asimuth
Asimat
Divvation
Devvttoo
(c) Domestic sounds
(f) Door
Anmuth
Arimth
Devation
Devation
(g) Music
(h) Musical instruments
Adtmutth
Devation
+24
-04
(i) Faucet
Bell
(k) Knock
(1) Overall
Figure 7: The distribution of azimuth and elevation labels. Each point in the scatter figures represents a frame. The top and right figures are stacked histograms of azimuth and elevation.
25

---

result
person
personjo 85
Figure 8: An example of object detection results. While the object detector detects people, the detector does not detect sink, which is slightly to the right of the center.
26

---

100
Audio Vicual
00-
20
NA
NAA
0-
Clap
Laugh sounds
montitaps
speech
instr.
related milattd
(a) Class-aware localization error in FOA format
Aacio Vicual
PAcEE Dity
5
20-
0-
per speech
Male
Clap
Phone
Laugh
Dom Poonsteps Door
Music
Music
Paacet
Knock
Body von body
(b) Class-aware localization recall in FOA format
Aucio-Orr Oity
F 2.20l
1V
Fem spooch speech
Male
Clap
Phone
Laugh
Uom Footszeps Uoor
Music
Music Instr.
Foucet
sell
knock
Bedy van body related related
sounds
(c) Location-aware F-score MIC format
Aacio-hiiaal
100
40
M/A
MIA
N/A
NVA
Laugh sounds
sell
knock
body relused related
Bpocchspeech
instr.
(d) Class-aware localization error in MIC format
AuchoNlitall Aucto-Oity
20
d raucet
0
Hmm
Clap Phone
Laugh uom rootscaps Uoor
music
music
Knock
Body VOn body
(c) Class-aware localization recall in MIC format
Figure 9: SELD performance (C = 13) per sound event class in audio-visual and audio-only systems evaluated for the dev-set-test in STASS23. We found that few classes sometimes had zero localization recall, i.e., no true positive output in these classes. As statistics of such cases are less meaningful in localization error, we do not show bars and put N/A in the figures.
27


Reviews:
[{"review_id": "DGRQHBGWHJ", "reviewer": "Reviewer_zAcc", "summary": "", "questions": "", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ The collected Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23) dataset consists of multichannel audio data recorded with a microphone array, video data, and spatiotemporal annotation of sound events. It enables audio-visual research for spatial audio scene understanding.\n\n+ The proposed audio-visual sound event localization and detection (SELD) task is very interesting. It uses multichannel audio and visual information to estimate the temporal activation and direction of arrival (DOA) of target sound events.\n\n+ The paper is easy to follow. The significance of the dataset is clearly illustrated, and the details about the dataset collection are provided.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "bQ9sXIiBWM", "reviewer": "Reviewer_7Cu6", "summary": "", "questions": "", "limitations": "The author mentioned improving SELD performance as their limitation. In my opinion, some analysis of difficult cases can demonstrate in visual would be helpful to the following researchers. Also, the recordings contain a total of 13 target sound event classes, while the total amount of STARSS23 is a little bit scarce. Maybe the authors have plans to expand the dataset in the future. ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The idea of adding another modality input, visual data, to the existing STARSS22 dataset is intuitive. It is good for tackling the task of audio-visual sound event localization and detection. The visual information can benefit temporal activation and DOA, as audible information is usually apparent in the visual modality. This dataset could facilitate research in this field and is therefore an overall positive contribution to the audio, vision, and multimodal research communities.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "8dWWxSrulk", "reviewer": "Reviewer_pjsy", "summary": "", "questions": "", "limitations": "The main weakness of the proposed dataset is the limited novelty over the previous work STARSS22. It seems that the work is very incremental in that only visual information is intergrated by adding an additional camera, while other pipelines would remain the same. We understand it can be useful to develop a multimodal dataset for SELD, while the contribution of adding visual information looks to be not novel and significant enough to be considered as a NeurIPS publication.\n\nThe experimental results also show that the visual information may not be always useful for SELD as it harms the detection performance (table 3). While it can be claimed that this makes room for algorithm development, the work should at least demonstrate the necessity of using the visual information.\n\nAnother concern comes from the privacy issues. It is shown that the faces of participants are blurred in the released dataset, however, in practical usage the raw visual information will still be captured. It is not practical to always get permission of all persons that may appear in the camera.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The pipeline of the dataset construction is clear, feasible and reproducible. The baseline setups are also reasonable.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "wqymlmdqNC", "reviewer": "Reviewer_CWXv", "summary": "", "questions": "", "limitations": "The authors should discuss more potential tasks such as audio-visual cross-modal retrieval and classification .\n", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "13 target sound event classes are annotated. The classes follow loosely the Audioset ontology.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: In general, based on the reviewers' opinions, this work has made some contributions, and the overall rating tends to be positive. Therefore, I believe it is suitable for acceptance at our conference. However, a concern raised by multiple reviewers is the necessity of visual information. Due to the presence of experimental results that could potentially contradict the motivation and theory, the authors should consider providing additional analysis and explanations in the final camera-ready version.
Decision: Accept (Poster)