SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kazuki Shimada
kazuki.shimada@sony.com
Archontis Politis
Tampere University
archontis.politis@tuni.fi
Parthasaarathy Sudarsanam
Daniel Krause
Kengo Uchida
Sharath Adavanne
Aapo Hakala
Yuichiro Koyama
Sony Group Corporation
Naoya Takahashi
Shusuke Takahashi
Tuomas Virtanen
Yuki Mitsufuji
Sony AI, Sony Group Corporation
While direction of arrival (DOA) of sound events is generally estimated from multichannel audio data recorded in a microphone array, sound events usually derive from visually perceptible source objects, e.g., sounds of footsteps come from the feet of a walker. This paper proposes an audio-visual sound event localization and detection (SELD) task, which uses multichannel audio and video information to estimate the temporal activation and DOA of target sound events. Audio-visual SELD systems can detect and localize sound events using signals from a microphone array and audio-visual correspondence. We also introduce an audio-visual dataset, Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23), which consists of multichannel audio data recorded with a microphone array, video data, and spatiotemporal annotation of sound events. Sound scenes in STARSS23 are recorded with instructions, which guide recording participants to ensure adequate activity and occurrences of sound events. STARSS23 also serves human-annotated temporal activation labels and human-confirmed DOA labels, which are based on tracking results of a motion capture system. Our benchmark results demonstrate the benefits of using visual object positions in audio-visual SELD tasks. The data is available at https://zenodo.org/record/7880637.
Given multichannel audio input from a microphone array, a sound event localization and detection (SELD) system [1, 45, 8, 46] outputs a temporal activation track for each of the target sound classes along with one or more corresponding spatial trajectories, e.g., the direction of arrival (DOA) around the microphone array, when the track indicates activity. Such a spatiotemporal characterization of sound scenes can be used in a wide range of machine cognition tasks, such as inference on the type of environment, tracking of specific types of sound sources, acoustic monitoring, scene visualization systems, and smart-home applications. Recently neural network (NN)-based SELD
Figure 1: An audio-visual SELD system takes multichannel audio recording data and video data aligned with the audio data and estimates temporal activity and DOA of each class per frame. In a frame  $t$  of a recording in STARSS23, a knocking sound comes from a door while a domestic sound derives from a vacuum cleaner behind a microphone array, as shown in an overlay of  $360^{\circ}$  video and spatial acoustic power map.
systems [1, 45, 8, 46] show high localization and detection performance. These systems need data with activity and DOA labels of target sound events for training and evaluation. Because annotation of DOA labels is challenging in real sound scene recordings, most SELD datasets [2, 36, 35, 19, 28] consist of synthetic audio data, which are made by convolution of monaural sound event signals and multichannel impulse response signals with DOA labels. The Sony-TAu Realistic Spatial Soundscapes 2022 dataset (STARSS22) [39] tackles real sound scene recordings with DOA labels, which is based on tracking results of a motion capture (mocap) system. Currently, STARSS22 is the only SELD dataset with real sound scenes, including overlapping sound events, moving source events, and natural distribution of temporal activation and DOA, e.g., sounds of footsteps are relatively short and heard from a lower elevation. While this dataset is suitable for evaluating audio-only SELD systems in natural sound scenes, the dataset does not include other modality input, e.g., video data.
Sound events in real sound scenes originate from their sound source objects, e.g., speech comes from a person's mouth, sounds of footsteps are produced from the feet of a walker, and a knocking sound originates from a door. Such sound source object information is usually apparent in the visual modality. Video data aligned with audio recordings in SELD tasks have the potential to mitigate difficulties and ambiguities of the spatiotemporal characterization of the sound scene as audio-visual data improves source separation [59, 14, 58, 26] and speech recognition [32, 3, 34, 24]. Visible people in the video data can provide candidate positions of human body-related sounds. When a person walks in the video, tapping sounds are easily recognized as footsteps. To investigate how a visual feature input affects a SELD system, we propose an audio-visual SELD task, which uses audio and video data to estimate spatiotemporal characterization of sound events. The left side of Figure 1 shows an audio-visual SELD system, which takes multichannel audio recordings and video aligned with the audio recordings and outputs activity and DOA of target sound events in each frame. To tackle audio-visual SELD tasks, we need an audio-visual dataset consisting of multichannel audio, video, and activity and DOA labels of sound events per frame.
There is another interest in audio-visual sound source localization tasks [44, 12, 61, 4, 33], which takes monaural audio and images and estimates where the audio is coming from in the images. They focus on learning audio-visual semantic correspondence, not estimating physical DOA around a microphone array. While the audio-visual sound source localization datasets [44, 12, 61] are adequate to train NNs with audio-visual correspondence and evaluate localization performance in images, they are typically monophonic without spatial labels for the target sound events. Several datasets serve multichannel audio and video data with real sound scenes [27, 53, 54, 60, 56, 10, 40, 41, 50], whereas only a few audio-visual datasets have multichannel audio data with DOA labels of speakers around a microphone array [40, 41, 50]. While these datasets help to evaluate audio-visual speaker DOA estimation (DOAE) tasks, the evaluation focuses only on speech, not sound events such as musical instruments or footsteps.
To tackle audio-visual SELD tasks, we introduce an audio-visual dataset, the Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23), consisting of multichannel audio, video, and spatiotemporal annotations of sound events, i.e., activity and DOA labels per each frame. The right part of Figure 1 shows a still in a frame of a video in STARSS23, made from  $360^{\circ}$  video, spatial acoustic power
Table 1: We compare STARSS23 to other real sound scene datasets from three points of view: whether audio data is multichannel, whether video data is included, and what annotation type is.
Dataset	Multichannel audio	Video	Annotation
STARSS22 [39]	✓	-	Activity & DOA of sound events
VGG-SS [12]	-	✓	Bounding box
AVSBench [61]	-	✓	Segmentation map
YouTube-360 [27]	✓	✓	-
AVRI [41]	✓	✓	Activity & DOA of speech
STARSS23 (Ours)	✓	✓	Activity, DOA & distance of sound events
map generated from a microphone array, and sound event labels. The dataset contains over 7-hour recordings with 57 participants in 16 rooms with the spatiotemporal annotation as a development set. The participants are guided by generic instructions and suggested activities in the recording to induce adequate occurrences of the sound events and diversity of content. There are 13 classes of target sound events, such as speech, musical instruments, and footsteps. To reveal whether the dataset has a natural distribution of sound events, we analyze the dataset regarding frame coverage, overlap, and DOA distributions per each sound event class. We develop and test an audio-visual SELD system with STARSS23. To investigate the effects of audio-visual input, we present overall localization and detection performance and per-class results.
Sound event localization and detection SELDnet [1] is the first SELD method, which uses convolutional recurrent neural network (CRNN) to output activity and DOA separately. An activity-coupled Cartesian DOA (ACCCDOA) [45] vector, which embeds sound event activity information to the length of a Cartesian DOA vector, enables us to solve SELD tasks with a single output. While the two methods tackle overlaps from different classes, they cannot solve overlaps from the same class. Multi-ACCCDOA [46] is an ACCDOA extension, allowing models to output overlaps from the same class. To handle that case effectively, Multi-ACCCDOA incorporates auxiliary duplicating permutation invariant training (ADPIT) [46]. There are other SELD works about framework [8, 29, 55], network architecture [48], audio feature [30, 21], and data augmentation [51, 42].
To train or evaluate SELD methods, we need multichannel audio data with temporal activation and DOA labels. In synthetic multichannel audio datasets [2, 36, 35, 19, 28], DOA labels can be easily annotated because the data are made from multichannel impulse response signals with DOA labels. While the SECL-UMons and AVECL-UMons datasets [6, 5] tackled spatial recording with DOA labels, it is limited to isolated single event recordings or combinations of two simultaneous events, ignoring spatiotemporal information linking events in a natural scene. STARSS22 [39] tackled real spatial recording with temporal activation and DOA labels of each target class in natural scenes. Participants improvise natural scenes with a mocap system, whose tracking results are used for DOA labels. However, the dataset does not release video data. Therefore, it cannot be used to evaluate audio-visual SELD systems.
Audio-visual sound source localization There is broad interest in audio-visual sound source localization tasks [44, 12, 61, 4, 33]. Chen et al. have tackled unsupervised learning to localize sound sources in video and evaluated the method on the VGG-SS dataset [12], which annotates bounding boxes of sound sources for sound and video pairs. The AVSBench dataset [61] serves pixel-level audio-visual segmentation maps for videos over 23 class categories. Because the datasets do not have multichannel audio recordings, they cannot be applied to evaluating SELD tasks.
Audio-visual dataset with multichannel audio Several audio-visual datasets include multichannel audio data [27, 53, 54, 60, 56, 10, 40, 41, 50]. As many datasets are used for self-supervised learning [27, 54, 53] or non-localization tasks [60, 56, 10], there are no DOA labels. The YouTube-360 dataset [27] serves first-order ambisonics (FOA) signal and  $360^{\circ}$  video data without any labels for self-supervised learning.
Figure 2: The pipeline of data construction.
A few audio-visual datasets are collected for audio-visual DOAE tasks [40, 41, 50]. Qian et al. proposed an audio-visual DOAE system, which takes spectrograms and phase features from the audio input and face-bounding boxes from video input to estimate the DOA of each speaker [41]. The system was evaluated with the Audio-Visual Robotic Interface (AVRI) dataset, recorded using Kinect and a four-channel Respeaker array, along with activity and DOA labels. The audio-visual features are helpful for DOAE, and the dataset supports the evaluation of audio-visual speaker DOAE. However, the dataset is only for speech, not various sound events such as clapping and knocks. We summarize the comparison of STARSS23 with other real sound scene datasets in Table 1.
3 STARSS23 Dataset
STARSS23 contains multichannel audio and video recordings of sound scenes in various rooms and environments, together with temporal and spatial annotations of sound events belonging to a set of target classes. The dataset enables us to train and evaluate audio-visual SELD systems, which localize and detect sound events from multichannel audio and visual information. STARSS23 is available in a public research data repository<sup>1</sup> under the MIT license. There is also a demo video<sup>2</sup>.
The contents are recorded with short instructions, guiding participants in improvising sound scenes. The recordings contain a total of 13 target sound event classes. The multichannel audio data are delivered as two 4-channel spatial formats: FOA and tetrahedral microphone array (MIC). The video data are anonymized  $1920 \times 960$  equirectangular videos recorded by a  $360^{\circ}$  camera. The annotations of STARSS23 consist of temporal activation, DOA, and source distance of the target classes. STARSS23 is split into a development set and an evaluation set. The development set totals about 7 hours and 22 minutes, of which 168 clips were recorded with 57 participants in 16 rooms. The development set is further split into a training part (dev-set-train, 90 clips) and a testing part (dev-set-test, 78 clips) to support the development process. In the evaluation set, no publicly available annotations exist because the evaluation set is prepared for a competition, which is described in Appendix D.
STARSS23 improves a multichannel audio dataset, i.e., STARSS22 [39]. One of the critical differences is releasing video data aligned with multichannel audio data. While we maintain all the sessions of STARSS22, we add about 2.5 hours of material to the development set. STARSS23 also serves source distance labels of sound events as additional annotations. We follow the recording procedure in STARSS22, where video data are used only to check labels internally. Adding descriptions about video data and distance annotation, we show the data construction part as an audio-visual dataset.
3.2 Data Construction
As shown in Figure 2, STARSS23 is constructed in three steps: sound scene recording, data conversion, and annotation. We explain each step as follows.
1https://zenodo.org/record/7880637
$^{2}$ https://www.youtube.com/watch?v=ZtL-8wBYPow
Sound scene recording STARSS23 was created in Tampere, Finland, and Tokyo, Japan. Recordings at both sites shared the same process, organized in sessions corresponding to different rooms, soundmaking props, and participants. In each session, various clips were recorded with combinations of that session's participants acting simple scenes and interacting among themselves and with the sound props. The scenes were based on generic instructions on the desired sound events. The instructions were a rough guide to ensure adequate event activity and occurrences of the target sound classes in a clip. The left photo of Figure 2 shows that participants improvise following the instructions.
A set of 13 target sound event classes are selected to be annotated, based on the sound events captured adequately in the recorded scenes. The class labels are chosen to conform to the AudioSet ontology [17]. They are: female speech, male speech, clapping, telephone, laughter, domestic sounds, footsteps, door, music, musical instrument, water tap, bell, knock. Music, e.g., background music or pop music, is played by a loudspeaker in the room. On the other hand, musical instruments are played by participants, including acoustic guitar, piano, and others. Domestic sounds consist of vacuum cleaners, mechanical fans, and boiling, which have strongly directional and loud sounds. They can be distinguishable from natural background noise in sound scenes. The scenes also contain directional interference sounds, such as computer keyboard or shuffling cards that are not labeled.
As shown in the left photos of Figure 2, each scene was captured with audio-visual sensors, i.e., a high-resolution 32-channel spherical microphone array (Eigenmike em32<sup>3</sup>) with a height set at  $1.5\mathrm{m}$ , and a  $360^{\circ}$  camera (Ricoh Theta  $\mathrm{V}^{4}$ ) mounted  $10\mathrm{cm}$  above the microphone array. For each recording session, a suitable position of the Eigenmike and Ricoh Theta V was determined to cover the scene from a central place. We also captured the scenes with two additional sensors for annotation: a mocap system of infrared cameras surrounding the scene, tracking reflective markers mounted on the participants and sound sources of interest (Optitrack Flex  $13^{5}$ ), and wireless microphones mounted on the participants and sound sources, providing close-mic recordings of the main sound events (Røde Wireless Go II<sup>6</sup>). The origin of the mocap system was set at ground level on the exact position of the Eigenmike. In contrast, the mocap cameras were positioned at the room's corners.
Recording starts on all devices before the beginning of a scene and stops right after. A clapper sound initiated the acting, and it served as a reference signal for synchronization between the different types of recordings, including the mocap system that can record a monophonic audio side signal for synchronization. All types of recordings were manually synchronized based on the clapper sound and subsequently cropped and stored at the end of each recording session. The details of sound scene recording, e.g., generic instructions, sound events, and sensors, are summarized in Appendix A.1.
Data conversion The original 32-channel recordings were converted to two 4-channel spatial formats: FOA and MIC. Conversion of the Eigenmike recordings to FOA following the SN3D normalization scheme (or ambiX) was performed with measurement-based filters [37]. Regarding the MIC format, channels 6, 10, 26, and 22 of the Eigenmike were selected, corresponding to a nearly tetrahedral arrangement. Analytical expressions of the directional responses of each format can be found in [36]. Finally, the converted recordings were downsampled to  $24\mathrm{kHz}$ . The raw  $360^{\circ}$  video data were converted to an equirectangular format with  $3840\times 1920$  resolution at 29.97 frames per second, which was convenient to handle as planar video data. Based on the participant's consent, the visible faces of all recordings were blurred. Finally, the video with face-blurring was converted to a  $1920\times 960$  resolution.
Annotation Spatiotemporal annotations of the sound events were conducted manually by the authors and research assistants. As shown in the lower right part of Figure 2, there were four steps: a) annotate the subset of the target classes that were active in each scene, b) annotate the temporal activity of such class instances, c) annotate the position of each such instance when active, moreover, d) confirm the annotations. Class annotations (a) were observed and logged during each scene recording. Activity labels (b) were manually annotated by listening to the wireless microphone recordings. Because each wireless microphone would capture prominent sounds produced by the participant or source it was assigned to, onset, offsets, source, and class information of each event could be conveniently extracted. In scenes or instances where associating an event to a source was
<sup>3</sup>https://mhacoustics.com/products#eigenmike1
4https://theta360.com/en/about/theta/v.html
<sup>5</sup>https://optitrack.com/c cameras/flex-13/
<sup>6</sup>https://rode.com/en/microphones/wireless/wirelessgoii
Table 2: Frame coverage and max, mean, and the degree of polyphony globally and of each class separately on the dev-set-train of STARSS23. The mean polyphony is computed over active frames.
Global	Fem. speech	Male speech	Clap	Phone	Laugh	Dom. sounds	Footsteps	Door	Music	Music. instr.	Faucet	Bell	Knock
Frame coverage (% total frames)	82.5	28.4	31.4	0.53	1.02	2.89	18.6	2.17	0.70	22.2	2.32	0.58	0.99	0.06
Max. polyphony	6	2	3	2	1	4	2	3	1	2	2	1	1	1
Mean polyphony	1.47	1.05	1.06	1.06	1.00	1.25	1.03	1.02	1.00	1.20	1.28	1.00	1.00	1.00
Polyphony 1 (% active frames)	62.878	95.44	94.35	93.74	100	78.67	97.27	97.87	100	79.99	71.66	100	100	100
Polyphony 2	28.443	4.56	5.51	6.26	0	18.48	2.73	1.94	0	20.01	28.36	0	0	0
Polyphony 3	7.385	0	0.14	0	0	2.39	0	0.19	0	0	0	0	0	0
Polyphony 4	1.129	0	0	0	0	0.46	0	0	0	0	0	0	0	0
Polyphony 5	0.162	0	0	0	0	0	0	0	0	0	0	0	0	0
Polyphony 6	0.003	0	0	0	0	0	0	0	0	0	0	0	0	0
(a) Female speech
(b) Footsteps
Figure 3: The distribution of the azimuth and elevation. Each point in the scatter figure represents a frame. The top and right figures are stacked histograms of azimuth and elevation.
ambiguous purely by listening, annotators would consult the video recordings to establish the correct association. The temporal annotation resolution was set to 100 msec. After onset, offset, and class information of events was established for each source and participant in the scene, the positional annotations (c) were extracted for each such event by attaching tracking results to the temporal activity window of the event. Positional information was logged in Cartesian coordinates with respect to the mocap system's origin. The event positions were converted to spherical coordinates, i.e., azimuth, elevation, and distance, which are more convenient for SELD tasks. Then, the class, temporal, and spatial annotations were combined and converted to the text format used in the previous dataset [39]. The annotation details are summarized in Appendix A.2. Confirmation of the annotations (d) was performed by listening to the Eigenmike recording while watching a synthetic video. The video was the equirectangular one overlapped with the event activities, which were visualized as labeled markers positioned at their respective azimuth and elevation on the video plane. If a clip was not passed the confirmation, the clip was annotated again.
Having a natural distribution of sound events is beneficial to evaluate audio-visual SELD systems. We analyze the frame coverage, polyphony, and DOA per sound event classes on the dev-set-train of STARSS23. Table 2 shows frame coverage and max, mean, and distribution of polyphony globally and of each class separately. Regular classes in frames are female and male speech, music, and domestic sounds. These classes are also frequent in our daily lives. Musical instruments and laugh classes show a high mean polyphony of the same class, which are natural situations in jam sessions and conversations. Figure 3 shows the distribution of DOA with the axis of the azimuth and elevation. Regarding female speech in Figure 3a, the elevation distribution has a strong peak around -10 degrees, while that of azimuth seems uniformly distributed. Compared to the speech class, footsteps appear in lower azimuth than -10 degrees. See Appendix B for further data analysis, e.g., duration.
In this section, we examine an audio-visual SELD task with STARSS23. For evaluation, we set the dev-set-train for training and hold out the dev-set-test for validation.
Figure 4: An audio-visual SELD system uses a CRNN model with a multi-ACCDOA output. While multi-ACCDOA output can have several tracks, we show that with a single track for simplicity. We use an object detector to incorporate visual information, which outputs bounding boxes on target classes. The results are encoded to vectors along with azimuth and elevation. Then, the encoded vectors are embedded and concatenated before the GRU layer.
4.1 Audio-Visual SELD System
To build an audio-visual SELD system, we start with an audio-only system based on SELDnet [1] and multi-ACCDOA [46], which is widely used in audio-only SELD tasks [39, 43, 31]. To extend the audio-only systems to audio-visual, we merge visual and audio information in the middle of the network, following the audio-visual speaker DOAE work [41].
First, we summarize the audio-only SELD system. Let  $X_{\mathrm{a}} \in \mathbb{R}^{M \times F \times T}$  be an input consisting of  $M$ -channel  $F$ -dimensional  $T$ -frame audio features, e.g., amplitude spectrograms. Convolutional layers are used as an audio embedding network,  $\mathcal{F}_{\mathrm{a},\theta_{\mathrm{a}}}(\cdot)$ , where  $\theta_{\mathrm{a}}$  is the network parameters:
$$
\boldsymbol {E} _ {\mathrm {a}} = \mathcal {F} _ {\mathrm {a}, \theta_ {\mathrm {a}}} \left(\boldsymbol {X} _ {\mathrm {a}}\right), \tag {1}
$$
where  $E_{\mathrm{a}} \in \mathbb{R}^{K_{\mathrm{a}} \times T'}$  is an audio embedding.  $K_{\mathrm{a}}$  and  $T'$  are the length of an embedding vector and the number of time frames, respectively. The embedding is followed by an audio-only decoder network with parameter  $\theta_{\mathrm{ao}}$ ,  $\mathcal{F}_{\mathrm{ao}, \theta_{\mathrm{ao}}}(\cdot)$ , which consists of a gated recurrent unit (GRU) layer and a fully connected layer (FC):