SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\boldsymbol {P}} = \mathcal {F} _ {\mathrm {a o}, \theta_ {\mathrm {a o}}} (\boldsymbol {E} _ {\mathrm {a}}), \tag {2}
$$
where  $\hat{P} \in \mathbb{R}^{3 \times N \times C \times T''}$  is an estimated output of  $N$ -track  $C$ -class  $T''$ -frame multi-ACCDOA. Each class of the multi-ACCDOA output in track  $n$  and frame  $t$  is represented by a three-dimensional vector with Cartesian coordinates  $x, y,$  and  $z$ , as shown in the bottom of Figure 4. The length of the vector shows activity, and the direction of the vector indicates the DOA around the microphone array. To train the SELD system, we use mean squared error (MSE) between the estimated and target multi-ACCDOA outputs under the ADPIIT scheme [46]. In inference, when the length of the vector is greater than a threshold, the class is considered active.
Then, we prepare a visual embedding. As visual input, we use the corresponding video frame at the start of the audio features. With the corresponding image, an object detection module, e.g., YOLOX [15], outputs bounding boxes of potential objects on target classes, e.g., person class. Let us denote a bounding box as  $(u_b, v_b, w_b, h_b)$ , where  $u_b$  and  $v_b$  indicate the image position of the top-left corner, and  $w_b$  and  $h_b$  show the width and height of the bounding box. Each bounding box is encoded to two vectors  $\rho_{\mathrm{azi}}(u)$  and  $\rho_{\mathrm{ele}}(v)$  along the image's horizontal axis  $u$  and vertical axis  $v$ . We follow the same formulation in [41] using a Gaussian distribution:
$$
\rho_ {\mathrm {a z i}} (u) = \exp \left(\frac {- | u - \mu_ {u} | ^ {2}}{\sigma_ {u} ^ {2}}\right), \tag {3}
$$
where the center  $\mu_{u} = u_{b} + \frac{1}{2} w_{b}$  is the horizontal center of the bounding box, and the standard deviation  $\sigma_{u}$  is proportional to  $w_{b}$ . The vertical vector  $\rho_{\mathrm{ele}}(v)$  has the same formula as  $\rho_{\mathrm{azi}}(u)$  by replacing  $u$  with  $v$ ,  $\mu_{u}$  with  $\mu_{v} = v_{b} + \frac{1}{2} h_{b}$ , and  $\sigma_{u}$  with  $\sigma_{v}$  in Equation 3. These vectors are concatenated into a visual feature input  $X_{\mathrm{v}} \in \mathbb{R}^{2 \times B \times L}$ .  $B$  is the maximum number of bounding boxes, and  $L$  is the length of the vectors. When no bounding box is detected, the visual feature input is set to an all-zero tensor. The right part of Figure 4 depicts the visual feature input. FCs are used as a visual embedding network,  $\mathcal{F}_{\mathrm{v},\theta_{\mathrm{v}}}(\cdot)$ , where  $\theta_{\mathrm{v}}$  is the network parameters:
$$
\boldsymbol {E} _ {\mathrm {v}} = \mathcal {F} _ {\mathrm {v}, \theta_ {\mathrm {v}}} \left(\boldsymbol {X} _ {\mathrm {v}}\right), \tag {4}
$$
where  $E_{\mathrm{v}} \in \mathbb{R}^{K_{\mathrm{v}}}$  is a  $K_{\mathrm{v}}$ -dimensional visual embedding vector.
Using the visual embedding vector, we finally extend the audio-only system into an audio-visual one. The visual embedding vector is repeated  $T'$  times to align the both embeddings in the time axis. Then the repeated visual embedding  $E_{\mathrm{v}}' \in \mathbb{R}^{K_{\mathrm{v}} \times T'}$  and the audio embedding are concatenated and fed into an audio-visual decoder network with parameter  $\theta_{\mathrm{av}}$ ,  $\mathcal{F}_{\mathrm{av}, \theta_{\mathrm{av}}}(\cdot)$ , to get multi-ACCDOA output:
$$
\hat {\boldsymbol {P}} = \mathcal {F} _ {\mathrm {a v}, \theta_ {\mathrm {a v}}} (\boldsymbol {E} _ {\mathrm {a}}, \boldsymbol {E} _ {\mathrm {v}} ^ {\prime}). \tag {5}
$$
We used four joint localization and detection metrics [25, 38], which are widely-used in audio-only SELD tasks [46, 39, 31]. Two metrics are referred to as location-aware detection and are error rate  $(ER_{20^{\circ}})$  and F-score  $(F_{20^{\circ}})$  in one-sec non-overlapping segments. We consider a prediction as a true positive if the prediction and reference class are the same and the angle difference is below  $20^{\circ}$ .  $F_{20^{\circ}}$  is calculated from location-aware precision and recall, whereas  $ER_{20^{\circ}}$  is the sum of insertion, deletion, and substitution errors, divided by the total number of the references. The other two metrics are referred to as class-aware localization and are localization error  $(LE_{CD})$  in degrees and localization recall  $(LR_{CD})$  in one-sec non-overlapping segments, where the subscript refers to classification-dependent. Unlike location-aware detection, we do not use any threshold but estimate the difference between the correct prediction and reference.  $LE_{CD}$  expresses the average angular difference between the same class's predictions and references.  $LR_{CD}$  tells the true positive rate of how many of these localization estimates were detected in a class out of the total number of class instances. We used the macro mode of computation while the mode does not apply  $ER_{20^{\circ}}$  because it includes substitution errors between two classes. We first computed the metrics for each class and then averaged them for the other three metrics to obtain the final system performance. See Appendix C.1 for further details.
4.3 Experimental Setting
As audio features, multichannel amplitude spectrograms and inter-channel phase differences (IPDs) are used [46]. Input features are segmented to have a fixed length of 1.27 sec. To reduce the calculation cost of video, we use  $360 \times 180$  videos converted from the released  $1920 \times 960$  videos. As visual input, we extract the corresponding video frame at the start of the audio features. We use a pretrained YOLOX object detection model<sup>7</sup> to get bounding boxes of person class. Other classes, e.g., cell phone or sink, are not stably detected in our preliminary experiments with STARSS23 videos. The bounding box results are encoded to vectors along azimuth and elevation as in Sec.4.1. The maximum number of bounding boxes is  $B = 6$ . The vector size is  $L = 37 (= 36 + 1)$  to cover 360 degrees of azimuth per 10 degrees. While we tested convolutional neural network (CNN)-based visual feature extraction methods [47, 20], they performed worse than the object detection method.
To get audio embedding, we stack three convolutional layers with kernel size  $3 \times 3$ . We embed the visual encoded vectors with two FCs. The lengths of both embedding vectors are  $K_{\mathrm{a}} = K_{\mathrm{v}} = 64$ . The concatenated embeddings are processed with a bidirectional GRU layer with a hidden state size of 256. The number of target classes was  $C = 13$ . The number of tracks in the multi-ACCDOA format was fixed at  $N = 3$  maximum simultaneous sources. The threshold for activity was 0.3 to binarize predictions during inference.
The experiments are for the two formats: FOA and MIC. We compare the audio-visual SELD system with an audio-only system based on the same data split and implementation. The difference is the
$^{7}$ https://github.com/open-mmlab/mmdetection/blob/master/configs/yolox/yolox_tiny_8x8_300e_coco.py
Table 3: SELD performance ( $C = 13$ ) in audio-visual and audio-only systems evaluated for the dev-set-test in STASS23.
Format	System	ER20° ↓	F20° ↑	LECD ↓	LRCD ↑
FOA	Audio-Visual	1.03 ± 0.03	13.2 ± 0.6 %	51.6 ± 5.6 °	34.2 ± 1.0 %
Audio-Only	0.97 ± 0.05	14.8 ± 0.3 %	55.9 ± 4.0 °	35.1 ± 2.8 %
MIC	Audio-Visual	1.08 ± 0.04	10.0 ± 0.8 %	60.0 ± 3.3 °	27.6 ± 1.7 %
Audio-Only	1.04 ± 0.02	10.7 ± 2.1 %	66.9 ± 6.8 °	29.6 ± 3.0 %
Figure 5: Location-aware F-score per sound event class ( $C = 13$ ) in audio-visual and audio-only systems evaluated for the dev-set-test in STASS23.
presence or absence of video input. Because the visual input is bounding boxes of person class, we especially focus on five classes related to the human body, i.e., female and male speeches, clapping, laughing, and footsteps. To further investigate the effect of the visual input on the body-related classes, we add experiments where we set only the body-related classes as the target classes of the SELD systems. The systems are trained and evaluated with activities and DOAs of speeches, clap, laugh, and footsteps classes, i.e.,  $C = 5$ . The other settings are the same as the 13-class SELD experiments. Details on the experimental setting are in Appendix C.2. The code of training, inference, and evaluation is available in a GitHub repository<sup>8</sup> under the MIT license.
4.4 Experimental Results
Table 3 summarizes the 13-class SELD performance of the audio-visual and audio-only SELD systems in both audio formats. Compared with both formats, SELD systems with FOA format show better SELD performance than those with MIC format. In FOA format, while the audio-visual SELD system shows a slightly worse location-aware F-score, the audio-visual system exhibited better localization error with comparable localization recall. There is a similar trend of lower localization error in MIC format.
We investigate the location-aware F-score over 13 classes, considering both localization and detection aspects. Figure 5 shows the F-score per class in FOA format. We show the average score of the five body-related classes on the left of the figure. The audio-visual system demonstrates a higher location-aware F-score in the body-related. On the other hand, the audio-visual system performs worse in the average of the other classes, i.e., non-body-related. The results suggest that the visual input, i.e., bounding boxes of a person, contributes to localization and detection of body-related classes, whereas the visual input might limit the performance of non-body-related classes.
Table 4 shows the body-related classes only SELD performance in audio-visual and audio-only systems. The audio-visual SELD system outperforms the audio-only system in all metrics. The visual location information of people enables the audio-visual system to localize and detect body-related classes more accurately. Audio-only system in MIC format shows high standard deviation in localization error. It is because a few classes are sometimes set  $180^{\circ}$  as they had no true positive output. Even if we omit such cases, the audio-visual system still shows lower localization error.
8 https://github.com/sony/audio-visual-seld-dcase2023
Table 4: Body-related classes only SELD performance ( $C = 5$ ) in audio-visual and audio-only systems evaluated for the dev-set-test in STASS23.
Format	System	ER20° ↓	F20° ↑	LECD ↓	LRCD ↑
FOA	Audio-Visual	0.93 ± 0.08	22.4 ± 2.4 %	31.0 ± 3.6°	45.0 ± 5.0 %
Audio-Only	0.93 ± 0.02	18.0 ± 1.9 %	33.8 ± 0.8°	40.9 ± 5.5 %
MIC	Audio-Visual	0.95 ± 0.01	20.9 ± 1.9 %	28.9 ± 1.5°	37.4 ± 4.6 %
Audio-Only	0.97 ± 0.10	16.0 ± 3.7 %	58.4 ± 26.8°	30.9 ± 10.9 %
Our experimental results show that visual location information of people improves SELD performance in human body-related classes. If visual location information of people is provided, we should expect improvements in the human body-related classes, while we should not expect improvements in the rest. The results suggest that if visual information matches the target classes of an audio-visual SELD system, the audio-visual system performs better.
5 Discussion on Limitations and Future Work
Like any visual machine learning system, deploying audio-visual SELD systems might lead to privacy issues, e.g., capturing raw visual information, if the consent of the people being recorded is not considered. We also recognize that improved SELD capability could empower surveillance technology without appropriate regulation.
As shown in Table 2, STARSS23 is imbalanced and long-tailed. When considering it as an evaluation set, the dataset would be appropriate as it can reflect real soundscape, which is usually imbalanced and long-tailed. On the other hand, if we use it as a training set, the imbalance and long-tail make it difficult for a model to learn about rare categories. We can use machine learning algorithms [22, 7] or data sampling strategies [9] to tackle the data imbalance problem.
The amount of STARSS23 is relatively small as training data. We can use a multichannel audio-visual simulation platform, e.g., SoundSpaces 2.0 [11], for geometry-based audio rendering in 3D environments. Also, we can use multichannel audio-only data such as TAU-NIGENS Spatial Sound Events (TNSSE) datasets [2, 36, 35] to train an audio branch separately. Unlabeled multichannel audio-visual data, e.g., YouTube-360 [27], could help the pre-training for audio-visual SELD tasks.
There is room for developing visual features and model structure to improve the audio-visual SELD performance. While the visual location information of people improves the performance in the body-related classes, we need to develop practical visual features for the non-body-related classes. A more sophisticated model structure than the current concatenation could boost the performance. A recent work made the audio-visual SELD system better than the audio-only one in STARSS23, using decision-level audio-visual fusion with keypoints, e.g., feet or telephones [52].
While we apply STARSS23 for audio-visual SELD tasks, STARSS23 would be applied to other tasks, e.g., audio-visual sound source localization [44, 12, 61, 4, 33] or audio-visual cross-modal retrieval tasks [57]. Also, when large multi-modal models [18, 49] are extended to multichannel audio, STARSS23 is an appropriate audio-visual soundscape evaluation data for the models because STARSS23 has unique spatial annotations and multichannel audio data beyond stereo.
This paper attempts to broaden sound event localization and detection (SELD) to an audio-visual area by introducing an audio-visual SELD task. We present an audio-visual dataset, Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23), which consists of multichannel audio data, video data, and spatiotemporal annotation of sound events in natural sound scenes. Furthermore, we present quantitative evaluations for audio-visual SELD systems compared with audio-only systems and demonstrate the benefits of visual object positions. We still need to improve the SELD performance of various sound events using audio-visual data. Also, STARSS23 opens a wide range of future research on spatial audio-visual tasks, taking advantage of the well-organized audio-visual recording and detailed labels about spatial sound events.
We thank Akira Takahashi for his helpful code review and thank Atsuo Hiroe, Kazuya Tateishi, Masato Hirano, Takashi Shibuya, Yuji Maeda, and Zhi Zhong for valuable discussions about the data construction process.
The data collection and annotation at Tampere University have been funded by Google. This work was carried out with the support of the Centre for Immersive Visual Technologies (CIVIT) research infrastructure at Tampere University, Finland.
We show appendixes for data construction, data analysis, experiment, competition, social impact, and personal data handling. Finally, we answer the questions from Datasheets for Datasets [16].
A Data Construction
A.1 Sound Scene Recording
The generic instructions for recording participants include information about duration, participants, props, active classes, and description of sound scenes. The recording duration can be shorter or longer as they are rough instructions. We set the basic structure of sound scenes, e.g., people gathering on a sofa and discussing the weekend. Following the rough instructions, we leave the other details of sound scenes to the participants, e.g., they can walk wherever they want and talk about whatever they want without a fixed dialogue text. We show an example of the instructions:
Duration 3 min
Participants 3 people
Props Playing cards, mobile phone
Active classes Speech, laugh, mobile phone
- 3 people gather on the sofa and talk about the weekend.
- Propose to play a game of cards, shuffle, and distribute the cards.
- Mobile phone rings while playing, attend the call, walk around and talk for a few seconds, and laugh during the call.
- Come back, sit, and continue playing.
Sound scenes consist of target sound events, directional interference sounds, and background noise. Each target class contains diverse sounds, e.g., the speech classes include a few different languages, and the phone class has sounds from different mobile phones. In addition, several target classes correspond to super classes with some subclasses in the AudioSet ontology [17], e.g., the domestic sounds class contains the vacuum cleaner, mechanical fan, and boiling subclasses. We provide the subset of sounds encountered in the recordings for the target classes in the form of more specific AudioSet-related labels. The subset information is summarized in Table 5, which is a re-post of a table in the STARSS22 paper [39]. Directional interference sounds are derived from computer keyboards, shuffling cards, dishes, pots, and pans; however, they are not annotated. Natural background noise is mainly related to HVAC (heating, ventilation, and air-conditioning), ranging from low to high levels.
Spatial and temporal annotations relied on carefully mounting tracking markers and wireless microphones [39]. The tracking markers were mounted on independent sound sources such as mobile phones or hoovers. Head markers were also provided to the participants as headbands or hats. The head tracking results were reference points for all human body-related sound classes. Mouth position for speech and laughter classes, feet stepping position for footstep class, and hand position for clapping class were each approximated with a fixed translation from the head tracking result. Regarding clapping, participants were instructed to clap about  $20~\mathrm{cm}$  in front of their faces, while footsteps were projected from the head coordinates to floor level. Hence, the mounting positions were considered in the annotation process to translate the tracking data of each class into each sound source position. The wireless microphones were mounted to the lapel of each participant and to additional independent sound sources being far from the participants.
A.2 Annotation
At a certain time, the motion capture (mocap) system could not track the attached marker, e.g., when markers were out of the view of the mocap coverage, markers were moving too fast, or obstacles occluded markers. Whenever such misses were short, the tracking results were interpolated with Motive<sup>9</sup>. If such misses were long and interpolating the results was not possible, azimuth and
<sup>9</sup>https://optitrack.com/software/motive/
Table 5: Relation of target classes to specific AudioSet classes. Target classes not included in the table have a one-to-one relationship with the similarly named AudioSet ones.
Target Class	Related AudioSet subclasses
Telephone	Telephone bell ringing, Ringtone (no musical ringtones)
Domestic sounds	Vacuum cleaner, Mechanical fan, Boiling (produced by hoover, air circulator, water boiler)
Door, open or close	Combination of Door & Cupboard, open or close
Music	Background music & Pop music, (played by a loudspeaker in the room)
Musical instrument	Acoustic guitar, Marimba, Xylophone, Cowbell, Piano, Rattle (instrument)
Bell	Combination of sounds from hotel bell and glass bell, closer to Bicycle bell & single Chime
elevation of sound sources were annotated based on the  $360^{\circ}$  video data. For example, direction of arrival (DOA) of the door class was often annotated from the video data because many doors were outside of the view of the mocap. To calculate source distance in such cases, we also used information on room dimensions and installation positions from our recording log in addition to the video data. Any interpolated spatial annotations were visually checked in the confirmation videos.
A.3 Data Format
We use WAV file format for the audio data and MP4 file format for the video data. The metadata are tabulated and served in CSV file format. The sound event classes, DOAs, and distances are provided in the following format:
- frame number, active class index, source number index, azimuth, elevation, distance
with all labels served as integers. Frame, class, and source enumeration begin at 0. Frames correspond to a temporal resolution of 100 msec. Azimuth and elevation angles are given in degrees, rounded to the closest integer value, with azimuth and elevation being zero at the front, azimuth  $\phi \in [-180^{\circ}, 180^{\circ}]$ , and elevation  $\theta \in [-90^{\circ}, 90^{\circ}]$ . The azimuth angle increases counter-clockwise ( $\phi = 90^{\circ}$  at the left). Distances are provided in centimeters, also rounded to the closest integer value. The source index is a unique integer for each source in the scene. Note that each unique participant gets assigned one identifier, but not individual events produced by the same participant; e.g., a clapping event and a laughter event produced by the same person have the same identifier. Independent sources that are not participants (e.g., a loudspeaker playing music in the room) get a 0 identifier. Note that the source index and the source distance are only included as additional information that can be exploited during training. An example line could be as follows:
10, 1, 1, -50, 30, 181
which describes that in frame 10, an event of class male speech (class 1) belonging to one participant (source 1) is active at location  $(-50^{\circ}, 30^{\circ}, 181\mathrm{cm})$ .
B Data Analysis
B.1 Duration
Figures 6 illustrates the box plots of duration, i.e., how long a sound event lasts. The figure shows that there are classes with similar duration trends. Speech classes have a wide range of plots, whereas laughing has a shorter duration than speech. While phone and bell have similar medians, the box of phone is longer as phone calls repeat the recorded sound until it is answered. There are several classes with longer duration, e.g., domestic sounds, music, musical instruments, and faucet, whereas collision or tapping sounds such as door and knock are relatively short.
B.2 DOA
Figure 7 shows the distribution of DOAs of the rest of the classes not depicted in Figure 3. While DOAs of human-produced classes such as speech, laugh, clap, or footsteps are dispersed across the  $360^{\circ}$  plane, classes such as door, knock, or faucet result in specific discrete points in the plot due to their fixed position in the room. Music and bell classes also show similar trends to door class
Figure 6: Box plots of the event durations across classes. The vertical axis indicates frame numbers in a log scale. The center line of a box shows the median. The top and bottom lines show the first and third quartile, while the whiskers extend from the box by 1.5 times the interquartile range. Each plot represents a data point of duration.
as their sound sources are rarely moved. Domestic sounds, phone, and musical instrument classes show similar trends to the speech classes as the respective sources are sometimes still and other times moving.
C Experiment
C.1 Evaluation Metric
We describe details on evaluation metrics [25, 38]. The metrics are based on true positives,  $TP$ , and false positives,  $FP$ , determined not only by correct or wrong detections but also based on whether they are closer or further than a distance threshold ( $20^{\circ}$  in this paper) from the reference.
We associate  $P_{c}$  predicted events of class  $c$  with  $R_{c}$  reference events of class  $c$  for each class  $c \in [1, \dots, C]$  and each segment. We count false negatives for misses:  $FN_{c} = \max(0, R_{c} - P_{c})$ , and count false positives for extraneous predictions:  $FP_{c,\infty} = \max(0, P_{c} - R_{c})$ .  $K_{c}$  predictions are spatially associated with references based on the Hungarian algorithm:  $K_{c} = \min(P_{c}, R_{c})$ . The number can also be considered as the unthresholded true positives:  $TP_{c} = K_{c}$ . We apply the spatial threshold to move  $L_{c} (\leq K_{c})$  predictions with a value greater than the threshold to false positives:  $FP_{c, \geq 20^{\circ}} = L_{c}$ , and  $FP_{c} = FP_{c,\infty} + FP_{c, \geq 20^{\circ}}$ . We count the remaining matched estimates per class as true positives:  $TP_{c, \leq 20^{\circ}} = K_{c} - FP_{c, \geq 20^{\circ}}$ .