SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on those, we form the location-aware F-score and error rate. We perform macro-averaging of the location-aware F-score:  $F_{20^{\circ}} = \sum_{c}F_{20^{\circ},c} / C$ . Additionally, we evaluate localization accuracy through a class-aware localization error  $LE_{CD}$ , computed as the mean angular error of the matched true positives per class, and then macro-averaged:  $LE_{CD,c} = \sum_{k}\theta_{k} / TP_{c}$  for each segment with  $TP_{c} > 0$ , and with  $\theta_{k}$  being the angular error between the  $k$ -th matched prediction and reference, and after averaging across all frames that have any true positives,  $LE_{CD} = \sum_{c}LE_{CD,c} / C$ . When there are no true positive outputs in a class, to compute the macro average, we set  $180^{\circ}$  as the localization error in the class. Complementary to the localization error, we compute a localization recall metric per class, also macro-averaged:  $LR_{CD,c} = TP_{c} / (TP_{c} + FN_{c})$ , and  $LR_{CD} = \sum_{c}LR_{CD,c} / C$ . The localization error and recall are not thresholded to provide more varied complementary information to the location-dependent F-score, presenting localization accuracy outside the spatial threshold. For a more thorough analysis of the joint SELD metrics, please refer to [25, 38].
C.2 Experimental Setting
We add a few details on experimental settings. The number of channels on audio features is  $M = 7$ , consisting of 4ch amplitude spectrograms and 3ch IPDs. We apply the short-term Fourier transform (STFT) to a 1.27-sec audio signal with a 20-ms frame length and 10-ms frame hop. The number of frequency bins is  $F = 257$ , and the length of time frames is  $T = 128$ . We keep the input feature length to 128 frames during inference and set the shift length to 120 frames.
For visual features, we use the pre-trained YOLOX object detection model $^{10}$ , which is trained with the COCO dataset [23]. While COCO has 80 object classes, we focus on person, cell phone, and sink classes because they are strongly related to the 13 sound events in STARSS23. Only person class is stably detected in our preliminary experiments in STARSS23 videos. Therefore, we use the model to get bounding boxes for the person class. We show an example of object detection results in Figure 8. While people are detected by the object detector, a sink is not detected by the detector.
Instead of the current object detection method, we also experimented with two convolutional neural network (CNN)-based visual feature extraction methods, i.e., VGG-16 [47] and ResNet-18 [20]. While training losses of the CNN-based visual features are lower than the object detection method, validation results were worse than the object detection method. Bounding boxes of person class have simple but enough information on human positions. That would lead to generalization on human body-related classes. The CNN-based visual features retain richer visual information than bounding boxes, so they may require more training data to generalize. Visual features extracted from Transformer-based methods, e.g., ViT models [13], would face the same issues as CNN-based methods.
We use a batch size of 16 and the Adam optimizer with a weight decay of  $10^{-6}$  to train the audiovisual sound event localization and detection (SELD) system. The learning rate is set to 0.001. We validate and save model weights at every 1,000 iterations up to 20,000 iterations. We select a model that demonstrated the best aggregated SELD error,  $\mathcal{E}_{\text{SELL}}$ , calculated as
$$
\mathcal {E} _ {S E L D} = \frac {E R _ {2 0 ^ {\circ}} + \left(1 - F _ {2 0 ^ {\circ}}\right) + \frac {L E _ {C D}}{1 8 0 ^ {\circ}} + \left(1 - L R _ {C D}\right)}{4}. \tag {6}
$$
We report the average scores and error bars of five experiments.
The model parameter size is  $0.8\mathrm{M}$ . The model size is kept intentionally small for easier trials. A single GPU (e.g., GeForce GTX 1080 Ti) is used for training. The training takes around six hours.
C.3 Experimental Results
In addition to the F-score per class in first-order ambisonics (FOA) format in Figure 5, we show the other SELD metrics per class in both FOA and tetrahedral microphone array (MIC) formats. As shown in Figure 9c, the F-score in MIC format shows a similar trend as in FOA format. While the audio-visual SELD system performs worse in the F-score of non-body-related classes, the audio-visual system demonstrates a higher F-score in body-related classes. Figure 9a and 9d show that the audio-visual system demonstrates lower localization error of body-related classes in both formats. There is no significant trend of the localization recall between the two formats.
D Competition
STARSS23 has served as the development set and evaluation set for the SELD Task of the DCASE 2023 Challenge $^{11}$ , which aims to accelerate audio-only and audio-visual SELD research. The task participants use the development audio/video recordings and labels to train and validate their SELD systems in the development process. The evaluation recordings without labels are used to produce system outputs for the challenge evaluation phase. If researchers wish to compare their system against the submissions of the challenge, they will have directly comparable results if they use the evaluation data as their testing set. Also, the implementation of the audio-visual SELD system described herein, trained and evaluated with STARSS23, has served as the baseline method for the audio-visual track of the challenge.
E Social Impact
The STARSS23 enables research on audio-only and audio-visual SELD tasks, which form the backbone of various real-world applications on acoustic and audio-visual monitoring, intelligent home applications, or audio-visual machine perception. The dataset, together with the associated
10. https://github.com/open-mmlab/mmdetection/blob/master/configs/yolox/yolox_tiny_8x8_300e_coco.py  
11. https://dcasecommunity/challenge2023/task-sound-event-localization-and-detection-evaluated-in-real-spatial-sound-scenes
challenge, accelerates such research for research institutes and industry since it is the first of its kind based on real annotated recordings. Multiple research institutes, university laboratories, and industrial research and development groups have already shown interest in the dataset and its use, either as part of the DCASE challenge or outside of it in independent published studies. We expect the dataset to set the standard in the upcoming years in audio-visual SELD-related studies due to its unique spatial annotations from real tracked people and sound sources and its spatial audio content, which is becoming more and more relevant with multiple monitoring or smart home devices employing microphone arrays. The dataset also offers opportunities for cross-regional evaluation, with its recordings coming from two different sites geographically far apart. Of course, as with many strongly annotated datasets, the diversity of sound events is limited and cannot capture the conditions of many real-world application-specific scenes. However, we believe that it is a valuable contribution to the development and maturation of such systems, at which point we expect more application-specific SELD datasets to appear.
F Personal Data Handling
STARSS23 data was recorded with over 50 voluntary participants. Before the recording, we explain our research purpose, how we record the sound scenes, and how we treat and release the recording data. Regarding the recording process, an example of the generic instructions is in Appendix A. Our explanation is based on text format and verbal description. Participants can ask us questions related to recording and public release. We also explain potential risks, i.e., recording data containing personally identifiable information, and our Institutional Review Board (IRB) approvals. The personally identifiable information is raw speech and blurred faces. The participants are also instructed not to reveal personal information during the recordings, and limit themselves to conversations of generic topics. After the explanation and Q&A, when participants understand the purpose and risk of recording and release, each participant signs the consent form.
G Data Sheet
For dataset documentation, we take the questions from Datasheets for Datasets [16] and answer them.
- Q: For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.
A: This dataset is created to tackle audio-visual sound event localization and detection (SELD) tasks. While visual data about sound source objects can support SELD tasks, e.g., feet are a potential source of footsteps, the existing datasets did not contain the complete multichannel audio, video, and annotation set. STARSS23 serves real sound scene recording with multichannel audio, video data aligned with the audio, and spatiotemporal annotation of sound events. STARSS23 allows the incorporation of audio-visual correspondence into multichannel audio signal processing methods.
- Q: Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
A: Creative AI Laboratory (CAL) at Sony Group and Audio Research Group (ARG) at Tampere University.
- Q: Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
A: The data collection and annotation at Tampere University has received funding by Google.
Q: Any other comments?
A: N/A.
- Q: What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
A: Real sound scene recordings with multichannel audio data, video data aligned with the audio data, and annotations of temporal activation, the direction of arrival (DOA), and distance of sound events.
- Q: How many instances are there in total (of each type, if appropriate)?
A: The development set totals about 7 hours and 22 minutes, of which 168 clips were recorded with 57 participants in 16 rooms with annotation. The evaluation set totals about 3.5 hours and 79 clips without annotation.
- Q: Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
A: It contains all possible instances.
- Q: What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide a description.
A: Raw audio and video data, but we convert 32ch  $48\mathrm{kHz}$  audio recordings to 4ch  $24\mathrm{kHz}$ , and  $3840 \times 1920360^{\circ}$  video recordings to  $1920 \times 960$  equirectangular. We also conduct face-blurring on video data to anonymize identical information.
- Q: Is there a label or target associated with each instance? If so, please provide a description.
A: The dataset contains temporal activation, DOA, and source distance labels of 13 target sound event classes. The classes are female speech, male speech, clapping, telephone, laughter, domestic sounds, footsteps, door, music, musical instrument, water tap, bell, and knock.
- Q: Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
A: The evaluation set has no annotation because the set is used in a testing phase of the SELD task of the DCASE 2023 challenge. While almost all audio recordings in the development and evaluation set are accompanied by synchronized video recordings, only 12 audio recordings in the development set are missing videos (from fold3_room21_mix001.wav to fold3_room21_mix012.wav).
- Q: Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
A: In each clip, we use the same tag, e.g., foldX_roomY_mixZ. We use the tag for audio (.wav), video (.mp4), and labels (.csv).
- Q: Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
A: Yes. STARSS23 has the dev-set-train part and the dev-set-test part for the development process.
- Q: Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.
A: In addition to target sound events, the sound scene recordings contain directional interference sounds and background noise. That makes the dataset a more realistic situation. We confirm all the labels by listening to the audio and watching the video. If there are any errors, they would be negligible.
- Q: Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
A: Self-contained.
- Q: Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description. A: No.
- Q: Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
- Q: Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.
A: STARSS23 set female and male speech as two target sound event classes. The frame coverages of both classes are almost equal:  $28.4\%$  and  $31.4\%$ , respectively.
- Q: Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
A: As an audio-visual sound scene recording dataset, STARSS23 contains raw speech data. However, faces in video data are blurred, and the talk contents have no personal topics. So, it is hard to identify individuals. We also explain to participants the potential risk of identical information before recording. After the explanation, participants sign a consent form for recording when they understand the potential risk.
- Q: Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.
- Q: How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If the data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
A: Each clip of a sound scene is recorded in a room with participants and sound props. Participants improvise a scene following a generic instruction about the sound scene and event.
- Q: What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? How were these mechanisms or procedures validated?
A: Multichannel audio and video data are recorded with a 32ch microphone array and  $360^{\circ}$  camera, respectively. A motion capture system and wireless microphones are also recorded
for annotation. The specific recording equipment is Eigenmike em32 $^{12}$ , Ricoh Theta V $^{13}$ , Optitrack Flex 13 $^{14}$ , and Røde Wireless Go II $^{15}$  respectively.
- Q: If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
- Q: Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
A: Voluntary participants act in a sound scene, and authors record the scene.
- Q: Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
A: The first round of recordings was collected between September 2021 and April 2022. A second round of recordings was collected between November 2022 and February 2023.
- Q: Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.
A: Yes. We get approvals from our Institutional Review Board (IRB). Following the discussion about personally identifiable information, we conduct face-blurring on video data and explain to participants about potential risks, i.e., recording data containing identifiable information.
- Q: Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
A: From the individuals.
- Q: Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.
A: Yes. Before the recording, we explain our research purpose, how we record sound scenes, and how we treat and release the recording data. Our explanation is based on text format and verbal description. Participants can ask us questions related to recording and release.
- Q: Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
A: Yes. After our explanation and Q&A, when participants understand the purpose and risk of recording and release, each participant signs the consent form.
- Q: If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).
A: Yes. In such a case, they can contact authors.
- Q: Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.
12. https://mhacoustics.com/products#eigenmike1
13 https://theta360.com/en/about/theta/v.html
14 https://optitrack.com/c cameras/flex-13/
15https://rode.com/en/microphones/wireless/wirelessgoii
G.4 Preprocessing/Cleaning/Labeling
- Q: Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remaining questions in this section.
A: We convert audio and video data to the appropriate size. We also annotate temporal activation and DOA of sound events. After annotating, we confirm the labels by listening to the audio and watching the video.
- Q: Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
A: We saved "raw" (large size) audio and video data for future use. If one is interested in them, one can contact the authors.
- Q: Is the software that was used to preprocess/clean/label the data available? If so, please provide a link or other access point.
A: We use Python code for audio data conversion, and RICOH THETA $^{16}$  and FFmpeg $^{17}$  for video data conversion. To annotate temporal activation, Audacity $^{18}$  and REAPER $^{19}$  are used. Tracking results are used in Motive $^{20}$ .
- Q: Any other comments?
- Q: Has the dataset been used for any tasks already? If so, please provide a description.
A: No, the dataset has not yet been used for any scientific papers. This paper is the first to use the dataset.
- Q: Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.
A: Yes. We have the code repository of the SELD system: https://github.com/sony/ audio-visual-seld-dcase2023.
- Q: What (other) tasks could the dataset be used for?
A: Apart from audio-visual SELD tasks, one could use the dataset for audio-only SELD tasks, audio-visual speaker DOA estimation tasks, audio-visual sound source localization tasks, and audio-visual cross-modal retrieval tasks. Audio-visual sound source distance estimation could be another task.
- Q: Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other risks or harms (e.g., legal risks, financial harms)? If so, please provide a description. Is there anything a dataset consumer could do to mitigate these risks or harms?
- Q: Are there tasks for which the dataset should not be used? If so, please provide a description.
$^{16}$ https://theta360.com/en/about/application/pc.html
17 https://ffmpeg.org/
18. https://www.audacityteam.org/
19 https://www.reaper.fm/
20 https://optitrack.com/software/motive/
- Q: Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.
A: Yes. STARSS23 is publicly available at https://zenodo.org/record/7880637.
- Q: How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?
A: STARSS23 is distributed via https://zenodo.org/record/7880637. The DOI is "10.5281/zenodo.7709051".
- Q: When will the dataset be distributed?
A: STARSS23 is already distributed.
- Q: Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.