SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SOUNDCAM: A Dataset for Finding Humans Using Room Acoustics
Mason Wang\* Samuel Clarke\*1 Jui-Hsien Wang2 Ruohan Gao' Jiajun Wu
$^{1}$ Stanford University  $^{2}$ Adobe Research
A room's acoustic properties are a product of the room's geometry, the objects within the room, and their specific positions. A room's acoustic properties can be characterized by its impulse response (RIR) between a source and listener location, or roughly inferred from recordings of natural signals present in the room. Variations in the positions of objects in a room can effect measurable changes in the room's acoustic properties, as characterized by the RIR. Existing datasets of RIRs either do not systematically vary positions of objects in an environment, or they consist of only simulated RIRs. We present SOUNDCAM, the largest dataset of unique RIRs from in-the-wild rooms publicly released to date. $^{1}$  It includes 5,000 10-channel real-world measurements of room impulse responses and 2,000 10-channel recordings of music in three different rooms, including a controlled acoustic lab, an in-the-wild living room, and a conference room, with different humans in positions throughout each room. We show that these measurements can be used for interesting tasks, such as detecting and identifying humans, and tracking their positions.
The physical sound field of a room, or the transmission and reflection of sound between any two points within the room, is influenced by many factors, including the geometry of the room as well as the shape, position, and surface material of each object within the room [Kuster et al., 2004, Bistafa and Bradley, 2000, Cucharero et al., 2019]. For this reason, each room has a distinct sound field, and a change in a position of a particular object in the room will generally change this sound field in distinct ways [Mei and Mertins, 2010, Gotoz et al., 2021]. We present SOUNDCAM, a novel dataset to investigate whether this principle can be used for three distinct tasks for identifying humans and tracking their positions in diverse conditions, including in both controlled and in-the-wild rooms.
A room's sound field between a given source and listener location pair can be characterized by a room impulse response (RIR) between that pair. While there are many datasets of RIRs recorded in diverse environments, existing datasets mainly focus on densely characterizing the sound field of rooms by varying the locations of the sources and/or listeners in the environment. There are datasets which focus on isolating the effects of changing objects or their positions in the room, but they provide RIRs generated purely in simulated environments. Our SOUNDCAM dataset is the largest dataset of unique, real RIRs from in-the-wild rooms publicly released to date and specifically focuses on isolating the acoustic effects of changing the positions and identities of humans.
The ability to identify and/or track the position of a human in an indoor environment has many applications to ensure safe and high-quality user experiences for interactive applications such as
*indicates equal contribution
<sup>1</sup>The project page and dataset are available at https://masonlwang.com/soundcam/, with data released to the public under the MIT license.
Figure 1: Spectrograms visualizing the RIRs from the Treated Room (left column) and real Living Room (right column), either empty (top row) or with humans standing near the loudspeaker sound source (bottom row). The RIRs within each column are from the same speaker and microphone position. While the human's obstructing the direct path noticeably attenuates the intensity and duration of the RIR as measured by the microphone in each room, the Living Room has much stronger indirect paths for the sound to reach the microphone through reflections and thus shows less obvious effects.
virtual/augmented reality and smart home assistants. While many prior works have successfully tracked humans and objects using vision signals, we instead focus on identifying and tracking targets using audio signals. Sound waves undergo diffraction, making it possible to curve around obstacles (e.g., you can hear a speaker behind a sofa playing music even when you cannot see it). Audio signals also occupy a broad frequency spectrum, making them an excellent probe for the many surfaces in our everyday environment that demonstrate frequency-dependent reflection and absorption properties. For these reasons, audio signals can provide information about an environment's state which complements that provided by visual signals, to make tracking and detection more robust to edge cases such as with transparent objects [Singh and Nagla, 2019] or occlusion [Lindell et al., 2019]. Using high-resolution vision signals can also create privacy concerns in certain use cases such as in healthcare settings [Zhang et al., 2012, Chou et al., 2018], where acoustic features could provide more privacy. Conversely, using audio signals to track and identify humans may also have applications in covert surveillance, and we claim that this makes releasing a public dataset and benchmark to the academic community essential to mitigating the risk of malicious use on an unwitting party.
We thus collect a large dataset of 5,000 10-channel RIRs from human subjects standing in varied positions across a total of three controlled and in-the-wild rooms. Each room contains data on between 2 and 5 individuals, allowing users of SOUNDCAM to check how well their methods for tracking humans generalize to unseen humans, and to develop methods for identifying humans. In real-world applications, precisely measuring RIRs using non-intrusive signals in in-the-wild rooms may be difficult. However, natural sounds are often present in real rooms, from the noise of a television, a speaker playing music, a baby crying, or a neighbor hammering nails. These natural sounds provide additional clues about the presence, identities, locations, and activities of people inside the room, even those who are not making noise. In this paper, we focus on actively emitted natural sounds, so SOUNDCAM also includes 2,000 10-channel recordings of music with the same annotations as the RIR measurements, in a controlled and a real-world room. This data can be used for developing methods that use more general source signals as input.
Our results show that SOUNDCAM can be used to train learning-based methods to estimate a human's location to within  $30~\mathrm{cm}$  of error, using room impulse responses measured by multiple microphones. However, these methods perform worse when using RIRs measured from a single microphone, when using natural music as a source signal and when the layout of the room changes. When the source music is unknown to the model and even all 10 microphones are used, our best baseline model is able to detect the presence of a human in a room with only  $67\%$  accuracy using music. Our best baseline model for identification is able to correctly classify the human from a group of five  $82\%$  of the time, when using all 10 microphones. Since placing 10 microphones in a room is less realistic for real-world applications, our results also indicate that abating the number of microphones to 4, 2, or 1 generally reduces performance across tasks.
Estimating environment geometry with sound. Numerous works have shown how audio can be used to precisely measure salient aspects of room geometry, such as the positions of walls and furniture. Many analytical approaches require no training data in order to estimate the positions of planar walls in an environment [Dokmanić et al., 2013, Antonacci et al., 2012b]. In addition, several works use room impulse responses (RIRs) to localize inanimate acoustic reflectors [Antonacci et al., 2012a, Aprea et al., 2009, Tervo et al., 2012]. While these methods rely on some strong assumptions of the room geometry, only claiming to map out the geometry of large planar surfaces such as walls, learning-based approaches learn to estimate environment geometries without such assumptions. Purushwalkam et al. [2021] combined camera and microphone signals gathered by a robot exploring a simulated environment, using both actively emitted and passively received audio as inputs to a learning-based framework, which estimates entire floor plans from a video sequence of only a region of the floor plan. Gao et al. [2020] showed how an agent's binaural recordings of actively emitted chirps can be used to estimate depth maps of the agent's views within a simulated environment. Christensen et al. [2020] showed similar results in real environments. Both approaches learn from datasets of chirp recordings and their corresponding ground truth depth images, learning to estimate the depth of objects within the agent's line of sight. We show how acoustic information can even be used to estimate changes in the position of a human which is occluded from the line of sight of both the sound source and the microphone.
Tracking objects and humans with multimodal signals. Both passively and actively emitted sounds can be used to track and identify both objects and humans. Passive methods using sounds emitted by objects and humans while they are in motion have been used to track aircraft [Lo and Ferguson, 1999], motor vehicles [Gan et al., 2019], and human speakers [Crocco et al., 2017]. Rather than passively recording the sound emitted by the target, we record the sounds actively emitted from a speaker while a human remains silent, making active methods more relevant to our work. Yang et al. [2022] used the times of arrival (TOA) of chirping signals, emitted by specialty ultrasonic speakers, jointly with camera signals to estimate the metric scale joint pose of human bodies in household environments. They assume the bodies are within the lines of sight of both the camera and the speakers and microphones in their setup. Lindell et al. [2019] relaxed the assumption that an object must be within the acoustic line of sight by using a moving array of both microphones and co-located loudspeakers in a controlled acoustically-treated environment, emitting and recording the sounds which reflect against a wall to estimate the shape of an object behind another wall which obstructs the line of sight to the object. Our dataset includes recordings from both controlled and in-the-wild environments, including in contexts when the human is occluded from the microphone and/or speaker's line of sight. Other signals, such as radio [Adib et al., 2015, Zhao et al., 2018] and WiFi [Adib and Katabi, 2013] can be used to track human motion when the line of sight is obstructed, but these methods and that of Lindell et al. [2019] require specialized hardware setups which are not common in homes. We include recordings of both sine sweeps and popular music, played and recorded by inexpensive commodity hardware similar to what is already contained within many homes due to home assistant devices or media entertainment systems.
Large acoustic datasets. Many works have released large datasets of measurements of the acoustic properties of objects and environments. The Geometric-Wave Acoustic Dataset [Tang et al., 2022] consists of 2 million synthetic room impulses generated with a wave solver-based simulation of 18,900 different scenes within virtual house layouts. The authors demonstrate how the dataset can be used for data augmentation in tasks such as automated speech recognition (ASR) or source separation. Other datasets have included measurements of real room impulse responses. The voiceHome corpus [Bertin et al., 2016] features 188 8-channel impulse responses from 12 different rooms in 3 real homes, and the dEchorate dataset [Carlo et al., 2021] comprises nearly 2,000 recordings of impulse responses of the same acoustically treated environment in 11 different configurations with varying reverberation characteristics. All our measurements are real recordings, recorded in both acoustically treated and in-the-wild environments. More importantly, our dataset measures real impulse responses of rooms with humans in different annotated positions, whereas these prior works only measure rooms with inanimate objects.
Figure 2: Images and visualizations from the real living room. (Left) A photo of the room. (Middle) An aerial view of a 3D scan of the room. (Right) A visualization of the microphone, speaker, and human positions in our dataset. See Appendix A for visualizations of the other rooms.
3 The SOUNDCAM Dataset
We collect large datasets of both sine sweeps and music clips recorded by microphones in different rooms. While we include recordings of each room while it is empty, each recording for which a human is present is paired with calibrated depth images from multiple camera angles to annotate the position of the human standing in the room while preserving subject anonymity. In total, SOUNDCAM contains multichannel sine sweep recordings from humans standing in 5,000 unique positions across three different rooms, including at least 200 RIRs from each of 5 unique humans, with an additional 2,000 recordings from humans standing in unique positions with natural music being played in a room, as summarized in Table 1. SOUNDCAM includes the largest dataset of unique RIRs from in-the-wild rooms publicly released to date. Next, we introduce our audio setup, rooms where the recording is conducted, how the human pose data is collected, and our postprocessing steps.
Audio. We place a QSC K8.2 2kW Active Loudspeaker in a corner of each room, then place 10 Dayton Audio EMM6 microphones at different positions along the periphery. The 3D position of the speaker and each microphone is annotated, as shown in Figure 2. Maps of the microphone array layouts for additional rooms are included in Appendix A. The loudspeaker and microphones are connected to the same synchronized pair of MOTU 8M audio interfaces, which simultaneously play and record a signal from the loudspeaker and microphones respectively, in a time-synchronized fashion at  $48\mathrm{kHz}$ . Depending on the experiment, we play and record a different signal. For experiments requiring precise RIRs, we use a 10-second  $20\mathrm{Hz}-24\mathrm{kHz}$  sine sweep. For experiments that test the generalization of methods to natural signals, we select a 10-second clip from the Free Music Archive [Defferrard et al., 2016], balancing the main eight genres by sampling uniformly randomly within each one. For both the sine sweeps and the music sweeps, we also record the four seconds of silence after each signal is played, to allow for the decay of the recording to be captured. The longest RT60 Reverberation Time of any of our rooms is 1.5 seconds, so the final two seconds of the recording we presume to be complete silence except for environmental and human noises.
Rooms. We collect separate datasets from three rooms with diverse geometry, acoustic properties, and function. The first is an acoustically treated room without direct forced ventilation. Within the room, we collect subsets of data in each of two different static configurations: 1) with the room empty, except for recording equipment, and 2) with four fabric divider panels placed in the room, in order to test for robustness to occlusions and changes in static elements in the room. The second room is a living room in a real house (shown in Figure 2), and the third is an untreated conference room with standard furniture. We collect a textured 3D scan of each room. Additional images and floor plans of each room are included in Appendix A.
Humans and pose. Our dataset includes recordings of sine sweeps and music from different humans, with the number of unique humans varying by room. We collect a 3D scan of each human, with texture and fine details omitted to preserve anonymity. We record some sine sweeps and/or music from each room while it is empty, as well as recordings where at most one human is present in each recording. For each recording with a human, the human selects a new position in the room, based on following a pattern that optimizes for even coverage of the reachable space. The human then stands straight with their arms at their sides, facing the same direction for each recording. We
RIR (Sine Sweep)	Music
Humans × Recordings	Empty	Humans × Recordings	Empty
Treated Room	1×1000 + 4×200	1000	1×1000	1000
Treated Room w/ Panels	1×1000	100	0	0
Living Room	1×1000 + 1×100	100	1×1000 + 1×100	0
Conference Room	1×1000 + 1×100	100	0	0
Table 1: Number of 10-channel recordings by room, unique human, and signal type.
Treated Room	Number of Microphones
10	4	2	1
kNN on Levels	54.4 (66.9)	81.1 (57.9)	104.3 (63.5)	135.1 (57.4)
Linear Regression on Levels	112.1 (58.8)	130.6 (48.5)	133.4 (47.5)	133.8 (47.3)
VGGish (pretrained)	93.6 (53.1)	106.5 (60.2)	97.8 (55.9)	95.8 (55.9)
VGGish (multichannel)	18.1 (13.1)	17.2 (16.2)	20.4 (15.4)	71.6 (50.4)
Time of Arrival	148.1 (106.2)	133.0 (103.0)	244.0 (117.8)	307.6 (102.7)
Treated Room w/ Panels
kNN on Levels	30.1 (29.0)	61.5 (55.1)	98.7 (72.5)	134.6 (58.0)
Linear Regression on Levels	105.7 (39.0)	117.9 (56.7)	136.3 (60.0)	138.2 (51.2)
VGGish (pretrained)	92.5 (60.5)	79.9 (52.6)	82.1 (53.5)	104.8 (62.2)
VGGish (multichannel)	13.1 (11.2)	20.7 (19.3)	19.6 (18.2)	45.7 (42.2)
Time of Arrival	163.7 (118.0)	1528 (119.4)	240.7 (121.1)	314.3 (102.0)
Living Room
kNN on Levels	61.6 (59.6)	93.2 (77.1)	123.0 (80.4)	157.3 (60.0)
Linear Regression on Levels	125.0 (62.7)	154.5 (54.7)	165.3 (54.2)	168.4 (53.5)
VGGish (pretrained)	142.2 (84.9)	147.0 (86.9)	151.0 (84.4)	151.9 (91.3)
VGGish (multichannel)	27.9 (22.0)	23.6 (15.0)	26.3 (21.3)	42.0 (30.3)
Time of Arrival	229.9 (150.9)	222.5 (129.7)	244.7 (128.5)	308.7 (133.0)
Table 2: Localization error of each model using sine sweep-based RIRs from varying numbers of microphones in different environments. Errors are in centimeters and "mean (stdev)" format.
position three Azure Kinect DK RGBD cameras in three corners of each room and capture RGBD images from each camera immediately before recording each sine sweep. To preserve the anonymity of our subjects, we do not release RGB images of them, and instead only release depth images and joint positions.
Postprocessing. In order to estimate a room impulse response (RIR) from our recordings of sine sweeps, we deconvolve the signal audio file, either sine sweep or music, from each of the microphone recordings. For annotating human pose, we use the Azure Kinect's 3D body tracking API to extract an estimate of the skeleton joint positions in the 3D coordinate frame of each camera. We then calibrate each camera's coordinate frame to the same room frame using an Aruco marker hanging in the middle of each room. We label each recording with the median of each camera's estimate of the position of the pelvis in the room coordinate frame, excluding cameras that could not successfully capture the body. See Figure 2 for a visualization of the distribution of human position annotations in the Living Room, with additional rooms shown in Appendix A. Each example's input consists of the individual simultaneous recordings from each microphone combined together in the same ordering, while the label is the estimated x-y position of the pelvis in the room coordinate frame. We release both raw recordings and postprocessed features for all our data. The number of recordings we have for each room, configuration, and unique human is summarized in Table 1.
Our dataset can be used for multiple interesting tasks in learning from acoustics, including localizing (Sec. 4.1), identifying (Sec. 4.2), and detecting (Sec. 4.3) humans. We describe the formulation, baselines, and results for each task.
Treated Room	Number of Microphones
10	4	2	1
kNN on Levels	133.5 (51.0)	133.5 (48.4)	135.8 (51.9)	133.3 (49.2)
Linear Regression on Levels	133.7 (46.9)	133.9 (48.8)	133.8 (47.4)	133.7 (47.3)
VGGish (pre-trained)	164.8 (79.0)	148.0 (82.0)	147.5 (79.4)	155.8 (79.4)
VGGish (multichannel)	31.4 (23.9)	44.6 (34.0)	48.6 (39.2)	106.3 (79.6)
Time of Arrival	232.4 (95.7)	232.2 (96.9)	219.8 (95.9)	232.0 (97.7)
Living Room
kNN on Levels	167.2 (56.5)	170.0 (59.8)	169.3 (65.7)	168.7 (54.3)
Linear Regression on Levels	125.0 (62.7)	154.5 (54.7)	165.3 (54.2)	168.4 (53.5)
VGGish (pre-trained)	186.6 (89.4)	191.4 (99.9)	199.7 (96.8)	189.9 (100.9)
VGGish (multichannel)	25.6 (18.4)	40.3 (38.6)	43.1 (41.6)	82.2 (53.8)
Time of Arrival	297.0 (133.0)	298.1 (131.0)	281.5 (127.1)	301.7 (132.6)
Table 3: Localization error of each model using music-based RIRs from varying numbers of microphones in different environments. Errors are in centimeters and "mean (stdev)" format.
4.1 Human Localization
We investigate whether different learning-based and analytical models can localize a human's position in a room from different sounds, including the RIR from a sine sweep, the RIR estimated from music signals, or a recording when the speaker is silent. In each case, we truncate to the first two seconds of each signal, as we found that the RIR had decayed by at least 60 decibels in each room within at most  $\sim 1.5$  seconds. Note that this task differs from prior works which either assume the tracked object is not silent and is therefore a sound source [Gan et al., 2019, Crocco et al., 2017, Lo and Ferguson, 1999], assume that the sound source and receiver(s) are roughly collocated [Christensen et al., 2020, Gao et al., 2020, Lindell et al., 2019], or fuse estimates from both vision and ultrasonic audio [Yang et al., 2022].
Baselines We test both learning-based and analytical baselines for this task, as detailed below.
- kNN on Levels: We use k-Nearest Neighbors (kNN) based on the overall RMS levels from each microphone in the input, summarizing each signal by a scalar per microphone.
- Linear Regression on Levels: This is another simple learning-based baseline similar to kNN on levels except that we use Linear Regression models on the same summarized inputs.
- VGGish (pre-trained): We use the popular VGGish architecture [Hershey et al., 2017] with weights pre-trained on the AudioSet classification task [Gemmke et al., 2017]. This pretrained model requires reducing signals into a single channel by averaging and downsampling signals to a sample rate of  $16\mathrm{kHz}$  to use as input. We use the outputs of the final hidden layer as inputs to three additional layers, with a final linear layer estimating normalized coordinates.
- VGGish (multichannel): This baseline is the same as the previous one except that we use a VGGish model that preserves the channels at full  $48\mathrm{kHz}$  sample rate and shares the same weights for each channel. We combine the hidden features across all channels with three additional layers similar to the pre-trained version and train this entire model from scratch.
- Time of Arrival: We use an analytical method based on time of arrival (TOA), similar to that described in [Yang et al., 2022]. For each input RIR, we subtract an RIR of the empty room which is the mean of multiple trials, then use the peaks in the absolute difference to find intersecting ellipsoids based on loudspeaker and microphone positions.
Additional details and hyperparameters of each baseline are included in Appendix B.
Results We first test each baseline on the RIRs derived from sine sweep recordings in each room. To test the influence of using multiple microphones, we also ablate the number of microphones/channels of each RIR from 10 to 4, 2, and 1. The resulting localization errors are shown in Table 2. For all our results, we define error as the average distance in centimeters between the ground truth location and the predicted location, across the test set. We perform the same experiments for each baseline on
Number of Microphones
10	4	2	1
kNN on Levels	163.6 (78.8)	166.7 (82.8)	164.2 (79.5)	164.9 (85.4)
Linear Regression on Levels	148.9 (59.6)	147.2 (59.7)	147.1 (59.8)	147.0 (59.9)
VGGish (pre-trained)	202.9 (96.9)	195.5 (94.8)	187.0 (96.7)	205.7 (100.9)
VGGish (multichannel)	50.7 (39.7)	70.5 (57.0)	67.1 (67.2)	118.9 (74.2)
Time of Arrival	304.5 (111.4)	301.2 (108.5)	282.8 (113.3)	305.2 (112.3)
Table 4: Localization error of models using music-derived RIRs in the Living Room, trained on data from one human and tested on data from another. Errors are in cm and "mean (stdev)" format.