SlowGuess/NeurIPS_2023_OCR huggingface dataset:
RIRs derived from the music subset we have collected in the Treated Room and the Living Room, with resulting localization errors shown in Table 3. In both cases, the analytical method performs quite poorly, but performs especially poorly on the music-derived RIR. Deconvolving a natural music signal to derive the RIR is rather ill-posed versus deconvolving a monotonic sine sweep, so there are more likely to be spurious peaks in the RIR in the time domain, which is used by the analytical method. Further work can develop methods to estimate room acoustics given natural music signals in a way that is useful for the localization task. The poor performance of linear regression shows that the function relating the volume levels of the recording to the location is non-linear. The kNN baseline can better characterize a nonlinear landscape and thus performs better. When using more than two microphones, kNN on levels also rather consistently outperforms pre-trained VGGish, which condenses all channels to one through a mean operation before passing them through the CNN. While this suggests that the separate channels have complementary information, the improvement in the performance of pre-trained VGGish relative to kNN with one or two microphones suggests that there is important information in the spectrograms of the RIRs which is discarded when only using their levels. The multichannel VGGish takes the best of both approaches, using all channels and the full spectrograms, and consistently outperforms all models on this task. Furthermore, its performance relative to the pre-trained version on single-channel RIRs suggests that it is important to use the full  $48\mathrm{kHz}$  sample rate rather than downsampling to  $16\mathrm{kHz}$ .
Generalization To broaden the scope of potential applications of this task, we investigate the ability of our models to generalize to data outside of the training distribution. For these tasks, we only test the learning-based baselines, since the analytical time of arrival method requires no training data.
First, we use models trained on data from one human in a particular room and test generalizations to one or more other humans in the same room. Results from training models on music-derived RIRs from one human and testing on music-derived RIRs from another human in the Living Room are shown in Table 4. Though performance drops relative to training and testing on the same human, the multichannel VGGish is still able to make somewhat accurate predictions. Future work should test generalization on a more comprehensive distribution of humans of different shapes, sizes, ages, etc.
Next, we test the generalization between different layouts of furniture in the same room. Though SOUNDCAM could theoretically be used to benchmark generalization among different rooms, each room varies significantly in structure and therefore requires using a unique layout of microphones and speakers to record data, so this task is not well-defined for our baselines. On the other hand, for the Treated Room, our recordings with and without the panels both used the same placement and arrangement of speakers and microphones, with only the static contents of the room varying between recordings. We thus train each baseline model on the sine sweep RIRs from the Treated Room and test on RIRs from the Treated Room with the fabric divider panels in it, and vice versa. None of the baselines are particularly robust to the introduction of these fabric panels, with each model's mean error being at least a meter for each amount of microphones. This unsolved issue of generalization to minor changes in layout in the room could severely restrict the downstream practical applications of these models. Detailed results are included in Appendix C.1.
4.2 Human Identification
We also investigate whether learning-based methods can correctly classify RIRs by the identity of the human in the room. We use the RIRs from our Treated Room, where we collect at least 200 RIRs for each of the five humans. A visualization of the unique positions of each human in the room is shown at the right of Figure 3. Further, to determine whether the RIR is indeed useful
Figure 3: (Left) An aerial view of a 3D scan of the Treated Room. (Right) A visualization of the microphone, speaker, and unique human positions within the Treated Room for our human identification subset.
Sine Sweep RIR	Number of Microphones
10	4	2	1
kNN on Levels	65	46	42	28
VGGish (pretrained)	72	60	39	31
VGGish (multichannel)	82	81	76	64
Silence
kNN on Levels	39	20	17	20
VGGish (pretrained)	20	20	20	20
VGGish (multichannel)	20	20	20	20
Table 5: Classification accuracy  $(\%)$  in identifying among five humans from both sine sweep RIRs and silence in the Treated Room.
Music RIR	Number of Microphones
10	4	2	1
kNN on Levels	57	58	56	54
VGGish (pretrained)	65	62	70	80
VGGish (multichannel)	99	100	100	98
Music Raw
kNN on Levels	51	55	49	51
VGGish (pretrained)	53	53	51	55
VGGish (multichannel)	75	77	71	55
Silence
kNN on Levels	59	54	54	52
VGGish (pretrained)	55	47	54	53
VGGish (multichannel)	50	50	50	50
Table 6: Classification accuracy (%) in detecting the presence of a human in the Treated Room using either a music-derived RIR, a raw recording of music, or a recording of silence.
for identification, or if the human can be identified merely by using other spurious noises, such as distinctive breathing patterns or external temporally-correlated noises, we also test on recordings from each human standing in the same positions as the RIRs, but while the speaker is silent. We randomly split the train and test data such that both have a balanced ratio of recordings from each human.
Baselines We use similar baselines to those used in Section 4.1, with the exception that we omit Linear Regression and our analytical Time of Arrival since they are both specifically suited to regression and localization, respectively. The VGGish-based models use the same structure as those in localization except that the last layer uses a Softmax to predict the likelihood scores for each class.
Results Table 5 shows the results. We see strong evidence that the actively-measured RIR indeed has important information for identification, and that the passively-recorded noises from the room or emitted by the humans while standing in presumed silence are generally not enough for our baselines to identify. Once again, our multichannel VGGish baseline consistently outperforms other baselines.
4.3 Human Detection
We investigate whether recordings from a room can be used to detect whether a single human is in the room or not. To do so, we compare recordings of a room with a human in it to those of the empty room. We observe that sine sweep-derived RIRs of an empty room are quite similar between trials. When a human is introduced into the room, the sine sweep-derived RIR becomes different enough to make the task of detecting the presence of a human from a sine sweep trivial. Instead, we focus on the task of using natural music signals to detect the presence of a human. We use data from our Treated Room, where we have 1,000 recordings, each of music played in the room with and without a human. We devise three different subtasks based on different inputs used to detect a human in the room: 1) an RIR derived from music, 2) a segment of the raw recording of the music, and 3) a segment of a recording of silence. Note that the first subtask requires having access to the time-synchronized signal that is being played in order to deconvolve out an estimate of the RIR, and the second does not assume such knowledge of the signal being played.
Baselines We use the same baselines as in the identification task in Section 4.2, with the exception that the final output of the VGGish-based models is that of a Sigmoid rather than a Softmax.
Results We show results for each of the different signals in Table 6. Once again, silence is not enough for our models to significantly outperform random guessing. Our multichannel VGGish baseline is able to detect a human in the room from these music-derived RIRs almost perfectly. However, knowledge of the signal being played seems to be essential to our baselines, since their performance on the music-derived RIR is much better than that on the raw music without knowledge of the underlying signal. In many real-world scenarios, access to the signal being played or emitted by the sound source is not available.
We presented SOUNDCAM, a large dataset of unique room impulse responses (RIRs) and recorded music from a controlled acoustic lab and in-the-wild rooms with different humans in positions throughout each room. We have demonstrated that our dataset can be used for tasks related to localizing, identifying, and detecting humans. Our results show that while each of these tasks can be solved rather robustly when training on large datasets and testing on data from within the same distribution, there are still unsolved problems in generalizing to unseen humans as well as unseen rooms. These unsolved problems may hamper the practical real-world applications of these tasks.
The main limitations of our dataset are the breadth of environments, humans, and poses represented. While we collect data from two in-the-wild rooms, these rooms cannot possibly represent the wide distribution of rooms from different regions and socioeconomic backgrounds. The five humans in our dataset similarly cannot capture the diversity of age, sizes, and shapes of humans in the world. Finally, each human assumes the same pose facing the same direction to remove minor differences in pose as a variable, whereas in real-world applications, humans assume diverse poses as they go about their activities. While a wider diversity of human shapes and poses would be beneficial for widespread real-world applications, SOUNDCAM is a significant first step in developing such applications.
As future work, we first plan to augment the diversity of humans and their poses in our dataset. Our dataset currently includes more annotations than needed for our tasks, such as 3D scans of each room, and could thus be used for additional novel tasks, such as prediction of RIR from simulation. We also hope to augment the dataset to vary the number of people in the room, so that the dataset could be used for tasks of quantifying, identifying, or locating multiple people in a single room using sound. Finally, the failure of our baselines in learning from raw audio signals rather than RIRs suggests additional applications in estimating room acoustics from arbitrary source signals or even techniques which do not assume knowledge of the source signal, such as blind deconvolution.
Acknowledgments. We thank Mert Pilanci and Julius O. Smith for valuable discussions. This work is supported in part by NSF CCRI #2120095, ONR MURI N00014-22-1-2740, Adobe, Amazon, and the Stanford Institute for Human-Centered AI (HAI). The work was done in part when S. Clarke was an intern at Adobe.
The supplementary materials consist of:
A. Details of the rooms used in the dataset.
B. Details of the baseline implementations and their hyperparameters.
C. Additional results from experiments in the human localization tasks.
D. Details on the procedures for data collection and preprocessing.
E. A datasheet with important metadata about the dataset.
A Room Details
We include photos and further details of each room.
A.1 Acoustically Treated Room
The Treated Room is rectangular, approximately  $4.9 \times 5.1$  meters and 2.7 meters in height. Each of its walls are covered in 15-cm-thick melamine foam panels, each having a noise reduction coefficient (NRC) of 1.26. The ceiling tiles are fiberglass with an NRC of 1.0, and the floor is carpeted with standard office carpet. Photos of the room in its two different configurations, empty and with fabric divider panels, are shown in Figures 4 and 5, respectively. Diagrams of the positions of the Human 1 subsets from the room in each configuration are shown in Figure 6. The microphones and speaker remained in the same positions for both subsets.
Figure 4: Images from the Treated Room in its empty configuration.
Figure 5: Images from the Treated Room in its configuration with fabric panels.
Figure 6: Diagrams of the positions of speaker, microphones, and Human 1 in Treated Room subsets in (Left) empty configuration (representing both sine sweep and music subsets) and (Right) configuration with fabric divider panels.
Figure 7: Images from the Living Room.
A.2 Living Room
The Living Room is in a real household with an open layout, i.e., the room does not have specific walls delineating it from parts of the rest of the house, including a kitchen and a stairway. The floor is covered in tile, and nearby walls and the vaulted ceiling are composed of drywall. The room contains a television screen and entertainment system, a coffee table, and a fabric sofa, with other furniture outside the boundaries of the area used for data collection also affecting the acoustic signature. In addition to the photo of the room and the diagram of the microphone, speaker, and human positions included in Figure 2 of the main manuscript, Figure 7 shows additional photos of the room.
A.3 Conference Room
The Conference Room is rectangular, approximately  $6.7 \times 3.3$  meters and 2.7 meters in height. Three of its walls are composed of drywall, while the remaining wall (on a short side of the rectangle) is mostly composed of glass, with drywall trim and a wooden door with a glass window. The ceiling is closed with standard office ceiling tiles, and the floor is covered with standard office carpet. The room has a large monitor screen on one wall, a large whiteboard on another wall, and a long wooden table pushed against the wall with the monitor, with wheeled chairs surrounding it. Photos from the Conference Room are shown in Figure 8, and the diagram of speaker, microphone, and human positions in the room are shown at the right of Figure 9.
Figure 8: Images from the Conference Room.
Figure 9: (Left) An aerial view of a 3D scan of the Conference Room. (Right) Diagram of the microphone, speaker, and unique positions of Human 1 in the Conference Room.
A.4 RT60 Reverberation Times
The reverberation time for each room is measured from the empty room's impulse response, from which we calculate the time for the sound to decay by 60 dB (RT60) using the Schroeder method [Schroeder, 1965]. The RT60 measurements for each of the rooms/ configurations are shown in Table 7. Since there are 10 microphones from which RT60s can be measured, we report the mean, minimum, and maximum RT60 across the 10 microphone locations.
Mean RT60(s)	Min RT60(s)	Max RT60 (s)
Treated Room	0.158	0.112	0.264
Treated Room w/ Panels	0.158	0.111	0.248
Living Room	1.121	1.022	1.170
Conference Room	0.581	0.541	0.608
Table 7: Overall RT60 Reverberation Times for each of the rooms/ configurations. The second column shows the minimum RT60 across all microphones. The third column shows the maximum RT60 across all microphones.
Figure 10: Octave-band RT60 reverberation time measurements from each of our four rooms/ configurations. The solid line indicates the average RT60 for each frequency band across all microphone locations, while the shaded region represents the standard deviation.
Sweep Level (dB)	Silence Level (dB)	Content of Silence
Treated Room	-3.76	-42.76	Microphone noise
Living Room	-2.57	-44.67	Microphone noise, occasional vehicle
Conference Room	0.00	-30.60	Microphone noise, ventilation noise
Table 8: RMS levels of the recordings during the sweep (first 10 seconds) and silence (last 2 seconds). Levels are measured in dB, relative to the sweep level in the conference room, which was the loudest. Lower dB levels mean the silence is quieter relative to the sound being played.
In addition, we provide octave-band RT60s in Figure 10. We observe that reverberation times differ significantly across frequency bands, and that the relationship between frequency and reverberation time is different for each room.
A.5 Silence Levels and Content
The last 2 seconds of every recording are presumed to be silent, since recording continues for 4 seconds after the sweep/music signal plays from the speaker, and the longest RT60 measured in any of our rooms at any frequency was under 1.5 seconds. Table 8 shows the RMS level when the sweep is playing (first 10 seconds), relative to the RMS level of the presumed silence (last 2 seconds). RMS levels are measured relative to the sweep level in the conference room. The levels are averaged across all microphone recordings.
B Baseline Details and Hyperparameters
We include further details of each baseline in addition to important hyperparameters.
B.1 Levels-based Baselines
For all baselines based on volume levels, each room impulse response from each microphone was truncated to two seconds, and condensed into a single number representing the RMS volume level
of the room impulse response. Thus, for baselines that use all 10 microphones, each data point is summarized by 10 features, one for each microphone.
Each feature for the data points is normalized to have a mean of 0 and a standard deviation of 1 across the training set. We use the validation set of 100 datapoints to determine the best-k for k-nearest neighbors, with k ranging from 1-100, and report the average error in centimeters on the test set. The linear regression baseline does not use the validation set, and fits a linear model that minimizes squared error on the training set.
For baselines using raw music recordings (as opposed to room impulse responses, or room impulse responses estimated from music), a 2-second interval from the middle of the recording (from 5 to 7 seconds) is taken instead.
For baselines using silence, we examine the raw recording of the sine sweep that was used to measure the room impulse response. The sine sweep plays for 10 seconds in this recording, followed by 4 seconds of silence. For our baselines, the last two seconds of this raw audio recording are taken and assumed to be silent, since the longest RT60 Reverberation Time of any of our rooms at any frequency was measured to be 1.5 seconds.
B.2 Deep Neural Network-based Baselines
Because the pretrained VGGish model from [Hershey et al., 2017] was previously trained on AudioSet [Gemmke et al., 2017], the authors had designed the model around some key assumptions about the input audio, including the sample rate and the number of channels. We accordingly conducted some preprocessing of our input audio before before passing it to the pretrained VGGish. First, we downsample the audio from  $48\mathrm{kHz}$  to  $16\mathrm{kHz}$ , to match the sampling rate on which the VGGish was pretrained. Next, if the input includes audio from multiple microphones, we collapse the audio to a single channel by taking the mean across all channels. We then truncate the input audio to 30950 samples (or 1.93 seconds) to match the pretrained network's input size. The audio is then transformed into a log-mel spectrogram, which is passed to a deep convolutional neural network that generates a 128-dimensional representation of the audio. We train a 3-layer, fully connected neural network with ReLU activations following hidden layers on these embeddings, to predict an  $x,y$  location, a vector of class probabilities, or a single number representing the probability of human presence in the room. The dimension of each hidden layer is 256.
For the multichannel VGGish baselines, we do not downsample or reduce the channels of the input audios as part of the preprocessing and instead compute the log-mel spectrogram on each input channel at full sample rate. For these baselines, the input to the neural network is an  $N$ -channel tensor of stacked spectrograms from each channel, where  $N$  is the number of audio channels. The neural network for multichannel VGGish is based on the same overall structure as that of the pretrained VGGish in terms of the number of layers and size of the convolutional kernels at each layer, other than the first convolutional layer which is resized to accommodate inputs of more than one channel. We randomly initialize all weights at the beginning of training.
Thus, the key differences between the multichannel VGGish and the pretrained VGGish are that 1) the weights in the CNN of the pretrained VGGish's weights are frozen to those from AudioSet, while the weights of the multichannel VGGish are not, and randomly initialized, 2) the input is collapsed to a single channel in the pretrained VGGish via mean reduction, and 3) the input is reduced to a sampling rate of  $16\mathrm{kHz}$  in the pretrained VGGish while the multichannel VGGish maintains the full sample rate (48kHz).
All VGGish-based models are trained for 1000 epochs using the Adam optimizer with a learning rate of  $1 \times 10^{-4}$  and a batch size of 32.
B.3 Time of arrival Baseline
Our time of arrival baseline is based on principles described in [Yang et al., 2022], which contains more details about the principles we used. As a brief summary, the room's impulse response is modeled as the sum of two room impulse responses - the impulse response of the empty room, and the impulse response resulting from the additional reflections induced by the person in the room.
To isolate the reflections generated by the person, we subtract the empty room impulse response from the impulse response obtained when the room has a person in it. This provides us with a time-domain
Figure 11: Visualizations of the intersecting spatial heatmaps from the Time-of-Arrival baseline, using the positions of the two microphones closest to the speaker from two-channel impulse responses in the Treated Room. (Left) A success case and (Right) a failure case.
Time of arrival	Center Prediction
Treated Room	96.2	136.0
Treated Room w/ Panels	89.3	144.3
Living Room	225.9	176.6
Conference Room	136.0	159.0
Table 9: Median error in centimeters for the time of arrival method and the center-of-room prediction method. The results are using 4-channel RIRs measured from sine sweeps recorded in each of the rooms/ configurations.
representation of the person's reflections, which is referred to by Yang et al. [2022] as the "pose kernel". This representation can be mapped to a spatial encoding of where the person might be standing based on the speed of sound and the positions of the sound source and listener.
In order to obtain a measurement of the empty room impulse response, we compute the average of the 100 room impulse responses measured in the empty room. Averaging in the time domain is possible due to the time-alignment procedure described in Appendix D.1.
In order to determine a person's location from one- or many-channel room impulse responses, we compute a pose kernel for each impulse response. We take the envelope of each pose kernel, and use them to compute a spatial heatmap of possible locations. These spatial heatmaps are computed on an  $xy$  grid with a resolution of  $5\mathrm{cm}$ , covering the space within the walls of the room.
When the impulse responses are from multiple channels, these spatial heatmaps are intersected by multiplying them together. The predicted location is the location with the maximum value on the resultant heatmap.
A success and a failure case for the time of arrival method are visualized in Figure 11.