SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is worth noting that failure cases for this baseline are often catastrophic, resulting in large errors which inflate the mean error. In Table 9 we provide the median test error of the time of arrival baseline for each room. The results are shown for the 4 microphone case, where RIRs are estimated from sine sweeps. The median errors are from the exact same experiments as those in Table 2. We compare it to the method that predicts the center of the room each time. As we can see, the time of arrival baseline outperforms center-of-room guessing by a substantial margin in the Treated Room and the Treated Room with Panels, and performs worse than it in the Living Room. It slightly outperforms center-of-room guessing in the Conference Room. This suggests that this method is based on delicate assumptions and may not be robust to the challenges of uncontrolled environments in the wild. Since this approach has the advantage of not requiring training examples, these results motivate future work to make this approach more robust to reverberant rooms.
Number of Microphones
10	4	2	1
kNN on Levels	26.0 (35.9)	64.0 (53.4)	117.2 (80.3)	150.6 (73.0)
Linear Regression on Levels	97.7 (62.6)	116.3 (63.1)	129.6 (67.7)	153.2 (71.6)
VGGish (pretrained)	101.1 (76.5)	102.9 (75.6)	92.5 (69.4)	120.0 (87.5)
VGGish (multichannel)	24.0 (29.9)	20.7 (28.4)	27.5 (32.4)	44.4 (42.4)
Time of Arrival	181.9 (105.8)	152.7 (87.4)	218.1 (114.4)	382.9 (134.7)
Table 10: Localization error of each model using sine sweep-based RIRs from varying numbers of microphones in the Conference Room. Errors are in centimeters and "mean (stdev)" format.
Train Base → Test w/ Panels	Number of Microphones
10	4	2	1
kNN on Levels	146.3 (77.4)	162.6 (109.2)	149.1 (85.1)	135.7 (55.2)
Linear Regression on Levels	126.3 (46.0)	131.8 (53.3)	138.1 (48.3)	138.3 (47.6)
VGGish (pretrained)	163.1 (85.6)	168.9 (74.4)	157.3 (80.5)	163.8 (75.9)
VGGish (multichannel)	132.6 (107.8)	126.0 (97.4)	135.1 (71.2)	163.6 (75.0)
Train w/ Panels → Test Base
kNN on Levels	182.5 (111.0)	170.5 (88.5)	175.4 (86.7)	135.7 (56.7)
Linear Regression on Levels	367.8 (274.2)	151.4 (92.3)	134.9 (59.3)	135.0 (49.3)
VGGish (pretrained)	219.3 (96.8)	146.2 (72.8)	151.4 (73.0)	169.2 (90.6)
VGGish (multichannel)	114.8 (68.7)	133.9 (47.8)	175.1 (91.2)	143.0 (77.2)
Table 11: Localization error of each model when trained on recordings of a human from the treated room in one arrangement and tested on recordings from the same human in the same room in the other arrangement. Errors are in centimeters with "mean (stdev)" format.
B.4 Microphone Ablations
For baselines experiments that only use one microphone, the furthest microphone from the speaker is selected. For baseline experiments that use two microphones, the closest microphone to the speaker and the furthest microphone from the speaker are selected. For baselines that use four microphones, microphones from each corner of the room are selected.
B.5 Resources used
In order to run our baseline experiments, we used the GPU clusters provided to our research group. The kinds of GPUs are NVIDIA GeForce RTX 2080 Ti, Titan RTX, GeForce RTX 3090, A40, A5000, and A6000. Altogether, the baseline experiments shown in the paper consumed approximately 300 GPU hours.
C Additional Baseline Results
We include additional results from the localization task to first show the performance of each baseline on generalizing to a different configuration of the room and next show the effects of downsampling recordings on our multichannel VGGish baseline.
C.1 Additional Results on Human Localization
We show additional quantitative results from experiments described in Section 4.1 of the main manuscript. In addition to the results shown in Table 2 of the main manuscript, the results of our baselines on localizing a human from sine sweep-based room impulse responses (RIRs) within the Conference Room are shown in Table 10. The results of training on sine sweep-based RIRs from the empty Treated Room and testing on sine sweep-based RIRs from the same human in the Treated Room with fabric divider panels, and vice versa, are shown in Table 11.
Number of Data Points	Localization Error (cm)
25	166.5 (61.2)
50	112.9 (78.3)
100	67.0 (51.1)
200	52.2 (37.9)
300	34.3 (27.0)
400	42.1 (33.6)
500	31.5 (24.1)
800	27.9 (22.0)
Table 12: Localization error of the Multichannel VGGish model in the living room when trained on various numbers of data points. The models are trained on 10-channel RIRs. Errors are in centimeters with "mean (stdev)" format.
Model	Number of Microphones
10	4	2	1
VGGish (multichannel, 48 kHz)	18.1 (13.1)	17.2 (16.2)	20.4 (15.4)	71.6 (50.4)
VGGish (multichannel, 16 kHz)	18.6 (15.4)	21.8 (16.1)	26.8 (23.1)	49.4 (54.3)
Table 13: Localization errors of the multichannel VGGish with the full  $48\mathrm{kHz}$  sampling rate, and the multichannel VGGish when the input audio is downsampled to  $16\mathrm{kHz}$ . The models are trained and evaluated on RIRs from the Treated Room. Errors are in centimeters with "mean (stdev)" format.
RIRs from Treated Room, RIR	Number of Microphones
10	4	2	1
VGGish (multichannel, 48 kHz)	82	81	76	64
VGGish (multichannel, 16 kHz)	63	60	51	42
Table 14: Classification Accuracy (%) in identifying among five humans using RIRs in the Treated Room for the multichannel VGGish, using the full  ${48}\mathrm{{kHz}}$  sampling rate and downsampling to  ${16}$  kHz.
In addition, we perform a data ablation study testing sample efficiency on the localization task for the multi-channel VGGish in the living room. Training on just 100 10-channel RIRs, we are able to localize the human to within  $68~\mathrm{cm}$  in the test set. These results are shown in Table 12.
C.2 Effect of Sample Rate on Baseline Performance
Since VGGish (pretrained) is pretrained on audio recordings with a sample rate of  $16\mathrm{kHz}$ , we were interested in seeing if the higher,  $48\mathrm{kHz}$  sample rate used in our dataset is justified by isolating the influence of changing sample rate on our best-performing model, the multichannel VGGish. Table 13 shows results for the human localization task. We see that the model trained and evaluated on  $48\mathrm{kHz}$  RIRs outperforms the model trained and evaluated on RIRs downsampled to  $16\mathrm{kHz}$  in all conditions except for the single-microphone case. Table 14 shows the classification accuracy for both versions of the VGGish on the 5-person human identification task. These results both suggest that the higher sample rate is an important advantage for extracting task-relevant insights from these recordings.
D Additional Details on the Data Collection Procedure and Preprocessing
We describe aspects of our data collection procedure and processing in more details, including the logistics of how the human is directed to move between sine sweeps (App. D.1), the estimation of the room impulse responses (RIR) from recordings (App. D.2), and collecting recordings of music (App. D.3).
D.1 Collection Procedure
Room impulse responses in the dataset are measured by playing and recording sine sweeps. The details of how room impulse responses are obtained from these recordings are described in (App. D.2). If there is a person in the room, they are directed (by a beep) to select a location to stand in before the sine sweep begins playing. Another beep tells them that they should stand still, and two seconds later, their location and pose are captured by the RGBD cameras in the room. Then, the sine sweep is played from the speaker and recorded from the 10 microphones simultaneously. If we are capturing recorded music as well, a short pause is given after the sine sweep is finished recording, and then a 10-second music clip is played and recorded as well. Below, we have details on the processing techniques used to obtain the room impulse response, and to ensure our data is properly time-aligned.
D.2 Measuring the Room Impulse Response
In order to measure a room impulse response, we had the speaker play a logarithmic sine sweep from  $20\mathrm{Hz}$  to  $24,000\mathrm{Hz}$  for 10 seconds, followed by 4 seconds of silence. This sine sweep was recorded from each of the ten microphones. At the same time that the sine sweep is being sent from the audio interface to the speaker, a loopback signal is also being sent from the audio interface's output to one of its inputs. This loopback signal is used to estimate and correct for the latency in the system.
To compute the room impulse response  $r[t]$ , we take
$$
r [ t ] = I F F T \left(\frac {F F T (a [ t ])}{F F T (l [ t ])}\right)
$$
Where  $FFT$  and  $IFFT$  are the Fast-Fourier Transform and its inverse,  $a[t]$  is the digital recording of the sine sweep, and  $l[t]$  is the digital loopback signal. Observe that we deconvolve the loopback signal from the recording, instead of deconvolving the source signal sent to the speaker from the recording. The loopback signal is assumed to be the same as the source signal, but delayed in time in an amount equal to the latency of the system. Deconvolving from a delayed copy of the source signal instead of directly from the source signal corrects for the delay in the system. The last 0.1 seconds of the 14-second room impulse response is removed to eliminate anti-causal artifacts.
In case cases where the room impulse responses needs to be estimated from music, the procedure is identical, but the sine sweep is replaced by music.
D.3 Recording Music
We selected 1000 different clips of music from the 8 top-level genres included in the small version of the Free Music Archive dataset provided by Defferrard et al. [2016]. The genres included are "International", "Pop", "Rock", "Electronic", "Hip-Hop", "Experimental", "Folk", and "Instrumental". The clips are selected randomly, and the same number of clips are selected from each genre. Each clip is truncated to 10 seconds, and padded by four seconds of silence at the end to allow for the music to decay. In order to keep the volume level between clips consistent, the music clip being played is normalized to have the same root mean square across all data instances. The music is played by the speaker and recorded at the same time from each of the 10 microphones. In order to correct for the latency in the system, a loopback signal is recorded at the same time. The loopback signal is thresholded to determine when the music began playing from the interface, which is used to estimate the delay in the system. All music recordings are adjusted by this delay and saved in a separate file. The raw music recordings (without adjustment) are included, as well as the loopback signals.
E SOUNDCAM Datasheet
Following the guidelines suggested in Gebru et al. [2021], we document details of the SOUNDCAM dataset below.
E.1 Motivation
We created SOUNDCAM to develop methods for tracking, identifying, and detecting humans using room acoustics. SOUNDCAM allows for the training and testing of these methods in controlled and real-world environments. We describe these three tasks and provide baseline methods for them. Our results demonstrate that all of these tasks are unsolved under certain conditions. To our knowledge, no other publicly available dataset can be used to test methods in tracking, identifying, or detecting humans using acoustic information from real rooms.
Who created the dataset (e.g., which team, research group) and on behalf of which entity? (e.g., company, institution, organization)?
SOUNDCAM was created by a collaboration among the authors on behalf of their organizations, Stanford University and Adobe Inc.
The National Science Foundation and Adobe Inc.
E.2 Composition
What do the instances that comprise the dataset represent? (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
The types of data instances in SOUNDCAM can be split into four categories:
1. Empty Room Impulse Responses. These instances contain a measurement of the room impulse response of an empty room (i.e. without a human in it), as measured from 10 microphones.
2. Room Impulse Responses, From Rooms With a Human in Them. These instances contain a measurement of the room impulse response in a room, with a person standing at a specific, measured location. Each instance contains 1) a measurement of the room impulse response as measured from 10 microphones; 2) a measurement of the person's location in the room, determined by the pelvis joint location as estimated by three RGBD cameras; 3) the locations of the 17 joints given by the Azure Kinect's body tracking tool [Microsoft, 2022]; and 4) the anonymized identifier of the person present in the room.
3. Recordings of Music in an Empty Room. These instances contain a recording of a 10-second music clip playing in an empty room, as measured from 10 microphones. We vary the music clips between datapoints, selecting them from [Defferrard et al., 2016]. The source music clip is included in each datapoint, as well as a number identifying the source music clip.
4. Recordings of Music, in a Room With a Human Standing in It. These instances contain a recording of a 10-second music clip playing in a room with a person standing in it. Each instance contains 1) a recording of the music playing in the room as measured from 10 microphones; 2) a measurement of the person's pelvis location; 3) the locations of the 17 joints given by the Azure Kinect's body tracking tool; 4) the source music clip, as well as a number identifying it; and 5) the anonymized identifier of the person present in the room. The same 1000 music clips that are used across rooms, as well as when we record music in the empty room.
In addition, we provide a 3D scan of the each room in which we collected data, along with the 3D locations of each microphone and of the speaker. We also include calibrations for each of the 10 microphones we used for measurements. We also include a rough, anonymity-preserving scan of each person represented in the dataset in the same poses they assumed while we collected the recordings.
We summarize this in Table 1 of the main manuscript.
Does the dataset contain all possible instances, or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the
sample representative of the larger set (e.g. geographic coverage) If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
The dataset represents some imperfect sampling at a number of levels. First, the locations of the humans in the room are sampled from the set of all possible feasible locations that someone could be standing in the room, though each human may have been more biased toward certain regions. We validate how representative each sample is by providing visualizations of the rooms and the corresponding human positions in Figures 2, 6, and 9. The five humans who volunteered as subjects could not possibly represent the diversity of the global population from which we sampled them. We would need at least an order of magnitude more distinct humans to approach this ideal. We sampled the music clips for our recordings from the 'fma-medium' version of the Free Music Archive Defferrard et al. [2016] dataset, sampling uniformly within each of the eight major genres.
We provide a description of each data instance's annotation in an answer to a previous question. Both "raw" data and preprocessed data are included in the dataset. For instance, each room impulse response is measured by playing a sine sweep from the speaker. The recording of this sine sweep is also included in the dataset alongside the RIR produced through preprocessing this recording. More detailed descriptions of the "raw" data included in the dataset are discussed below in Appendix E.4.
For the localization task, the labels are the planar  $x, y$  location of the human's pelvis joint. For the detection task, the label is if the person is in the room or not. For the identification task, the label is the person's identity. We document all of these labels for each data instance.
Everything is included for each individual instance. No annotation is missing for any particular instance.
Are the relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
Yes, across each dimension of potential relationships, the relationships are made explicit by the annotations. For example, each of the 1000 song clips used is given a number, so data points using the same song clip can be matched. Clips with the same human in the room can similarly be matched.
Are there recommended datasplits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
The train-valid/test split used for our baseline experiments are provided as a part of the dataset. These splits were done randomly, in a 80/10/10 fashion. For generalization experiments, we recommend holding out all of the data from the human, room, or signal to which the generalization is being tested.
See Appendix E.4 below.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, overtime; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
The dataset is self-contained. For each of the song clips used in the dataset, we provide the song ID from the Free Music Archive as well as the song's genre.
The dataset does not contain confidential information.
The dataset does not include any offensive or insulting aspects to our knowledge. However, it is possible that a small number of the songs we randomly sampled from the Free Music Archive Defferrard et al. [2016] may contain mildly crude language of which we are not aware. None of our subjects notified us as such.
The dataset does not identify any subpopulations. The humans used in the dataset are between the ages of 22 and 30, between  $157~\mathrm{cm}$  and  $183~\mathrm{cm}$  tall, and represent a diverse set of genders and ethnicities. However, we do not specifically identify the subpopulations to which any individual subject belongs.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset?
No, the 3D scans of each individual are not detailed enough to identify any of the people, and we omit RGB data which could be used to distinguish other features.
The dataset does not contain such sensitive information.
E.3 Collection Process
The details of the data collection procedure are described in Appendix D.1. The data was directly observable.
In order to collect our data, we used 10 omnidirectional Dayton Audio EMM6 microphones placed on microphone stands throughout the periphery of the room. The location of each microphone stand is shown in Figures 2, 6, and 9. These 10 microphones are routed to a synchronized pair of MOTU 8M audio interfaces, which simultaneously record and play the sine sweep, at a sampling rate of  $48\mathrm{kHz}$ . Python's sounddevice module was used to play and record the audio.
The music was sampled uniformly from the 'fma-medium' version of the Free Music Archive Deffer-rard et al. [2016].
Each human's location within the room is sampled from the set of feasible standing locations inside of that room given all obstructions (for instance, furniture), and within the boundaries of the microphones and cameras. We instructed each subject to move in a serpentine fashion in even steps to cover the entire room as uniformly as possible. Throughout the collection process, we reviewed the subjects' coverage of the room frequently (every  $\sim 50$  datapoints) to ensure uniform coverage. The subjects made several passes across the room in each dataset, and we sometimes directed the subjects to move randomly. These procedures ensure uniform coverage while also reducing correlation between location and time.
All of those involved in the data collection process were students at Stanford University, who graciously volunteered their time to us.
Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created. Finally, list when the dataset was first published.
We collected the dataset between May 5, 2023, and June 4, 2023, which should be commensurate with the timestamps on the raw files. Preprocessed files and compressed versions of files may have later timestamps.
The exemption specified in CFR 46.104 (d) (3) (i) (A) applies to our research. The exemption states that:
(i) Research involving benign behavioral interventions in conjunction with the collection of information from an adult subject through verbal or written responses (including data entry) or audiovisual recording if the subject prospectively agrees to the intervention and information collection and at least one of the following criteria is met:
(A) The information obtained is recorded by the investigator in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects;
We collected the data directly.
We notified the individuals that we were collecting data from them as we requested their participation. An example of the wording for this request is below.
The individuals consented to the collection and use of their data. We asked all subjects if they would like to help us collect data, described the intentions of the project, and described the time and physical tasks we would require of them to participate. An example request is, "Would you have some time to help us collect data for a dataset on how room acoustics vary with humans in different positions? We would need you to stand silently in different positions in a room for about two hours total."
In addition, we have written documentation from all participants explicitly confirming their consent to the data collection procedure and to the data release. The participants acknowledged that they had the option to discontinue the data collection process at any time, and are also given the chance to opt out of the dataset and have their data removed.
The form that they signed can be viewed at https://masonlwang.com/soundcam/SoundCamConsentForm.pdf
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate)
We gave all subjects the choice to leave the data collection process and withdraw the data they provided at any time.
We have not conducted a formal analysis, though we made design choices which were careful to consider subject privacy (e.g. omitting RGB images) and physical protection (e.g. providing hearing protection and allowing breaks as needed during data collection).
E.4 Preprocessing/Cleaning/Labeling
We describe how the room impulse responses (RIRs) were derived in Appendix D.2 and describe how the music recordings were processed in Appendix D.3.
Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)?