Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio
Authors: Xudong XU
Dejan Markovic
Jacob Sandakly
Todd Keebler
Steven Krenn
Alexander Richard
Venue: NeurIPS 2023 spotlight (2023)
Keywords: sound field
spatial audio
virtual humans
human body
body modeling
TL;DR: We present a model that can generate accurate 3D sound fields of human bodies from headset microphones and body pose as inputs.

Abstract: While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community. To close this gap, we present a model that can generate accurate 3D spatial audio for full human bodies. The system consumes, as input, audio signals from headset microphones and body pose, and produces, as output, a 3D sound field surrounding the transmitter's body, from which spatial audio can be rendered at any arbitrary position in the 3D space. We collect a first-of-its-kind multimodal dataset of human bodies, recorded with multiple cameras and a spherical array of 345 microphones. In an empirical evaluation, we demonstrate that our model can produce accurate body-induced sound fields when trained with a suitable loss. Dataset and code are available online.

Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio
Xudong Xu*
Dejan Marković Meta Reality Labs Research de dejaammrkovicemmtt com
Shanghai AI Laboratory
judongxu9710@@mm com
Jacob Sandakly
Todd Keebler
Steven Krenn
Meta Reality Labs Research j jasandakly@meta com
Meta Reality Labs Research
Meta Reality Labs Research
toddkeebler com
stevenkrenn@metaa com
Alexander Richard Meta Reality Labs Research r cha rda l ex@meta...... com
Abstract
While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community. To close this gap, we present model that can generate accurate 3D spatial audio for full human bodies. The system consumes,
input, audio signals from headset microphones and body pose, and produces, as output, a 3D sound field surrounding the transmitter's body, from which spatial audio can be rendered at any arbitrary position in the 3D space. We collect a first-of-its-king multimodal dataset of human bodies, recorded with multiple cameras and spherical array of 345 microphones. In an empirical evaluation, we demonstrate that our model can produce accurate body-induced sound fields when trained with suitable loss. Dataset and code are available online
Introduction
Throughout all of our history, humans have been at the center of the stories we tell: starting from early cave paintings, over ancient tales and books, to modern media such movies, TV series, computer games, or, more recently, virtual and augmented realitics. Not surprisingly, digital representations of the human body have therefore long been research topic in community Guan al.) [2009],
Loper et al. 2015]. Kanazawa et al. (2018). With the rise of computer -eepurtt-eeeeetted imagery the film industry and the availability of powerful computing consumer devices, visual representations of 3D human bodies have seen great advances recently, leading to extremely lifelike and photorealistic
body models Bagautdinov et al. [2021 Zheng al. 2023
However, while the visual representation of human bodies has made steady progress, its acoustic counterpart has been largely neglected. Yet, human sensing is intrinsically multi-modal, and visual and acoustic components are intertwined in our perception the world around us. Correctly modeling 3D sound that matches the visual scene is essential to the feeling of presence and immersion in
Work done during internship at Meta Reality Labs Research, Piusburgh, Pitsuughh PA, USA.
com http:/thwww......ooooooooooooo/a
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

input niss frowe VR houdioo
oid0o decode:
sound feld rendeee
body pose
[0.,0.) Benget position on sphere
predict sound larget microphonny
order oustooii
X - [V
Figure 1: System Overview. The model consumes audio from microphones on headset and body pose and encodes them into latent audio and pose embeddings. An audio decoder that is conditioned on a target microphone position then generates an audio signal as it would sound from the given target position. Note that the decoder can only render target microphones that lie a sphere around the transmitter as this is the capture setup for the training data. In order to model the full sound field any arbitrary position, we predict the signal at N (K 1)2 microphone positions on sphere surrounding the body and compute the K --th order ambisomic sound field from these signals.
3D environment Hendrix and Barfield [996]. In this work, propose method close this gap between the visual and acoustic representation of the human body. We demonstrate that accurate 3D spatial sound can be generated from head-mounted microphones and human body pose.
In particular, we consider an AR/VR telepresence scenario where people interact as full-body avatars. In this setting, available inputs include egocentric audio data from the head-mounted microphones (it common for headsets to employ microphone arrays), as well as body pose used to drive the avatar, obtained either using external sensors from the headset itself (the current generation of headsets are able to track the upper body hands). The goal is to model the user-generated sound field so that be correctly rendered arbitrary positions in the virtual space.
Existing approaches to sound spatialization, both traditional signal processing Savioja et al. 1999 and in neural sound modeling [Richard et al. 202 2022] operate under the strict assumption that the sound to be spatialized has known location and also recorded at this location, without interference through other sounds. These assumptions First, the location of the sound is unknown, i.e., a finger snap could come from the right hand as well as the left hand. Second, the sound not be recorded at the location where it is produced. In particular, the microphones are typically head-mounted (as in an AR/VR device), such that finger snaps footsteps
in stark contrast with the task address in this work.
not recorded at the location of their origin. Third, it is usually not possible to record a clean signal for each sound source separately. Body sounds like footsteps or hand- uroddprdduued sounds often co-oceur with speech and are therefore superimposed in the recorded signal.
To overcome these challenges, we train multi-modal network that leverages body pose to disam- biguate the origin of different sounds and generate the correctly spatialized signal. Our method consumes, as input, audio from seven head-mounted microphones and body pose. As output, the sound field surrounding the body is produced. In other words, our method is able to synthesize 3D spatial sound produced by the human body from only head-mounted microphones and body pose, as shown in Figure I
Our major contributions are (1) a novel method that allows rendering accurate 3D sound fields for human bodies using head-mounted microphones and body pose input, (2) an extensive empirical evaluation demonstrating the importance of body pose and well-designed loss, and (3) first-of- its-kind dataset of multi-view human body data paired with spatial audio recordings from array with 345 microphones. We recommend watching the supplemental video before reading the paper.
2 Related Work
Vistao-Visulllla Learning. In our physical world, audio and video are implicitly and strongly connected.
By exploiting the multi-modal correlations, plenty of self-supervised learning methods are proposed for representation learning and significantly outperform those approaches learning from single
modality Owens et al. (2066) Korbar et al. (2018) Alwassel et al. 2020 Tian et al. 2020 Xiao et al.

---

2020]
Patrick et al.
[202]]. Morgado et al. [2020] 2021a b Besides, such a close cross modality
correlation is also shown to boost the performance in audio-visual navigation Chen et al.
2020
2021].
Gan et al. 2020
2022 speech enhancement Owens and Efros
2018).
Yang et al. 2022
audio source localization Qian et al. [200), Hu et al. [2020]. ian et al. [2021 Diang et al.
2022
Mo and Morgado
2022 and separation Afouras et al. 2018). Ephrat et al. [018 Zhao et al.
2018
Gao et al. [2018] Gao and Grauman 2021J. Among these works, the most related papers to ours
Tocus on audio-driven gesture synthesis Ginosar al. 2019]. Lee et al. [2019]. Ahuja al. [020]]
Yoon et al. 2020], which aims to generate a smooth sequence the human skeleton coherent with
the given audio. While Ginosar et al. (2019), Ahuja et al. [2020]. . Yoon et al. (2020) focus the
translation from speech to conversational gestures based on the cross cross-modal relationships, [ee et al.
[2019] explore a synthesis- byniilsb--aallsiii learning framework to generate dancing sequence from the input music. Different from them, we are interested the opposite direction, Le., reconstructing the 31) sound field around the human from the corresponding body pose.
Hisually-guider Audio Spatialization. Visually-guided audio spatialization audio into 3D spatial audio under the guidance of visual information. There is increasing number of data-driven methods for this task, where crucial spatial information is obtained from the corresponding videos. Morgado et al. (2018). Li et al. 2018 and Huang et al.] (2019] leverage 360 videos to generate the binaural audio and concurrently show the capability of localizing the active sound sources. Besides, Gao and Grauman (219) regards the mixture of left and right channels as pscudo mono audio and utilizes the visual features to predict the left-right difference with a Iearnable U-Net.
transform monaural
After that, a line of papers Lu et al. (2199] Yang et al. (2020), Zhou et al. (2020) . Garg et al. [2021]. Xu et al. [202T] follow a similar framework and achieve better spatialization quality with different novel ideas. However, these methods fail to model time delays and reverberation effects, making the binaural audio less realistic. As for this problem, [Richard et al. [202T] propose to spatialize speech with WaveNet van den Oord al. 2016] conditioned on the source and listener positions explicitly, Despite the remarkable progress in audio spatialization, these approaches are all limited to binaural audio first-order ambisonics and thus fail to recover the whole sound field. In contrast, our method
strives to generate the whole 3D sound field by using the corresponding 3D human body pose.
3 Dataset
Setting. In this work we focus on mod- eling speech and body generated sounds considering an AR/VR telepresence sce- nario. general, is desiderable model these sounds in an anechoic environment, iee. without acoustics of specific room, which allows to add arbitrary room rever- berations using traditional signal process- ing methods (e.g., convolution with room impulse responses). For this reason we col- lect the dataset in an anechoic chamber. We note that the user's real environment can be noisy and reverberant, which could af- fect the input audio signals from the head- mounted microphones. There is huge corpus of research around denoising and dereverberation and are not addressing these problems in this paper.
Figure Capture stage with five body-tracking cameras and 345 microphones on sphere around the participant.
Data capture. We capture corpus of paired audio-visual data in anechoic chamber with 5 Kinects for body tracking and 345 microphones arranged on a sphere around the center of the capture stage. The cameras capture at 30 frames per second and audio is recorded at 48kHz. The Kinects and 345 microphones are co-calibrated such that they share the same world coordinate system. During the capture, the participants standing in the center of the stage while wearing head-mounted device with seven microphones to gather input audio.
In total, we collect 4.4h of human body sounds and speech from eight different participants. We capture each participant in three different settings: standing and wearing light clothes like a T-Shirt that does not produce a lot of noise while moving; wearing heavy clothes like jackets or coats that
3

---

produce audible noise while moving; and sitting in light clothing. For each setting, participants are instructed perform variety of body sounds such as clapping, snapping, tapping their body, footsteps, or scratching themselves, as well as speaking. The full capture script can be found in the supplemental material. Overall, 50% of the data are non-speech body sounds and 50% are speech. Note that handling both kinds of data poses an additional challenge for our approach: speech is typically a dense and structured signal, while body sounds are sparse, impulsive, and contain rich high-frequency components.
Data processing. We extract body poses from each camera using OpenPose Cao et al. [011 and obtain 3D keypoints via triangulation of the poses from the five camera views. The audio signals from all microphones (345 microphones on the spherical array and seven head-mounted microphones) and the Kinects are senessycchoonzzed using an IRIG time code. Technical details about audio-visual time synchronization can be found in the supplemental material.
Pose-Guided 3D Spatial Audio Generation
Producing full 3D sound fields inherently requires multimodal learning from both audio input and pose input, as neither modality alone carries the full spatial sound information. We formalize the problem and describe the system outlined in Figure [ that enable effective learning from the information-deficien input data.
with focus network architecture and loss
4.1 Problem Formulation
Let a1:T (u1,..., .TT) ) be an audio signal of length T captured by the head-mounted microphones, where T indicates the number of samples, and each de E Ci represents the value of the waveform at time / for each of the Cin input channels. Let further P1:5 = (P1, ................................ ..P))) be the temporal stream of body pose, where each ps E /x3 represents the 3D coordinates each of the J body joints at the visual frame Note that audio signals a and pose data P1:s of the same duration, but
sampled at a different rate. In this work, audio is sampled at 48kHz and pose at 30 fps, so there is one new pose frame for every 1,600 audio samples.
Ideally, we would like to learn a transfer function T that maps from the head- mounted microphone signals and body pose directly to the audio signal 11.7 at any arbitrary coordinate (zz, y, 2) in the 3D space,
(r, T
T(x: (x, U, 2, a1:r, PI:S).
However, it is in practice impossible to densely populate a 3D space with microphones and record the necessary training data. Instead, our capture stage is spherical microphone array and data can only be measured on the surface of this sphere, i.e. at microphone positions defined by their polar coordinates (r, o). Since the radius r is the same for every point on the sphere, a microphone position is uniquely defined by its azimuth 0 and elevation @ Leveraging the recordings of the microphones on the sphere as supervision, we model a transfer function
(1r Φb)) T T(2;, di, aI.T, PI:s)
that maps from input audio a and body pose P1:s to the signal recorded by microphone which is located at (0,, Φ.) on the sphere.
In order to render sound at an arbitrary position (x, in 3D space, we predict (1. Φ) for all microphone positions (0i, Φi) on the sphere, and encode them into harmonic sound field coeffi- cients Samarasinghe and Abhayapala (0122). from which the signal S can then be rendered at any arbitrary spatial position (x,, y, 2), see Section 4.4 for details.
4.2 Network Architecture
In this section, we describe how we implement the transfer function defined in Equation 2 At the core of the model are an audio encoder and pose encoder which project both input modalities onto a latent space, followed by an audio decoder that has a WaveNet yan den Oord et al. (0166 like
2This is true for perfectly spherical microphone array. In practice, the microphones are mounted imperfectly and slight deviations from the surface of the sphere occur, which neglect here.

---

a1:T
Dillted
gride
ConelD
CoavID
relu
fout
6 (0,, 9b)
P
ConvID
1
mio position
(2..
FC
j-th block
Syrr (1r 92)
Mdudio decodor blocks
Figure 3: Network Architecture. The model consists of an encoder for each of the input modalities and stack of M WaveNet-like audio decoding blocks. In the j-th block, the input the conditioning features fc and the audio encoding fa (j = 1) or the output of the previous block f (j > 1), respectively. The final oulput is an average-pooles accumulation of all block's 's outputs forr forwarded through two convolutional layers.
architecture and generates the target sound $1.7 at given microphone position (0,, 6.). The model architecture is outlined in Figure E
As training data, we use recordings from the capture stage described in Section where have fully synchronized and paired data of headset input, body pose, and target microphones. In other words, our training data are tuples (a1:7, P1:s; (0,, oi)) for each microphone position (6,, Ø,) on the spherical array.
Audio Encoder. Acoustic propagation through space at the speed of sound causes time delay between the emitting sound source (e.g., hand), the recording head-mounted microphones, and the actual target microphone on the spherical microphone array. It has been found beneficial to apply time-warping, í.e., to shift the input signal by this delay, before feeding into network Richard et al. [202T]. In case, as sound might emerge from various locations the body, we create copy of the input signal for multiple body joints and shift the input as if it originated from each of these joints. We then concatenate all these shifted input copies along the channel axis and feed them into linear projection layer to obtain the audio encoding fa that contains information about the time-delayed audio signal from all possible origins, see the supplemental material for details.
Pose Encoder. While a person is making some sound, their body poses P1:$ give strong cues when and where the sound is emitted. Additionally, sound interacts with the body and propagates differently through space depending on body pose. Hence, the signal 8 at target microphone position (0,, o depends on the body pose at that time. For the input pose sequence Pi:s. the 3D coordinates of each body joint are first encoded to latent features and the temporal relation is aggregated via two layers of temporal convolutions with kernel size 5. Then, we concatenate all body joint features and use an MLP to fuse them into a compact pose feature fp-
Target Microphone Position. In addition to audio and pose encodings, the decoder depends on the position (0,, oe) of the target microphone. We first map the microphone location into continuous Cartesian space
(x, =) (0.)(0.) ().) sin (i,)))))))))) (o))) cos((:))
and project a higher-dimensional embedding using a learnable, fully-connected layer.
Pose encoding and target microphone position encoding are then concatenated into conditioning feature vector which upsampled to 48kHz and, together with the warped input audio, passed to the audio decoder.
Following prior works yan den Oord al. (016]. Richard et al. [202]) we stack M WaveNet-like spatial audio generation blocks as illustrated Figure in which the input audio features fa are transformed into the desired output signal (1,7((, Φi) under the guidance of the conditioning pose and position features fe- Each block is a gated network combining information from audio and conditioning features. In the j-th block, the audio feature ʃá passes through dilated Conv ID layer and is added to conditioning features processed by Conv layers in both filter and gate branches, which are then integrated after the tanh and sigmoid activation functions:
Audio Decoder.
z = tanh (W) * få + V fe 1 oW fg Vj * fc);
5

---

where * denotes convolution operator, @ indicates element-wise multiplication. o is a sigmoid function, j is the index of the decoder block, f and g denote filter and gate, respectively, and W. V are both learnable convolution filters. A ReLU activation is applied to z which is then processed by two separated Conv 1D layers. One layer predicts the residual audio feature to update fa to få+1 for the next generation block, and another is responsible for producing the output audio feature fou All output features flr...MM from M blocks are average pooled and then decoded to raw audio waves $1: ; 0,) via Conv 1D layers. Finally, for training, the predicted audio is compared with the ground-truth recordings $1:r(0:: 02) using loss functions & (see Section 4.3
4.3 Loss Function
Optimizing for correct spatialization of body sounds is challenging due both the nature of the task (correct spatialization requires correct time alignment), and the nature of the signals we are dealing with (a mix of dense and structured speech with sparse, impulsive sounds such as clapping and snapping). To avoid speech distortions, a loss on the amplitude spectrogram is gen- erally preferred. On the other hand, such loss is unable to achieve accurate temporal alignment. Alternatively, the time domain l2 loss has been successfully used for neu- ral synthesis of binaural speech RRicaard et al. [0221] However, in our experiments we found that Eg tends to highly attenuate impulsive sounds. One issue is that small shifts of impulsive signals cause big spikes in & error in combination with the fact that available information is not sufficient for a perfect sample-level alignment (body keypoints give only a rough indication of a possible origin of sound). To address this issue we propose variation of & that is more forgiving small shift offsets. The loss, referred here a shift-l2, is computed over sliding windows with higher penalty applied to higher offsets.
(-
( 12- .......... (1)...... (a)
(1 +1-+
Figure 4: Illustration of the shift-f2 loss computation.
We start by dividing the estimated signal â(t) into N segments of size L as illustrated in Figure 4 For each segment n. we compute a weighed & error against sliding window from a reference signal a()), with sliding offset going from - L and L. The error is computed as
ä(nL +) - annL +
m (T)
(4)
v on min( σa) + 8
where a indicates the standard deviation of the signal, and is small value used to avoid numerical
issues. The result is set of 2L values __ = (-L).... elt) (O)...., elTh) (L) representing
l2 errors normalized by the signals energy and computed over set of shift offsets. After penalizing larger offsets by using a window function W = a (1 [w) w0).... w ([)))), with w(T) denoting the Blackman window of length 2L + and a being the penalty coefficient, we select the
(n) + 1)(W + - 11] By design this value can be 0 only if the &2 error is 0 at shift r = O. Finally, the loss value is obtained by averaging the results of each of N segments
minimum value min
(I
(II (n) + + + +1)
shift-l2
min
(5)
N
0=
We use I.= 128, = 100, 5= 0.001. To reduce the distortions of speech data, we combine shift-
with a multiscale STFT loss Yamamoto et al. 2021 computed for window sizes of 256, 128, 64, 32.
4.4 Sound Field Rendering
Given the audio signals each microphone and the knowledge of microphone locations, we encode the 3D sound field in the form of harmonic sound field coefficients following Samarasinghe and Abhayapala 2012). The maximum harmonic order is limited by the number of microphones as K LVN For our system with 345 microphones this allows sound field encoding up to 17-th harmonic order. More details can be found in the supplemental material.

---

Table The role of body pose is critical to solving the task. Both, on speech and non-speech signals, we see significant improvements when the model has access to the full body pose as opposed to no pose or headpose only.
non-speech
speech
pose information
SDR 1
amplitude / phase
SDR
amplitude |
phase
no pose
2.518
0.823
0.4644
6.682
9.007
1.362
headpose only
2.848
0.809
0.461
9.264
7.093
1.190
full body pose
3.004
0.784
0.454
9.580
7.059
1.184
right
front
left
zoom-in
ground truth
DO pose
headpoee only
full body pose
Figure 5: Without pose, the audio signal always originates from the center of the 3D space. With only headpose, the model always predicts the head the sound source location. Only when provided full body pose, the model is able to correctly localize the sound source.
Given the encoded harmonic sound field coefficients Bum (T, J), where 7 and f are, respectively, time and frequency bins, the sound pressure at any point x = (r. 0, Φ) in space can be decoded as Williams 1999
t(x,t) iSTFT Σ
Brm (T, h) (Kr) (0, の)
(6)
where k 2x f /soun is the wave number, Usound is the speed of sound; Ynm (0, Φ) represents the spherical harmonic of order and degree (angular spatial component), and h, (kr) is rith-order spherical Hankel function (radial spatial component).
Experiments
Dataset. All the experiments are conducted on our collected dataset. We the recordings into clips of one second in length without overlapping, resulting in 15,822 clips in total. The dataset partitioned into rain/validation/test sets of 12.285/1,776/1,777
clips, respectively. The consists of 961 non-speech clips and 800 speech clips. To the best of our knowledge, this is the first
we polar coordinates here instead of Cartesian coordinates for simpler notation in Equation 6

---

Table Besides pose, multiple head- wrad-mounted microphones provide source of spatial information. Increasing the number of microphones from one to seven leads to substantial improvements in output quality. Note that all experiments here use full body pose.
non-speech
speech
# input mics
SDR
amplitude | phase 4
SDR
amplitude |
phase |
1
018848
0.8-19
0.478
8.259
7.956
1.235
3
2.338
0.812
0.458
9.526
7.276
1.199
7
1.004
0.784
0.454
9.580
7.059
1.184
right
front
left
zoom-in
ground truth
input mic
input mics
Figure 6: Body pose alone not always enough to disambiguate the correct origin of sound. Using multiple microphones helps infer the missing spatial information.
dataset of its kind that contains multimodal data of speech and body sounds, which paves the way for more comprehensive exploration of modeling the 3D spatial sound of humans.
Implementation Details. In the encoders, the pose is mapped onto 32-d embedding fp. while the audio input a is mapped to 128-d features fa after tirp-nar,ing, For the audio decoder, we use M 10 blocks, in which each Conv 1D layer has 128 channels and kernel size 3, and the dilated Conv 1D layer starts with a dilation size of and doubles the size after each block. To transform the output features from 128 channels to a single channel audio, we employ two sequential Conv ID layers with ReLU activations. During training, we fetch 4 audio clips from the training set as a batch and randomly select 38 target microphones from the 345 available target microphones in the capture stage, ie., total of 152 1s audio clips are processed in each forward pass. We use Adam to train our model for 125 epochs with learning rate 0.002. When training with shift- and multiscale STFT, multiply the multiscale STFT loss with weight 100. All experiments are run 4 NVIDIA Tesla A100 GPUs and each needs about 48 hours to finish.
(SDR), & error on the amplitude spectrogram, and angular error of the phase spectrogram. While SDR measures the quality of the signal as whole, amplitude and phase eπrors show a decomposition of the signal into two components. High amplitude error indicates a mismatch in energy compared the ground truth, while high phase errors particularly occur when the delays of the spatialized signal are wrong. Note that the angular phase error has an upper bound of x/2 1.571. We report amplitude errors amplified by a factor of 1000 to remove leading zeros.
Evaluation Metrics. We report results on three metrics, momaalddstortioonoo
5.1
The Role of Body Pose
The ablation of pose information is studied as shown in Table I We use all seven head-mounted microphones as input and vary the pose information, removing all the poses or only using the head pose. As mentioned in Section the non-speech data is significantly different from the speech data,
8

---

Table 3: Loss ablation. The proposed shift-&z loss in combination with a multiscale STFT loss outperforms other losses most metrics.
non-speech
speech
loss
SDR t amplitude
phase
SDR
amplitude J
phase
2.557
1.064
0.449
7.610
10.594
1.093
multiscale STFT
-.222
0.797
0.557
-5.077
6.898
1.902
multiscale STFT + &2
2.394
0.936
0.465
8.566
8.477
1.204
shift-22
2.956
0.910
0.447
8.899
9.716
1.136
shift-f2 mult. STFT
3.004
0.784
0.454
9.580
7.059
1.184
right
front
left
z0om-in
ground truth
shift-F2
f/
mult. SIFT
mult. STFT 22
shift-t2 + mult. STFT
Figure 7: While commonly used losses like ly and multiscale STFT underestimate the sound field energy and have spatial alignment errors, the proposed shift-f2 loss enables accurate sound field reconstruction. Fine-grained details further improve when paired with a multiscale STFT loss.
and thus we report quantitative results on speech and non-speech data separately. In Table our model with full body pose outperforms variants with no pose or headpose only on all three metrics because full body pose provides important position information of sound sources. Compared the no pose version, the headpose leads to a huge improvement in speech data since the headpose itself is capable providing necessary position information on the speech case.
Figure 5 iillustrates how these quantitative differences manifest in the sound field. In the case of finger snap above the head, shown in the figure, the sound source estimated the center of the 3D space if no pose information is available. With only headpose, the model puts all sound sources in the head region where speech would be produced. Finally, when provided with full body pose, the model is able to correctly locate the snap at the right hand location above the head.
5.2 The Role of Head-Mounted Microphones
We also study the impact brought by the number of input microphones. Table lists the results of our model trained with 1, 3, and head-mounted input microphones respectively. As can be observed, our model with 3 microphones shows clear improvement over that with only input microphone since, complementary to pose information, some spatial information can inferred from 3-channel input audio but monaural audio implies nothing about the space. Compared to input mics, microphones can further boost the overall performance.
Figure 6 iillustrates challenging example where body motion alone does not give a clue whether the finger snap originates from the left the right hand. With single microphone as input, the model can only guess, and in the example shown in the figure, it places the origin of sound in the wrong hand. In contrast, using microphones as input, the model is able estimate the direction of arrival of the sound and, consequently, place its origin in the correct hand.

---

Table 4: Our method outperforms baselines both quantitatively and qualitatively.
non-speech
speech
methods
SDR t
amplitude +
phase 4
SDR
amplitde +
phase 4
Only time warping Ours w/o time warping Ours
0.016
1.031
0.502
11217
20.675
1.532
2.671
0.808
0.461
8.986
7.691
1.239
3.004
0.784
0.454
9.580
7.059
1.184
5.3 Accurate Sound Field Learning with Shift-L2
In order to evaluate the effectiveness of our new loss function proposed in Section we train our model with different loss functions. As shown Table we compare the new loss function shift-E2 with traditional losses like & or multiscale STFT Yamamoto al. 2021] Although not always the best on all three metrics, this new loss empowers our model to achieve a more balanced performance, unlike traditional loss functions. Moreover, if combined with multiscale STFT, it can further improve the SDR value while lowering the amplitude error, on both speech and non-speech data.
A qualitative evaluation of the different losses in Figure 7 shows the deficiencies of lo-loss and multi- scale STFT loss. With darker colors (red and blue) representing high energy and green representing low energy, both losses severely underestimate the energy in the predicted signal. The proposed shift-&2 loss, on the contrary, matches the ground truth signal well. When combined with a multiscale STFT loss, we observe even more fidelity and better sound field reconstruction in the fine grained details, see the zoomed-in example for shift-lz + multiscale STFT Figure 7
5.4 Comparison to baselines
Being the first to address the problem, there is no existing work that can serve as a meaningful baseline. Existing approaches to sound spatialization (Savioja et al. [1999], Morgado et al. (2018), Richard et al.] [202T) rely on knowledge of the sound source location or are limited to first-order ambisonic signals- recall that we model 17-th order signals for high enough spatial resolution. We therefore compare our approach to a naive baseline, where the input signal is naively time-warped from the head position towards the receiver microphone the sphere, with distance attenuation applied as well. Additionally, we run our system without time-warping to demonstrate the impact of time-warping. As shown in Table 4 on all three metries, our model demonstrates substantial improvements over the baseline. The results from our model without time-warping are inferior to the model with warping
Additional visualization examples, including failure cases, can be found in the supplemental material.
Conclusions
Our pose-guided sound rendering system is the first approach of its kind that can successfully render 3D sound fields produced by human bodies, and therefore enriches recent progress in visual body models by its acoustic counterpart. Enabling this multimodal experience is critical to achieving immersion and realism in 3D applications such as social VR.
Limitations. While the results of our approach are promising, still comes with considerable limitations that require deeper investigation in future work. First, the approach is limited to rendering sound fields of human bodics alone. General social interactions have frequent interactions of humans and objects, which affect sound propagation and the characteristics of the sound itself. Moreover, as the system is built upon an ambisonic sound field representation. it produces accurate results in the far field, yet fails in the near field when entering a trust region too close to the body that is modeled. Generally, the model assumes free-field propagation around the body, which is violated if two bodies close to each other and cause interactions and diffractions in the sound field. Lastly, the system relies on powerful GPUs to render the sound field efficiently, and not yet efficient enough to run on commodity hardware with low computing resources such consumer VR device.
Nevertheless, correctly modeling 3D spatial audio is necessary step for building truly immersive virtual humans and our approach is critical step towards this goal.
10

---

References
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech
enhancement. Proc. Interspeech 2018, pages 3244-33222 3248, 2018
Chaitanya Ahuja,
Dong Won Lee, Yukiko Nakano, and Louis-Phillipe Morency. Style transfer co-speech gesture animation: A multi-speaker conditional-mixx approach. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 248-265. Springer, 2020.
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems, 33:9758-9770 2020.
Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih, Driving-signal aware full-hody avatars. ACM Transactions on Graphics (TOG), 1014) 2021.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh, Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017.
Changan Chen, Unnat Jain, Carl Schissler. Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Computer Vision-ECCU 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 17-36. Springer, 2020.
Changan Chen, Sagnik Majumder, Al-Halah Ziad, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. In ICLR, 2021.
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen the cocktail party: a speakke--nddeeenden audio-visual model for specch separation. ACM Transactions Graphics (TOG), 37(4):1-11, 2018.
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9701-9707. IEFF, 2020.
Chuang Gan, Xiaoyu Chen, Phillip Isola, Antonio Torralba, and Joshua Tenenbaum. Noisy agents: Self- supervised exploration by predicting auditory events. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9259-9265. IEEE, 2022.
Ruohan Gao and Kristen Grauman. 2.5 d visual sound. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 324-333, 2019.
Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency. In 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 15490-15500. IEEE, 2021.
Ruohan Gao. Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35-53, 2018.
Rishabh Garg, Ruohan Gao, and Kristen Grauman. Geometry- aware multi-task learning for binaural audio generation from video. In BMVC, 2021.
Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. In Proceedings of the IEEE/CVF Conference Computer Vision and Putiern Recognition, pages 3497-3506. 2019.
Peng Guan, Alexander Weiss, Alexandru O Balan, and Michael J Black. Estimating human shape and pose from single image. 2009 IEEE 12th International Conference on Computer Vision, pages 1381 1388. JEEE, 2009.
Claudia Hendrix and Woodrow Barfield. The sense of presence within auditory virtual environments. Presence: Teleoperators and Virtual Environments, 5(3):290-301, 1996.
Di Hu, Rui Qian, Minyue Jiang. Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self- supervised audiovisual matching, Advances in Neural Information Processing Systems, 13:10077-10087, 2020.

---

Haikun Huang, Michael Solah. Dingzeyu Li, and Lap-Fai Yu. Audible panorama: Automatic spatial audio generation for panorama imagery. In Proceedings of the 2019 CHIl conference on human factors in computing systems, pages 1-11, 2019.
Hao Jiang, Calvin Murdock, and Vamsi Krishna Ithapu. Egocentric deep multi-channel audio-visual active speaker localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- 2022.
tion, pages
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7122-7131, 2018.
Bruno Korbar, Du Tran, and Lorenzo Torresani, Cooperative learning of audio and video models from self- supervised synchronization. Advances in Neural Information Processing Systems, 31, 2018.
Hsin- Ying Lce, Xiaodong Yang, Ming- Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. Dancing music. Advances in neural information procesing systems, 32, 2019.
Dingzeyu Li. Timothy R Langlois, and Changxi Zheng. Scene-aware audio for 360 videos. ACM Transactions on Graphics (TOG), 37(4): 12, 2018.
Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
Yu-Ding Lu, Hsin- Lee, Hung- Yu Tseng, and Ming-Hsuan Yang. Self-supervised audio spatialization with correspondence classifier. In 2019 IEEE International Conference on Image Processing (ICIP), pages 3347-3351. IEEE, 2019.
Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 218-234. Springer. 2022.
Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. Advances in neural information processing systems, 31, 2018.
Pedro Morgado. Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. Advances Neural Information Processing Systems, 33:4733-4744, 2020.
Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visue instance discrimination. Pru- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12934-12945, a02
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12475-12486, 2021b.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV), pages 631-648, 2018
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralha. Ambient sound provides supervision for visual learning. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14 2016, Proceedings. Part I 14, pages 801-816. Springer, 2016.
Mandela Patrick, Yuki M. Asano, Polina Kuznetsova, Ruth Fong, João Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations, 2021.
Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 292-308. Springer, 2020.
Alexander Richard, Dejan Markovic, Israel D Gebru, Steven Krenn, Gladstone Alexander Butler, Fernando Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. International Conference on Learning Representations, 2021.
Alexander Richard, Peter Dodds, and Vamsi Krishna Ithapu. Deep impulse responses: Estimating and parameter- izing filters with deep networks. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3209-3211 3213. IEEE, 2022.
12

---

P. N. Samarasinghe and T. D. Abhayapala. 3D spatial soundfield recording over large regions. In Proc. Int. Workshop on Acoustic Signal Enhancement (IWAENC), 2012.
Lauri Savioja, Jyri Huopaniemi, Tapio Lokki, and Ritta Väänänen. Creating interactive virtual acoustic environments. Journal the Audio Engineering Society, 4779):675-705 1999.
Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part III 16, pages 436 454. Springer, 2020.
Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. Proceedings of the IEEE/CVF Conference on Computer Vixion and Puttern Recognition, pages 2745-2754, 2021.
Aäron van Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalch- brenner, Andrew Senior, and Koray Kavukeuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop, pages 125-125,20 2016.
E. G. Williams. Fourier Acoustics. Academic Press, 1999.
Fanyi Xiao, Yong Jae Lee. Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020.
Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15485-1544 15494, 2021.
Ryuichi Yamamoto, Eunwoo Song, Min-Jae Hwang, and Jae-Min Kim. Parallel waveform synthesis based on generative adversarial networks with voicing-aware aware conditional discriminators. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6039 6043, IEEE,
2021.
Karren Yang. Bryan Russell, and Justin Salamon. Telling left from right: Learning spatial conresoondence of sight and sound. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9932-9941, 2020.
Karren Yang. Dejan Markovic, Steven Krenn, Vasu Agrawal, Alexander Richard. Audio visual speech codecs: Rethinking audio-visua speech enhancement by re-synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2022.
Youngwoo Yoon, Bok Cha, Joo- Moo-Hang Lee, Minsu Jang, Jacycon Lec, Jachong Kim, and Geehyuk Lec. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6):1 16, 2020.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference computer vision (ECCV), pages 570-586, 2018.
Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu. Avatarrex: Real-time expressive full-body avatars. ACM Transactions on Graphics (TOG), 42(4), 2023.
Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-sterco: Visually guided stereo- phonic audio generation by associating source separation. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XII 16, pages 52-69. Springer, 2020.
13


Reviews:
[{"review_id": "0xXxBaLIK7", "reviewer": "Reviewer_1NH5", "summary": "The paper tackles the task spatializing a mixture of speech and body sounds at different points in a sphere around a human body without explicitly recording or knowing the sound source location. Towards that goal, the paper captures a new dataset of humans speaking and making different body sounds in a very controlled and extensive setup. The paper proposes a model that takes as input the time-warped audio recorded by a VR headset and the body pose sequence, and renders the audio at different points on the sphere conditioned on the azimuth and elevation of the points. Furthermore, the paper proposes a novel and intuitloss function to accurately model both speech and short-lived body sounds. The paper compares its model with a heurisitcal baseline, evaluates different versions of the loss function, and also reports results for different microphone channel count and an ablation of the model. \n\n\nPost author-reviewer discussion: I have read the rebuttal. The responses are detailed and answer my questions. I have increased my score.", "questions": "I would like to request the authors to address the questions/concerns I raised in 'Weaknesses', in the rebuttal.", "limitations": "The paper discusses its limitations, and I can't think of any serious negative societal implications of the work that need to be discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed task is interesting.\n\n2. The proposed loss is intuitive and seems to be well-designed.\n\n3. The dataset could also be a valuable contribution for the community.", "weaknesses": "1. a) The dataset capture seems to be done in just 1 environment. There is no discussion or experiments on the model's generalization to other environments.\n   b) The capture setup seems to be pretty complicated and could limit the applicability of the model to novel environments if training needs to be redone for new environments.\n\n2. a) The capture scene seem to be simple in the sense that it doesn't contain other objects, is single-room in nature, etc. How would the model fare for more complex realworld scenes?\n\n    b) Would the model be able to accurately render sounds at locations that don't have a path of direct sounds from the source? \n\n3. How is the source location in L150-1 determined?\n\nMinor:\n1. The shift-l2 loss doesn't seem to perform by itself in most cases (as also pointed out by the authors) even though the design is pretty intuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Questions/concerns:\n\n1. a) & b) Generalization to other environments\n\nWe would like to clarify that we are considering a VR telepresence application where people interact in a VR space as full-body avatars. In this scenario we can distinguish between two environments: (1) the real physical environment the user is in while driving their avatar, and (2) the virtual space where the interaction is taking place.\nFor (2), it is desirable to model the spatial body sounds in an anechoic environment, which allows to add arbitrary room reverberations using traditional signal processing methods (e.g., convolution with room impulse responses).\nFor (1), the situation is more challenging: a user’s real environment (the “input environment”) can be noisy and reverberant. There is a huge corpus of research around denoising and we are not addressing the problem in this paper. Yet, we did a test to see how our model behaves with inputs from a highly reverberant room. An example is shown in the rebuttal PDF. The example shows that, although the model is trained on anechoic data, the origin of sound is correctly spatialized in the reverberant case as well. However, we note that high levels of undesired reverberation are present, and in general a dereverberation step would be required in this application scenario.\n\n2. a) Simple scene with no objects\n\nSince this is the first work to address the problem of sound field prediction from egocentric data and considering that humans are main sources of sound in a telepresence system, we decided to focus on speech and body sounds in the current work. We plan to extend the proposed framework to human-object interaction by collecting a new dataset consisting of people interacting with different kinds of objects in future work.\n\n2. b) Sound field reconstruction inside regions without free-field propagation? \n\nSince the network is trained only on observations that surround the acoustic scene (the microphone array encompasses the capture space), reconstructions inside any region that violates the free-field assumption are not going to be accurate. We mention this as a limitation of the proposed method in the Conclusions section. \n\n3. How is the source location in L150-1 determined? \n\nSource localization is performed implicitly by the network. We do perform the geometric warping to help the network with the sound propagation delay, but since the origin of sound is unknown, we perform the warping from six body joints that are most frequent sound emitters and feed all warped signals as inputs to the network. More details can be found in the section A Time Warping of supplementary material.\n"}, {"review_id": "A8kj4chf7h", "reviewer": "Reviewer_of5R", "summary": "This work introduces an approach for generating spatial audio from a 3D human pose and microphones placed close to the subject’s head, e.g. on a VR/AR headset, as is the case here. The authors first collect a multimodal dataset that contains 3D bodies and audio, recorded using multiple Kinects for body tracking and 345 microphones for audio acquisition. The dataset contains multiple participants in different outfits and poses, generating different sounds following a pre-defined script. To generate spatial audio, the authors propose a model that receives as input the audio recorded from the headset microphones and the 3D body pose and predicts the audio signal recorded at a microphone of the spherical capture array. Using the audio at each microphone and the locations of each microphone, the authors compute harmonic sound field coefficients that best represent the sound field. ", "questions": "- Do you use only body keypoints? Or do you include hand keypoints from OpenPose? \n- I believe that a comment or discussion on the granularity of the representation of the human body would be useful. The ablation clearly shows that a more detailed representation of the human body helps improve the quality of the predicted sound field. I would like to hear the authors' answer on the following questions:\n  - Is the robustness of the estimated keypoints important? Do noisy keypoints significantly affect the quality of the sound field?\n  - Is the lower body important for sound quality? \n  - Would a finer representation of the subject help? For example, would including hand or face keypoints from OpenPose help?\nSome implementation details that should be clarified:\n- How was the size of the window in loss function chosen?\n- How was the training sampling strategy selected? Maximization of GPU memory usage?\n\n- Finally, regarding the performance of the model, it would be useful to have a visualization of the distribution of errors around the sphere of microphones. Is there a subset of microphones where the audio signal is better reconstructed?\n", "limitations": "The authors present the limitations of the proposed dataset and method. They acknowledge that the current work is limited to modeling sound effects for a single person in an artificial scene and that it cannot model sound close to the body.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Important problem with large impact: The authors tackle an important problem for AR/VR applications that has not yet received a lot of attention. Generating accurate sound fields promises to increase immersion and the realism of virtual experiences. The proposed dataset will be an important benchmark for the community and open up new and exciting avenues of research.\n- Novelty: The proposed dataset is an important contribution with unique data that are not publicly available. \n- A loss that is well-motivated by prior-work and adapted to the problem structure that leads to clear benefits.\n- A simple model that merges different types of information, using well-proven components.\n- Writing: The paper is well-written and easy to follow.\n- Experiments: The experimental analysis is clear, highlights the effects of each design choice and provides insights into what makes the method work. \n", "weaknesses": "- The related work section lacks a discussion of recent methods that model sound fields using neural network fields, such as:\n  - [Learning Neural Acoustic Fields](https://www.andrew.cmu.edu/user/afluo/Neural_Acoustic_Fields/).\n  - [INRAS: Implicit Neural Representation for Audio Scenes](https://openreview.net/forum?id=7KBzV5IL7W)\n\nThese could also be useful baselines for the model, both in terms of accuracy and in terms of computation compared to the proposed method. Furthermore, the following datasets could be a useful discussion point for the related work section:\n  - SoundSpaces: Audio-Visual Navigation in 3D Environments: A discussion of whether a similar dataset as the one proposed in this work could be constructed by simulating audio propagation from artificial bodies.\n  - Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis: I believe the related work section could benefit from a comparison with this paper in terms of granularity of pose representation and sound field, number of persons and type of conversations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\n\nWe thank the reviewer for suggestions that would help make the related works section more comprehensive. We will add a paragraph that discusses mentioned works. NAF and INRAS both leverage implicit representations for sound field modeling, but they focus on room impulse responses and require precise sound source locations. SoundSpaces also focuses on room acoustics while we’re interested in audio propagation around the human body insulated from room reverberation. Besides, SoundSpaces is a synthetic dataset and the fidelity of the sound field may be bounded by the adopted room acoustics modeling algorithm. Talking With Hands 16.2M is a large-scale dataset of body-finger motion and conversational speech. Apart from body poses, the dataset captures finger motion as well. Talking With Hands 16.2M uses only  two directional microphones, which are not enough for sound field modeling. \n\nQuestions:\n\n1. Are only body keypoints used? Would including hand or face keypoints help?\n\nWe are using body keypoints only in this work. While we plan to include detailed finger pose as well, this will require improvements to the capture system. We found the quality of finger tracking from the five Kinects used in our setup insufficient to be helpful.\n\n2. Impact of noisy keypoints & importance of lower body?\n\nIn the attached PDF we added an ablation study with noisy keypoints (Gaussian noise with 10cm standard deviation) and without the lower body. As expected, performance degrades in both cases, although the noise has a bigger impact overall, while lower body influence is more marginal. \n\n3. Size of the window in loss function?\n\nGiven the size of the capture dome (diameter = ~2.44m, microphone tip-to-tip) there exist an upper limit of the sound propagation delay that can occur in this setting between two extreme positions in the dome, which is around 340 samples (48000 * 2.44 / 343). Since the network deals with the time-differences-of-arrival between the input headset microphones and output capture dome microphones, the actual upper limit that the network needs to account for is roughly the half, i.e., ~170 samples. We choose 128 as the closest power of two.   \n\n4. Sampling strategy during training – maximization of GPU memory usage?\n\nYes, the sampling strategy maximizes the GPU memory usage: for each input sequence we randomly select 38 out of 345 microphones on the capture dome; selecting a higher number leads to out-of-memory failures during training.  \n\n5. Spatial distribution of errors?\n\nWe added a visualization of the distribution of errors around the sphere of microphones to the attached PDF. All participants were facing mostly towards -25 deg azimuth (opposite to the entrance of the dome) and we notice the best performance is achieved in that region in terms of SDR and phase loss. The amplitude loss shows the opposite pattern, but we note that, unlike phase that does not depend on signal energy and SDR that is a normalized value, amplitude loss may be higher in the frontal region simply because the signal energy for speech is generally higher in the frontal direction. \n"}, {"review_id": "GfhvFWtg8D", "reviewer": "Reviewer_k1Fj", "summary": "The paper introduces a novel model for a novel task to render a 3D spatial sound from human body motion and audio collected by headset microphones. The authors also present a novel dataset containing human body motion and audio for this task. The model takes encoded audio, pose features, and target microphone position to render the corresponding spatial sound. A novel shifted-l2 loss is also proposed to address the issue of big spikes in l2 error due to the small shifts of impulsive signals. Experiments and results demonstrate the effectiveness of the proposed method.", "questions": "1. Why do you use only 1s audio clips during training? Is there a bottleneck that avoids having a longer audio sequence during training? Or is it simply because of the high sampling rate (48khz)?\n\n2. Would it be possible to have data with multi-person and human-object interaction in the future? What might be the challenges to record these data?\n\n3. I am interested in the hyper-parameter choices for the shifted-l2 loss. How is the L=128 varied if the sampling rate differs?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The motivation and objectives are clear. To my best knowledge, it is indeed the first paper that aims to render 3D spatial sound from a human body pose.\n\n2. The dataset itself is valuable. It could unlock various interesting research projects in the future.\n\n3. The system is simple but effective. The proposed shifted-l2 loss is interesting and could be useful in other audio applications as well.\n\n4. Analyses of the roles of body pose and head-mounted microphones are well-done in the ablation studies.\n\n5. The phase is important and I am glad that the authors consider it in the optimization.", "weaknesses": "1. While extensive quantitative evaluations and analyses are done, it is often challenging to understand the gap in the perceptual level. It would be great if there is a subjective evaluation to demonstrate both the importance of the proposed task/approach and the contribution of the technical components.\n\n2. It would also be nice to have more samples in the supplementary material for a subjective evaluation of the proposed approach.\n\n3. I wonder whether any failure cases are appearing in the current exploration stage.\n\n4. audio and visual data time synchronization is often challenging but the details to achieve that are missing.\n\n5. As the authors mentioned in the limitation, the current approach could not handle spatial sound for human-object interaction, mainly for far-field modeling, and require intensive computation resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\n\n1. Subjective evaluation \n\nWe agree that subjective evaluations are important. At the same time, the proposed work deals with sound spatialization, and it is not straightforward to perceptually evaluate spatialization quality. Study participants should be able to see the scene in 3D and an effective way to evaluate the correctness of the sound spatialization should be devised. To our knowledge there is no standard procedure for this kind of evaluations. Additionally, complex user studies in VR are not repeatable by the research community and therefore add limited value for other researchers.\n\n2. & 3. Additional samples + failure cases \n\nWe agree that additional samples could be useful for the readers. We will include more examples in the supplementary material of the final manuscript, also showcasing the cases where the model does not perform well. \n\n4. Audio-visual time synchronization\n\nThe audio system is a distributed array of MADI-based analog-to-digital converters. Each analog-to-digital converter is phase-locked to each other via the same WordClock signal. The video system is a distributed array of Kinect Azure cameras. Each Kinect is set to “subordinate mode” where the unit will only take an image when they get a digital high signal from the sync-in port. The Kinect array is in a “star” pattern where one master unit sends a digital high signal out to a distribution amplifier. The master Kinect digital high signal is then propagated to all of the subordinate Kinects. We also record the digital high signal into our analog-to-digital audio converters so we have correspondence between a visual frame and an audio sample. We will include these technical details in the supplementary material of the final manuscript.\n\nQuestions:\n\n1. Why 1 second duration?\n\nBecause of the high sampling rate (48 kHz) and the fact that for each input sequence (1-7 headset mics) we predict multiple output signals (38 randomly selected mics from the capture stage), there is a memory bottleneck that prohibits using longer sequences. One-second-long sequences worked fine in our case. We would also like to point out that since the receptive field of the network is 2047 samples, there are no benefits of using a longer sequence. \n\n2. multi-person and human-object interaction data?\n\nWe plan to extend the proposed framework to human-object interaction by collecting a new dataset consisting of people interacting with different kinds of objects. At least in theory, no network architecture or capture system changes are needed for this extension. On the other hand, multi-person interaction is more challenging: no information about the sound field in-between the people would be available since the microphones surround the scene. Adding additional microphones in-between the participants would be challenging and would significantly restrict the body movement.  \n\n3. Hyper-parameter choice: why L = 128 and its dependence on sampling frequency?\n\nGiven the size of the capture dome (diameter = ~2.44m, microphone tip-to-tip) there exist an upper limit of the sound propagation delay that can occur in this setting between two extreme positions in the dome, which is around 340 samples (48000 * 2.44 / 343). Since the network deals with the time-differences-of-arrival between the input headset microphones and output capture dome microphones, the actual upper limit that the network needs to account for is roughly the half, i.e., ~170 samples. We choose 128 as the closest power of two.   \n\n"}, {"review_id": "4GdIG1VTCq", "reviewer": "Reviewer_R8ko", "summary": "This paper deals with the problem of predicting sound fields around human bodies and proposes a method that exploits binaural audio signals and human body motions. The proposed method first encodes binaural audio signals and human poses and then decodes them into audio signals that are supposed to be captured by surrounding microphones. Finally, the proposed method renders the predicted signals into sound fields. The proposed method also introduces a loss function that is robust against small time shifts of time-domain audio signals. Experimental evaluations with originally collected datasets demonstrate that the proposed method captured the ground-truth sound fields reasonably well and it requires full body poses instead of only head poses for obtaining good predictions.", "questions": "None.", "limitations": "I could not find any negative societal impacts of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The problem dealt with in this paper is novel as far as I know. Although sound field prediction is one of the standard tasks in audio signal processing, sound field prediction with blindly moving microphones would be novel and challenging. This paper solves this problem by (1) assuming that sound is emitted from a human and the human wears microphones and (2) integrating pose estimation of the target human.\n\n2. The proposed method is technically sound. Since the current problem setting is sensitive to time shifts, the proposed method handles audio signals in the time domain, which is not so popular in multi-modal and cross-modal analysis with deep learning. The proposed method achieves time-domain processing by incorporating WaveNet-like decoders.\n\n3. The newly introduced loss function is also technically sound. Time-domain signals are more sensitive to time shifts than frequency-domain ones. The proposed loss function named the shift-\\ell_2 loss effectively mitigates this problem while maintaining its discriminability.\n\n4. The experimental evaluations demonstrate that\n(1) the proposed method reasonably worked well,\n(2) full-body poses contributed to the performance improvement of sound source localization compared with head poses, and\n(3) the proposed loss function was effective for non-speech sounds.", "weaknesses": "1. The current experimental evaluations do not contain comparisons with other existing methods. I understand that the current problem setting is novel and thus there are no existing methods that can be directly applied to this problem. However, several previous methods for sound field prediction have already been known, and experimental comparisons with one of those methods will be informative for readers.\n\n2. The problem setting seems to be too extreme. Sound field prediction and reconstruction are one of significant problems in audio signal processing. However, the current problem setting requires (1) humans as sound sources, (2) microphones worn by humans, and (3) external video cameras. In particular, the first constraint seems to be too strict. Some justifications and potential applications will be required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Comparison with previous methods for sound field prediction\n\nWe recognize that a comparison with other methods would be beneficial. However, as pointed out by the reviewer as well, the sound field reconstruction from egocentric data is a novel problem and no existing methods can be directly applied to this setting. Moreover, we are not aware of any sound field prediction methods that can be adapted in a straightforward fashion without involving substantial modifications. If the reviewer has any specific papers in mind, we are happy to consider them for the final manuscript. \n\n2. Problem setting – justifications and potential applications\n\nWe are considering VR telepresence application where people interact in VR space as full-body avatars. In this application the constraint 2 (microphones worn by humans) is easily satisfied as it is common for VR headsets to employ microphone arrays. Regarding the constraint 1 (humans as sound sources): since this is the first work to address the problem of sound field prediction from egocentric data and considering that humans are main sources of sound in a telepresence system, we decided to focus on speech and body sounds in the current work. We plan to extend the work to human-object interaction in future. Finally, regarding the constraint 3 (external video cameras), the proposed method requires pose as input, which can be obtained from external cameras, or from the headset itself (note: the current generation of headsets such as Meta Quest 2 are able to track the upper body / hands while the lower body remains a challenge). We will more clearly highlight this application scenario in the final manuscript. \n\n"}]

Author rebuttal:
We would like to thank all the reviewers for taking the time to review the paper and their help in improving the quality of the manuscript with their constructive feedback. We replied to each reviewer’s comments and suggestions individually below. Attached to this rebuttal is also a PDF that includes: (1) a figure with examples of model outputs using data from anechoic chamber and a reverberant room; (2) a table with ablation study that shows impact of keypoint robustness and granularity of the representation of the human body; and (3) a figure showing the distribution of errors around the sphere of microphones.


Meta-review: This paper received all accept ratings and the provide rebuttal resolved the concerns raised by the reviewers. In particular, the pros include: (1) solving a novel and interesting problem, (2) contributing a new dataset, (3) a new and effective loss, (4) good writing. Given that, I recommend accept for spotlight.
Decision: Accept (spotlight)