Title: Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio
Venue: NeurIPS 2023
Accepted: yes
Keywords: sound field

spatial audio

virtual humans

human body

body modeling
Abstract: While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community. To close this gap, we present a model that can generate accurate 3D spatial audio for full human bodies. The system consumes, as input, audio signals from headset microphones and body pose, and produces, as output, a 3D sound field surrounding the transmitter's body, from which spatial audio can be rendered at any arbitrary position in the 3D space. We collect a first-of-its-kind multimodal dataset of human bodies, recorded with multiple cameras and a spherical array of 345 microphones. In an empirical evaluation, we demonstrate that our model can produce accurate body-induced sound fields when trained with a suitable loss. Dataset and code are available online.

Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio

Abstract

While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community. To close this gap, we present a model that can generate accurate 3D spatial audio for full human bodies. The system consumes, as input, audio signals from headset microphones and body pose, and produces, as output, a 3D sound field surrounding the transmitter's body, from which spatial audio can be rendered at any arbitrary position in the 3D space. We collect a first-of-its-kind multimodal dataset of human bodies, recorded with multiple cameras and a spherical array of 345 microphones. In an empirical evaluation, we demonstrate that our model can produce accurate body-induced sound fields when trained with a suitable loss. Dataset and code are available online. 1

1 Introduction

Throughout all of our history, humans have been at the center of the stories we tell: starting from early cave paintings, over ancient tales and books, to modern media such as movies, TV series, computer games, or, more recently, virtual and augmented realities. Not surprisingly, digital representations of the human body have therefore long been a research topic in our community Guan et al. [2009], Loper et al. [2015], Kanazawa et al. [2018]. With the rise of computer-generated imagery in the film industry and the availability of powerful computing on consumer devices, visual representations of 3D human bodies have seen great advances recently, leading to extremely lifelike and photorealistic body models Bagautdinov et al. [2021], Zheng et al. [2023].

However, while the visual representation of human bodies has made steady progress, its acoustic counterpart has been largely neglected. Yet, human sensing is intrinsically multi-modal, and visual and acoustic components are intertwined in our perception of the world around us. Correctly modeling 3D sound that matches the visual scene is essential to the feeling of presence and immersion in a

…

input mics

(e.g., from VR headset)

body pose

audio

encoder

pose

encoder

audio

decoder

target mic position

on sphere

sound field

renderer

sound field

K-th order ambisonics

predict sound for N

target microphones

…

…

a1

a2

ai

aN

(0i, Φi)

3D environment Hendrix and Barfield [1996]. In this work, we propose a method to close this gap between the visual and acoustic representation of the human body. We demonstrate that accurate 3D spatial sound can be generated from head-mounted microphones and human body pose.

In particular, we consider an AR/VR telepresence scenario where people interact as full-body avatars. In this setting, available inputs include egocentric audio data from the head-mounted microphones (it is common for headsets to employ microphone arrays), as well as body pose used to drive the avatar, obtained either using external sensors or from the headset itself (the current generation of headsets are able to track the upper body / hands). The goal is to model the user-generated sound field so that it be correctly rendered at arbitrary positions in the virtual space.

Existing approaches to sound spatialization, both in traditional signal processing Savioja et al. [1999] and in neural sound modeling Richard et al. [2021, 2022], operate under the strict assumption that the sound to be spatialized has a known location and is also recorded at this location, without interference through other sounds. These assumptions are in stark contrast with the task we address in this work. First, the location of the sound is unknown, i.e. , a finger snap could come from the right hand as well as the left hand. Second, the sound can not be recorded at the location where it is produced. In particular, the microphones are typically head-mounted (as in an AR/VR device), such that finger snaps or footsteps are not recorded at the location of their origin. Third, it is usually not possible to record a clean signal for each sound source separately. Body sounds like footsteps or hand-produced sounds often co-occur with speech and are therefore superimposed in the recorded signal.

To overcome these challenges, we train a multi-modal network that leverages body pose to disambiguate the origin of different sounds and generate the correctly spatialized signal. Our method consumes, as input, audio from seven head-mounted microphones and body pose. As output, the sound field surrounding the body is produced. In other words, our method is able to synthesize 3D spatial sound produced by the human body from only head-mounted microphones and body pose, as shown in Figure 1.

Our major contributions are (1) a novel method that allows rendering of accurate 3D sound fields for human bodies using head-mounted microphones and body pose as input, (2) an extensive empirical evaluation demonstrating the importance of body pose and of a well-designed loss, and (3) a first-ofits-kind dataset of multi-view human body data paired with spatial audio recordings from an array with 345 microphones. We recommend watching the supplemental video before reading the paper.

2 Related Work

Audio-Visual Learning. In our physical world, audio and video are implicitly and strongly connected. By exploiting the multi-modal correlations, plenty of self-supervised learning methods are proposed for representation learning and significantly outperform those approaches learning from a single modality Owens et al. [2016], Korbar et al. [2018], Alwassel et al. [2020], Tian et al. [2020], Xiao et al.

[2020], Patrick et al. [2021], Morgado et al. [2020, 2021a,b]. Besides, such a close cross-modality correlation is also shown to boost the performance in audio-visual navigation Chen et al. [2020, 2021], Gan et al. [2020, 2022], speech enhancement Owens and Efros [2018], Yang et al. [2022], audio source localization Qian et al. [2020], Hu et al. [2020], Tian et al. [2021], Jiang et al. [2022], Mo and Morgado [2022] and separation Afouras et al. [2018], Ephrat et al. [2018], Zhao et al. [2018], Gao et al. [2018], Gao and Grauman [2021]. Among these works, the most related papers to ours focus on audio-driven gesture synthesis Ginosar et al. [2019], Lee et al. [2019], Ahuja et al. [2020], Yoon et al. [2020], which aims to generate a smooth sequence of the human skeleton coherent with the given audio. While Ginosar et al. [2019], Ahuja et al. [2020], Yoon et al. [2020] focus on the translation from speech to conversational gestures based on the cross-modal relationships, Lee et al. [2019] explore a synthesis-by-analysis learning framework to generate a dancing sequence from the input music. Different from them, we are interested in the opposite direction, i.e. , reconstructing the 3D sound field around the human from the corresponding body pose.

Visually-guided Audio Spatialization. Visually-guided audio spatialization is to transform monaural audio into 3D spatial audio under the guidance of visual information. There is an increasing number of data-driven methods for this task, where crucial spatial information is obtained from the corresponding videos. Morgado et al. [2018], Li et al. [2018] and Huang et al. [2019] leverage 360 ◦ videos to generate the binaural audio and concurrently show the capability of localizing the active sound sources. Besides, Gao and Grauman [2019] regards the mixture of left and right channels as pseudo mono audio and utilizes the visual features to predict the left-right difference with a learnable U-Net. After that, a line of papers Lu et al. [2019], Yang et al. [2020], Zhou et al. [2020], Garg et al. [2021], Xu et al. [2021] follow a similar framework and achieve better spatialization quality with different novel ideas. However, these methods fail to model time delays and reverberation effects, making the binaural audio less realistic. As for this problem, Richard et al. [2021] propose to spatialize speech with a WaveNet van den Oord et al. [2016] conditioned on the source and listener positions explicitly. Despite the remarkable progress in audio spatialization, these approaches are all limited to binaural audio or first-order ambisonics and thus fail to recover the whole sound field. In contrast, our method strives to generate the whole 3D sound field by using the corresponding 3D human body pose.

3 Dataset

Setting. In this work we focus on modeling speech and body generated sounds considering an AR/VR telepresence scenario. In general, it is desiderable to model these sounds in an anechoic environment, i.e. , without acoustics of a specific room, which allows to add arbitrary room reverberations using traditional signal processing methods ( e.g. , convolution with room impulse responses). For this reason we collect the dataset in an anechoic chamber. We note that the user's real environment can be noisy and reverberant, which could affect the input audio signals from the headmounted microphones. There is a huge corpus of research around denoising and dereverberation and we are not addressing these problems in this paper.

Data capture. We capture a corpus of paired audio-visual data in an anechoic chamber with 5 Kinects for body tracking and 345 microphones arranged on a sphere around the center of the capture stage. The cameras capture at 30 frames per second and audio is recorded at 48kHz. The Kinects and 345 microphones are co-calibrated such that they share the same world coordinate system. During the capture, the participants are standing in the center of the stage while wearing a head-mounted device with seven microphones to gather input audio.

In total, we collect 4.4h of human body sounds and speech from eight different participants. We capture each participant in three different settings: standing and wearing light clothes like a T-Shirt that does not produce a lot of noise while moving; wearing heavy clothes like jackets or coats that

produce audible noise while moving; and sitting in light clothing. For each setting, participants are instructed to perform a variety of body sounds such as clapping, snapping, tapping their body, footsteps, or scratching themselves, as well as speaking. The full capture script can be found in the supplemental material. Overall, 50% of the data are non-speech body sounds and 50% are speech. Note that handling both kinds of data poses an additional challenge for our approach: speech is typically a dense and structured signal, while body sounds are sparse, impulsive, and contain rich high-frequency components.

Data processing. We extract body poses from each camera using OpenPose Cao et al. [2017] and obtain 3D keypoints via triangulation of the poses from the five camera views. The audio signals from all microphones (345 microphones on the spherical array and seven head-mounted microphones) and the Kinects are time-synchronized using an IRIG time code. Technical details about audio-visual time synchronization can be found in the supplemental material.

4 Pose-Guided 3D Spatial Audio Generation

Producing full 3D sound fields inherently requires multimodal learning from both audio input and pose input, as neither modality alone carries the full spatial sound information. We formalize the problem and describe the system outlined in Figure 1, with a focus on network architecture and loss that enable effective learning from the information-deficient input data.

4.1 Problem Formulation

Let a 1: T = ( a 1 , . . . , a T ) be an audio signal of length T captured by the head-mounted microphones, where T indicates the number of samples, and each a t ∈ R C in represents the value of the waveform at time t for each of the C in input channels. Let further p 1: S = ( p 1 , . . . , p S ) be the temporal stream of body pose, where each p s ∈ R J × 3 represents the 3D coordinates of each of the J body joints at the visual frame s . Note that audio signals a 1: T and pose data p 1: S are of the same duration, but sampled at a different rate. In this work, audio is sampled at 48kHz and pose at 30 fps, so there is one new pose frame for every 1,600 audio samples.

Ideally, we would like to learn a transfer function T that maps from the head-mounted microphone signals and body pose directly to the audio signal s 1: T at any arbitrary coordinate ( x, y, z ) in the 3D space,

However, it is in practice impossible to densely populate a 3D space with microphones and record the necessary training data. Instead, our capture stage is a spherical microphone array and data can only be measured on the surface of this sphere, i.e. at microphone positions defined by their polar coordinates ( r, θ , φ ) . Since the radius r is the same for every point on the sphere, a microphone position is uniquely defined by its azimuth θ and elevation φ . 2 Leveraging the recordings of the microphones on the sphere as supervision, we model a transfer function

that maps from input audio a 1: T and body pose p 1: S to the signal recorded by microphone i which is located at ( θ i , φ i ) on the sphere.

In order to render sound at an arbitrary position ( x, y, z ) in 3D space, we predict s 1: T ( θ , φ ) for all microphone positions ( θ i , φ i ) on the sphere, and encode them into harmonic sound field coefficients Samarasinghe and Abhayapala [2012], from which the signal s 1: T can then be rendered at any arbitrary spatial position ( x, y, z ) , see Section 4.4 for details.

4.2 Network Architecture

In this section, we describe how we implement the transfer function defined in Equation (2). At the core of the model are an audio encoder and a pose encoder which project both input modalities onto a latent space, followed by an audio decoder that has a WaveNet van den Oord et al. [2016] like

Dilated

Conv1D

Conv1D

audio decoder blocks

Conv1D

j-th block

.

audio

encoder

pose

encoder

FC

target mic position

…

Dilated

Conv1D

Conv1D

Conv1D

mean

Conv1D

Conv1D

architecture and generates the target sound s 1: T at a given microphone position ( θ i , φ i ) . The model architecture is outlined in Figure 3.

As training data, we use recordings from the capture stage described in Section 3, where we have fully synchronized and paired data of headset input, body pose, and target microphones. In other words, our training data are tuples ( a 1: T , p 1: S , s 1: T ( θ i , φ i )) for each microphone position ( θ i , φ i ) on the spherical array.

Audio Encoder. Acoustic propagation through space at the speed of sound causes a time delay between the emitting sound source ( e.g. , a hand), the recording head-mounted microphones, and the actual target microphone on the spherical microphone array. It has been found beneficial to apply time-warping , i.e. , to shift the input signal by this delay, before feeding it into a network Richard et al. [2021]. In our case, as sound might emerge from various locations on the body, we create a copy of the input signal for multiple body joints and shift the input as if it originated from each of these joints. We then concatenate all these shifted input copies along the channel axis and feed them into a linear projection layer to obtain the audio encoding f a that contains information about the time-delayed audio signal from all possible origins, see the supplemental material for details.

Pose Encoder. While a person is making some sound, their body poses p 1: S give strong cues when and where the sound is emitted. Additionally, sound interacts with the body and propagates differently through space depending on body pose. Hence, the signal s 1: T at a target microphone position ( θ i , φ i ) depends on the body pose at that time. For the input pose sequence p 1: S , the 3D coordinates of each body joint are first encoded to latent features and the temporal relation is aggregated via two layers of temporal convolutions with kernel size 5 . Then, we concatenate all body joint features and use an MLP to fuse them into a compact pose feature f p .

Target Microphone Position. In addition to audio and pose encodings, the decoder depends on the position ( θ i , φ i ) of the target microphone. We first map the microphone location into a continuous Cartesian space

( x, y, z ) = ( cos( θ i ) sin( φ i ) , sin( θ i ) sin( φ i ) , cos( φ i ) ) and project it to a higher-dimensional embedding using a learnable, fully-connected layer.

Pose encoding and target microphone position encoding are then concatenated into a conditioning feature vector f c which is upsampled to 48kHz and, together with the warped input audio, passed to the audio decoder.

Audio Decoder. Following prior works van den Oord et al. [2016], Richard et al. [2021], we stack M WaveNet-like spatial audio generation blocks as illustrated in Figure 3, in which the input audio features f a are transformed into the desired output signal s 1: T ( θ i , φ i ) under the guidance of the conditioning pose and position features f c . Each block is a gated network combining information from audio and conditioning features. In the j -th block, the audio feature f j a passes through a dilated Conv1D layer and is added to conditioning features processed by Conv1D layers in both filter and gate branches, which are then integrated after the tanh and sigmoid activation functions:

where ∗ denotes a convolution operator, /circledot indicates element-wise multiplication, σ is a sigmoid function, j is the index of the decoder block, f and g denote filter and gate, respectively, and W , V are both learnable convolution filters. A ReLU activation is applied to z which is then processed by two separated Conv1D layers. One layer predicts the residual audio feature to update f j a to f j +1 a for the next generation block, and another is responsible for producing the output audio feature f j out . All output features f 1 ,...,M out from M blocks are average pooled and then decoded to raw audio waves ˆ s 1: T ( φ i , θ i ) via two Conv1D layers. Finally, for training, the predicted audio is compared with the ground-truth recordings s 1: T ( φ i , θ i ) using loss functions L (see Section 4.3).

4.3 Loss Function

Optimizing for correct spatialization of body sounds is challenging due to both the nature of the task (correct spatialization requires correct time alignment), and the nature of the signals we are dealing with (a mix of dense and structured speech with sparse, impulsive sounds such as clapping and snapping). To avoid speech distortions, a loss on the amplitude spectrogram is generally preferred. On the other hand, such loss is unable to achieve accurate temporal alignment. Alternatively, the time domain /lscript 2 loss has been successfully used for neural synthesis of binaural speech Richard

a(t)

n + 1

n+2

L

e(n)(-L)

Ie2n)(0)

em)(L)

[T)(.()(..(T-)(]

N1

a(t)

n=0

0

et al. [2021]. However, in our experiments we found that /lscript 2 tends to highly attenuate impulsive sounds. One issue is that small shifts of impulsive signals cause big spikes in /lscript 2 error in combination with the fact that available information is not sufficient for a perfect sample-level alignment (body keypoints give only a rough indication of a possible origin of sound). To address this issue we propose a variation of /lscript 2 that is more forgiving to small shift offsets. The loss, referred here a shift/lscript 2 , is computed over sliding windows with higher penalty applied to higher offsets.

We start by dividing the estimated signal ˆ a ( t ) into N segments of size L as illustrated in Figure 4. For each segment n , we compute a weighed /lscript 2 error against a sliding window from a reference signal a ( t ) , with sliding offset τ going from -L and L . The error is computed as

∣ ∣ where σ indicates the standard deviation of the signal, and δ is small value used to avoid numerical issues. The result is a set of 2 L +1 values L ( n ) 2 = [ /lscript ( n ) 2 ( -L ) , . . . , /lscript ( n ) 2 (0) , . . . , /lscript ( n ) 2 ( L ) ] representing /lscript 2 errors normalized by the signals energy and computed over a set of shift offsets. After penalizing larger offsets by using a window function W = α (1 -[ w ( -L ) , . . . , w (0) , . . . , w ( L )]) , with w ( τ ) denoting the Blackman window of length 2 L +1 , and α being the penalty coefficient, we select the minimum value as min [ ( L ( n ) 2 +1)( W +1) -1 ] . By design this value can be 0 only if the /lscript 2 error is 0 at shift τ = 0 . Finally, the loss value is obtained by averaging the results of each of N segments

We use L = 128, α = 100 , δ = 0.001. To reduce the distortions of speech data, we combine shift/lscript 2 with a multiscale STFT loss Yamamoto et al. [2021] computed for window sizes of 256, 128, 64, 32.

4.4 Sound Field Rendering

Given the audio signals at each microphone and the knowledge of microphone locations, we encode the 3D sound field in the form of harmonic sound field coefficients following Samarasinghe and Abhayapala [2012]. The maximum harmonic order K is limited by the number of microphones N as K = /floorleft √ N /floorright1 . For our system with 345 microphones this allows sound field encoding up to 17 -th harmonic order. More details can be found in the supplemental material.

right

left

front

zoom-in

ground truth

no pose

headpose only

full body pose

?

Given the encoded harmonic sound field coefficients β nm ( τ , f ) , where τ and f are, respectively, time and frequency bins, the sound pressure at any point x = ( r, θ , φ ) in space 3 can be decoded as Williams [1999]

where k = 2 π f/v sound is the wave number, v sound is the speed of sound; Y nm ( θ , φ ) represents the spherical harmonic of order n and degree m (angular spatial component), and h n ( kr ) is n th-order spherical Hankel function (radial spatial component).

5 Experiments

Dataset. All the experiments are conducted on our collected dataset. We cut the recordings into clips of one second in length without overlapping, resulting in 15,822 clips in total. The dataset is partitioned into train/validation/test sets of 12,285/1,776/1,761 clips, respectively. The test set consists of 961 non-speech clips and 800 speech clips. To the best of our knowledge, this is the first

<latexi sh

1_b

64="0

Qv

BjA

GH

TMR

IPp

w

>

/

c

VDLS

N

FJ3U

2

du

o

gq

7KCf ym

Y

O

9

+

zn

8

5

k

r

X

W

E

Z

<latexi sh

1_b

64="M

PA

vqSD

532

kZGEL

>

B9X

c

V

J

gN

O

W

K

Cj

d

8R

0

p

uof

U

+Q

H

T

r

/

F

w

n

I

7

m

Y

y

z

<latexi sh

1_b

64="dm8TQy

q

j

S

FH

VKO/Y

E

>A

B

X

c

DJ

gN

GL

R

w

9

I

k

C

U

3

Zz

o

+v

7

n

r

5

0

W

f

2

p

P

u

M

<latexi sh

1_b

64="

NjFER

CHwg

y

d

5

>A

B/n

c

VDJS

O2

W

Q

pz

8

L

Kk

u

0YM

f

9

3q

G

Z

v

I

o

P

X

U

r

7

<latexi sh

1_b

64="McJz uX

VO

oYSjH2

vfTn

>A

B83

DL

gN

E

y

r

U

9

P

K

Q

I5

Z

m

d

p

7

w

F

k

G

q

W

0

C

R

/

+

<latexi sh

1_b

64="A7

+C2zVW

nT5

Gv

K

YQ

>

B83

c

DJSgN

EO

X

jP

d

R

L

Z

k

m

0

y

9

I

/

o

U

Mu

f

r

w

H

p

F

q

<latexi sh

1_b

64="WYqC

gTIE

FyJ+

8d

K0

>A

B

n

VDLS

N

r

fU

9

oP

2

5

H

m

v

Z

M

w

3

k

G

j

X

u

R

O

/

<latexi sh

1_b

64="Q

zwT

3

7k

R

DY

dE

I

m

C

>A

B9X

c

V

gM

PSF

v

WN

H

JU

8pO

Kf

0u

+

y

/

G

Z

q

2

r

L

o

5

n

j

non-speech

speech

# input mics

SDR

↑

amplitude

↓

phase

↓

SDR

↑

amplitude

↓

phase

↓

1

0

.

848

0

.

849

0

.

478

8

.

259

7

.

956

1

.

235

3

2

.

338

0

.

812

0

.

458

9

.

526

7

.

276

1

.

199

7

3

.

004

0

.

784

0

.

454

9

.

580

7

.

059

1

.

184

ground truth

right

left

front

zoom-in

1 input mic

7 input mics

dataset of its kind that contains multimodal data of speech and body sounds, which paves the way for a more comprehensive exploration of modeling the 3D spatial sound of humans.

Implementation Details. In the encoders, the pose is mapped onto a 32 -d embedding f p , while the audio input a 1: T is mapped to 128 -d features f a after time-warping. For the audio decoder, we use M = 10 blocks, in which each Conv1D layer has 128 channels and kernel size 3 , and the dilated Conv1D layer starts with a dilation size of 1 and doubles the size after each block. To transform the output features from 128 channels to a single channel audio, we employ two sequential Conv1D layers with ReLU activations. During training, we fetch 4 audio clips from the training set as a batch and randomly select 38 target microphones from the 345 available target microphones in the capture stage, i.e. , a total of 152 1 s audio clips are processed in each forward pass. We use Adam to train our model for 125 epochs with learning rate 0 . 002 . When training with shift/lscript 2 and multiscale STFT, we multiply the multiscale STFT loss with weight 100. All experiments are run on 4 NVIDIA Tesla A100 GPUs and each needs about 48 hours to finish.

Evaluation Metrics. We report results on three metrics, signal-distortion-ratio (SDR), /lscript 2 error on the amplitude spectrogram, and angular error of the phase spectrogram. While SDR measures the quality of the signal as a whole, amplitude and phase errors show a decomposition of the signal into two components. High amplitude error indicates a mismatch in energy compared to the ground truth, while high phase errors particularly occur when the delays of the spatialized signal are wrong. Note that the angular phase error has an upper bound of π / 2 ≈ 1 . 571 . We report amplitude errors amplified by a factor of 1000 to remove leading zeros.

5.1 The Role of Body Pose

The ablation of pose information is studied as shown in Table 1. We use all seven head-mounted microphones as input and vary the pose information, removing all the poses or only using the head pose. As mentioned in Section 3, the non-speech data is significantly different from the speech data,

<latexi sh

1_b

64="0

Qv

BjA

GH

TMR

IPp

w

>

/

c

VDLS

N

FJ3U

2

du

o

gq

7KCf ym

Y

O

9

+

zn

8

5

k

r

X

W

E

Z

<latexi sh

1_b

64="D

I

j

HL

8g

C

/J

+c z2

ok0

>A

B

3

V

S

N

F

U

vW

du

q

T

7K

f

ymQ

Y

O

G

R

p

X

P

n

Z

9

r

E

w

M

5

<latexi sh

1_b

64="HPqU

zJBukZ

Qd

O+

E

f5/

c

>A

V

NS8

3

W

X

v

0

w

R

7

o

MF

K

p

2

m

T

C

j

Y

G

r

n

y

L

9

g

D

I

<latexi sh

1_b

64="McJz uX

VO

oYSjH2

vfTn

>A

B83

DL

gN

E

y

r

U

9

P

K

Q

I5

Z

m

d

p

7

w

F

k

G

q

W

0

C

R

/

+

<latexi sh

1_b

64="A7

+C2zVW

nT5

Gv

K

YQ

>

B83

c

DJSgN

EO

X

jP

d

R

L

Z

k

m

0

y

9

I

/

o

U

Mu

f

r

w

H

p

F

q

<latexi sh

1_b

64="WYqC

gTIE

FyJ+

8d

K0

>A

B

n

VDLS

N

r

fU

9

oP

2

5

H

m

v

Z

M

w

3

k

G

j

X

u

R

O

/

<latexi sh

1_b

64="Q

zwT

3

7k

R

DY

dE

I

m

C

>A

B9X

c

V

gM

PSF

v

WN

H

JU

8pO

Kf

0u

+

y

/

G

Z

q

2

r

L

o

5

n

j

non-speech

speech

loss

SDR

↑

amplitude

↓

phase

↓

SDR

↑

amplitude

↓

phase

↓

/lscript

2

2

.

557

1

.

064

0

.

449

7

.

610

10

.

594

1

.

093

multiscale STFT

-

2

.

222

0

.

797

0

.

557

-

5

.

077

6

.

898

1

.

902

multiscale STFT +

/lscript

2

2

.

394

0

.

936

0

.

465

8

.

566

8

.

477

1

.

204

shift-

/lscript

2

2

.

956

0

.

910

0

.

447

8

.

899

9

.

716

1

.

136

shift-

/lscript

2

+ mult. STFT

3

.

004

0

.

784

0

.

454

9

.

580

7

.

059

1

.

184

/lscript

2

mult. STFT +

/lscript

2

mult. STFT

ground truth

shift-

/lscript

2

shift-

/lscript

2

+ mult. STFT

right

left

front

zoom-in

and thus we report quantitative results on speech and non-speech data separately. In Table 1, our model with full body pose outperforms variants with no pose or headpose only on all three metrics because full body pose provides important position information of sound sources. Compared to the no pose version, the headpose leads to a huge improvement in speech data since the headpose itself is capable of providing necessary position information on the speech case.

Figure 5 illustrates how these quantitative differences manifest in the sound field. In the case of a finger snap above the head, as shown in the figure, the sound source is estimated at the center of the 3D space if no pose information is available. With only headpose, the model puts all sound sources in the head region where speech would be produced. Finally, when provided with full body pose, the model is able to correctly locate the snap at the right hand location above the head.

5.2 The Role of Head-Mounted Microphones

We also study the impact brought by the number of input microphones. Table 2 lists the results of our model trained with 1 , 3 , and 7 head-mounted input microphones respectively. As can be observed, our model with 3 microphones shows a clear improvement over that with only one input microphone since, complementary to pose information, some spatial information can be inferred from a 3 -channel input audio but monaural audio implies nothing about the space. Compared to 3 input mics, 7 microphones can further boost the overall performance.

Figure 6 illustrates a challenging example where body motion alone does not give a clue whether the finger snap originates from the left or the right hand. With a single microphone as input, the model can only guess, and in the example shown in the figure, it places the origin of sound in the wrong hand. In contrast, using 7 microphones as input, the model is able to estimate the direction of arrival of the sound and, consequently, place its origin in the correct hand.

<latexi sh

1_b

64="oOGPS

Lg

+

F

v

7/rI

5

8

>A

B

X

c

VD

N

E

z

M

H

Y

d3

j0

C

U

y

J

m

n

Z

R

k

w

T

K

q

2

W

p

Q

f

u

9

<latexi sh

1_b

64="QGr0D

SONT

cK

v

k

E

APq

I

>

B+n

V

L

FJ3U

2

du

o

g

X

Z

j

z

5

Y

H

f

7/8

pm

W

w

9

R

M

C

y

<latexi sh

1_b

64="0

Qv

BjA

GH

TMR

IPp

w

>

/

c

VDLS

N

FJ3U

2

du

o

gq

7KCf ym

Y

O

9

+

zn

8

5

k

r

X

W

E

Z

<latexi sh

1_b

64="McJz uX

VO

oYSjH2

vfTn

>A

B83

DL

gN

E

y

r

U

9

P

K

Q

I5

Z

m

d

p

7

w

F

k

G

q

W

0

C

R

/

+

<latexi sh

1_b

64="A7

+C2zVW

nT5

Gv

K

YQ

>

B83

c

DJSgN

EO

X

jP

d

R

L

Z

k

m

0

y

9

I

/

o

U

Mu

f

r

w

H

p

F

q

<latexi sh

1_b

64="k

/w uL

V8B

POZ25+

QR

K

o

>A

C

H

c

DJSgN

E

W

GM

jU

T3I

z

d

7

F

q

9

m

0

r

X

y

fv

Y

p

n

<latexi sh

1_b

64="WYqC

gTIE

FyJ+

8d

K0

>A

B

n

VDLS

N

r

fU

9

oP

2

5

H

m

v

Z

M

w

3

k

G

j

X

u

R

O

/

<latexi sh

1_b

64="Q

zwT

3

7k

R

DY

dE

I

m

C

>A

B9X

c

V

gM

PSF

v

WN

H

JU

8pO

Kf

0u

+

y

/

G

Z

q

2

r

L

o

5

n

j

<latexi sh

1_b

64="MWg/T

7C5

Su

ZG

8

q

B

>A

c

VDJ

N

EO2

v

0

wYp

F

Rz

K

Um

n

P

f

k

3

d

9

Q

I

L

y

+

<latexi sh

1_b

64="Ay

9DZNGfL

Jcr

Yk

3T

Rq

I

>

CEX

V

B

O

j

v

2z

o

M

d

pW

u

Q

/

78

F

K

n

U

Pm

w

H

+

0

5

5.3 Accurate Sound Field Learning with Shift/lscript 2

In order to evaluate the effectiveness of our new loss function proposed in Section 4.3, we train our model with different loss functions. As shown in Table 3, we compare the new loss function shift/lscript 2 with traditional losses like /lscript 2 or multiscale STFT Yamamoto et al. [2021]. Although not always the best on all three metrics, this new loss empowers our model to achieve a more balanced performance, unlike traditional loss functions. Moreover, if combined with multiscale STFT, it can further improve the SDR value while lowering the amplitude error, on both speech and non-speech data.

A qualitative evaluation of the different losses in Figure 7 shows the deficiencies of /lscript 2 -loss and multiscale STFT loss. With darker colors (red and blue) representing high energy and green representing low energy, both losses severely underestimate the energy in the predicted signal. The proposed shift/lscript 2 loss, on the contrary, matches the ground truth signal well. When combined with a multiscale STFT loss, we observe even more fidelity and better sound field reconstruction in the fine grained details, see the zoomed-in example for shift/lscript 2 + multiscale STFT in Figure 7.

5.4 Comparison to baselines

Being the first to address the problem, there is no existing work that can serve as a meaningful baseline. Existing approaches to sound spatialization Savioja et al. [1999], Morgado et al. [2018], Richard et al. [2021] rely on knowledge of the sound source location or are limited to first-order ambisonic signals - recall that we model 17-th order signals for high enough spatial resolution. We therefore compare our approach to a naive baseline, where the input signal is naively time-warped from the head position towards the receiver microphone on the sphere, with distance attenuation applied as well. Additionally, we run our system without time-warping to demonstrate the impact of time-warping. As shown in Table 4, on all three metrics, our model demonstrates substantial improvements over the baseline. The results from our model without time-warping are inferior to the model with time-warping.

Additional visualization examples, including failure cases, can be found in the supplemental material.

6 Conclusions

Our pose-guided sound rendering system is the first approach of its kind that can successfully render 3D sound fields produced by human bodies, and therefore enriches recent progress in visual body models by its acoustic counterpart. Enabling this multimodal experience is critical to achieving immersion and realism in 3D applications such as social VR.

Limitations. While the results of our approach are promising, it still comes with considerable limitations that require deeper investigation in future work. First, the approach is limited to rendering sound fields of human bodies alone. General social interactions have frequent interactions of humans and objects, which affect sound propagation and the characteristics of the sound itself. Moreover, as the system is built upon an ambisonic sound field representation, it produces accurate results in the far field, yet fails in the near field when entering a trust region too close to the body that is modeled. Generally, the model assumes free-field propagation around the body, which is violated if two bodies are close to each other and cause interactions and diffractions in the sound field. Lastly, the system relies on powerful GPUs to render the sound field efficiently, and is not yet efficient enough to run on commodity hardware with low computing resources such as a consumer VR device.

Nevertheless, correctly modeling 3D spatial audio is a necessary step for building truly immersive virtual humans and our approach is a critical step towards this goal.

References

Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech enhancement. Proc. Interspeech 2018 , pages 3244-3248, 2018.

Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency. Style transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16 , pages 248-265. Springer, 2020.

Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Selfsupervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems , 33:9758-9770, 2020.

Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih. Driving-signal aware full-body avatars. ACM Transactions on Graphics (TOG) , 40(4):1-17, 2021.

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR , 2017.

Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16 , pages 17-36. Springer, 2020.

Changan Chen, Sagnik Majumder, Al-Halah Ziad, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. In ICLR , 2021.

Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation. ACM Transactions on Graphics (TOG) , 37(4):1-11, 2018.

Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 9701-9707. IEEE, 2020.

Chuang Gan, Xiaoyu Chen, Phillip Isola, Antonio Torralba, and Joshua B Tenenbaum. Noisy agents: Selfsupervised exploration by predicting auditory events. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 9259-9265. IEEE, 2022.

Ruohan Gao and Kristen Grauman. 2.5 d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 324-333, 2019.

Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15490-15500. IEEE, 2021.

Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 35-53, 2018.

Rishabh Garg, Ruohan Gao, and Kristen Grauman. Geometry-aware multi-task learning for binaural audio generation from video. In BMVC , 2021.

Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3497-3506, 2019.

Peng Guan, Alexander Weiss, Alexandru O Balan, and Michael J Black. Estimating human shape and pose from a single image. In 2009 IEEE 12th International Conference on Computer Vision , pages 1381-1388. IEEE, 2009.

Claudia Hendrix and Woodrow Barfield. The sense of presence within auditory virtual environments. Presence: Teleoperators and Virtual Environments , 5(3):290-301, 1996.

Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems , 33:10077-10087, 2020.

Haikun Huang, Michael Solah, Dingzeyu Li, and Lap-Fai Yu. Audible panorama: Automatic spatial audio generation for panorama imagery. In Proceedings of the 2019 CHI conference on human factors in computing systems , pages 1-11, 2019.

Hao Jiang, Calvin Murdock, and Vamsi Krishna Ithapu. Egocentric deep multi-channel audio-visual active speaker localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10544-10552, 2022.

Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7122-7131, 2018.

Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from selfsupervised synchronization. Advances in Neural Information Processing Systems , 31, 2018.

Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. Dancing to music. Advances in neural information processing systems , 32, 2019.

Dingzeyu Li, Timothy R Langlois, and Changxi Zheng. Scene-aware audio for 360 videos. ACM Transactions on Graphics (TOG) , 37(4):1-12, 2018.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG) , 34(6):1-16, 2015.

Yu-Ding Lu, Hsin-Ying Lee, Hung-Yu Tseng, and Ming-Hsuan Yang. Self-supervised audio spatialization with correspondence classifier. In 2019 IEEE International Conference on Image Processing (ICIP) , pages 3347-3351. IEEE, 2019.

Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII , pages 218-234. Springer, 2022.

Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. Advances in neural information processing systems , 31, 2018.

Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. Advances in Neural Information Processing Systems , 33:4733-4744, 2020.

Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12934-12945, 2021a.

Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12475-12486, 2021b.

Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 631-648, 2018.

Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 801-816. Springer, 2016.

Mandela Patrick, Yuki M. Asano, Polina Kuznetsova, Ruth Fong, João F. Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations, 2021.

Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16 , pages 292-308. Springer, 2020.

Alexander Richard, Dejan Markovic, Israel D Gebru, Steven Krenn, Gladstone Alexander Butler, Fernando Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. In International Conference on Learning Representations , 2021.

Alexander Richard, Peter Dodds, and Vamsi Krishna Ithapu. Deep impulse responses: Estimating and parameterizing filters with deep networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3209-3213. IEEE, 2022.

N. Samarasinghe and T. D. Abhayapala. 3D spatial soundfield recording over large regions. In Proc. Int. Workshop on Acoustic Signal Enhancement (IWAENC) , 2012.

Lauri Savioja, Jyri Huopaniemi, Tapio Lokki, and Ritta Väänänen. Creating interactive virtual acoustic environments. Journal of the Audio Engineering Society , 47(9):675-705, 1999.

Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 436-454. Springer, 2020.

Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2745-2754, 2021.

Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop , pages 125-125, 2016.

G. Williams. Fourier Acoustics . Academic Press, 1999.

Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740 , 2020.

Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15485-15494, 2021.

Ryuichi Yamamoto, Eunwoo Song, Min-Jae Hwang, and Jae-Min Kim. Parallel waveform synthesis based on generative adversarial networks with voicing-aware conditional discriminators. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6039-6043. IEEE, 2021.

Karren Yang, Bryan Russell, and Justin Salamon. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9932-9941, 2020.

Karren Yang, Dejan Markovic, Steven Krenn, Vasu Agrawal, and Alexander Richard. Audio-visual speech codecs: Rethinking audio-visual speech enhancement by re-synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG) , 39(6):1-16, 2020.

Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV) , pages 570-586, 2018.

Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu. Avatarrex: Real-time expressive full-body avatars. ACM Transactions on Graphics (TOG) , 42(4), 2023.

Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16 , pages 52-69. Springer, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with the problem of predicting sound fields around human bodies and proposes a method that exploits binaural audio signals and human body motions. The proposed method first encodes binaural audio signals and human poses and then decodes them into audio signals that are supposed to be captured by surrounding microphones. Finally, the proposed method renders the predicted signals into sound fields. The proposed method also introduces a loss function that is robust against small time shifts of time-domain audio signals. Experimental evaluations with originally collected datasets demonstrate that the proposed method captured the ground-truth sound fields reasonably well and it requires full body poses instead of only head poses for obtaining good predictions.\n\nSTRENGTHS:\n1. The problem dealt with in this paper is novel as far as I know. Although sound field prediction is one of the standard tasks in audio signal processing, sound field prediction with blindly moving microphones would be novel and challenging. This paper solves this problem by (1) assuming that sound is emitted from a human and the human wears microphones and (2) integrating pose estimation of the target human.\n\n2. The proposed method is technically sound. Since the current problem setting is sensitive to time shifts, the proposed method handles audio signals in the time domain, which is not so popular in multi-modal and cross-modal analysis with deep learning. The proposed method achieves time-domain processing by incorporating WaveNet-like decoders.\n\n3. The newly introduced loss function is also technically sound. Time-domain signals are more sensitive to time shifts than frequency-domain ones. The proposed loss function named the shift-\\ell_2 loss effectively mitigates this problem while maintaining its discriminability.\n\n4. The experimental evaluations demonstrate that\n(1) the proposed method reasonably worked well,\n(2) full-body poses contributed to the performance improvement of sound source localization compared with head poses, and\n(3) the proposed loss function was effective for non-speech sounds.\n\nWEAKNESSES:\n1. The current experimental evaluations do not contain comparisons with other existing methods. I understand that the current problem setting is novel and thus there are no existing methods that can be directly applied to this problem. However, several previous methods for sound field prediction have already been known, and experimental comparisons with one of those methods will be informative for readers.\n\n2. The problem setting seems to be too extreme. Sound field prediction and reconstruction are one of significant problems in audio signal processing. However, the current problem setting requires (1) humans as sound sources, (2) microphones worn by humans, and (3) external video cameras. In particular, the first constraint seems to be too strict. Some justifications and potential applications will be required.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nI could not find any negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel model for a novel task to render a 3D spatial sound from human body motion and audio collected by headset microphones. The authors also present a novel dataset containing human body motion and audio for this task. The model takes encoded audio, pose features, and target microphone position to render the corresponding spatial sound. A novel shifted-l2 loss is also proposed to address the issue of big spikes in l2 error due to the small shifts of impulsive signals. Experiments and results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The motivation and objectives are clear. To my best knowledge, it is indeed the first paper that aims to render 3D spatial sound from a human body pose.\n\n2. The dataset itself is valuable. It could unlock various interesting research projects in the future.\n\n3. The system is simple but effective. The proposed shifted-l2 loss is interesting and could be useful in other audio applications as well.\n\n4. Analyses of the roles of body pose and head-mounted microphones are well-done in the ablation studies.\n\n5. The phase is important and I am glad that the authors consider it in the optimization.\n\nWEAKNESSES:\n1. While extensive quantitative evaluations and analyses are done, it is often challenging to understand the gap in the perceptual level. It would be great if there is a subjective evaluation to demonstrate both the importance of the proposed task/approach and the contribution of the technical components.\n\n2. It would also be nice to have more samples in the supplementary material for a subjective evaluation of the proposed approach.\n\n3. I wonder whether any failure cases are appearing in the current exploration stage.\n\n4. audio and visual data time synchronization is often challenging but the details to achieve that are missing.\n\n5. As the authors mentioned in the limitation, the current approach could not handle spatial sound for human-object interaction, mainly for far-field modeling, and require intensive computation resources.\n\nQUESTIONS:\n1. Why do you use only 1s audio clips during training? Is there a bottleneck that avoids having a longer audio sequence during training? Or is it simply because of the high sampling rate (48khz)?\n\n2. Would it be possible to have data with multi-person and human-object interaction in the future? What might be the challenges to record these data?\n\n3. I am interested in the hyper-parameter choices for the shifted-l2 loss. How is the L=128 varied if the sampling rate differs?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces an approach for generating spatial audio from a 3D human pose and microphones placed close to the subject’s head, e.g. on a VR/AR headset, as is the case here. The authors first collect a multimodal dataset that contains 3D bodies and audio, recorded using multiple Kinects for body tracking and 345 microphones for audio acquisition. The dataset contains multiple participants in different outfits and poses, generating different sounds following a pre-defined script. To generate spatial audio, the authors propose a model that receives as input the audio recorded from the headset microphones and the 3D body pose and predicts the audio signal recorded at a microphone of the spherical capture array. Using the audio at each microphone and the locations of each microphone, the authors compute harmonic sound field coefficients that best represent the sound field.\n\nSTRENGTHS:\n- Important problem with large impact: The authors tackle an important problem for AR/VR applications that has not yet received a lot of attention. Generating accurate sound fields promises to increase immersion and the realism of virtual experiences. The proposed dataset will be an important benchmark for the community and open up new and exciting avenues of research.\n- Novelty: The proposed dataset is an important contribution with unique data that are not publicly available. \n- A loss that is well-motivated by prior-work and adapted to the problem structure that leads to clear benefits.\n- A simple model that merges different types of information, using well-proven components.\n- Writing: The paper is well-written and easy to follow.\n- Experiments: The experimental analysis is clear, highlights the effects of each design choice and provides insights into what makes the method work.\n\nWEAKNESSES:\n- The related work section lacks a discussion of recent methods that model sound fields using neural network fields, such as:\n  - [Learning Neural Acoustic Fields](https://www.andrew.cmu.edu/user/afluo/Neural_Acoustic_Fields/).\n  - [INRAS: Implicit Neural Representation for Audio Scenes](https://openreview.net/forum?id=7KBzV5IL7W)\n\nThese could also be useful baselines for the model, both in terms of accuracy and in terms of computation compared to the proposed method. Furthermore, the following datasets could be a useful discussion point for the related work section:\n  - SoundSpaces: Audio-Visual Navigation in 3D Environments: A discussion of whether a similar dataset as the one proposed in this work could be constructed by simulating audio propagation from artificial bodies.\n  - Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis: I believe the related work section could benefit from a comparison with this paper in terms of granularity of pose representation and sound field, number of persons and type of conversations.\n\nQUESTIONS:\n- Do you use only body keypoints? Or do you include hand keypoints from OpenPose? \n- I believe that a comment or discussion on the granularity of the representation of the human body would be useful. The ablation clearly shows that a more detailed representation of the human body helps improve the quality of the predicted sound field. I would like to hear the authors' answer on the following questions:\n  - Is the robustness of the estimated keypoints important? Do noisy keypoints significantly affect the quality of the sound field?\n  - Is the lower body important for sound quality? \n  - Would a finer representation of the subject help? For example, would including hand or face keypoints from OpenPose help?\nSome implementation details that should be clarified:\n- How was the size of the window in loss function chosen?\n- How was the training sampling strategy selected? Maximization of GPU memory usage?\n\n- Finally, regarding the performance of the model, it would be useful to have a visualization of the distribution of errors around the sphere of microphones. Is there a subset of microphones where the audio signal is better reconstructed?\n\nLIMITATIONS:\nThe authors present the limitations of the proposed dataset and method. They acknowledge that the current work is limited to modeling sound effects for a single person in an artificial scene and that it cannot model sound close to the body.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the task spatializing a mixture of speech and body sounds at different points in a sphere around a human body without explicitly recording or knowing the sound source location. Towards that goal, the paper captures a new dataset of humans speaking and making different body sounds in a very controlled and extensive setup. The paper proposes a model that takes as input the time-warped audio recorded by a VR headset and the body pose sequence, and renders the audio at different points on the sphere conditioned on the azimuth and elevation of the points. Furthermore, the paper proposes a novel and intuitloss function to accurately model both speech and short-lived body sounds. The paper compares its model with a heurisitcal baseline, evaluates different versions of the loss function, and also reports results for different microphone channel count and an ablation of the model. \n\n\nPost author-reviewer discussion: I have read the rebuttal. The responses are detailed and answer my questions. I have increased my score.\n\nSTRENGTHS:\n1. The proposed task is interesting.\n\n2. The proposed loss is intuitive and seems to be well-designed.\n\n3. The dataset could also be a valuable contribution for the community.\n\nWEAKNESSES:\n1. a) The dataset capture seems to be done in just 1 environment. There is no discussion or experiments on the model's generalization to other environments.\n   b) The capture setup seems to be pretty complicated and could limit the applicability of the model to novel environments if training needs to be redone for new environments.\n\n2. a) The capture scene seem to be simple in the sense that it doesn't contain other objects, is single-room in nature, etc. How would the model fare for more complex realworld scenes?\n\n    b) Would the model be able to accurately render sounds at locations that don't have a path of direct sounds from the source? \n\n3. How is the source location in L150-1 determined?\n\nMinor:\n1. The shift-l2 loss doesn't seem to perform by itself in most cases (as also pointed out by the authors) even though the design is pretty intuitive.\n\nQUESTIONS:\nI would like to request the authors to address the questions/concerns I raised in 'Weaknesses', in the rebuttal.\n\nLIMITATIONS:\nThe paper discusses its limitations, and I can't think of any serious negative societal implications of the work that need to be discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}