Learning Neural Acoustic Fields

1 INTRODUCTION . The sound of the ball leaving the bat , as much as its visible trajectory , tells us whether the hit is likely to be a home run or not . Our experience of the world around us is rich and multimodal , depending on integrated input from all of five sensory modalities – vision , touch , smell , hearing , and taste . Understanding underlying scene structure relies on a continuous and diverse set of inputs : the visual appearance and geometry of the physical world , the way sounds bound off of or are blocked by surfaces , the way odors diffuse across the local environment , and the way objects feel as we walk on them or touch them . Recent progress in implicit neural representations has enabled the construction of continuous , differentiable representations of the visual world directly from raw image observations ( Sitzmann et al. , 2019 ; Mildenhall et al. , 2020 ; Niemeyer et al. , 2020 ; Yariv et al. , 2020 ) . These models typically utilize a neural renderer in combination with a learned implicit representation to jointly capture and render images of a scene . By leveraging the multiview consistent nature of visual observations , these methods can infer images of the same scene from novel viewpoints once trained on sparse views . However , our perception of the physical world is informed not only by our visual observations , but also by the acoustics of environmental sounds . When we are in a large room , we expect to hear highly reverberant echos ( e.g. , as in a cathedral ) . Moreover , given only the acoustic properties of sounds in this space , we may infer a variety of other properties of the environment , for example , the location of the emitted sound or the underlying room layout . As a preliminary step in learning the acoustic properties of scenes , we explore modeling the underlying impulse response of audio reverberations . Past work has explored capturing the underlying acoustics of a scene ( Raghuvanshi & Snyder , 2014 ; 2018 ; Chaitanya et al. , 2020 ) . These models , however , require manually designing acoustic functions which , critically , prevent such approaches from being applied to arbitrary scenes . In this work , we extend this approach by constructing an implicit neural representation which captures , in a generic manner , the underlying acoustics of a scene . In particular , following ( Raghuvanshi & Snyder , 2014 ) , we define the acoustic modeling problem as modeling the impulse-response a listener receives given a sound emitted at an emitter location ( as illustrated in Figure 1 ) and across all possible emitter- listener pairs in a scene . This learned representation inherently captures the pattern of all acoustic reverberations in a scene . Learning a representation of scene acoustics poses several challenges compared to the visual setting . First , how do we represent , in high fidelity , the underlying impulse response at each emitter-listener position ? While we may represent the visual appearance of a scene with an underlying threedimensional vector , an acoustic reverberation ( represented as an impulse response ) can consist of over 20000 values and , thus , is significantly harder to capture . Second , how do we learn an acoustic neural representation that densely generalizes to novel emitter-listener locations ? In the visual setting , ray-tracing can enforce view consistency across large portions of a visual scene ( modulo occlusions ) . While in principal , in a similar manner , we may reflect acoustic `` rays '' in our scene to obtain an impulse response , a intractable number of rays are necessary to obtain the desired representation . To address both challenges , we propose Neural Acoustic Fields ( NAFs ) . To capture the complex signal representation of impulse responses , NAFs encode and represent an impulse-response in the Fourier frequency domain . Motivated by the strong influence of nearby geometry on anisotropic reflections ( Raghuvanshi & Snyder , 2018 ) , we propose to condition NAFs on local geometric information present at both the listener and emitter locations when decoding the impulse response . In our framework , local geometric information is learned directly from impulse responses . Such a decomposition facilitates the transfer of local information captured from training emitter-listener pairs to novel combinations of emitters and listeners By modeling the dense acoustic fields of an environment , NAFs are a powerful method to extract structural information about the scene , which we demonstrate is useful across a variety of different tasks . In the cross-modal setting , we demonstrate how the learned acoustic structure can be utilized to aid learned visual representations , improving novel view synthesis . Further , by directly utilizing the acoustic information in NAFs , we demonstrate how one can localize different sound sources . In summary , we present Neural Acoustic Fields ( NAFs ) , a neural implicit representation which captures the underlying acoustics of a scene in a compact and spatially continuous fashion . We show that NAFs are able to outperform baselines in modeling scene acoustics , and provide detailed analysis of the design choices in NAFs . We further illustrate how the structure learned by NAFs , can improve cross-modal generation of novel visual views of a scene . Finally , we illustrate how the continuous nature of NAFs enable the downstream application of sound source localization . 2 RELATED WORK . Audio Field Coding There is a rich history of encoding methods for 3D spatial audio . These approaches largely fall into two categories . The first approach encodes the sound field at a usercentric location by capturing the sound from spatially distributed sources ( Gerzon , 1973 ; Breebaart et al. , 2005 ; Pulkki , 2007 ) . While they may leverage perceptual cues that create the sense of spatial audio , they do not allow the listener to freely traverse the scene and experience sound from different locations . The second approach aims to model the sound heard as a listener moves in a scene ( Raghuvanshi & Snyder , 2014 ; Mehra et al. , 2014 ; Raghuvanshi & Snyder , 2018 ; Chaitanya et al. , 2020 ) . Since the complete acoustic field of a scene is computationally prohibitive to simulate in real time , and expensive to store in full fidelity , these methods have relied on a handcrafted encoding of the acoustic field , prioritizing efficiency above reproduction fidelity . Our work allows a listener to move and experience sounds that come from anywhere in a scene , and can represent the acoustic field continuously at high fidelity by directly learning from data . Implicit representation Our approach towards modeling the underlying acoustics a scene relies on the use of a neural implicit representations . Implicit representations have emerged as a promising representation of 3D geometry Niemeyer et al . ( 2019 ) ; Chen & Zhang ( 2019 ) ; Park et al . ( 2019 ) ; Saito et al . ( 2019 ) and appearance Sitzmann et al . ( 2019 ) ; Mildenhall et al . ( 2020 ) ; Niemeyer et al . ( 2020 ) ; Yariv et al . ( 2020 ) of a scene . Compared to traditional discrete representations , implicit representations are a continuous mapping capable of capturing data at an `` infinite resolution '' . Jiang et al . ( 2020 ) proposed a grid based representation for implicit scene reconstruction , while more recently DeVries et al . ( 2021 ) has adopted spatial conditioning for 3D image synthesis , where in both settings , the grid enables a higher-fidelity encoding of the scene . Our work also leverages local grids to model acoustics , but as an inductive bias and way to generalize to novel inputs . Audio-Visual Learning Our work is also closely related to joint modeling of vision and audio . By leveraging the correspondence between vision and audio , work has been done to learn unsupervised video and audio representations ( Aytar et al. , 2016 ; Arandjelovic & Zisserman , 2017 ) , localize objects that emit sound ( Senocak et al. , 2018 ; Zhao et al. , 2018 ) , and jointly use vision and audio for navigation ( Chen et al. , 2020 ) . Recent work aims to propose plausible reverberations or sounds from image input ( Singh et al. , 2021 ; Du et al. , 2021 ) , these approaches model the phase-free logmagnitude STFT using either convolution or implicit functions , which we also utilize . Different from them , our work leverages the geometric features learned by modeling acoustic fields to improve the learning of 3D view generation . 3 METHODS . We are interested in learning a generic acoustic representation of an arbitrary scene , which can capture the underlying sound propagation of arbitrary sound sources across both seen and unseen locations in a scene . We first review relevant background information towards modeling environment reverberations . We then describe Neural Acoustic Fields ( NAFs ) , a neural field which we show can capture , in a generic manner , the acoustics of arbitrary scenes . We further discuss how we can parameterize NAF in a manner so that it can capture acoustics property even at unseen sound sources and listener positions . Finally , we discuss the underlying implementation details of our model . 3.1 BACKGROUND ON ENVIRONMENTAL REVERBERATION . The sound emitted by a sound source undergoes decay , occlusion , and scattering due to both the geometric and material properties of a scene . For a fixed location pair ( q , q′ ) , we define the impulseresponse at a listener position q , as the sound pressure p ( t ; q , q′ ) induced by an impulse at q′ . Such behavior can be concisely and elegantly modeled utilizing the linear wave equation ( Pierce , 2019 ) : [ 1 c2 ∂2 ∂t2 −∇2 ] p ( t , q , q′ ) = δ ( t ) δ ( q − q′ ) , ( 1 ) where c is the speed of sound , p is the sound pressure , ( q , q′ ) being the listener and emitter location respectively , and δ the Dirac delta representing the forcing function , where we refer to sound pressure p ( t ; q , q′ ) as the impulse-response observed at listener position q . Given an accurate model of the impulse-response p ( t ; q , q′ ) described in Eqn ( 1 ) , we may model audio reverberation of any sound waveform s ( t ) emitted at q′ , by computing the response r ( t , q , q′ ) at listener location q by querying the continuous field and using temporal convolution : r ( t ; q , q′ ) = s ( t ) ~ p ( t ; q , q′ ) ( 2 ) 3.2 NEURAL ACOUSTIC FIELDS . We are interested in constructing a continuous representation of the underlying acoustics of a scene , which may specify the reverberation patterns of an arbitrary sound source . The parameterization of an impulse-response introduced in Section 3.1 provides us with a method to model audio propagation when given an omnidirectional listener and emitter . To construct a model of a directional listener , we further model the 3D head orientation θ ∈ R2 , and ear k ∈ { 0 , 1 } ( binary left or right ) of a listener , in addition to the spatial position q ∈ R3 of the listener and q′ ∈ R3 of the emitter . We may then model the time domain impulse response v using a neural field Φ which takes as input the listener and emitter parameters : Φ : R8 × { 0 , 1 } → RT , ( q , θ , k , q′ ) → Φ ( q , θ , k , q′ ) = v ( 3 ) Directly outputting the impulse-response v ∈ RT with a neural network is difficult to its high dimensional ( over 20000 elements ) and chaotic nature . A naïve solution would be further add t as an additional argument our neural field , but we found that such a solution worked poorly , due to the highly non-smooth representation of the waveform ( see Table A2 ) . We instead encode the impulseresponse utilizing a short-time Fourier transform ( STFT ) to create a log-magnitude spectrogram denoted vSTFT , which we find to be significantly more amenable to neural network prediction , due to the smooth nature of the frequency space . In Figure 3 we show spectrograms for ground truth impulse responses and those learned by our network . Thus , our parameterization of NAF is a neural field Ω that is trained to estimate the impulse response function φ , and outputs vSTFT for a given time and frequency coordinate : Ω : R10 × { 0 , 1 } → R , ( q , θ , k , q′ , t , f ) → Ω ( q , θ , k , q′ , t , f ) ≈ vSTFT ( t , f ) ( 4 ) We train our model using MSE loss between the generated and ground truth log-spectrograms vSTFT : LNAF = ‖Ω ( q , θ , k , q′ , t , f ) − vSTFT ( t , f ) ‖2 ( 5 ) across spectrogram coordinates t and f . In our qualitative results we utilize Griffin-Lim reconstruction to recover v from vSTFT similar to prior work ( Du et al. , 2021 ) . While such an approximate reconstruction ignores the underlying phase of the impulse response , phase free models are typically used in spatial acoustic modeling ( Pulkki , 2007 ; Raghuvanshi & Snyder , 2018 ; Singh et al. , 2021 ) . Such approximations are used because the phase locking cutoff in humans occurs at around 1∼ 2kHz ( Shinn-Cunningham et al. , 2000 ; Joris & Verschooten , 2013 ) , below which time delay provides important spatial cues . We demonstrate in our qualitative results that NAFs with this approximation may still accurately model realistic spatial acoustic effects . Alternative methods for waveform recovery exist , including methods that can be learned from data ( Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ) . These approaches are orthogonal to our work , and we leave this exploration to a future study .

Summary: This paper makes use of the NeRF idea on RIRs (coined here as NAFs).  The main idea is relatively straightforward, sample a room from multiple emitter/receiver locations and learn to predict the resulting RIR using a neural net that receives as input only the involved locations, and some positional information.  It is shown that this approach learns something, and it is used in a few demonstrative examples such as sound localization and enhancing NeRF estimation.