Learning Neural Acoustic Fields

1 INTRODUCTION . The sound of the ball leaving the bat , as much as its visible trajectory , tells us whether the hit is likely to be a home run or not . Our experience of the world around us is rich and multimodal , depending on integrated input from all of five sensory modalities – vision , touch , smell , hearing , and taste . Understanding underlying scene structure relies on a continuous and diverse set of inputs : the visual appearance and geometry of the physical world , the way sounds bound off of or are blocked by surfaces , the way odors diffuse across the local environment , and the way objects feel as we walk on them or touch them . Recent progress in implicit neural representations has enabled the construction of continuous , differentiable representations of the visual world directly from raw image observations ( Sitzmann et al. , 2019 ; Mildenhall et al. , 2020 ; Niemeyer et al. , 2020 ; Yariv et al. , 2020 ) . These models typically utilize a neural renderer in combination with a learned implicit representation to jointly capture and render images of a scene . By leveraging the multiview consistent nature of visual observations , these methods can infer images of the same scene from novel viewpoints once trained on sparse views . However , our perception of the physical world is informed not only by our visual observations , but also by the acoustics of environmental sounds . When we are in a large room , we expect to hear highly reverberant echos ( e.g. , as in a cathedral ) . Moreover , given only the acoustic properties of sounds in this space , we may infer a variety of other properties of the environment , for example , the location of the emitted sound or the underlying room layout . As a preliminary step in learning the acoustic properties of scenes , we explore modeling the underlying impulse response of audio reverberations . Past work has explored capturing the underlying acoustics of a scene ( Raghuvanshi & Snyder , 2014 ; 2018 ; Chaitanya et al. , 2020 ) . These models , however , require manually designing acoustic functions which , critically , prevent such approaches from being applied to arbitrary scenes . In this work , we extend this approach by constructing an implicit neural representation which captures , in a generic manner , the underlying acoustics of a scene . In particular , following ( Raghuvanshi & Snyder , 2014 ) , we define the acoustic modeling problem as modeling the impulse-response a listener receives given a sound emitted at an emitter location ( as illustrated in Figure 1 ) and across all possible emitter- listener pairs in a scene . This learned representation inherently captures the pattern of all acoustic reverberations in a scene . Learning a representation of scene acoustics poses several challenges compared to the visual setting . First , how do we represent , in high fidelity , the underlying impulse response at each emitter-listener position ? While we may represent the visual appearance of a scene with an underlying threedimensional vector , an acoustic reverberation ( represented as an impulse response ) can consist of over 20000 values and , thus , is significantly harder to capture . Second , how do we learn an acoustic neural representation that densely generalizes to novel emitter-listener locations ? In the visual setting , ray-tracing can enforce view consistency across large portions of a visual scene ( modulo occlusions ) . While in principal , in a similar manner , we may reflect acoustic `` rays '' in our scene to obtain an impulse response , a intractable number of rays are necessary to obtain the desired representation . To address both challenges , we propose Neural Acoustic Fields ( NAFs ) . To capture the complex signal representation of impulse responses , NAFs encode and represent an impulse-response in the Fourier frequency domain . Motivated by the strong influence of nearby geometry on anisotropic reflections ( Raghuvanshi & Snyder , 2018 ) , we propose to condition NAFs on local geometric information present at both the listener and emitter locations when decoding the impulse response . In our framework , local geometric information is learned directly from impulse responses . Such a decomposition facilitates the transfer of local information captured from training emitter-listener pairs to novel combinations of emitters and listeners By modeling the dense acoustic fields of an environment , NAFs are a powerful method to extract structural information about the scene , which we demonstrate is useful across a variety of different tasks . In the cross-modal setting , we demonstrate how the learned acoustic structure can be utilized to aid learned visual representations , improving novel view synthesis . Further , by directly utilizing the acoustic information in NAFs , we demonstrate how one can localize different sound sources . In summary , we present Neural Acoustic Fields ( NAFs ) , a neural implicit representation which captures the underlying acoustics of a scene in a compact and spatially continuous fashion . We show that NAFs are able to outperform baselines in modeling scene acoustics , and provide detailed analysis of the design choices in NAFs . We further illustrate how the structure learned by NAFs , can improve cross-modal generation of novel visual views of a scene . Finally , we illustrate how the continuous nature of NAFs enable the downstream application of sound source localization . 2 RELATED WORK . Audio Field Coding There is a rich history of encoding methods for 3D spatial audio . These approaches largely fall into two categories . The first approach encodes the sound field at a usercentric location by capturing the sound from spatially distributed sources ( Gerzon , 1973 ; Breebaart et al. , 2005 ; Pulkki , 2007 ) . While they may leverage perceptual cues that create the sense of spatial audio , they do not allow the listener to freely traverse the scene and experience sound from different locations . The second approach aims to model the sound heard as a listener moves in a scene ( Raghuvanshi & Snyder , 2014 ; Mehra et al. , 2014 ; Raghuvanshi & Snyder , 2018 ; Chaitanya et al. , 2020 ) . Since the complete acoustic field of a scene is computationally prohibitive to simulate in real time , and expensive to store in full fidelity , these methods have relied on a handcrafted encoding of the acoustic field , prioritizing efficiency above reproduction fidelity . Our work allows a listener to move and experience sounds that come from anywhere in a scene , and can represent the acoustic field continuously at high fidelity by directly learning from data . Implicit representation Our approach towards modeling the underlying acoustics a scene relies on the use of a neural implicit representations . Implicit representations have emerged as a promising representation of 3D geometry Niemeyer et al . ( 2019 ) ; Chen & Zhang ( 2019 ) ; Park et al . ( 2019 ) ; Saito et al . ( 2019 ) and appearance Sitzmann et al . ( 2019 ) ; Mildenhall et al . ( 2020 ) ; Niemeyer et al . ( 2020 ) ; Yariv et al . ( 2020 ) of a scene . Compared to traditional discrete representations , implicit representations are a continuous mapping capable of capturing data at an `` infinite resolution '' . Jiang et al . ( 2020 ) proposed a grid based representation for implicit scene reconstruction , while more recently DeVries et al . ( 2021 ) has adopted spatial conditioning for 3D image synthesis , where in both settings , the grid enables a higher-fidelity encoding of the scene . Our work also leverages local grids to model acoustics , but as an inductive bias and way to generalize to novel inputs . Audio-Visual Learning Our work is also closely related to joint modeling of vision and audio . By leveraging the correspondence between vision and audio , work has been done to learn unsupervised video and audio representations ( Aytar et al. , 2016 ; Arandjelovic & Zisserman , 2017 ) , localize objects that emit sound ( Senocak et al. , 2018 ; Zhao et al. , 2018 ) , and jointly use vision and audio for navigation ( Chen et al. , 2020 ) . Recent work aims to propose plausible reverberations or sounds from image input ( Singh et al. , 2021 ; Du et al. , 2021 ) , these approaches model the phase-free logmagnitude STFT using either convolution or implicit functions , which we also utilize . Different from them , our work leverages the geometric features learned by modeling acoustic fields to improve the learning of 3D view generation . 3 METHODS . We are interested in learning a generic acoustic representation of an arbitrary scene , which can capture the underlying sound propagation of arbitrary sound sources across both seen and unseen locations in a scene . We first review relevant background information towards modeling environment reverberations . We then describe Neural Acoustic Fields ( NAFs ) , a neural field which we show can capture , in a generic manner , the acoustics of arbitrary scenes . We further discuss how we can parameterize NAF in a manner so that it can capture acoustics property even at unseen sound sources and listener positions . Finally , we discuss the underlying implementation details of our model . 3.1 BACKGROUND ON ENVIRONMENTAL REVERBERATION . The sound emitted by a sound source undergoes decay , occlusion , and scattering due to both the geometric and material properties of a scene . For a fixed location pair ( q , q′ ) , we define the impulseresponse at a listener position q , as the sound pressure p ( t ; q , q′ ) induced by an impulse at q′ . Such behavior can be concisely and elegantly modeled utilizing the linear wave equation ( Pierce , 2019 ) : [ 1 c2 ∂2 ∂t2 −∇2 ] p ( t , q , q′ ) = δ ( t ) δ ( q − q′ ) , ( 1 ) where c is the speed of sound , p is the sound pressure , ( q , q′ ) being the listener and emitter location respectively , and δ the Dirac delta representing the forcing function , where we refer to sound pressure p ( t ; q , q′ ) as the impulse-response observed at listener position q . Given an accurate model of the impulse-response p ( t ; q , q′ ) described in Eqn ( 1 ) , we may model audio reverberation of any sound waveform s ( t ) emitted at q′ , by computing the response r ( t , q , q′ ) at listener location q by querying the continuous field and using temporal convolution : r ( t ; q , q′ ) = s ( t ) ~ p ( t ; q , q′ ) ( 2 ) 3.2 NEURAL ACOUSTIC FIELDS . We are interested in constructing a continuous representation of the underlying acoustics of a scene , which may specify the reverberation patterns of an arbitrary sound source . The parameterization of an impulse-response introduced in Section 3.1 provides us with a method to model audio propagation when given an omnidirectional listener and emitter . To construct a model of a directional listener , we further model the 3D head orientation θ ∈ R2 , and ear k ∈ { 0 , 1 } ( binary left or right ) of a listener , in addition to the spatial position q ∈ R3 of the listener and q′ ∈ R3 of the emitter . We may then model the time domain impulse response v using a neural field Φ which takes as input the listener and emitter parameters : Φ : R8 × { 0 , 1 } → RT , ( q , θ , k , q′ ) → Φ ( q , θ , k , q′ ) = v ( 3 ) Directly outputting the impulse-response v ∈ RT with a neural network is difficult to its high dimensional ( over 20000 elements ) and chaotic nature . A naïve solution would be further add t as an additional argument our neural field , but we found that such a solution worked poorly , due to the highly non-smooth representation of the waveform ( see Table A2 ) . We instead encode the impulseresponse utilizing a short-time Fourier transform ( STFT ) to create a log-magnitude spectrogram denoted vSTFT , which we find to be significantly more amenable to neural network prediction , due to the smooth nature of the frequency space . In Figure 3 we show spectrograms for ground truth impulse responses and those learned by our network . Thus , our parameterization of NAF is a neural field Ω that is trained to estimate the impulse response function φ , and outputs vSTFT for a given time and frequency coordinate : Ω : R10 × { 0 , 1 } → R , ( q , θ , k , q′ , t , f ) → Ω ( q , θ , k , q′ , t , f ) ≈ vSTFT ( t , f ) ( 4 ) We train our model using MSE loss between the generated and ground truth log-spectrograms vSTFT : LNAF = ‖Ω ( q , θ , k , q′ , t , f ) − vSTFT ( t , f ) ‖2 ( 5 ) across spectrogram coordinates t and f . In our qualitative results we utilize Griffin-Lim reconstruction to recover v from vSTFT similar to prior work ( Du et al. , 2021 ) . While such an approximate reconstruction ignores the underlying phase of the impulse response , phase free models are typically used in spatial acoustic modeling ( Pulkki , 2007 ; Raghuvanshi & Snyder , 2018 ; Singh et al. , 2021 ) . Such approximations are used because the phase locking cutoff in humans occurs at around 1∼ 2kHz ( Shinn-Cunningham et al. , 2000 ; Joris & Verschooten , 2013 ) , below which time delay provides important spatial cues . We demonstrate in our qualitative results that NAFs with this approximation may still accurately model realistic spatial acoustic effects . Alternative methods for waveform recovery exist , including methods that can be learned from data ( Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ) . These approaches are orthogonal to our work , and we leave this exploration to a future study .

Summary: This paper introduces the concept of Neural Acoustic Fields, which is an implicit representation to capture how sound propagates in a pysical scene. Using a NeRF-like network architecture, the model is able to predict room impulse response given the listener and source locations as input. The idea is new and interesing, but there are multiple overclaims in the paper. Some interesting tasks are demonstrated like sound source location, cross-modal generation, etc.