Learning Signal-Agnostic Manifolds of Neural Fields

1 Introduction . Every moment , we receive and perceive high-dimensional signals from the world around us . These signals are in constant flux ; yet remarkably , our perception system is largely invariant to these changes , allowing us to efficiently infer the presence of coherent objects and entities across time . One hypothesis for how we achieve this invariance is that we infer the underlying manifold in which perceptual inputs lie [ 1 ] , naturally enabling us to link high-dimensional perceptual changes with local movements along such a manifold . In this paper , we study how we may learn and discover a low-dimensional manifold in a signal-agnostic manner , over arbitrary perceptual inputs . Manifolds are characterized by three core properties [ 2 , 3 ] . First , a manifold should exhibit data coverage , i.e. , all instances and variations of a signal are explained in the underlying low-dimensional space . Second , a manifold should be locally metric , enabling perceptual manipulation of a signal by moving around the surrounding low-dimensional space . Finally , the underlying structure of a manifold should be globally-consistent ; e.g . similar signals should be embedded close to one another . Existing approaches to learning generative models , such as GANs [ 4 ] , can be viewed as instances of manifold learning . However , such approaches have two key limitations . First , low-dimensional latent codes learned by generative models do not satisfy all desired properties for a manifold ; while the underlying latent space of a GAN enables us to perceptually manipulate a signal , GANs suffer from mode collapse , and the underlying latent space does not cover the entire data distribution . Second , existing generative architectures are biased towards particular signal modalities , requiring custom architectures and losses depending on the domain upon which they are applied – thereby preventing us from discovering a manifold across arbitrary perceptual inputs in a signal-agnostic manner . 1Code and additional results are available at https : //yilundu.github.io/gem/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As an example , while existing generative models of images are regularly based on convolutional neural networks , the same architecture in 1D does not readily afford high-quality modeling of audio signals . Rather , generative models for different domains require significant architecture adaptations and tuning . Existing generative models are further constrained by the common assumption that training data lie on a regular grid , such as grids of pixels for images , or grids of amplitudes for audio signals . As a result , they require uniformly sampled data , precluding them from adequately modeling irregularly sampled data , like point clouds . Such a challenge is especially prominent in cross-modal generative modeling , where a system must jointly learn a generative model over multiple signal sources ( e.g. , over images and associated audio snippets ) . While this task is trivial for humans – we can readily recall not only an image of an instrument , but also a notion of its timbre and volume – existing machine learning models struggle to jointly fit such signals without customization . To obtain signal-agnostic learning of manifolds over signals , we propose to model distributions of signals in the function space of neural fields , which are capable of parameterizing image , audio , shape , and audiovisual signals in a modalityindependent manner . We then utilize hypernetworks [ 5 ] , to regress individual neural fields from an underlying latent space to represent a signal distribution . To further ensure that our distribution over signals corresponds to a manifold , we formulate our learning objective with explicit losses to encourage the three desired properties for a manifold : data coverage , local linearity , and global consistency . The resulting model , which we dub GEM2 , enables us to capture the manifold of a variety of signals – ranging from audio to images and 3D shapes – with almost no architectural modification , as illustrated in Figure 1 . We further demonstrate that our approach reliably recovers distributions over cross-modal signals , such as images with a correlated audio snippet , while also eliciting sample diversity . We contribute the following : first , we present GEM , which we show can learn manifolds over images , 3D shapes , audio , and cross-modal audiovisual signals in a signal-agnostic manner . Second , we demonstrate that our model recovers the global structure of each signal domain , permitting easy interpolation between nearby signals , as well as completion of partial inputs . Finally , we show that walking along our learned manifold enables us to generate new samples for each modality . 2 Related Work . Manifold Learning . Manifold learning is a large and well-studied topic [ 2 , 3 , 6–10 ] which seeks to obtain the underlying non-linear , low-dimensional subspace in which naturally high-dimensional input signals lie . Many early works in manifold learning utilize a nearest neighbor graph to obtain such a low-dimensional subspace . For instance , Tenenbaum et al . [ 2 ] employs the geodesic distance between nearby points , while Roweis et al . [ 3 ] use locally linear subregions around each subspace . Subsequent work has explored additional objectives [ 7–10 ] to obtain underlying non-linear , low-dimensional subspaces . Recently , [ 11–13 ] propose to combine traditional manifold learning algorithms with deep networks by training autoencoders to explicitly encourage latents to match embeddings obtained from classical manifold learning methods . Further , [ 14 ] propose to utilize a heat kernel diffusion process to uncover the underlying manifold of data . In this work , we show how we may combine insights from earlier works in manifold learning with modern deep learning techniques to learn continuous manifolds of data in high-dimensional signal spaces . Drawing on these two traditions enables our model to smoothly interpolate between samples , restore partial input signals , and further generate new data samples in high-dimensional signal space . 2short for GEnerative Manifold learning Learning Distributions of Neural Fields . Recent work has demonstrated the potential of treating fully connected networks as continuous , memory-efficient implicit ( or coordinate-based ) representations for shape parts [ 15 , 16 ] , objects [ 17–20 ] , or scenes [ 21–24 ] . Sitzmann et al . [ 25 ] showed that these coordinate-based representations may be leveraged for modeling a wide array of signals , such as audio , video , images , and solutions to partial differential equations . These representations may be conditioned on auxiliary input via conditioning by concatenation [ 17 , 26 ] , hypernetworks [ 21 ] , gradient-based meta-learning [ 27 ] , or activation modulation [ 28 ] . Subsequently , they may be embedded in a generative adversarial [ 28 , 29 ] or auto-encoder-based framework [ 21 ] . However , both of these approaches require modality-dependent architecture choices . Specifically , modification is necessary in the encoder of auto-encoding frameworks , or the discriminator in adversarial frameworks . In contrast , we propose to represent the problem of learning a distribution as one of learning the underlying manifold in which the signals lie , and show that this enables signal-agnostic modeling . Concurrent to our work , Dupont et al . [ 30 ] propose a signal-agnostic discriminator based on pointwise convolutions , which enables discrimination of signals that do not lie on regular grids . In contrast , we do not take an adversarial approach , thereby avoiding the need for a convolution-based discriminator . We further show that our learned manifold better captures the underlying data manifold and demonstrate that our approach is able to capture manifolds across cross-modal distributions , which we find destabilizes training of [ 30 ] . Generative Modeling . Our work is also related to existing work in generative modeling . GANs [ 4 ] are a popular framework used to model modalities such as images [ 31 ] or audio [ 32 ] , but often suffer from training instability and mode collapse . VAEs [ 33 ] , another class of generative models , utilize an encoder and decoder to learn a shared latent space for data samples . Recently autoregressive models have also been used for image [ 34 ] , audio [ 35 ] , and multi-modal text and image generation [ 36 ] . In contrast to previous generative approaches , which require custom architectures for encoding and decoding , our approach is a modality-independent method for generating samples distribution and presents a new view of generative modeling : one of learning an underlying manifold of data . 3 Parameterizing Spaces of Signals with Neural Fields . An arbitrary signal I in RD1×···×Dk can be represented efficiently as a function Φ which maps each coordinate x ∈ Rk to the value v of the feature coordinate at that dimension . Φ : Rk → R , x→ Φ ( x ) = v ( 1 ) Such a formulation enables us to represent any signal , such as an image , shape , or waveform – or a set of signals , like images with waveforms – as a continuous function Φ . We use a three-layer multilayer perceptron ( MLP ) with hidden dimension 512 as our Φ . Only the input dimension of Φ is varied , depending on the dimensionality of the underlying signal – all other architectural choices are the same across signal types . We refer to Φ as a neural field . Parameterizing Spaces over Signals . A set of signals may be parameterized as a subspace F of functions Φi . To efficiently parameterize functions Φ , we represent each Φ using a low-dimensional latent space in Rn and map latents to F via a hypernetwork Ψ : Ψ : Rn → F , z → Ψ ( z ) = Φ , ( 2 ) To regress Φ with our hypernetwork , we predict individual weights WL and biases bL for each layer L in Φ . Across all experiments , our hypernetwork is parameterized as a three-layer MLP ( similar to our Φ ) , where all weights and biases of the signal representation Φ are predicted in the final layer . Directly predicting weights WL is a high-dimensional regression problem , so following [ 37 ] , we parameterize W via a low-rank decomposition as WL = WLs σ ( WLh ) , where σ is the sigmoid function . WLs is shared across all predicted Φ , while W L h = A L×BL is represented as two low-rank matrices AL and BL – and is regressed separately for each individual latent z . 4 Learning Structured Manifold of Signals . Given a training set C = { Ii } Ni=1 , of N distinct arbitrary signals Ii ∈ RD1×···×Dk , our goal is to learn – in a signal-agnostic manner – the underlying low-dimensional manifoldM⊂ RD1×···×Dk . In particular , a manifoldM [ 2 , 3 ] , over signals I , is a locally low-dimensional subspace consisting and describing all possible variations of I . A manifold has three crucial properties . First , it must embed all possible instances of I. Second , M should be locally metric around each signal I , enabling us to effectively navigate and interpolate between nearby signals . Finally , it must be locally perceptually consistent ; nearby points onM should be similar in identity . To learn a manifoldM with these properties , we introduce a loss for each property : LRec , LLLE , and LIso , respectively . LRec ensures that all signals I are embedded insideM ( Section 4.1 ) , LLLE forces the space near each I to be locally metric ( Section 4.2 ) , and LIso encourages the underlying manifold to be perceptually consistent ( Section 4.3 ) . As such , our training loss takes the form : LTotal = LRec + LLLE + LIso ( 3 ) Please see Fig . 2 for an overview of the proposed manifold learning approach . We utilize equal weighting across each introduced loss term . We discuss each component of the loss in more detail below .

Summary: This work introduces GEM - a model which learns the low-dimensional latent manifold structure of a dataset in a modality-independent way. GEM’s objective function is designed to encourage the learning of manifolds which exhibit the following properties, namely they: cover the data, are locally metric, and are globally consistent.  The model is trained on four datasets with different modalities (image, audio, 3D shape, and cross-modal image and audio). Qualitative and quantitative evaluations suggest that GEM can be used to do linear interpolation between points on the learned manifold, it can generate reconstructions of corrupted samples as well as generate new samples.