Learning Signal-Agnostic Manifolds of Neural Fields
Authors: Yilun Du
Katherine M. Collins
Joshua B. Tenenbaum
Vincent Sitzmann
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural Fields
Coordinate-based
Implicit Function
Manifold
Multimodal
Generative Model
Cross-Modal

Abstract: Deep neural networks have been used widely to learn the latent structure of datasets, across modalities such as images, shapes, and audio signals. However, existing models are generally modality-dependent, requiring custom architectures and objectives to process different classes of signals. We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains in a modality-independent manner. We cast our task as one of learning a manifold, where we aim to infer a low-dimensional, locally linear subspace in which our data resides. By enforcing coverage of the manifold, local linearity, and local isometry, our model -- dubbed GEM -- learns to capture the underlying structure of datasets across modalities. 
We can then travel along linear regions of our manifold to obtain perceptually consistent interpolations between samples, and can further use GEM to recover points on our manifold and glean not only diverse completions of input images, but cross-modal hallucinations of audio or image signals.  Finally, we show that by walking across the underlying manifold of GEM, we may generate new samples in our signal domains.

Learning Signal-Agnostic Manifolds of Neural Fields
Yilun Du' yilundu@mit.edd edu
Katherine Collins!.
Joshua B. Tenenbaum 1,2,3 jbt t. edu
Vincent Sitzmann' si tamann@mit. edu
kati emcemi edu
MIT CSAIL
MIT CBMM https://yilundu github.iot.o..o io/gem/
MIT BCS
Abstract
Deep neural networks have been used widely to learn the latent structure of datasets, across modalities such as images, shapes, and audio signals. However, existing models are generally modality-dependent, requiring custom architectures and objectives process different classes of signals. We leverage neural fields capture the underlying structure in image, shape, audio and cross- audiovisual domains a
manner. We cast our task as one of learning a manifold, where we aim to infer low-dimensional, locally linear subspace in which our data resides. By enforcing coverage of the manifold, local linearity, and local isometry, our model - dubbed GEM learns to capture the underlying structure of datasets across modalities. We can then travel along linear regions our manifold obtain perceptually consistent interpolations between samples, and can further use GEM to recover points on our manifold and glean not only diverse completions of input images, but cross-modal hallucinations of audio image signals. Finally, we show that by walking across the underlying manifold of GEM, may generate new samples in signal domains'.
Introduction
Every moment, we receive and perceive high-dimensional signals from the world around us. These signals are in constant flux; yet remarkably, our perception system is largely invariant to these changes, allowing to efficiently infer the presence of coherent objects and entities across time. One hypothesis for how we achieve this invariance is that we infer the underlying manifold which perceptual inputs lie [1], naturally enabling us to link high-dimensional perceptual changes with local movements along such a manifold. In this paper, we study how we may learn and discover a low-dimensional manifold in signal-agnostic manner, over arbitrary perceptual inputs.
Manifolds are characterized by three core properties [2, 3]. First, a manifold should exhibit data coverage. i.e., all instances and variations of signal are explained in the underlying dimeenniion space. Second, a manifold should be locally metric, enabling perceptual manipulation of a signal by moving around the surrounding low-dimensional space. Finally, the underlying strueture of manifold should be consisly- e.g. similar signals should be embedded close to one another.
Existing approaches to learning generative models, such as GANs [4], can be viewed as instances of manifold learning. However, such approaches have two key limitations. First, low-dimensional latent codes learned by generative models do not satisfy all desired properties for a manifold; while the underlying latent space a GAN enables us to perceptually manipulate a signal, GANs suffer from mode collapse, and the underlying latent space does not cover the entire data distribution. Second, existing generative architectures are biased towards particular signal modalities, requiring custom architectures and losses depending the domain upon which they are applied thereby preventing us from discovering a manifold across arbitrary perceptual inputs in a
manner.
Code and additional results are available at httpp:://yllmtmi werlllllduugiiin github. io/gem/.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

As an example, while existing generative models of images are regularly based on convolutional neural networks, the same architecture in ID does not readily afford high-quality modeling of audio signals. Rather, generative models for different domains require significant architecture adaptations and tuning. Existing generative models are further constrained by the common assumption that training data lie on a regular grid, such as grids of pixels for images, or grids of amplitudes for audio signals. As a result, they require uniformly sampled data, precluding them from adequately modeling irregularly sampled data, like point clouds. Such challenge is especially prominent in cross-modal generative modeling, where a system must jointly learn a generative model over multiple signal sources (e.g., over images and associated audio snippets). While this task is trivial for humans can readily recall not only an image of an instrument, but also a notion of its timbre and volume- existing machine learning models struggle to jointly fit such signals without customization.
To obtain signal-agnostic learning of manifolds over signals, we propose to model distributions of signals in the function space of neural fields, which are capable of parameterizing image, au- dio, shape, and audiovisual signals a modality- independent manner. We then utilize hypernet- works [5], to regress individual neural fields from underlying latent space represent signal distribution. To further ensure that our dis- tribution over signals corresponds to a manifold, we formulate our learning objective with explicit losses to encourage the three desired properties for a manifold: data coverage, local linearity, and global consistency. The resulting model, which we dub GEM enables us to capture the manifold of variety of signals ranging from audio to images and 3D shapes with almost no architectural modification, as illustrated in Fig- ure 1. We further demonstrate that our approach reliably recovers distributions over cross-modal signals, such as images with correlated audio snippet, while also eliciting sample diversity.
Cr.
Latent Manifold
CS. 90
KC (1,,
Figure I: GEM Icarns low-dimensional latent mani- fold over signals. Given cross-modal signal, latents in GEM mapped, using hypernetorr v. into neural networks di and Φ2. dn represents a image by mapping each pixel position (z, y) to its associated color c(z, y). d represents an audio spectrogram by mapping each pixel position (u, v) to its intensity w(u, v). This en- ables GEM he applied in a domain agnostic manner across separate (multi-modal) signals, by utilizing a sep- arate function & for each mode of a signal.
We contribute the following: first, we present GEM, which we show can learn manifolds over images, 3D shapes, audio, and cross-modal audiovisual signals in a signal-agnostic manner. Second, we demonstrate that our model recovers the global structure of each signal domain, permitting easy interpolation between nearby signals, as well as completion of partial inputs. Finally, we show that walking along our learned manifold enables us to generate new samples for each modality.
2 Related Work
Manifold Learning. Manifold learning is a large and well-studied topic [2, 3, 6-10] which seeks to obtain the underlying non-linear, lww-dinensional subspace in which naturally nigh-dimensional input signals lic. Many carly works in manifold learning utilize a nearest neighbor graph to obtain such a low-dimensional subspace. For instance, Tenenbaum et al. [2] employs the geodesic distance between nearby points, while Roweis et al. [3] use locally linear subregions around each subspace. Subsequent work has explored additional objectives [7-10] to obtain underlying non-linear, low-dimensional subspaces. Recently, [11-13] propose to combine traditional manifold learning algorithms with deep networks by training autoencoders to explicitly encourage latents to match embeddings obtained from classical manifold learning methods. Further, [14] propose to utilize a heat kernel diffusion process to uncover the underlying manifold of data. In this work, we show how we may combine insights from earlier works in manifold learning with modern deep learning techniques to learn continuous manifolds of data in -ich-iimensional signal spaces. Drawing on these two traditions enables our model to smoothly interpolate between samples, restore partial input signals, and further generate new data samples in -iseesinenii signal space.
short for GEnerative Manifold learning
2

---

Learning Distributions of Neural Fields. Recent work has demonstrated the potential of treating fully connected networks as continuous, memory-effiiien implicit (or coordinate- (orddnnate-bsee)) representa- tions for shape parts [15, 16), objects [17-20], or scenes [21-24] Sitzmann et al. [25] showed that these coordinate- representations may be leveraged for modeling wide array of signals, such as audio, video, images, and solutions to partial differential equations. These representations may be conditioned on auxiliary input via conditioning by concatenation [17, 26], hypernetworks [21], gradient-based meta-learning [27], or activation modulation [28]. Subsequently, they may be em- framework [21]. However, both of these approaches require modality- -epeeddent architecture choices. Specifically, modification is necessary in the encoder of auto-encoding frameworks, or the discriminator in adversarial frameworks. In contrast, propose to represent the problem of learning a distribution as one of learning the underlying manifold in which the signals lie, and show that this enables signal-agnostic modeling. Concurrent
bedded in a generative adversarial [28, 29] or auto-encoder-bant
work, Dupont al. [30] propose signal-agnostic discriminator based on pointwise convolutions, which enables discrimination of signals that do not lie on regular grids. In contrast, we do not take an adversarial approach, thereby avoiding the need for convolution-bass -based discriminator. We further show that our learned manifold better captures the underlying data manifold and demon- strate that our approach is able capture manifolds across cross-modal distributions, which we find destabilizes training of [30].
Generative Modeling. Our work is also related to existing work in generative modeling. GANs [4] are a popular framework used to model modalities such as images [31] or audio [32], but often suffer from training instability and mode collapse. VAEs [33], another class of generative models, utilize an encoder and decoder to learn a shared latent space for data samples. Recently autoregressive models have also been used for image [34], audio [35], and multi-modal text and image generation [36]. In contrast to previous generative approaches, which require custom architectures for encoding and decoding, our approach modality-indeenncee method for generating samples distribution and presents a new view of generative modeling: one of learning an underlying manifold of data.
3 Parameterizing Spaces of Signals with Neural Fields
An arbitrary signal I in RD xx can be represented efficiently as a function t which maps each coordinate x E R to the value of the feature coordinate at that dimension.
R x ((x)
(1)
Such a formulation enables us to represent any signal, such as an image, shape, or waveform - or set of signals, like images with waveforms multilayer perceptron (MLP) with hidden dimension 512 as our C. Only the input dimension of 4 is varied, depending the dimensionality of the underlying signal - all other architectural choices are the same across signal types. We refer to & as a neural field.
continuous function T. We three-layer
Parameterizing Spaces over Signals. A set of signals may be parameterized as subspace F of functions 2,. To efficiently parameterize functions Q we represent each & using a low-dimensional latent space in F" and map latents to F via hypernetwork :::
V : R" z - 1(2)
(2)
To regress & with our hypernetwork, we predict individual weights W and biases bf for each layer C in Φ. Across all experiments, our hypernetwork is parameterized as three-layer MLP (similar to our )), where all weights and biases of the signal representation & are predicted in the final layer. Directly predicting weights W is high-dimensional regression problem, following [37], parameterize W via low-rank decomposition as W = W. O (W,,), where a is the sigmoid function. W.r is shared across all predicted 4 while W.r = A x B is represented as two low-rank matrices A and BC - and is regressed separately for each individual latent z.
Learning Structured Manifold of Signals
Given a training set C {I.) of N distinet arbitrary signals I, R P Dr, our goal is to leam in signal-agnostic manner the underlying low-dimensional manifold M c g D Dix In
3

---

(a) Reconstruction Loss
(b) Local Isometry Loss
(c) LLE Loss
(x)) d())
#2 ~ 1 T2l
21 - Σ, a,2
c(
LESK Linsrr TEASTRLL
LinE
Leak
21
Σ, Ega==
Hypernetwork
Latent Manifold
Figure 2: GEM learns to embed an arbitrary signal I in latent manifold via three losses: (a) reconstruction loss which encourages neural network o, decoded from through a hypernetwork, to match training signal I each coordinate æt (b) local isometry loss to enforce perceptual consistency, which encourages distance between nearby latents 21 and 22 be porportional to their distance signal space Ij, I2, (c) a LLE loss which encourages a latent 21 to be represented as a convex combination a, of its neighbors.
particular, a manifold M [2, 3], over signals I, is a locally low-dimensional subspace consisting and describing all possible variations of I. A manifold has three crucial properties. First, it must embed all possible instances I. Second, M should be locally metric around each signal I, enabling us to effectively navigate and interpolate between nearby signals. Finally, it must be locally perceptually consistent; nearby points on M should be similar identity. To learn a manifold M with these properties, introduce loss for each property: LRees LLLE, and Liso respectively. Liee ensures that all signals I are embedded inside M (Section 4. 4.1), LII.R. forces the space near each I to be locally metric (Section 4.2), and Lisn encourages the underlying manifold be perceptually consistent (Section 4.3). As such, our training loss takes the form:
Croul = Lase + LLEE + Liso
(3)
Please see Fig. 2 for an overview of the proposed manifold learning approach. We utilize equal weighting across each introduced loss term. We discuss each component of the loss in more detail below.
4.1 Data Coverage via Auto-Decoding
Given parameterization of a subspace F (as introduced in Section 3) which represents our manifold M, we must ensure that V, our hypernetwork, can effectively cover the entire subspace in which our signal I lies. A common approach to learn subspaces over signals is to employ a GAN [4], whereby discriminator trains hypernetwork (z) to generate signals that are indistinguishable from real examples. However, an issue with such an approach is that many I may be missing in the resultant mapping.
Here, to ensure that our manifold covers all instances, we utilize the auto-decoder framework [17,21]. 21]. We learn separate latent 2, for each training signal I, and train models utilizing the loss
Lree T (z) [==
(4)
over each possible input coordinate By explicitly learning reconstruct each training point in latent space, we enforce that our latent space, and thus our manifold M, covers training signals Z.
4.2 Local Metric Consistency via Linear Decomposition
A manifold M is locally metric around a signal I if there exist local linear directions of variation on the manifold along which the underlying signals vary in perceptually coherent way. Inspired by [3], we enforce this constraint by encouraging our manifold to consist of a set of local convex linear regions R, our learned latent space.
We construct these convex linear regions R, using autodecoded latents 2; (Section 4.1). Each latent z; defines a convex region, R obtained by combining nearest latent neighbors x!,..., as illustrated in Figure 2.
To ensure that our manifold M consists of these convex local regions, during training, we enforce that each latent z; can be represented in local convex region, R,, e.g. that it can be expressed as

---

set of weights wi, such that 2, = Eg w/ and Σg wi = 1. Given a 2, we may solve for given w; by minimizing the objective 2|z - Σ3 n 2112 = w G, w, with respect to wi. In the above expression, G, is the Gram matrix, and is defined as G; kk", where k is a block matrix, with row j corresponding to k; 2, x Adding the constraint that all weights add up one via a Legrange multiplier, leads to the following optimization objective:
(5) where 1 corresponds to a vector of all ones. Finding the optimal wi in the above expression then corresponds to the best projection of 2; the linear region R; it defines. Taking gradients of the above expression, we find that wi GT11, 1, where À is set to ensure that elements of w: sum up to
[(w,, X) = G,w, + A(1'
The mapped latent 2 = Σ, w z corresponds to the best projection 2; on R,,. To enforce that z; lies in R2, we enforce that this projection
LLE = V (= [x))]
(6)
also decodes to the training signal Z, where we differentiate through intermediate matrix operations. To encourage w; to be positive, we incorporate an additional I.1 penalty to negative weights.
4.3 Perceptual Consistency through Local Isometry
Finally, a manifold M should be perceptually consistent. In other words, two signals that are perceptually similar should also be closer in the underlying manifold, according distance metric on the respective latents, than signals that are perceptually distinct. While in general the MSE distance between two signals is not informative, when such signals relatively similar to another, their MSE distance does contain useful information.
Therefore, we enforce that the distance between latents z; and zj in our manifold M are locally isometric to the MSE distance of the corresponding samples I₁ and Ij. when relative pairwise distance of samples is small. On the lowest 25% of pairwise distances, utilize the following loss:
L.o
||a (z:
2,)ll I I,ll
Such local isometry may alternatively be enforced by regularizing the underlying Jacobian map.
4.4 Applications
By learning downstream applications, each of which we explore in this work.
manifold M, GEM permits the following signal-agnostic
Interpolation. By directly interpolating between the learned latents z; and z, representing each signal, we can interpolate in a perceptually consistent manner between neighboring signals in M. Further, latents for novel signals may be obtained by optimizing Lase on the signal.
Conditional Completion. Given a partial signal î, we can obtain a point on the manifold M by optimizing latent using LRee LILE. Decoding the corresponding latent z then allows to complete the missing portions of the image.
Sample Generation. Our manifold M consists of a set of locally linear regions R. in underlying latent space z. As such, new samples in M can be generated by walking along linear regions R,, and sampling points each such region. To sample random point within R2, we generate a between O and 1, and compute a latent 2, az; + (1 a) z + B (0, 1) (corresponding to random sample in the neighborhood of 2,) and project onto the underlying region R,.
Learning Manifolds of Data
We validate the generality GEM by first showing that our model is capable of fitting diverse signal modalities. Next, we demonstrate that our approach captures the underlying structure across these signals; we are not only able to cluster and perceptually interpolate between signals, but inpaint to complete partial ones. Finally, we show that we can draw samples from the learned manifold of each signal type, illustrating the power of GEM be used a signal-agnostic generative model. We re-iterate that nearly identical architectures and training losses used across separate modalities.
5

---

StyleGAN2
GEM
Ground Truth
Modality
Model
MSE
PSNR f
Images
Heat Kemel
0.0365
14.99
VAE
0.0331
15.12
Images
FDN
0.0060
22.59
StyleGAN2
0.0044
24.03
3D Shapes
GEM
0.0025
26.53
VAE
0.0147
18.87
FDN
0.0050
23.68
Audio
StyleGAN2
0.0015
28.52
GEM
0.0011
29.98
StyleGAN2
Ground
FDN
0,0296
16.52
ShapeNet
GEM
0.0153
21.32
Audio visuml
VAE
0.0193
17.23
FDN
0.0663
11.78
Image and Audio
StyleGAN2
0.0063
22.36
GEM
0.0034
24.38
Figure 3: Test-time reconstruction. Comparison of GEM against baselines on the task of fitting test signals
over diverse set of signal modalities: images, 3D shapes, and audio-visual, respectively. GEM (center) achieves significantly better reconstruction performance qualitatively and quantitatively StyleGAN2 or FDN. indicating better coverage of the data manifold across all modalities. Results are run across one seed, but we report results across three seeds in the appendix, and find limited overall variance.
test signals than
Datasets. We evaluate GEM on four signal modalities: image, audio, 3D shape, and cross-modal image and audio signals, respectively. For the image modality, we investigate performance on CelebA-HQ dataset [38] fit on 29000 64 x 64 training celebrity images, and test on 1000 64 x 64 test images. To study GEM behavior on audio signals, we use the NSynth dataset [39], and fit on training set of 10000 one-second 16kHz sounds clips of different instruments playing, and test of 5000 one-second 16kHz sound clips. We process sound clips into spectrograms following [32]. For the 3D shape domain, we work with the ShapeNet dataset from [40]. We train 35019 training shapes at 64 x 64 x 64 resolution and test on 8762 shapes at 64 x 64 x 64 resolution. Finally, for the cross-modal image and audio modality, Sub-UR URMP dataset [41]. We train on 9800 images at 128 x 128 resolution and 0.5 second 16kHz audio clips, which are also processed into spectrograms following [32] and test 1080 images and associated audio clips. We provide additional cross-modal results in the supplement.
utilize the cello image and audio recordings from the
We benchmark GEM against existing manifold learning approaches. Namely, we compare our approach with that of StyleGAN2 [31], VAE [42], and concurrent work FDN of [30], as well as heat kernel manifold learning [14]. We utilize the authors' provided codebase for StyleGAN2, the Pytorch VAE library for the VAE, and the original codebase of FDN [30], which the authors graciously provided. We train all approaches with a latent dimension of 1024, and re-scale the size of the VAE to ensure parameter counts are similar. We report model architecture details the appendix. We need to change the architecture of both the VAE and StyleGAN2 baselines, respectively, in order to preserve proper output dimensions, depending on signal modality. However, we note that identical architectures readily can and are used for GEM. For fitting cross-modal audiovisual signals, all models utilize two separate decoders, with each decoder in GEM identical to each other. Architectural details provided in the appendix.
Setup and Baselines.
5.1 Covering the Data Manifold
We first address whether model is capable of covering the underlying data distribution of each modality, compared against our baselines, and study the impact of our losses on this ability.
Modality Fitting. We quantitatively measure reconstruction performance on test signals in Figure and find the GEM outperforms each of our baselines. the moosl domain particular, we find that FDN training destabilizes quickly and struggles to adequately fit examples. We illustrate test signal reconstructions Figure 3 and Figure 4. Across each modality, we find that GEM obtains the sharpest samples. In Figure 3 we find that while StyleGAN2 can reconstruct certain CelebA-HQ
"hapss/github.com/AAniitxXX VAE

---

FDN
GEM
Data
GEM
GPM
Data
Modality
LuE
Luo
MSE 4
PSNR 1
Keyboard
No
No
0.0028
26.11
Image
Yes
No
0.0025
26.53
Yes
Yes
0.0024
26.69
No
No
0.0020
27.63
Audio
No
Yes
0.0014
29.02
Organ
Yes
Yes
0.0011
29.98
No
No
0.0422
16.23
Shapes
Yes
No
0.0323
17.43
Yes
Yes
0.0153
21.32
Mallet
Image and Audio
No
No
0.0070
21.72
Yes
No
0.0058
22.57
Yes
Yes
0.0034
24.38
Figure 4: Test-time audio reconstruction. GEM (center) Table 1: Ablation. Impact of Luo and CLUE on achieves better spectrogram recovery than FDN (left), even test signal reconstruction performance. Ablating recovering fine details in the original signals.
each loss demonstrates that both components enable better fitting of the underlying test distribution.
100
GEM (No Regi
50
(CIIK)
Figure 6: t-SNE Structure. [- SNE plot of NSynth audio clip manifold embeddings colored by corresponding pitch. Pitch seperated by t-SNE.
Figure Nearest Neighbor Interpolation. Interpolation between nearest neighhors GEM, in images and 3D shape domains. Our model able to make perceptually continuous interpolations.
images well, fails badly on others, indicating a lack of data coverage. Our results indicate that our approach best captures the underlying data distribution.
Ablations. In Table 1, we assess the impact two of our proposed losses CLLE and Liso at enabling tignasignal reconstruction, and find to both help improve the resultant test construction. We posit that both losses serve to regularize the underlying learned data manifold, enabling GEM to more effectively cover the overall signal manifold.
Applications Other Models. While we apply CRee LLLE. Liso GEM, our overall losses may be applied more generally across different models, provided the model maps an underlying latent space output domain. We further assess whether these losses improve the manifold captured by StyleGAN2 via measuring reconstruction performance on test CelebA-HQ images. We find that while a base StyleGAN2 model obtains a reconstruction error of MSE 0.0044 and PSNR 24.03, the addition of Cree improves test reconstruction to MSE 0.0041 I and PSNR 24.29 with LLLE losses further improving test reconstruction MSE 0.0038 and PSNR 24.61. contrast, we found that Lis leads to a slight regression in performance. We posit that both LR and LLLE serve to structure the underlying StyleGAN2 latent space, while the underlying Gaussian nature the StyleGAN2 latent space precludes the need for Li... Our results indicate the generality of our proposed losses towards improving the recovery of data manifolds.
5.2 Learning the Structure of Signals
Next, explore the extent which the manifold learned by GEM captures the underlying global and local structure inherent to each signal modality. Additionally, we probe our model's understanding of individual signals by studying GEM's ability reconstruct partial signals.
Global Structure. We address whether GEM captures the global structure of signal distribution by visualizing the latents learned by GEM the NSynth dataset with t-SNE [43]. We find in Figure 6

---

Mask
Completions
Ground Truth
Qaery Image
Nearest Neighhor
GEM (No Rog)
GEM
Figure 7: Diverse Image Inpainting (Left)., GEM generates multiple possible completions of partial image. Across completions, skin and lip hues are completed correctly and eyeglasses are reconstructed consistently. Nearest Neighbors (Right). Nearest neighbors in the manifold of GEM with and without regularization. With regularization, neighbors latent space correspond perceptually similar images.
Inpul Imagee
Completed Audio
GT Audio
Impul Andio
Completed Emage
GT Image
Figure 8: Audio Hallucinations (Left). GEM generates multiple possible audio spectrogram completions of input image. Generations are cross-modal. For instance, conditioned image lifted bow, empty spectrograms are generated; likewise, higher frequency spectrograms generated when the bow on high strings, and lower frequency oncs on lower strings. Image Hallucinations (Right). Given an input spectrogram, GEM generates multiple possible corresponding image completions, again highlighting the cross-modality of the manifold. When the input spectrogram is emply, GEM generates images having the bow off the cello, and depending the frequency composition of the spectrogram, the bow position is sampled along different strings.
that the underlying pitch of signal maps onto the resulting t-SNE. We further visualize the inferred connectivity structure our manifold in Figure (left) and visualize the nearest neighbors in latent space of the autodecoded latents (right), finding that nearby latents correspond to semantically similar faces further supporting that GEM learns global structure. Note, we find that without either LLEE or Lis,, the underlying connectivity of our manifold is significantly poorer.
Local Structure. We next probe whether GEM learns densely connected manifold, e.g. one which allows us to interpolate between separate signals, suggestive of capturing the local manifold structure. In Figure 5, visualize nearest neighbor interpolations in manifold and observe that the combination LLTE and LTs enables us to obtain perceptually smooth interpolations over both image and shape samples. We provide additional interpolations and failure cases in the supplement.
Signal Completion. To investigate the ability of GEM understand individual signals, we assess the recovery of full signal when only subset of such signal is given. In Figure 7, consider the case inpainting CelebA-HO faces, where a subset the face is missing, We observe that GEM is able to obtain several different perceptually consistent inpaintings of face, with individual completions exhibiting consistent skin and lip colors, and restoring structural features such as eyeglasses. We provide additional examples and failure cases in the supplement.
Additionally, we consider signal completion the cross-modal audiovisual domain. Here, we provide only input image and ask GEM to generate possible audio spectograms. As seen in Figure 8,

---

FDN Generations
StyleGAN Gencrations
GEM Generations
Model
FID
Procision f
Recall
VAB
175.33
0.799
0,001
StyleGAN
0.618
0.481
FDN [30]
13.46
0.577
0,397
GEM
30.42
0.642
0.502
ShapeNet
Coverage 7
MMD
Latent GAN [40]
0.389
0.00117
FDN [30]
0,341
0.0021
GFM (Ours)
0.409
0.0014
Table 2: Generation Performance. Performance
Figure 9: Audio Generations. GEM (right) generates sharp audio samples compared FDN and StyleGAN2
of GEM and baselines signal generation.
FDN Generations
GEM Generntions
IMNET
GEM
3D Shupe Generation
FDN
StyleGAN2
GEM
Audiovioul Generation
GEM Generation Neurest Latent Neighbors
Figure 10: Image Generations (Top). GEM (right) generates comparable im- ages to FDN. Nearest Neighbors (Bot- tom). Illustration of nearest neighbors in latent space of generations.
Figure 11: Generations. (right) produces reasonable samples across the shape and audiovisual modalities. In contrast, audiovisual generations from StyleGAN2 exhibit noise, while FDN generates samples in both modalities.
observe that GEM is able to generate empty spectograms when the bow is not on the cello, as well as different spectrograms dependent on the position of the bow. Alternativeyy, in Figure 8, we provide only an input audio spectrogram and ask GEM to generate possible images. In this setting, we find that generated images have the bow off the cello when the input spectrogram is empty (with the bow missing due to GEM blurring the uncertainty of all possible locations of the bow), and at different positions depending on the input audio spectrogram. In contrast, we found that all baselines were unable to fit the audiovisual setting well, as discussed in Section 5.1 and shown in Figure 11.
Generating Data Samples
Finally, investigate the ability of GEM generate new data samples.
show random samples drawn from GEM and compare these to our baselines. We consider images in Figure 10, shapes in Figure 11, audio snippets in Figure and audiovisual inputs in Figure 11. While we find that GEM performs comparably to FDN in the image regime, model significantly outperforms all baselines and audiovisual modalities. We further display nearest latent space neighbors generations on CelebA-HQ in Figure 10 and find that our generations are distinct from those in the training dataset.
Qualitative Generations.
domains of audio, shape,
Next, provide quantitiative evaluations of generations in Table on image and shape modalities. We report the FID [44], precision, and recall [45] metrics on CelebA- HQ x 64. We find that GEM performs better than StyleGAN2 and FDN on precision and recall, but worse in terms of FID. We note that our qualitative generations in Figure 10 are comparable to those of FDN; however, we find that our approach obtains high FID scores due the inherent sensitivity of the FID metric to blur. Such sensitivity to bluriness has also been noted in [46], and
Quantitative Comparisons.

---

find that even our autodecoded training distribution obtains an FID of 23.25, despite images appearing near perceptually perfect (Figure 3). On ShapeNet, we report coverage and MMD metrics from [47] using Chamfer Distance. To evaluate generations, we sample 8762 shapes (the size of test Shapenet dataset) and generate 2048 points following [40]. We compare generations from GEM with those of FDN and latent GAN trained on IMNET (using the provided code in [40]). We find that GEM outperforms both approaches on the task of 3D shape generation.
Conclusion
We have presented GEM, an approach to learn modality-indeeendee manifold. We demonstrate how model enables to interpolate between signals, complete partial signals, and further generate new signals. A limitation of our approach is that while underlying manifolds are recovered, they are not captured with high-fidelity. We believe that promising direction of future work involves the pursuit of further engineered inductive biases, and general structure, which enable the recovery of high-fidelity manifolds of the plethora of signals in the world around us. We emphasize, however, that as our manifold is constructed over a dataset; GEM may learn to incorporate and propogate existing prejudices and biases present in such data, posing risks for mass deployment of our model. Additionally, while our work offers exciting avenues for cross-modal modeling and generation, we note that GEM has the potential to be used create enhanced "deep fakes" and other forms of synthetic media.
This project supported by DARPA under CW3031624 (Transfer, Augmen- tation and Automatic Learning with Less Labels), the Singapore DSTA under DST000CC12000022 (New Representations for Vision), as well as ONR MURI under N00014- 1--18112 1-2846. We would like to thank Bill Freeman and Fredo Durand for giving helpful comments on the manuscript. Yilun Du is supported by an NSF graduate research fellowship.
Acknowledgements
References
[1] Scbastian H. Scung and Danicl Lee. The manifold ways of perception. Science, 290:2268 2269, 2000.
[2] Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Sci., 290(5500):2319 2323, 2000. 1, 2,
[3] Sam T Roweis and Lawrence Saul. Nonlinear dimensionality reduction by locally linear embedding. Sci., 290(5500):2333-255
2000. 2,
[4] Ian Goodfellow. Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014. 1, 3,4
[5] David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. In Proc. ICLR, 2017.
[6] Bernhard Schölkopf. Alexander J. Smola, and
Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neurul Comput., 1095):1299-1399 1998. 2
[7] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representa- tion. Neural Comput., (333)1333-1396 June 2003. 2
Ronald R. Coifman and Stéphane Lafon. Diffusion maps. Applied and Computational Harmonic Analysis, 21(1):5-30, 2006. Special Issue: Diffusion Maps and Wavelets.
[9] David I.. Donoho and Carrie Grimes. Hessian eigenmaps: Locally lincar embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10:5591-5596 2003.
[10] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE Journal of Machine Learning Research, 9:2579-260S. 2008. 2
[11] Kui Jia, Lin Sun, Shenghua Gao, Zhan Song, and Bertram E. Shi. Laplacian auto-encoders: An explicit learning of nonlinear data manifold. Newrocomputing, 160:250-260, 2015.
[12] Gal Mishne, Uri Shaham, Alexander Cloninger, and Israel Cohen. Diffusion nets. Applied and Computa- tional Harmonic Analysis, (7(2))259-285 2019.
[13] Gautam Pai, Ronen Talmon, Alex Bronstein, and Ron Kimmel. Dimal: Deep isometric manifold learning using sparse geodesic sampling, 2018, 2
[14] Yufan Zhou, Changyou Chen, and Jinhui Xu. Learning manifold implicitly via explicit heat- hernel learning. arXiv preprint arXiv:2010.01761, 2020. 2,
10

---

[15] Kyle Genova, Forrester Cole. Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proc. ICCV, pages 7154-7164, 2019. 3
[16] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Deep structured implicit functions. arXiv preprint arXiv:1912.06120 2019. 3
[17] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR, 2019. 3,4
[18] Mateusz Michalkiewicz, Jhony Pontes, Dominic Jack, Mahsa Baktushmotlagh,, and Anders Eriksson. Implicit surface representations as layers in neural networks. In Proc. ICCV, pages 4743-4752, 2019.
[19] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proc. CVPR, 2020.
[20] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In Proc. ICML, 2020.
[21] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awwre neural scene representations. In Proc. NeurIPS 20/9, 2019. 3,4
[22] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang. Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proc. CVPR, pages 6001-6010, 2020.
[23] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Proc. ECCV, 2020.
[24] Rohan Chabra, Jan Eric Lenssen, Eddy llg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. arXiv preprint arXiv:2003.10983, 2020. 3
[25] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions, 2020. 3
[26] Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proc. CVPR, 2019. 3
[27] Vincent Sitzmann, Eric R Chan, Richard Tucker, Noah Snavely, and Gordon Weizstein. Metasdf: Meta- learning signed distance functions. Proc. NeurlPS, 2020. 3
[28] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. Proc. CVPR, 2020. 3
[29] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Proc. NeurIPS, 2020. 3
[30] Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of functions. arXiv preprint 202iv:2102.044:
102. 04776, 2021. 6,9
[31] Tero Karras, Samuli Laine. Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 3,6
[32] Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis, 2019. 3,6
[33] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. In NeurlPS, 2014. 3
[34] Aaron van den Oord, Nal Kalchhrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. Proc. ICML, 2016.
[35] Aaron van den Oord, Sander Dicleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016.
[36] Aditya Ramesh, Mikhail Paylov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. shot text-to-image generation, 2021. 3
[37] Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. arXiv preprint arXiv:201 1202,, 2020. 3
[38] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2017. 6
[39] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. Neural audio synthesis of musical notes with wavenet autoencoders, 2017. 6
[40] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proc. CVPR, pages 5938, 2019. 6, 9,
11

---

[41] Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chenliang Xu. Deep cross-modal audio-visual generation, 2017. 6
[42] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014. 6
[43]
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(1):2579-
2008. 7
[44]
Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6666 2017.
[45] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. CoRR, 1bs/1904.06991, 2019. 2019. 9
[46] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2,
2019. 9
[47] Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas J Guibas. Learning representations and generative models for 3d point clouds. arXiv preprint arXiv: 2017.
12


Reviews:
[{"review_id": "t4syT7PNbDy", "reviewer": "Reviewer_C9Je", "summary": "This paper proposes GEM model to capture the underlying structure of datasets across modalities. It can obtain perceptually consistent interpolations between samples, and can further recover points on the manifold and realize audio/image signals completion. Besides, it can also generates new samples. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The paper focuses on manifold learning and proposes an interesting idea with various interesting applications. \n+ The proposed model GEM with the local isometry loss and the LLE loss outperforms the three baselines in most cases both quantitatively and qualitatively.\n\nWeakness:\n- Missing comparisons to other baselines in the literature, such as Learning Manifold Implicitly via Explicit Heat-Kernel Learning. \n\nQuestion:\nIn the ablation study table, it shows that the local isometry loss and the LLE loss are quite effective with GEM structure. Of noting, StyleGANv2 sometimes outperforms the GEM without the local isometry loss and the LLE loss. I am wondering how these losses work with other models, say StyleGANv2? \nHow does the visualization in Fig.3 for GEM look like without the local isometry loss and the LLE loss?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any potential negative societal impact of their work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "hno8Md5WBWf", "reviewer": "Reviewer_w67Z", "summary": "This paper proposes a modality-agnostic implicit generative model. A majority of the existing generative models involve architecture adept in a particular domain (e.g., convolutional layers for images), often requiring careful design and incurring difficulties in dealing with multi-modal data. To tackle this, the authors propose a generative manifold learning (GEM) scheme, which faithfully reflects the manifold hypothesis by utilizing hypernetwork-based implicit modeling. While implicit modeling had been proposed in previous work, unique to GEM is the use of the proposed local isometry and LLE loss functions. Such loss design removes the need of auxiliary networks (e.g., discriminator for GANs), which might require modality-dependent desgin, making the training of the model as well be completely modality-agnostic.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper is well organized and reads easily. The considered topic of finding a generic architecture that can be shared across different modalities is interesting and important. However, the proposed method seems to have the following weak point: \n\n* The model seems to be good at reconstruction, but weak at generation. \n\nSince the latent space structure/distribution is of a very flexible form, the reconstruction should be good by design. However, the cost from which is difficulty in obtaining samples from such a complicated latent distribution. \n\nGenerating samples by interpolating two points, as suggested in Sec. 4.4, could be too crude: 1) linear approximations of a manifold can have large error, especially when the number of training data is small. 2) samples obtained from such a process does not have a gurantee to converge to the true underlying density in probabilistic sense. As a result, generated samples can be inaccurate.\n\nThis might partially explain the results in Table 2 and Figure 10. The proposed model shows a FID score far worse than StyleGAN2 and the generated images are blurry. It however shows better recall, as the interpolation-based sampling scheme seems to be advantageous in covering all the data (almost perfect if reconstruction error is low).\n\n---\n\nAll in all, it is understandable that the suggested framework and the loss functions are a solution to the modality-agnostic training; GANs need a discriminator, which usually requires modality-dependant design. However, considering the aforementioned issues, and given that the the loss functions themselves are not directly connected to the modality-independance, there seems to be an alternative method. For example, one could employ the techniques from adversarial auto-encoders (A. Makhzani et al., 2015), which can enforce the latent distribution to be Gaussian, resolving the aforementioned issues, and still be modality-agnostic. \n\n\n\nMinor:\n\n1) (line 278) Isn't the (non-)blurriness indeed an important property we want to measure from the generated samples? \n2) The loss in Eq. (7) seems to be closely related to the Jacobian of the generative map. Is there a particular reason that Eq. (7) is used instead of Jacobian?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors properly described the limitations and the societal impact in Sec. 6. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_OYJoqvARXF", "reviewer": "Reviewer_oxSK", "summary": "This work introduces GEM - a model which learns the low-dimensional latent manifold structure of a dataset in a modality-independent way. GEM’s objective function is designed to encourage the learning of manifolds which exhibit the following properties, namely they: cover the data, are locally metric, and are globally consistent.  The model is trained on four datasets with different modalities (image, audio, 3D shape, and cross-modal image and audio). Qualitative and quantitative evaluations suggest that GEM can be used to do linear interpolation between points on the learned manifold, it can generate reconstructions of corrupted samples as well as generate new samples. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- To the best of my knowledge, this is a novel approach to manifold learning which can easily be applied to data from different modalities.\n- The evaluation protocol is extensive and is both qualitative and quantitative. It includes visualizations of interpolation between signals, signal generation, and signal inpainting (for images), as well as hallucinations of one modality given another. \n- The paper is organized and written clearly. \n\n### Room for Improvement and Questions\n1. The presented quantitative results are missing error bars with respect to the number of random seeds used even though the answer to this question on the Checklist is [YES].\n2. How were the hidden layer size (of 512) and number of layers in the implicit function $\\Phi$ and hypernetwork $\\Psi$ chosen? Have the authors tried other values for these hyperparameters and measured their effect on the reported results? \n3.  How was hyperparameter search performed for the benchmarks (StyleGAN2, FDN, ShapeNet)?\n4. Since GEM’s loss is a sum of three different components, how are the relative weights of each loss component determined?\n5. I noticed that in the Image Hallucination visualization in Fig 8 for an empty spectrogram the bow is very blurry or missing. Do the authors have a guess of what the reason for this might be? \n\n### Nitpicks\n- Line 72: “embeddings $\\it{obtained}$”\n- Figure 1: “represents $\\it{an}$ audio spectrogram”\n- Line 102: there is an extra “method”\n- Line 106: “coordinate” \n- Line 111: “the same across signal types” (without “for”)\n- Line 120: $W^{L_{h}}$ should be $W^{L}_{h}$\n- Line 129: there is an extra “in”\n- Line 172: “than signals that $\\it{are}$ perceptually distinct” \n\n### Rebuttal Period Update\nI've decided to increase my score to 6 after reading the author's response to my questions and concerns. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes GEM, an approach for multi-modal manifold learning, which is novel and with interesting results. I found the rebuttal very well done and I think it clarified many open issues with the approach, although not to the degree to have a total agreement of the reviewers. Nevertheless I find the clarifications sufficient and I am inclined to accept the work. 

Decision: Accept (Poster)