Learning Signal-Agnostic Manifolds of Neural Fields
Year: 2021

Abstract:
Deep neural networks have been used widely to learn the latent structure of datasets, across modalities such as images, shapes, and audio signals. However, existing models are generally modality-dependent, requiring custom architectures and objectives to process different classes of signals. We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains in a modality-independent manner. We cast our task as one of learning a manifold, where we aim to infer a low-dimensional, locally linear subspace in which our data resides. By enforcing coverage of the manifold, local linearity, and local isometry, our model — dubbed GEM — learns to capture the underlying structure of datasets across modalities. We can then travel along linear regions of our manifold to obtain perceptually consistent interpolations between samples, and can further use GEM to recover points on our manifold and glean not only diverse completions of input images, but cross-modal hallucinations of audio or image signals. Finally, we show that by walking across the underlying manifold of GEM, we may generate new samples in our signal domains1.

heading: 1 Introduction
text: Every moment, we receive and perceive high-dimensional signals from the world around us. These signals are in constant flux; yet remarkably, our perception system is largely invariant to these changes, allowing us to efficiently infer the presence of coherent objects and entities across time. One hypothesis for how we achieve this invariance is that we infer the underlying manifold in which perceptual inputs lie [1], naturally enabling us to link high-dimensional perceptual changes with local movements along such a manifold. In this paper, we study how we may learn and discover a low-dimensional manifold in a signal-agnostic manner, over arbitrary perceptual inputs.
Manifolds are characterized by three core properties [2, 3]. First, a manifold should exhibit data coverage, i.e., all instances and variations of a signal are explained in the underlying low-dimensional space. Second, a manifold should be locally metric, enabling perceptual manipulation of a signal by moving around the surrounding low-dimensional space. Finally, the underlying structure of a manifold should be globally-consistent; e.g. similar signals should be embedded close to one another.
Existing approaches to learning generative models, such as GANs [4], can be viewed as instances of manifold learning. However, such approaches have two key limitations. First, low-dimensional latent codes learned by generative models do not satisfy all desired properties for a manifold; while the underlying latent space of a GAN enables us to perceptually manipulate a signal, GANs suffer from mode collapse, and the underlying latent space does not cover the entire data distribution. Second, existing generative architectures are biased towards particular signal modalities, requiring custom architectures and losses depending on the domain upon which they are applied – thereby preventing us from discovering a manifold across arbitrary perceptual inputs in a signal-agnostic manner.
1Code and additional results are available at https://yilundu.github.io/gem/.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
As an example, while existing generative models of images are regularly based on convolutional neural networks, the same architecture in 1D does not readily afford high-quality modeling of audio signals. Rather, generative models for different domains require significant architecture adaptations and tuning. Existing generative models are further constrained by the common assumption that training data lie on a regular grid, such as grids of pixels for images, or grids of amplitudes for audio signals. As a result, they require uniformly sampled data, precluding them from adequately modeling irregularly sampled data, like point clouds. Such a challenge is especially prominent in cross-modal generative modeling, where a system must jointly learn a generative model over multiple signal sources (e.g., over images and associated audio snippets). While this task is trivial for humans – we can readily recall not only an image of an instrument, but also a notion of its timbre and volume – existing machine learning models struggle to jointly fit such signals without customization.
To obtain signal-agnostic learning of manifolds over signals, we propose to model distributions of signals in the function space of neural fields, which are capable of parameterizing image, audio, shape, and audiovisual signals in a modalityindependent manner. We then utilize hypernetworks [5], to regress individual neural fields from an underlying latent space to represent a signal distribution. To further ensure that our distribution over signals corresponds to a manifold, we formulate our learning objective with explicit losses to encourage the three desired properties for a manifold: data coverage, local linearity, and global consistency. The resulting model, which we dub GEM2, enables us to capture the manifold of a variety of signals – ranging from audio to images and 3D shapes – with almost no architectural modification, as illustrated in Figure 1. We further demonstrate that our approach reliably recovers distributions over cross-modal
signals, such as images with a correlated audio snippet, while also eliciting sample diversity.
We contribute the following: first, we present GEM, which we show can learn manifolds over images, 3D shapes, audio, and cross-modal audiovisual signals in a signal-agnostic manner. Second, we demonstrate that our model recovers the global structure of each signal domain, permitting easy interpolation between nearby signals, as well as completion of partial inputs. Finally, we show that walking along our learned manifold enables us to generate new samples for each modality.

heading: 2 Related Work
text: Manifold Learning. Manifold learning is a large and well-studied topic [2, 3, 6–10] which seeks to obtain the underlying non-linear, low-dimensional subspace in which naturally high-dimensional input signals lie. Many early works in manifold learning utilize a nearest neighbor graph to obtain such a low-dimensional subspace. For instance, Tenenbaum et al. [2] employs the geodesic distance between nearby points, while Roweis et al. [3] use locally linear subregions around each subspace. Subsequent work has explored additional objectives [7–10] to obtain underlying non-linear, low-dimensional subspaces. Recently, [11–13] propose to combine traditional manifold learning algorithms with deep networks by training autoencoders to explicitly encourage latents to match embeddings obtained from classical manifold learning methods. Further, [14] propose to utilize a heat kernel diffusion process to uncover the underlying manifold of data. In this work, we show how we may combine insights from earlier works in manifold learning with modern deep learning techniques to learn continuous manifolds of data in high-dimensional signal spaces. Drawing on these two traditions enables our model to smoothly interpolate between samples, restore partial input signals, and further generate new data samples in high-dimensional signal space.
2short for GEnerative Manifold learning
Learning Distributions of Neural Fields. Recent work has demonstrated the potential of treating fully connected networks as continuous, memory-efficient implicit (or coordinate-based) representations for shape parts [15, 16], objects [17–20], or scenes [21–24]. Sitzmann et al. [25] showed that these coordinate-based representations may be leveraged for modeling a wide array of signals, such as audio, video, images, and solutions to partial differential equations. These representations may be conditioned on auxiliary input via conditioning by concatenation [17, 26], hypernetworks [21], gradient-based meta-learning [27], or activation modulation [28]. Subsequently, they may be embedded in a generative adversarial [28, 29] or auto-encoder-based framework [21]. However, both of these approaches require modality-dependent architecture choices. Specifically, modification is necessary in the encoder of auto-encoding frameworks, or the discriminator in adversarial frameworks. In contrast, we propose to represent the problem of learning a distribution as one of learning the underlying manifold in which the signals lie, and show that this enables signal-agnostic modeling. Concurrent to our work, Dupont et al. [30] propose a signal-agnostic discriminator based on pointwise convolutions, which enables discrimination of signals that do not lie on regular grids. In contrast, we do not take an adversarial approach, thereby avoiding the need for a convolution-based discriminator. We further show that our learned manifold better captures the underlying data manifold and demonstrate that our approach is able to capture manifolds across cross-modal distributions, which we find destabilizes training of [30].
Generative Modeling. Our work is also related to existing work in generative modeling. GANs [4] are a popular framework used to model modalities such as images [31] or audio [32], but often suffer from training instability and mode collapse. VAEs [33], another class of generative models, utilize an encoder and decoder to learn a shared latent space for data samples. Recently autoregressive models have also been used for image [34], audio [35], and multi-modal text and image generation [36]. In contrast to previous generative approaches, which require custom architectures for encoding and decoding, our approach is a modality-independent method for generating samples distribution and presents a new view of generative modeling: one of learning an underlying manifold of data.

heading: 3 Parameterizing Spaces of Signals with Neural Fields
text: An arbitrary signal I in RD1×···×Dk can be represented efficiently as a function Φ which maps each coordinate x ∈ Rk to the value v of the feature coordinate at that dimension.
Φ : Rk → R, x→ Φ(x) = v (1)
Such a formulation enables us to represent any signal, such as an image, shape, or waveform – or a set of signals, like images with waveforms – as a continuous function Φ. We use a three-layer multilayer perceptron (MLP) with hidden dimension 512 as our Φ. Only the input dimension of Φ is varied, depending on the dimensionality of the underlying signal – all other architectural choices are the same across signal types. We refer to Φ as a neural field.
Parameterizing Spaces over Signals. A set of signals may be parameterized as a subspace F of functions Φi. To efficiently parameterize functions Φ, we represent each Φ using a low-dimensional latent space in Rn and map latents to F via a hypernetwork Ψ:
Ψ : Rn → F , z → Ψ(z) = Φ, (2)
To regress Φ with our hypernetwork, we predict individual weights WL and biases bL for each layer L in Φ. Across all experiments, our hypernetwork is parameterized as a three-layer MLP (similar to our Φ), where all weights and biases of the signal representation Φ are predicted in the final layer. Directly predicting weights WL is a high-dimensional regression problem, so following [37], we parameterize W via a low-rank decomposition as WL = WLs σ(WLh ), where σ is the sigmoid function. WLs is shared across all predicted Φ, while W L h = A
L×BL is represented as two low-rank matrices AL and BL – and is regressed separately for each individual latent z.

heading: 4 Learning Structured Manifold of Signals
text: Given a training set C = {Ii}Ni=1, of N distinct arbitrary signals Ii ∈ RD1×···×Dk , our goal is to learn – in a signal-agnostic manner – the underlying low-dimensional manifoldM⊂ RD1×···×Dk . In
particular, a manifoldM [2, 3], over signals I, is a locally low-dimensional subspace consisting and describing all possible variations of I. A manifold has three crucial properties. First, it must embed all possible instances of I. Second,M should be locally metric around each signal I, enabling us to effectively navigate and interpolate between nearby signals. Finally, it must be locally perceptually consistent; nearby points onM should be similar in identity. To learn a manifoldM with these properties, we introduce a loss for each property: LRec, LLLE, and LIso, respectively. LRec ensures that all signals I are embedded insideM (Section 4.1), LLLE forces the space near each I to be locally metric (Section 4.2), and LIso encourages the underlying manifold to be perceptually consistent (Section 4.3). As such, our training loss takes the form:
LTotal = LRec + LLLE + LIso (3) Please see Fig. 2 for an overview of the proposed manifold learning approach. We utilize equal weighting across each introduced loss term. We discuss each component of the loss in more detail below.

heading: 4.1 Data Coverage via Auto-Decoding
text: Given a parameterization of a subspace F (as introduced in Section 3) which represents our manifold M, we must ensure that Ψ, our hypernetwork, can effectively cover the entire subspace in which our signal I lies. A common approach to learn subspaces over signals is to employ a GAN [4], whereby a discriminator trains a hypernetwork Ψ(z) to generate signals that are indistinguishable from real examples. However, an issue with such an approach is that many I may be missing in the resultant mapping.
Here, to ensure that our manifold covers all instances, we utilize the auto-decoder framework [17, 21]. We learn a separate latent zi for each training signal Ii and train models utilizing the loss
LRec = ‖Ψ(zi)(x)− Ii(x)‖2 (4) over each possible input coordinate x. By explicitly learning to reconstruct each training point in our latent space, we enforce that our latent space, and thus our manifoldM, covers all training signals I .

heading: 4.2 Local Metric Consistency via Linear Decomposition
text: A manifoldM is locally metric around a signal I if there exist local linear directions of variation on the manifold along which the underlying signals vary in a perceptually coherent way. Inspired by [3], we enforce this constraint by encouraging our manifold to consist of a set of local convex linear regionsRi in our learned latent space. We construct these convex linear regionsRi using autodecoded latents zi (Section 4.1). Each latent zi defines a convex region, Ri – obtained by combining nearest latent neighbors z1i , . . . ,z j i , as illustrated in Figure 2.
To ensure that our manifoldM consists of these convex local regions, during training, we enforce that each latent zi can be represented in a local convex region,Ri, e.g. that it can be expressed as a
set of weights wi, such that zi = ∑ j w j iz j i and ∑ j w j i = 1. Given a zi we may solve for a given
wi by minimizing the objective ‖zi − ∑ j w j iz j i ‖2 = wTi Giwi with respect to wi. In the above expression, Gi is the Gram matrix, and is defined as Gi = kkT , where k is a block matrix, with row j corresponding to kj = zi− zji . Adding the constraint that all weights add up to one via a Legrange multiplier, leads to the following optimization objective:
L(wi, λ) = wTi Giwi + λ(1Twi − 1) (5) where 1 corresponds to a vector of all ones. Finding the optimal wi in the above expression then corresponds to the best projection of zi on the linear region Ri it defines. Taking gradients of the above expression, we find that wi = λ2G −1 i 1, where λ is set to ensure that elements of wi sum up to
1. The mapped latent z′i = ∑ j w j iz j i corresponds to the best projection zi onRi. To enforce that zi lies inRi, we enforce that this projection LLLE = ‖Ψ(z′i)(x)− Ii(x)‖2, (6)
also decodes to the training signal Ii, where we differentiate through intermediate matrix operations. To encourage wi to be positive, we incorporate an additional L1 penalty to negative weights.

heading: 4.3 Perceptual Consistency through Local Isometry
text: Finally, a manifold M should be perceptually consistent. In other words, two signals that are perceptually similar should also be closer in the underlying manifold, according to a distance metric on the respective latents, than signals that are perceptually distinct. While in general the MSE distance between two signals is not informative, when such signals are relatively similar to one another, their MSE distance does contain useful information.
Therefore, we enforce that the distance between latents zi and zj in our manifoldM are locally isometric to the MSE distance of the corresponding samples Ii and Ij , when relative pairwise distance of samples is small. On the lowest 25% of pairwise distances, we utilize the following loss:
LIso = ‖α ∗ ‖(zi − zj)‖ − ‖Ii − Ij‖‖. (7) Such local isometry may alternatively be enforced by regularizing the underlying Jacobian map.

heading: 4.4 Applications
text: By learning a modality-independent manifold M, GEM permits the following signal-agnostic downstream applications, each of which we explore in this work.
Interpolation. By directly interpolating between the learned latents zi and zj representing each signal, we can interpolate in a perceptually consistent manner between neighboring signals inM. Further, latents for novel signals may be obtained by optimizing LRec on the signal.
Conditional Completion. Given a partial signal Î, we can obtain a point on the manifoldM by optimizing a latent z using LRec + LLLE. Decoding the corresponding latent z then allows us to complete the missing portions of the image.
Sample Generation. Our manifoldM consists of a set of locally linear regionsRi in a underlying latent space z. As such, new samples inM can be generated by walking along linear regions Ri, and sampling points each such region. To sample a random point withinRi, we generate α between 0 and 1, and compute a latent ẑi = αzi + (1− α) ∗ zji + β ∗ N (0, 1) (corresponding to a random sample in the neighborhood of zi) and project it onto the underlying regionRi.

heading: 5 Learning Manifolds of Data
text: We validate the generality of GEM by first showing that our model is capable of fitting diverse signal modalities. Next, we demonstrate that our approach captures the underlying structure across these signals; we are not only able to cluster and perceptually interpolate between signals, but inpaint to complete partial ones. Finally, we show that we can draw samples from the learned manifold of each signal type, illustrating the power of GEM to be used a signal-agnostic generative model. We re-iterate that nearly identical architectures and training losses are used across separate modalities.
Datasets. We evaluate GEM on four signal modalities: image, audio, 3D shape, and cross-modal image and audio signals, respectively. For the image modality, we investigate performance on the CelebA-HQ dataset [38] fit on 29000 64× 64 training celebrity images, and test on 1000 64× 64 test images. To study GEM behavior on audio signals, we use the NSynth dataset [39], and fit on a training set of 10000 one-second 16kHz sounds clips of different instruments playing, and test of 5000 one-second 16kHz sound clips. We process sound clips into spectrograms following [32]. For the 3D shape domain, we work with the ShapeNet dataset from [40]. We train on 35019 training shapes at 64× 64× 64 resolution and test on 8762 shapes at 64× 64× 64 resolution. Finally, for the cross-modal image and audio modality, we utilize the cello image and audio recordings from the Sub-URMP dataset [41]. We train on 9800 images at 128× 128 resolution and 0.5 second 16kHz audio clips, which are also processed into spectrograms following [32] and test on 1080 images and associated audio clips. We provide additional cross-modal results in the supplement.
Setup and Baselines. We benchmark GEM against existing manifold learning approaches. Namely, we compare our approach with that of StyleGAN2 [31], VAE [42], and concurrent work FDN of [30], as well as heat kernel manifold learning [14]. We utilize the authors’ provided codebase for StyleGAN2, the PytorchVAE library 3 for the VAE, and the original codebase of FDN [30], which the authors graciously provided. We train all approaches with a latent dimension of 1024, and re-scale the size of the VAE to ensure parameter counts are similar. We report model architecture details in the appendix. We need to change the architecture of both the VAE and StyleGAN2 baselines, respectively, in order to preserve proper output dimensions, depending on signal modality. However, we note that identical architectures readily can and are used for GEM. For fitting cross-modal audiovisual signals, all models utilize two separate decoders, with each decoder in GEM identical to each other. Architectural details are provided in the appendix.

heading: 5.1 Covering the Data Manifold
text: We first address whether our model is capable of covering the underlying data distribution of each modality, compared against our baselines, and study the impact of our losses on this ability.
Modality Fitting. We quantitatively measure reconstruction performance on test signals in Figure 3 and find the GEM outperforms each of our baselines. In the cross-modal domain in particular, we find that FDN training destabilizes quickly and struggles to adequately fit examples. We illustrate test signal reconstructions in Figure 3 and Figure 4. Across each modality, we find that GEM obtains the sharpest samples. In Figure 3 we find that while StyleGAN2 can reconstruct certain CelebA-HQ
3https://github.com/AntixK/PyTorch-VAE
images well, it fails badly on others, indicating a lack of data coverage. Our results indicate that our approach best captures the underlying data distribution.
Ablations. In Table 1, we assess the impact of two of our proposed losses – LLLE and LIso – at enabling test-signal reconstruction, and find to both help improve the resultant test construction. We posit that both losses serve to regularize the underlying learned data manifold, enabling GEM to more effectively cover the overall signal manifold.
Applications to Other Models. While we apply LRec , LLLE, LIso to GEM, our overall losses may be applied more generally across different models, provided the model maps an underlying latent space to an output domain. We further assess whether these losses improve the manifold captured by StyleGAN2 via measuring reconstruction performance on test CelebA-HQ images. We find that while a base StyleGAN2 model obtains a reconstruction error of MSE 0.0044 and PSNR 24.03, the addition of LRec improves test reconstruction to MSE 0.0041 and PSNR 24.29 with LLLE losses further improving test reconstruction to MSE 0.0038 and PSNR 24.61. In contrast, we found that LIso leads to a slight regression in performance. We posit that both LRec and LLLE serve to structure the underlying StyleGAN2 latent space, while the underlying Gaussian nature of the StyleGAN2 latent space precludes the need for LIso. Our results indicate the generality of our proposed losses towards improving the recovery of data manifolds.

heading: 5.2 Learning the Structure of Signals
text: Next, we explore the extent to which the manifold learned by GEM captures the underlying global and local structure inherent to each signal modality. Additionally, we probe our model’s understanding of individual signals by studying GEM’s ability to reconstruct partial signals.
Global Structure. We address whether GEM captures the global structure of a signal distribution by visualizing the latents learned by GEM on the NSynth dataset with t-SNE [43]. We find in Figure 6
that the underlying pitch of a signal maps onto the resulting t-SNE. We further visualize the inferred connectivity structure of our manifold in Figure 7 (left) and visualize the nearest neighbors in latent space of the autodecoded latents (right), finding that nearby latents correspond to semantically similar faces – further supporting that GEM learns global structure. Note, we find that without either LLLE or LIso, the underlying connectivity of our manifold is significantly poorer. Local Structure. We next probe whether GEM learns a densely connected manifold, e.g. one which allows us to interpolate between separate signals, suggestive of capturing the local manifold structure. In Figure 5, we visualize nearest neighbor interpolations in our manifold and observe that the combination of LLLE and LIso enables us to obtain perceptually smooth interpolations over both image and shape samples. We provide additional interpolations and failure cases in the supplement.
Signal Completion. To investigate the ability of GEM to understand individual signals, we assess the recovery of a full signal when only a subset of such a signal is given. In Figure 7, we consider the case of inpainting CelebA-HQ faces, where a subset of the face is missing. We observe that GEM is able to obtain several different perceptually consistent inpaintings of a face, with individual completions exhibiting consistent skin and lip colors, and restoring structural features such as eyeglasses. We provide additional examples and failure cases in the supplement.
Additionally, we consider signal completion in the cross-modal audiovisual domain. Here, we provide only an input image and ask GEM to generate possible audio spectograms. As seen in Figure 8, we
IMNET Latent GAN GEM
FDN
observe that GEM is able to generate empty spectograms when the bow is not on the cello, as well as different spectrograms dependent on the position of the bow. Alternatively, in Figure 8, we provide only an input audio spectrogram and ask GEM to generate possible images. In this setting, we find that generated images have the bow off the cello when the input spectrogram is empty (with the bow missing due to GEM blurring the uncertainty of all possible locations of the bow), and at different positions depending on the input audio spectrogram. In contrast, we found that all baselines were unable to fit the audiovisual setting well, as discussed in Section 5.1 and shown in Figure 11.

heading: 5.3 Generating Data Samples
text: Finally, we investigate the ability of GEM to generate new data samples.
Qualitative Generations. We show random samples drawn from GEM and compare these to our baselines. We consider images in Figure 10, shapes in Figure 11, audio snippets in Figure 9 and audiovisual inputs in Figure 11. While we find that GEM performs comparably to FDN in the image regime, our model significantly outperforms all baselines on domains of audio, shape, and audiovisual modalities. We further display nearest latent space neighbors of generations on CelebA-HQ in Figure 10 and find that our generations are distinct from those in the training dataset.
Quantitative Comparisons. Next, we provide quantitiative evaluations of generations in Table 2 on image and shape modalities. We report the FID [44], precision, and recall [45] metrics on CelebAHQ 64× 64. We find that GEM performs better than StyleGAN2 and FDN on precision and recall, but worse in terms of FID. We note that our qualitative generations in Figure 10 are comparable to those of FDN; however, we find that our approach obtains high FID scores due to the inherent sensitivity of the FID metric to blur. Such sensitivity to bluriness has also been noted in [46], and we
find that even our autodecoded training distribution obtains an FID of 23.25, despite images appearing near perceptually perfect (Figure 3). On ShapeNet, we report coverage and MMD metrics from [47] using Chamfer Distance. To evaluate generations, we sample 8762 shapes (the size of test Shapenet dataset) and generate 2048 points following [40]. We compare generations from GEM with those of FDN and latent GAN trained on IMNET (using the provided code in [40]). We find that GEM outperforms both approaches on the task of 3D shape generation.

heading: 6 Conclusion
text: We have presented GEM, an approach to learn a modality-independent manifold. We demonstrate how our model enables us to interpolate between signals, complete partial signals, and further generate new signals. A limitation of our approach is that while underlying manifolds are recovered, they are not captured with high-fidelity. We believe that a promising direction of future work involves the pursuit of further engineered inductive biases, and general structure, which enable the recovery of high-fidelity manifolds of the plethora of signals in the world around us. We emphasize, however, that as our manifold is constructed over a dataset; GEM may learn to incorporate and propogate existing prejudices and biases present in such data, posing risks for mass deployment of our model. Additionally, while our work offers exciting avenues for cross-modal modeling and generation, we note that GEM has the potential to be used to create enhanced "deep fakes" and other forms of synthetic media.
Acknowledgements This project is supported by DARPA under CW3031624 (Transfer, Augmentation and Automatic Learning with Less Labels), the Singapore DSTA under DST00OECI20300823 (New Representations for Vision), as well as ONR MURI under N00014-18-1-2846. We would like to thank Bill Freeman and Fredo Durand for giving helpful comments on the manuscript. Yilun Du is supported by an NSF graduate research fellowship.

Review summaries:
This work introduces GEM - a model which learns the low-dimensional latent manifold structure of a dataset in a modality-independent way. GEM’s objective function is designed to encourage the learning of manifolds which exhibit the following properties, namely they: cover the data, are locally metric, and are globally consistent.  The model is trained on four datasets with different modalities (image, audio, 3D shape, and cross-modal image and audio). Qualitative and quantitative evaluations suggest that GEM can be used to do linear interpolation between points on the learned manifold, it can generate reconstructions of corrupted samples as well as generate new samples. 

This paper proposes GEM model to capture the underlying structure of datasets across modalities. It can obtain perceptually consistent interpolations between samples, and can further recover points on the manifold and realize audio/image signals completion. Besides, it can also generates new samples. 

This paper proposes a modality-agnostic implicit generative model. A majority of the existing generative models involve architecture adept in a particular domain (e.g., convolutional layers for images), often requiring careful design and incurring difficulties in dealing with multi-modal data. To tackle this, the authors propose a generative manifold learning (GEM) scheme, which faithfully reflects the manifold hypothesis by utilizing hypernetwork-based implicit modeling. While implicit modeling had been proposed in previous work, unique to GEM is the use of the proposed local isometry and LLE loss functions. Such loss design removes the need of auxiliary networks (e.g., discriminator for GANs), which might require modality-dependent desgin, making the training of the model as well be completely modality-agnostic.