DDMI: Domain-agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations
Authors: Dogyun Park
Sihyeon Kim
Sojin Lee
Hyunwoo J. Kim
Venue: ICLR 2024 poster (2024)
Keywords: Implicit neural representation
generative model
domain agnostic
diffusion model
Primary area: generative models
TL;DR: We propose a latent diffusion model that generates hierarchically decomposed positional embeddings of Implicit neural representations, enabling high-quality generation on various data domains.

Abstract: Recent studies have introduced a new class of generative models for synthesizing implicit neural representations (INRs) that capture arbitrary continuous signals in various domains. These models opened the door for domain-agnostic generative models, but they often fail to achieve high-quality generation. We observed that the existing methods generate the weights of neural networks to parameterize INRs and evaluate the network with fixed positional embeddings (PEs). Arguably, this architecture limits the expressive power of generative models and results in low-quality INR generation. To address this limitation, we propose Domain-agnostic Latent Diffusion Model for INRs (DDMI) that generates adaptive positional embeddings instead of neural networks' weights. Specifically, we develop a Discrete-to-continuous space Variational AutoEncoder (D2C-VAE) that seamlessly connects discrete data and continuous signal functions in the shared latent space. Additionally, we introduce a novel conditioning mechanism for evaluating INRs with the hierarchically decomposed PEs to further enhance expressive power. Extensive experiments across four modalities, \eg, 2D images, 3D shapes, Neural Radiance Fields, and videos, with seven benchmark datasets, demonstrate the versatility of DDMI and its superior performance compared to the existing INR generative models. Code is available at \href{https://github.com/mlvlab/DDMI}{https://github.com/mlvlab/DDMI}.

Published as a conference paper at ICLR 2024
DDMI: DOMAIN-AGNOSTIC LATENT DIFFUSION MODELS FOR SYNTHESIZING HIGH-QUALITY IM-
PLICIT NEURAL REPRESENTATIONS
Dogyun Park, Sihyeon Kim, Sojin Lee, Hyunwoo J. Kim
Department of Computer Science
Korea University
Scoul, South Korea {gg933, sh_bs15, soj in lee, jyunwoojkimjj@koree ac.
ABSTRACT
Recent studies have introduced new class of generative models for synthesizing implicit neural representations (INRs) that capture arbitrary continuous signals in various domains. These models opened the door for domain-agnoste genera- tive models, but they often fail achieve high-quality generation. We observed that the existing methods generate the weights of neural networks to parameterize INRs and evaluate the network with fixed positional embeddings (PEs). Arguably, this architecture limits the expressive power of generative models and results in low-quality INR generation. To address this limitation, we propose Domain- agnostic Latent Diffusion Model for INRs (DDMI) that generates adaptive posi- tional embeddings instead of neural networks' weights. Specifically, we develop peccete-to-continuee space Variational AutoEncoder (D2C- VAE) that seam- lessly connects discrete data and continuous signal functions in the shared latent space. Additionally, we introduce a novel conditioning mechanism for evaluat- ing INRs with the hierarchically decomposed PEs to further enhance expressive power. Extensive experiments across four modalities, e.g., 2D images, 3D shapes, Neural Radiance Fields, and videos, with seven benchmark datasets, demonstrate the versatility DDMI and its superior performance compared to the existing INR generative models. Code is available at http::/gghhub..cm////ab/b/DD
INTRODUCTION
Implicit neural representation (INR) is popular approach for representing arbitrary signals tinuous function parameterized by a neural network. INRs provide great flexibiiity and expressivity with simple neural network like small multi-layer perceptron (MLP). INRs virtually comain-agnostic representations that can be applied to a wide range of signals domains, such as im- age (Müller al., 2022; Tancik al., 2020; Sitzmann al., 2020), shape/scene model (Mescheder et al., 2019; Peng et al., 2020; Park et al., 2019), video reconstruction (Nam et al., 2022b; Chen al.. 2022), and novel view synthesis (Mildenhall al., 2021: Martin-Brualla et al., 2021; Park et al., 2021; Barron et al., 2021). Also, INR enables the continuous representation of signals at arbi- trary scales and complex geometries. For instance, given an INR of an image, applying zoom-in/out or sampling an arbitrary- arbitrary-reoolution image is readily achievable, leading to superior performance in super-resolution (Chen al., 2021; Xu et al., 2021). Lastly, INR represents signals with high quality leveraging the recent advancements in parametric positional embedding (PE) (Müller et al., 2022; Cao & Johnson, 2023).
Recent research has expanded its attention to INR generative models using Normalizing Flows (Dupont et al., 2022a), GANs (Chen Zhang, 2019; Skorokhodov et al., 2021; Anokhin et al., 2021), and Diffusion Models (Dupont et al., 2022a; Zhuang et al., 2023). Especially, Dupont et al. (2022a); Zhuang et al. (2023); Du et al. (2021); Dupont et al. (2022b) have focused on devel- oping generic framework that can be applied across different signal domains. This is primarily
"Corresoonding author.

---

Published as a conference paper at ICLR 2024
Image generation
shape generation
Text prompt: curved chair"
Text "A metallic folding chair"
STo013
Arbitraryrresoluiion image generation
Text-to-shape generation
peneration
Neural field
Figure Generation results DDML. Our DDMI generates high-quality samples across four distinct domains including image, shape, video, and Neural Radiance Fields. DDMI also shows remarkable results for applications like arbitrary-scale image generation or text-to-shape generation.
accomplished by modeling the distribution of INR 'weights' by GAN (Dupont et al., 2022b), latent diffusion model (Dupont et al., 2022a), or latent interpolation (Du et et., 2021). However, these mod- els often exhibit limitations in achieving high-quality results when dealing with large and complex datasets. Arguably, this is mainly due to their reliance on generating weights for an INR function with fixed PEs. This places substantial burden function weights capture diverse details in multiple signals, whereas the careful designs of PE (Müller et al., 2022; Chan et al., 2022; Cao & Johnson, 2023) have demonstrated greater efficiency and effectiveness in representing signals.
Therefore, in this paper, we propose Domain-agnostic Latent Diffusion Model for INRs (DDMI) that generates adaptive positional embeddings instead of neural networks' weights (see Fig. for conceptual comparison). Specifically, we introduce Discrete-to-comiinouss space Variational Au- toEncoder (D2C-VAE) framework with encoder that maps discrete data into the latent space and decoder that maps the latent space to continuous function space. D2C- VAE generates basis fields using the decoder network conditioned on a latent variable. This means we define sumple-ppecific basis functions for generating adaptive PEs, shifting the primary expressive power from MI.P to PE. Additionally, we propose two modules that further enhance the expressive capacity of INR: 1) Hi- erarchically Decomposed Basis Fields (HDBFs): we decompose the basis fields into multiple scales to better account for the multi-scale nature of signals. 2) Coarse-to-Fine Conditioning (CFC); we introduce novel conditioning method, where the multi-scale PEs from HDBFs progressively conditioned on MLP coarse-to- to-finn manner. Based on D2C-VAN VAE, we train the latent diffusion model on the shared latent space (see Fig. for the overall framework). Ultimately, model can generate high-quality continuous functions across a wide range of signal domains (see Fig. I). To summarize, contributions as follows:
We introduce the Domain-agnostic Latent Diffusion Model for INRs (DDMI), generative model synthesizing high-quality INRs across various signal domains.
We define Discrete to Continuous space Variational AutoEncoder (D2C-VAE) that gener- ates adaptive PEs and learns the shared latent space to connect the discrete data space and the continuous function space.
We propose sawrarcciiall--ecooppseee Basis Fields (HDBFs) and Coarse-to-Fine Condi- tioning (CFC) to enhance the expressive power.
Extensive experiments across four modalities and seven benchmark datasets demonstrate the versatility of DDMI. The proposed method significantly outperforms the existing INR generative models, demonstrating the efficacy of our proposed methods.
2 RELATED WORKS
wrR-based generative models. Several works have explored the use of INR in generative mod- eling to leverage its continuous nature and expressivity. Especially the INR generative models
2

---

Published as a conference paper at ICLR 2024
Discrete data X
Continuous function (
HDBF
Irage
Latent
+
Grid features
Encoder Bp
Decodar De
MI.P
Video
(C)
Lalent Diffusion
Zy
Forward process
Coarse-to- Conditioning
Zy
Zt
7-1
Figure 2: Overall pipeline of DDML. Discrete data and continuous function w connected in the shared latent space z (D2C-VAF). The decoder generates perarchically-Decompee Fields (HDBFs) given latent variable p represents the coarsest scale PE and p corresponds to the finest scale PE. The MLP returns the signal value for queried coordinate c using the Coarse-to- Fine Conditioning method. Latent diffusion model operates on the shared latent space. Note that we use a tri-plane latent variable for 3D and video, and a single plane for 2D image.
Basis
known for their ability to generate data at arbitrary scales with a single model. Thus, there has been a surge of recent works across multiple modalities. For 2D images, CIPS (Anokhin et al., 2021) and INR- UR-GAN (Skorokhodov et al., 2021) employ GANs synthesize continuous image functions. For 3D shape, recent studies (Nam et al., 2022a; Zheng et al., 2022; Li et al., 2023; Erkoç et al., 2023) have proposed generating shapes Signed Distance Functions (SDFs) using GANs diffusion models. Also, for videos, DIGAN (Yu et al., 2022) and StyleGAN-V (Skorokhodov et al., 2022) have introduced GAN-based architectures generate videos continuous spatio-temporal func- tions. However, since these models are designed for specific modalities, they cannot easily adapt to different types signals. Another line research has explored the domain-agnostic architectures for INR generations, such as GASP (Dupont et al., 2022b), Functa (Dupont et al., 2022a), GEM (Du al., 2021), and DPF (Zhuang et al., 2023). Zhuang et al. (2023) directly applies diffusion models to explicit signal fields to generate samples at the targeted modality, yet it faces a scalability issue when dealing with large-scale datasets. Others (Dupont et al., 2022b;a; Du et al., 2021; Koyuneu al, 2023) attempt to model the weight distribution of INRs with GANs, diffusion models, or latent interpolation. In contrast, we introduce a domain aggossicc generative model that generates adaptive positional embeddings instead of weights of MLP in INRs.
Latent diffusion model. Diffusion models (Ho et al., 2020; 2022; Song et al., 2021) have demon- strated remarkable success in generation tasks. They consistently achieve high-quality results and high distribution coverage (Dhariwal & Nichol, 2021), often outperforming GANs. However, their iterative reverse process, typically involving large number of steps (e.g., 1000 steps), renders them significantly slower and inefficient compared to the implicit generative models like VAES (Kingma & Welling, 2013) and GANs. To alleviate this limitation, recent works (Rombach al., 2022; Vahdat et al., 2021) 1) have proposed learning the data distribution in oow-dimensional latent space, which offers computational efficiency. Latent diffusion models (LDMs) strike a favorable balance between quality and efficiency, making them an attractive option for various applications. Our work adopts latent space approach for designing the computationall-fficce generative model.
METHODOLOGY
We present a Domain-agnostic latent Diffusion Model for synthesizing high-quality Implicit neural representations (DDMI). In order to learn to generate continuous functions (INRs) from discrete data, e.g., images, we propose a novel VAE architecture D2C-VAE that maps discrete data to con- tinuous functions via shared latent space in Sec. 3.1. To enhance the quality of INR generation, introduce a coarse-to to to-fine conditioning (CFC) mechanism for evaluating INRs with the generated hierarchicallyydecomppee basis fields (HDBFs). Sec. 3.2 outlines the two-stage training proce-
3

---

Published as a conference paper at ICLR 2024
dures of the proposed method. As with existing latent diffusion models (Rombach et al., 2022), D2C-VAE learns the shared latent space in the first stage. In the second stage, the proposed frame- work trains diffusion model in the shared latent space while keeping the other networks fixed. The overall pipeline is illustrated in Fig. 2.
3.1 DDMI
Let w, ④ denote a continuous function repre- senting arbitrary signal and its approximation by neural networks respectively, where $2 is a continuous function space. Given a spatial spatiotemporal co- ordinate c E RT,, and its corresponding signal value R", training data x can be seen the evaluations of a continuous function at a set of coordinates, i.e., [c))]]=== w(c), where R R and / is the number of coordinates.
T(6]
(()
It
G
weight generation
(E)
D2C-VAE. We propose asymmetric VAE architec- ture, dubbed as Discrete-to-comtinuet
(())
space Varia- tional Auto-Encoder (DEC-VAE), to seamlessly con- nect a discrete data space and a continuous function space via the shared latent space. Specifically, the en- coder Eg maps discrete data x to the latent variable as 2D grid features, e.g. a 2D plane for images or a 2D tri-plane for 3D shapes and videos. The decoder D generates basis fields E, i.e., E = Dr where we define E as a set of dense grids consisting of generated basis vectors. Then, the positional embedding for the coordinate is computed by p (c; E), where function 7 performs bilinear interpolation on E, calcu- lating the distance-weighted average of its four near- est basis vectors at coordinate c. For tri tri-plane basis fields, an axis aligned orthogonal projection is applied beforehand (see Fig. 2). In this manner, the proposed method adaptively generates PFs according to different basis fields. Finally, the MI.P my returns the signal value given the positional embedding, i.e., f (p) (C). Fig. 3 shows the distinction between the proposed method and existing INR generative models. We observed that the PE generation improves the expressive power of INRs compared to weight generation approaches, see Sec. 4.
3 S~G G
G
generation
Figure 3: Comparison between weight generation and PE generation for INR generative models G. c is coordinate, p is a PE, y function that maps coor- dinates to PEs, wy is MLP, and @(c) is predicted signal value. For PE generation, sample basis fields E from G instead of The red line indicates the generation.
basis fields. Instead of relying on single-scale basis field, we pro- pose decomposing it into multiple scales better account for signals' multi-scale nature. To ef- ficiently generate multi-scale basis fields, we leverage the feature hierarchy of a single neural net- work. Specifically, the decoder D& outputs feature maps at different scale i, i.e., (z) (2)) n.....n). The feature maps undergo x convolution to match the feature dimensions across scales. We refer to these decomposed basis fields Then, we can compute multi-scale PEs {p') as p' >(; E`), for all i. In practice, we use three different scales of basis fields from three different levels of layers. Sec. 4.4 qualitatively validates the spatial frequencies in HDBFs of generated samples are decomposed into multiple levels. This demonstrates that each field dedicated learning a specific level of detail, leading to a more expressive representation.
Hferarchicalll--dcoom
Basis Fields (HDBFs).
Coarse-to-fne conditioning (CFC), naïve approach to using multi-scale positional embeddings from HDBFs for MLP To is concatenating them along channel dimensions. However, we found that it is suboptimal. Thus, we introduce a conditioning mechanism that gradually conditions MLP Ty coarse PEs to fine PEs. The intuition is to encourage the lower-scale basis field to focus on the details missing from the higher-scale basis field. To achieve this, we feed the PE from the lowest-scale basis field as input to the MLP block and then concatenate (or element- wise sum) its intermodiate output with the next PE from the higher-scale basis field. We continue this process for subsequent scales until reaching the n-th scale (see Fig. 2).
4

---

Published as a conference paper at ICLR 2024
3.2 TRAINING PROCEDURE AND INFERENCE
The training of DDMI involves two stages: VAE training and diffusion model training. In the first stage. VAC learns the shared latent space with an encoder Es map discrete data to latent vector Z and a decoder De to generate basis fields E. In the second stage, a diffusion model is trained the latent space learn the empirical distribution of latent vectors
D2C- training. We define training objective for D2C-VAE that maximizes the evidence lower bound (ELBO) of log-likelihood of the continuous function w with discrete data as:
log p(w) = log
Poi.zo (wlz) p(z)dz
(1)
Po.to (wlz)
log
(z(x)) g p(z) dz
(2)
(eirool)l)) p((2) q0l (z)(2)x) dz
2 log
(3)
go(zlxx
90 [00(2/x)
ga(z)x) log (w0) - -log
(4)
p(z)
(x(((())) [log Po.gP (w|z)] DKL (4o (z))))))pp((
(5)
where the inequality in Eq. is by Jensen's inequality. 2o (z(2)x) is the approximate posterior, p(z) a prior, and Pe.mo (x(z) is the likelihood. The first term in Eq. 5 measures the reconstruction loss,
and the KL divergence between the posterior and prior distributions p(z) encourages latent vectors follow the prior. However, since do not have observation w but only discrete data w()). we approximate Po.mo (w/z) by assuming coodinate-wis independence
Poi (wlz) Rs Po (w (c))zz
II Pv. (w)(l)))
(6)
DEC
where ((c) = Te(7(C, (z))) Thus, our training objective in Eq. 5 can be approximated as Egg (x/x) Plog (w/z)) DKL (go (20(2)))
Lo.vo(()) =
R (x)x)
DxL((((2>x)(lp(2)) ((()))
(8)
LCE
Since the reconstruction loss varies depending on the number of coordinates, ie., |c). in practice, we train D2C- -AE with the re-weighted objective function given as:
(x) E. (z)x))cc [log Po.me (w())()) X Dx
where Ay balances the two losses, and p(z) is standard normal distribution.
Diffusion model training. Following existing LDMs (Vahdat et al., 2021; Rombach et al., 2022; Ho al., 2020). the forward diffusion process is defined in the learned latent space as Markov chain with pre-defined Gaussian kernels g( 2. 1) == (2,; v1 3, 3,1). The forward diffusion process is given as q(zi.T (z))) 112 ( where T indicates the total number of diffusion steps and 3e is pre-defined noise schedule that satisfies q(zr) ~ N N((T; 0, I). The reverse diffusion process is also defined in the latent space P. = p(zr) II Po (o (zz). where pap (zo) (2z;; M4p (Ze, (), pi is parameterized by
is LDM prior. The Gaussian kernels (.p neural network php (Zt, and of is the fixed variances. The reverse diffusion process (z (2) is
trained with the following noise prediction using reparameterization trick:
[z(z) Ez,,e,t [w(t)]le cz t)]
(10)
Inference. Continuous function generation involves the reverse diffusion process Pe, D2C- decoder D&. and read-out MLP тe. First, latent Zo ~ Pio (zo) generated by iteratively conducting
ancestral sampling, 74-1 Var
9r Vi-ai e
+ ocS, where Ç _N(0, I) and p(zr)
(2 I). Then, the latent 2n is fed to D2C-VAL decoder D to generate HDBFs E. Finally, combining HDBFs with MLP va learned in the first stage, the proposed method parameterizes a continuous function w-.).
5

---

Published as a conference paper at ICLR 2024
Table 1: FID results on CelebA-HQ
Table 3: FID results on AFHQv2 Dog.
6F
128
258
384
128"
156"
384'
<Discrete representation>
Diieencte representation
T.DM (Rombach et al, 2022))
5:51
BAGN (Karres 20201o
6.73
LSGM (Vabdat et al., 2021)
7.22
Consiinous representation:>
Demain-spesiffe
<Contimouu representation.>
INR-GAN (Skorkhhokkv et al., 2021)
31,27
Domain-spocifee
CIPS (Amokhin et oll, 2021)
26,95
23.93
28.97
INR INR-GAN (Skomkhodoy et 2021)
2021)
10.3
Demain-agnoslte
Domain-agnostic
CIPS (Anokhin al, 2021)
15.41
13.53
DIMI (Ours)
(upoon al.,
10.81
8.54
78
11.47
Functa (Dupont
2022n)
40.4
Table 4: FID results on AFHQv2 Cat.
GEM (Du at, 2021)
30.4
GASP (Dupont al,, 20226)
13.5
19.2
DPF (Zhonng et 2025)
13.2
128'
250
381"
DDMI (Ours)
9.74
8,73
7.25
10.44
<Discrete ropresetation>>
Table 2: Precision and Recall results.
StyleGAN (Kamas et 20200)
3:25
<Contimouss representation>
Domain specific
CelebA-Ho P1
A
INR-GIAN (Storokhodow nt, 2021)
11.2
CCnntinaous representation:
R
PT
RT
CIPS (Anokhin 2021)
7.85
7.35
11.8
INR-GAN (Storoshnlov at, al, 1021)
0.671
0.335
0.719
0.281
Domainagroelic
CIPS (Anokhin et 2021)
0.682
0.287
0,716
0.117
GASP (Dugont et al., 20226)
17.48
DDMI (Ours)
0.734
0.408
0808
0.367
DDMI (Ours)
5,88
4.27
7.94
4 EXPERIMENTS
We evaluate the effectiveness and versatility of DDMI through comprehensive experiments across diverse modalities, including 2D images, 3D shapes, and videos. We assume multivariate normal distribution for the likelihood function 4s (w((cc) |z) for images and videos and Bernoulli distribution for shapes (occupancy functioon. For all domains, the multivariate normal distribution used for the posterior PV.,, x). For the type of latent variables for each domain, the encoder maps input data to the latent variable as 2D grid features, egg., a single 2D plane for images using 2D CNN- based encoder (Ho et al., 2020) or a 2D tri-plane for 3D shapes following Conv-ONET (Peng al., 2020) and videos as outlined in Timesformer (Bertasius et al., 2021). Then, we use a 2D CNN-based decoder D to convert latent variable z into basis fields E. Additional experiments (e.g.. NeRF) and more implementation details, evaluation, and baselines are provided in the supplement.
4.1 2D IMAGES
Datasets and baselines. For images, we evaluate models on AFHQV2 Cat and Dog (Choi et al., 2020) and CelebA-HQ dataset (Karras et al., 2018) with a resolution of 2562. We com- pared our method with three groups of models: 1) Domain-agnosti INR generative models such as Functa (Dupont et al., 2022a), GEM (Du et al., 2021), GASP (Dupont et al., 2022b), and DPF (Zhuang et al., 2023), which are the primary baselines, 2) Domain-specific INR generative models that are specifically tailored for image generation like INR-GAN (Skorokhodov et al., 2021) and CIPS (Anokhin et al., 2021). Apart from DPF, which generates the explicit signal field, every baseline operates weight generation, whereas ours opts for PE generation. 3) Discrete represen- tation based generative models for reference. We provide results from tftt-of-theear generative models (Vahdat et al., 2021; Rombach et al., 2022; Karras et al., 2020a) that learn to generate dis- crete images.
Quantitative analysis. We primarily measure FID (Heusel et al., 2017), following the setup in (Sko- rokhodov et al., 2021). Tab. I, 3, and 4 showcase the consistent improvement of DDMI over pri- mary baselines for multiple resolutions. For instance, Tab. 1 shows that compared to DPF, the recent domain- aggasii INR generative model, our approach achieves an FID score of 9.74 as opposed to 13.2 on the felebA-HQ at resolution of 642. Moreover, on AFHQv2 Cat in Tab. 4, DDMI demon- strates superior performance over CIPS, specifically developed for arbitrary scale image generation, achieving an average FID improvement of 2.97 across three different resolutions. In Tab. 2, we com- pare precision and recall with image-targeted INR generative models. Here, precision and recall are indicative of fidelity and diversity, respectively. DDMI exhibits a significant advantage over both baselines (Skorokhodov et al., 2021; Anokhin et al., 2021). )). demonstrating superior performance for both CelebA-HH .Hoba-HHaaaa and AFHQv2 Cat datasets. We provide additional results on CIFARIO (Krizhevsky et al., 2009) and Lsun Churches eYu al., 2015) in Tab. 11 of the supplement.

---

Published as a conference paper at ICLR 2024
64
128
256
256
5124
1024
ScaleParty (Discrete representation)
(a) CIPS (domain-specific INR)
642
128
256
256
5124
10244
(h) Ours
(b) Ours
Figure 4: Comparison between DDMI and
Figure 5: Comparison between DDMI and CIPS on arbitrary-scale generation.
ScaleParty on arbitrary-scale generation.
Qualitative analysis. For further validation, we generate images at arbitrary scales and conduct comparisons with notable models: ScaleParty (Ntavelis al., 2022), a recent generative model designed for multi-resolution discrete images, and CIPS, an image-targeted INR generative model. DDMI demonstrates an impressive capability to consistently generate clearer images across various resolutions, from low to high. In Fig. 4, our model excels generating images with preserved facial structure, whereas Ntavelis for lower-resolution cases. Also, in Fig. 5, DDMI succeeds in capturing high-frequency details across images of varying resolutions, while Anokhin et (2021) tends lack finer details as the resolution increases.
(2022) struggles to maintain the global structure of the image
4.2 3D SHAPPS
Datasets and baselines. For shapes, we adopt the ShapeNet dataset (Chang et al., 2015) with two settings: a single-class dataset with 4K chair shapes and a multi-class dataset comprising 13 classes with 35K shapes, following the experimental setup in (Peng et al., 2020). We learn the shape as the occupancy function (Mescheder et al., 2019) R + {0, 1}. where it maps 3D coordinates to occupancy values, e.g., 0 or Still, domain-agnostic INR generative models from Sec. 4.1 are primary baselines, where also compare with Nomainn-spectio INR generative models for 3D shapes like 3D-LDM (Nam et al., 2022a) and SDF-Diffusion (Shim et al., 2023). We also include results from generative models for generating discrete shapes, such as point clouds.
Quantitative analysis. We conduct extensive experiments with unconditional and conditional shape generation. Beginning with unconditional shape generation (Tab. 5), we measure fidelity and diver- sity using Minimum Matching Distance (MMD) and Coverage (Achlioptas et al., 2018). DDMI sur- passes both domain-agnostic and domain-specifie baselines, achieving the best MMD for the single chair class (1.5) and multi-class (1.3) settings. Moreover, we attain the highest COV in multi-class shape generation, showcasing our ability to generate diverse shapes with high fidelity.
Next, we provide text-guided shape generation results on Text2Shape (T2S) dataset (Chen et al., 2019), comprising 75K paired examples text and shapes on chairs and tables. For training, we utilize pre-trained CLIP text encoder (Radford et al., 2021) for encoding text prompt t into em- bedding 7t)) and condition it to LDM (z( T()) by cross-attention (Rombach et al., 2022). For generation, the text-conditioned score is derived using classifier-free guidance (Ho & Salimans, 2022): é0 (2, f, (t)) = e c0(2,, t, 0) + w (co((2, (t),t)((t)))) (t)) - cØ(2,, t, t)). where 0 and w indicates an empty prompt and guidance scale, respectively. We measure classification accuracy, CLIP-S (CILP similarity score), and Total Mutual Difference (TMD) to evaluate against shape generative mod- els (Mittal et al., 2022: Li al., 2023; Liu et al., 2022). Tab. 7 illustrates the strong performance of

---

Published as a conference paper at ICLR 2024
Table 5: Generation results on 3D shapes.
Table 6: Generation results on videos.
Chair MMD
COVT
MMD'
Cluss
COV
SkyTinlappse FVD
<Diserele r
PVD et
0.41
<Discrete regresentation
DPM3D (Lao & 2021)
1.3
0.567
VideoGPT (Yan al, 2021)
222.7
MoCoGAN (Tulyakov ct 2018)
206.6
<Continoous eppresestttion>
MaCuGAN HD (Tian et 2021)
164.1
LatenGAN (Chen Zhang, 20199
Domain specific
1.7
0.38y
IVDM (He et al,, 2022)
95.2
3D LDM (Nam 2022a)
1,68
0.426
PVDM (Yu et al. 2023)
SDF-SryleGAA (Zhong et 2022)
1.9
0.41
1.55
0,398
SDF-Diffsiin
2023)
0.498
<oontinuous representation
HyperDiflusion (Erkoç
0.530
Domain specific
GASP (Dupont et al., 20226)
agnostic
2.5
0.353
0.341
DIGAN (Yu et al., 2022)
83.11
GEM (Da et al, 2021)
14
0,409
StyleGAN-V (Skorokbodov 2022)
79.52
DPF (2hnang ol., 2022)
1.5
0.419
Domain-agnosse
DDMI (Ours)
13
0.50
0.421
DDMI (Ours)
66.25
are on Acnonyen (Uppeer 2011
Table 7: Text-to-shape generation results.
Table 8: Ablation study.
Ace
CLIPST
TMD J
Domain-specific
Generation target
First Stage (PSNR)
Second Stage
IMLE (Liu 2022)
34.79
0.891
HDBI's
CRC
(FID)
Auto-SDF (Mittal et 2022)
83.88
0.581
PE
32.72
8.54
Diffusion-SDF (Li et al. 2023)
88.56
28.63
0.169
PE PE
33.17
8.23
Damain-agnostic
33.56
7.82
DDMI (Ours, ur
91.30
30.30
0.204
DDMI, indicating our method generates high-fidelity shapes (accuracy and CLIP-S) with a diversity level comparable to baselines (TMD).
Qualitative analysis. Fig. and Fig.6 present qualitative results. Fig. 6 displays visualizations of text-guided shape generations, demonstrating our model's consistent generation of faithful shapes given text conditions (e.g., "a two-layered table") over Diffusion-SDD (Li et al., 2023). Fig. incor- porates the comprehensive results, including unconditional shape generation, text-conditioned shape generation, and unconditional Neural Radiance Field (NeRF) generation. Especially for NeRF, we train DDMI with SRN Cars dataset (Sitzmann et al., 2019). Specifically, encode point clouds to the 2D-triplane HDBFs, allowing MIPP to read out triplane features at queried coordinate and ray direction into color and density via neural rendering (Mildenhall et al., 2021). For sults, including a qualitative comparison between Functa (Dupont et al., 2022a) and ours on NeRF generation, see Fig. 15 in the supplement.
4.3 VIDEOS
Datasets and baselines. For videos, the SkyTimelapse dataset (Xiong et al., 2018) and preprocess each video to have 16 frames and resolution of 2562, following the conventional setup used in recent video generative models (Skorokhodov et al., 2022; Yu al., 2023). We learn 2D videos as continuous function maps spatial-temporal coordinates to corresponding RGB values, i.e,, w R + R., We compare our results with domain-speciffe INR generative models (Yu et al., 2022: Skorokhodov et al., 2022) well discrete video generative models like LVDM (He et al., 2022) and PVDM (Yu et al., 2023).
Quantitative analysis. Tab. illustrates the quantitative results of 2D video generation. We em- play Fréchet Video Distance (FVD) (Unterthiner et al., 2018) as the evaluation metric, following StyleGAN-V (Skorokhodov al., 2022). DDMI shows competitive performance to the most recent diffusion model (PVDM (Yu et al., 2023)), which is specifically trained on discrete video (pixels). This validates the effectiveness of our design choices in enhancing the expressive power of INR. We also provide qualitative results in Fig. and 14, where the generated videos exhibit realistic quality in each frame and effectively capture the motion across frames, demonstrating the versatility of our model in handling not only spatial but also temporal dimensions.
4.4 ANALYSIS
8

---

Published as a conference paper at ICLR 2024
A two layered table
rocking chair
Diffusion-SDP
DiffusionSDD SDF
Figure 6: Qualitative comparison on text-guided shape generation. We present comparison of the generation results produced by Diffusion-SDF and DDMI for given text prompt. DDMI excels in generating intricate details while preserving smooth surfaces. contrast, Diffusion-SDD struggles to capture fine details and often produces less polished surfaces.
Decomposition of HDBFs. In Fig. 7, we ana- lyze the role of different scales HDBFs rep- resenting signals. Specifically, we zero out all HDBFs except one during generation and observe the results in both spatial and spectral domains. When (a) E² and = are zeroed out, the generated image contains coarser details, such colors, in- dicating that the first E1 focuses on larger-scale components. In contrast, (c) with the third E our model generates images of high-frequence details, such as whiskers and furs. Results in the spectral domain after Fourier transform also match the tendency in the spatial domain, as the spectra of (c) exhibit high magnitudes in the high- frequency domains, whereas those of (a) have high magnitudes in the low-frequency domains. The analysis indicates that HDBFs effectively de- compose basis fields to capture the signals of dif- ferent scales.
Spatial Domain
Antertssngu
Fraquency Domais
Imago
Figure 7: HDBFs analysis. The upper two rows show generated images whereas the bot- one shows the spectral magnitude after ap- plying Fourier transform. The far left column indicates the generated image. Other columns indicate the generated image with (a): =² (b): E' =3, and (c): E| zeroed out, respec- tively. We employ histogram equalization for better visualization.
Ablation. Here, conduct ablation study evaluate the impact each component in DDMI. We train DDMI using various configurations on the AFHQv2 Cat dataset with 384 resolution. Refer to Tab. for the specific configurations. The baseline refers to D2C-VAE without HDBFs and CFC. We incrementally introduce compo- nents and present the Peak Signal-to-Noise Ratio (PSNR) and FID at the end of the first and second training stages. The results, outlined in Tab. 8, reveal gradual increase in PSNR and a corresponding decrease in FID scores as we incorporate additional components. This underscores the effectiveness of these components in enhancing the fidelity and realism of INR.
CONCLUSION
In this paper, we propose DDMI, domain-agnostic latent diffusion model designed to synthesize high-quality Implicit Neural Representations (INRs) across various signal domains. Our approach defines the Discrete-to-comiimuous space Variational AutoEncoder (D2C-VAE), which generates Po- sitional Embeddings (PEs) and establishes seamless connection between discrete data and contin- wous functions. Leveraging this foundation and our novel coarse-to to-fine conditioning mechanism with Hierarchically Decomposed Basis Fields (HDBFs), extensive experiments across wide range of domains have consistently demonstrated the versatility and superior performance of DDMI when compared to existing INR-based generative models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported in part by ICT Creative Consilience program (ITT22024-22200000181))) supervised by the IIP,, the National Research Foundation of Korea (NRF) grant funded by the Ko- rea government (MSIT) (NRF-2023R) 2023R1A2C2003333) and the Virtual Engineering Platform Project (Grant No. P0022336), funded by the Ministry of Trade, Industry & Energy (MoTIE, South Korea).
REFERENCES
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International Conference Machine Learn- ing, 2018.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Ko- rzhenkov. Image generators with conditionlll-indeeendeen pixel synthesis. In Conference on Computer Vision and Pattern Recognition, 2021.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Bruall and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In International Conference on Computer Vision, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-tima time attention all you need for video understanding? In International Conference on Machine Learning, 2021.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition, 2023.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes, arXiv preprint arXiv:2301.0963: 09632, 2023.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- 3d generative adversarial networks. In Conference on Computer Vision and Pattern Recognition, 2022.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 152.0001,, 2015.
Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In 14th Asian Conference on Computer Vision, 2019.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Conference on Computer Vision and Pattern Recognition, 2021.
Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In Conference Computer Vision and Pattern Recognition, 2022.
Zhiqin Chen and Hao Zhang, Learning implicit fields for generative shape modeling. In Conference on Computer Vision and Pattern Recognition, 2019.
Yunjey Choi, Youngjung Uh. Jacjun Yoo, and Jung-War Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Conference on Computer Vision and Pattern Recognition, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusson models beat gans on image synthesis. In Ad- vances Neural Information Processing Systems, 2021.
Yilun Du, M. Katherine Collins, B. Joshua Tenenbaum, and Vincent Sitzmann. Learning signal-
agnostic manifolds of neural fields. In Advances in Neural Information Processing Systems, 2021.
10

---

Published as a conference paper at ICLR 2024
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat like one. In International Conference Machine Learning, 2022a.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of func- tions. In International Conference Artificial Intelligence and Statistics, 2022b.
Clemens Eppner, Arsalan Mousavian, and Dieter Fox. Acronym: A large-scale grasp dataset based on simulation. In International Conference on Robotics and Automation, 2021.
Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generat- ing implicit neural fields with weight-space diffusion. In International Conference on Computer Vision, 2023.
Patrick Esser, Robin Rombach, and Bjöm Ommer. A note on data biases in generative models. In arXiv preprint arXiv:2012.02516 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Communications of the ACM, 2020.
Yingqing He, Tianyu Yang, Yong Zhang. Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high-fidelity video generation with arbitrary lengths. In arXiv preprint arXiv:221.13221,, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
Jonathan Ho and Tim Salimans. Classifier-fres diffusion guidance. In arXiv preprint arXiv:2207.12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. In The Journal of Machine Learning Research, 2022.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances Neural Information Processing Systems, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Conference on Computer Vision and Pattern Recognition, 2020b.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 1312v 14, 2013.
arXiv preprint
Batuhan Koyuneu, Pablo Sanchez-Martin, Ignacio Peis, Pablo M Olmos, and Isabel Valera. Vari- ational mixture of hypergenerators for learning distributions over functions. In arXiv preprint arXiv:2302.00623, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
11

---

Published as a conference paper at ICLR 2024
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, 2019.
Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdff: Text-to-shape via voxelized diffu- sion. In Conference on Computer Vision and Pattern Recognition, 2023.
Zhengzhe Liu, Yi Wang. Xiaojuan Qi, and Chi- Shii Fu. Towards implicit text-guided 3d shape generation. In Conference on Computer Vision and Pattern Recognition, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In arXiv preprint
1rXiv:1711.05101, 2017.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Confer- ence on Computer Vision and Pattern Recognition, 2021.
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovit- skiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Conference on Computer Vision and Pattern Recognition, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Conference on Computer Vision and Puttern Recognition, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Communica- lions of the ACM, 2021.
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Conference Computer Vision and Pattern Recognition, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. In ACM Transactions on Graphics, 2022.
Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00844 2022a.
Seonghyeon Nam, Marcus A Brubaker, and Michael S Brown. Neural image representations for multi-image fusion and layer separation. In European Conference on Computer Vision, 2022b.
Evangelos Ntavelis, Mohamad Shahbazi. Lason Kastanis, Radu Timofte, Martin Danelljan, and Luc Van Gool. Arbitrary-scale image synthesis. In Conference on Computer Vision and Pattern Recognition, 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf:
Learning continuous signed distance functions for shape representation. In Conference on Computer Vision and Pattern Recognition, 2019.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo Martin-Brualll Nerfies: Deformable neural radiance fields. In Conference on Computer Vision and Pattern Recognition, 2021.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision, 2020.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Conference on Computer Vision and Pattern Recognition, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
12

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition, 2022.
Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang. Chin- Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Conference on Computer Vision and Pattern Recognition, 2022.
Jachyeok Shim, Changwoo Kang, and Kyungdon Joo. Diffusion- signed distance fields for 3d shape generation. In Conference Computer Vision and Pattern Recognition, 2023.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d- structure-aware neural scene representations. In Advances in Neural Information Processing Systems, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 2020.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Conference on Computer Vision and Patlern Recognition, 2021.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: continuous video generator with the price, image quality and perks of stylegan2. In Conference Computer Vision and Puttern Recognition, 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Kridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Pro- cessing Systems, 2020.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In International Conference Learning Representations, 2021.
Patrick Tinsley, Adam Czajka, and Patrick Flynn. This face does not exist... but it might be yours! identity leakage in generative models. Winter Conference on Applications of Computer Vision, 2021.
Sergey Tulyakov, Ming-Yu Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Conference Computer Vision and Pattern Recognition, 2018.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 2018.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using stati dynamic generative adversarial networks. In Conference on Computer Vision and Pattern Recognition, 2018
13

---

Published as a conference paper at ICLR 2024
Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Ultrasr: Spatial encoding is missing key for implicit image function-base based arbitrary-scall super-resolution. arXiv preprint arX1: 2021.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. In arXiv preprint arXiv:2104. 10157, 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large- scale image dataset using deep learning with humans the loop. In arXiv 2015.
preprint arXiv:/506.088
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Woo Ha, and Jinwoo Shin. Generating videos with awnae implicit generative adversarial networks. International Conference Learning Representations, 2022.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Conference on Computer Vision and Pattern Recognition, 2023.
Xinyang Zheng, Yang Liu, Pengshuai Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based style- gan for 3d shape generation. In Computer Graphics Forum, 2022.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Conference on Computer Vision and Pattern Recognition, 2021.
Peiye Zhuang, Samira Abnar. Jiatao Gu, Alex Schwing, Joshua M Susskind, and Miguel Ángel Bautista. Diffusion probabilistic fields. In International Conference Learning Representations, 2023.
14

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
We here provide more details of our implementation, including the architecture of our models and hyperparameters. The comprehensive lists hyperparameters used in experiments are provided in Tab. 9.
Encoders and latent spaces. We use slightly different encoder backbones and structures for the latent variable depending the signal domains. For 2D image we encode it into latent variable as a 2D plane, using 2D CNN-based U-Net encoder (Ho et al., 2020).
For aD point cloud input E R3*N, consisting of N points in mys=-corddnnats,, we encode it into tri-plane latent variable z = {Zzy. Zyz; Zzz}. following Conv-ONet (Peng et al., 2020). Specifically, given an input point cloud, we employ PointNet (Qi et al., 2017) for feature extraction and apply an orthographic projection to map its feature onto three canonical planes. Features projecting onto the same pixel aggregated using local pooling operation, resulting in three distinct 2D grid features. We then apply a shared 2D CNN encoder to the projected features on grid features to obtain the tri-plane latent variables.
Lastly, for 2D video x, we encode it into a latent variable as a 2D tri-plane. We first convert into 3D features using TimesFormer (Bertasius et al., 2021) encoder. Then, we perform 2D projec- tion onto three canonical planes for latent variables = [Zzy. Bys Zaa] using three separate small Transformers (Vaswani al., 2017), respectively, following Yu al. (2023). Note that denotes temporal dimension.
2D CNN-based U-Net decoder D& convert latent variable into basis fields E. We maintain the structure of the basis fields to be consistent with the structure of the latent variables. For instance, in the case of 3D shapes, (Zyz), and E,z D (Zzz). The same
Decoders. For all domains,
compute tri tripplane basis fields as
E { E, where Eay = D& (Zzy); Eyz
procedure is applied to other data domains, e.g., E Exy for images and = [Eay E E,s] for
videos. As mentioned in the main paper, use three different scales of basis fields (E), =?. and E*) from our HDBFs.
MLP function. Given the basis fields E computed above, we compute positional embedding for given coordinate by bilinear interpolation that uses the distance distance-weighted average of the four nearest features of E\t from the coordinate c. We apply the same procedure to each scale of basis fields for a given coordinate to achieve p², and p² For tri-plane E, an axis-aligned orthogonal projection for each plane is applied beforehand. Then the positional embedding MLP-based residual blocks n, with we channels.
is fed into
Latent diffusion model. For the LDM backbone, we adopt the 2D Unet model LDM (Rombach al., 2022) and modify the hyperparameters.
B TRAINING AND INFERENCE
Our framework performs two-stage training. In the first stage, the proposed method learns the latent space by optimizing P2C-VAE. In the second stage, we optimize a diffusion model in the learned latent space. this work, all experiments are conducted on NVIDIA RTX3090 and V 1000 GPUs. We provide the hyperparameters of training in Tab. 10.
B.1 FIRST-STAGE TRAINING.
In the first stage, we optimize the encoder, decoder, and MLP of D2C-VAE using the AdamW (Loshchilov & Hutter, 2017) optimizer. We minimize the re-weighted ELBO objective of VAC in Eq. 9. We assume the standard normal prior distribution p(z) and a multivariate normal distribution for the posterior (z(x). The KL divergence loss between the posterior and the prior is calculated using the reparameterization trick Kingma & Welling (2013). For image, video, and NeRF, we assume that no (c)lz)) follows the multivariate normal distribution with isotropic
15

---

Published as a conference paper at ICLR 2024
Table Hyperparameters of models.
AFHQ CarDog
Co 256'
CIFARIO 32
Lsun Church
ShapeNet
SKY 256
256
128"
Chair
Multi Class
Encoder
latent shupe
2D plane
2D plane
2D plane 16
2D plane
2D tri-plane
2D tri-plane 64
2D tri-plane 64
dims, of
64
32 64
16
Decuder
channel multiplier
(4,2,1)
(4,2,1)
(4,2.1)
(4,2,1)
(4,2,1)
(4,2.1)
(8.4.2.1)
bead channel
64 2D plane (1.2.4)
64
64
64
32
32
64
shupe
plune
2D plane (1)
2D plane
tri-plane (1,2.4)
2D tri-plane (1.2.4)
2D tri-plane (1,2,4)
hasis field res.
(1,2.4)
(1.2.4)
MLP
256
256
256
256
256
256
256
Latent diffusion model
residual blocks
2
2
2
channel multiplier
(4,3,2.1)
(4,3,2.1)
(4.3,2.1)
(4.3.2.1)
(4,3.2.1)
(4,3.2.1)
(4,3,2.1)
head channel
256
256
128
256
256
256
256
atin res.
(2,4,8)
(2,4,8)
(2,4)
(2,48))
(2,4,))
(2,4,8)
(2,4.8)
Table 10: Training details
AFHQ Car/Dog 256'
CelebA-HQ 256
CIFARTO 32
Lsun Church 128
ShapeNet
SKY 256²
Chair
Multi Class
First stage training
learning rate
le-4
le-4
le-4
le-4
le-4
le44 AdamW
le-4 A
optimizer
AdamW
AdamW
AdamW
AdamW
AdamW
weight start
10
SN weight decay anneal
linear
lineur
linear
linear
lineur
linear
linear
A start
le-4
1e.4
le-4
le-4
le-4
1e-4
le-4
A₂ end
le-3
lo-3
le-3
1o-3
lo-3
anneal
linear
linear
linear
linear
linear
linear
linear
Ax anneal portion
0.9
0.9
0,9
0.9
0.9
0.9
0.9
Second stage training
learning rate optimizer
le-4
le-4
le-4 AdamW
2e-4
2e-4
le-4
Adam
AdamW
AdamW
AdamW
AdamW
AdamW
covariance
[[((c) (c) w(c) E.
Pejcng (c)lz)) = II
<Ec 7 exp
(11)
2g
For the occupancy functionn we assume the Bernoulli distribution:
Pv. (w(c)
II ()))(()) (1-)(c))w0) (()) (0,
(12)
In practice, we employ -osss for image, video, and NeRF, and binary cross-entropy for 3D shapes. We linearly increase the weight A₂ during training and apply a spectral normalization (SN) regular- ization for the encoder, following Vahdat et al. (2021).
Multi-scale training and scale injection. For training our model on images, we incorporate the multi-scale training scheme (Ntavelis et al., 2022) to further encourage the estimation of accurate W. our approach, we incorporate a scale variable s E that represents the sampling period of the coordinate set e for the discrete data x. Through the training of D2C-VAE with data of diverse scales, e.g., utilizing multi-resolution data or applying random crop augmentation, we enable INR to explore wider range of coordinate sets e in Eq. 9, enabling high-quality generation at arbitrary scales. In addition to multi-scale training, we incorporate a Scale Injection (SI) to enhance the performance of our framework further. We modulate the weights of each fully connected layer of MI.P with the scale variable 8, similar to style modulation in StyleGAN Karras et al. (2020b). The injection of scale information makes INR aware of the target scale. Specifically, the (i,j) entry of 4th layer weight ø² E mmxx is modulated to 8':
A'(s))
(s)
(13)
Es (b4,x Al(s)))
16

---

Published as a conference paper at ICLR 2024
where is small constant for numerical stability and A' indicates layer-wist mapping function. s undergoes Fourier embedding with a fully connected layer before the layer-wise mapping. In practice, we provide the encoder with full-frame resouutootttion images of size rxr (e.g.. 256x256) to make fixed-size latent variables. Then, for each batch of latent variables, we randomly select a resolution from predefined set of resolutions {r, 1.5 r; r) that the INR aims present in a discrete image. For images with a higher resolution than r, we randomly crop them to match the resolution r.
B.2 SECOND-STAGE TRAINING.
In the second stage, optimize LDM on the empirical distribution of the latent variables using the AdamW (Loshchilov & Hutter, 2017) optimizer. We minimize the noise prediction loss, discussed in the main paper, while the other networks trained in the first stage are frozen. In the case of tri- plane latent variables, e.g., 3D and video, we use a single 2D Unet model to denoise each latent plane and add attention layers that operate on the intermediate features of three latent planes follow- ing PVDM (Yu et al., 2023). This allows us to effectively model the dependency between planes and use the shared 2D Unet structure across different domains. The training objective of LDM in Eq. 10 is by reparameterization trick Pup (Z t)
9. - (Ze, t)), where ar := 3. you (2z Vi-a.
and ā := II..., ag. We mostly follow techniques proposed in the previous literature. Specifically, utilize mixed parameterization of score function (Vahdat et al., 2021). We found mixed param- eterization beneficial for training the latent space of INRs since we also regularize the posterior distribution towards standard normal distribution in the first stage. Following existing works, an exponential moving average (EMA) of the parameters of the LDM network with a 0.9999 decay rate.
B.3 INFERENCE.
For generation, we use reverse diffusion process with fixed number of steps T 1000 (DDPM sampling) for all experiments presented in the main paper unless stated otherwise. However, our model can also leverage recent advanced samplers discussed in Tab. 12.
C BASELINES AND EVALUATION METRICS
In this section, we provide brief overview of baselines and the evaluation metries employed to assess the performance of the model.
C.I 2D IMAGE
Baselines. CIPS (Anokhin al., 2021). INR-GAN (Skorokhodov et al., 2021), and ScaleParty (Ntavelis et al., 2022) incorporate adversarial training (Goodfellow et al., 2020) within the StyleGANv2 framework (Karras al., 2020b). These models utilize fixed single-scale Fourier features for PEs instead of constant input used in StyleGANv2 and modulate the activation function of the network with global latent variables. particular, ScaleParty (Ntavelis al., 2022) deploys a CNN-based architecture instead of an MLP and leverages multi-scale training to make the network aware the generation scale. VaMoH (Koyuncu et al., 2023) employs multiple that generate the weights of INRs with fixed PEs to capture different aspects of signals.
Fréchet Inception Distance (FID) (Heusel et al., 2017) is utilized to quantify the dissimilarity between distributions, typically used in the context of generative models. It operates by com- paring the mean and standard deviation of features extracted from the deepest layer of the Inception v3 neural network. In our evaluation, we compute the FID between all available real samples, up to a maximum of 50K, and 50K generated samples, following Karras et al. (2020b). FID helps gauge the quality and diversity of the generated samples by assessing their proximity to the real data distribution. Lower FID scores signify better agreement between the two distributions.
Improved precision and recall (P&R) (Kynkäääniemmi et al., 2019) measures the expected like- lihood of real (fake) samples belonging the support of fake (real) distribution. They approximate
17

---

Published as a conference paper at ICLR 2024
the support of distribution by constructing K-nearest neighbor hyperspheres around each sample. P&R typically represents fidelity and diversity of generative model, respectively. In our evalua- tion, we compute the P&R between all available real samples, up to maximum of 50K, and 50K generated samples.
C.2 3D SHAPE
Baselines. DPM3D (Luo & Hu, 2021) and PVD construct diffusion processes for point clouds by utilizing point cloud generators. LatentGAN (Chen & Zhang, 2019) and SDF-StyleGAN (Zheng al., 2022) both make use of global latent vector to represent global 3D shape features through adversarial training. Specifically, SDF-StyleGAN employs global and local discriminators to gen- erate fine details in the 3D shapes. 3D-LDM (Nam al., 2022a) learns the global latent vector of SDF through auto-decoder and trains diffusion models in the latent space. SDF-Diffusion (Shim ct al., 2023) introduces a diffusion model applied to voxel-based Signed Distance Fields (SDF) and another diffusion model for patch-wise SDF voxel super-resolution. AutoSDF (Mittal et al., 2022) combines a non-sequential autoregressive prior for 3D shapes conditioned on a text prompt. Diffusion-SDD SDD (Li et al., 2023) encodes point clouds into voxelized latent space and introduces voxelized diffusion model for text-guided generation.
Classification Accuracy involves the use of voxel-based classifier that has been pre-trained to classify objects into 13 different categories within the ShapeNet dataset (Chang et al., 2015), following the approach (Sanghi et al., 2022). It is employed to gauge the semantic authenticity of the generated samples, assessing how well the generated shapes align with the expected categories in ShapeNet.
CLIP similarity score (CLIP-S) employs the pre-trained CLIP model for measuring the corre- spondence between images and text descriptions by computing the cosine similarity. It evaluates how well the generated shape aligns with the intended text. We follow the evaluation protocol in Li al. (2023): render five different views for the generated shape measure CLIP-S for these rendered images and use the highest score obtained for each text description.
Total mutual difference (TMD). In order to measure TMD, we follow the protocols in Li et al. (2023). Specifically, we generate ten different samples for each given text description. Subsequently, we calculate the average Intersection over Union (IoU) score for each generated shape concerning the other 9 shapes. The metric then computes the average IoU score across all text queries. TMD serves measure of generation diversity for each specific text query.
C.3 VIDEO
Baselines. MoCoGAN (Tulyakov et al., 2018) decomposes motion and content in video generation by employing separate generators. DIGAN (Yu al., 2022) and StyleGAN-V (Skorokhodov al., 2022) introduce Implicit Neural Representation (INR)-based video generators with computationally efficient discriminators. VideoGPT (Yan al., 2021) encodes videos into sequences of latent vec- tors using VQ-VAE and learns autoregressive Transformers. PVDM (Yu et al., 2023) also encodes videos into a projected tri-plane latent space and subsequently learns a latent diffusion model. While our model shares similar structure with PVDM, it is primarily designed for generating fixed dis- crete pixels, which limits its adaptability to other domains. There are other recent works, such as VLDM (Blattmann al., 2023), that utilize diffusion models the spatio-temporal latent spaces. However, VLDM employs a 3D voxel-based latent space and a 3D-CNN--aaseeee network, which can be computationally more intensive. Furthermore, their work relies large-scale in-house dataset without open-source code availability, which hinders a fair comparison with our model.
Fréchet Video Distance (FID) is metric that calculates the Fréchet distance between the feature representations real and generated videos. To obtain appropriate feature representations, FVD employs pre-trained Inflated 3D Convnet. In our evaluation, compute FVD by comparing 2,048 real video samples with 2,048 fake samples, following the preprocessing protocol recommended in Skorokhodov et al. (2022).
18

---

Published as a conference paper at ICLR 2024
Table 11: Generation results on diverse datasets.
CIFARIO 32"
Lsun Churches 128"
FID
<iierrtte representation>
DDPM+ [Song et 2021]
2,47
StyleGAN-2 (Karra et 2020b)
11.07
3.78
LSGM (Valdat ot al,, 2021)
2.10
<Centinous representation
Domain specific
(Skorokhodov ctal.,
7.38
Domain agnostic
GEM (Du al., 2021)
23.83
DPF (Zhuang al, 2023)
15.1
DDMI (Ours)
4,53
5.12
D ADDITIONAL RESULTS
D.1. QUANTITATIVE RESULTS
CIFARIO and Lsun Church. To demonstrate that DDMI generalize effectively to datasets with more diverse global structures and classes, we have trained our model on the CIFAR10 10 and LSUN Church datasets. As shown in Tab. 11, DDMI achieves significant performance improve- ment over both domain-specific INR generative model (CIPS) and domain-agnostie INR generative models (GEM and DPF) both datasets. This result further affirms the efficacy of design choices, e.g. basie fields generation with D2C-VAE, HDBFs, and CFC, holds for datasets with diverse characteristics.
Efficient sampling with ODE solver. Recent advances in diffusion models, such as sam- pling via ODE (Ordinary Differential Equation) solvers Song al. (2021); Vahdat al. (2021); Karras et al. (2022), have shown promising results in reducing sampling time. We use an RK45 ODE solver, instead of ancestral sampling, similar Vahdat et al. (2021). By increasing the ODE solver error tolerances, we generate samples with lower NFEs (number of func- tion evaluations). In Tab. 12, report FID scores on AFHQv2 Cat and Dog datasets for three cases: 1) T=1000, 2) ODE solver and 3) ODE solver eтror tolerances of 10-5, The results demonstrate that DDMI framework per- forms efficient sampling while still achieving satis- factory performance (see Fig. 16 and 17 for qualita- tive results).
Table 12: NFE vs. FID.
AFHQv2 Cat FID
AFHQv2 Dog
Avg. NFE
FID
tolerances of 10-3
1000
4.27
8.54
50
5.32
11.04
25
5.93
14.23
D.2 QUALITATIVE RESULTS
Visualization of the nearest neighbor samples. In order to confirm that our model is not over- fitting to the training dataset, we employ visualization technique that displays the nearest training samples to the generated samples in the VGG feature space. This visualization is presented in Fig. 13.
Arbitrary-scale 2D image synthesis. Fig. 8, 9, and 10 show the generated 2D images at arbitrary resolutions, including 2562, 3842, and 512" These results demonstrate the ability our model to generate continuous representations with high quality.
3D shape generation. In Fig 12, we compare the qualitative results on 3D shape with two recent baselines: GASP (Dupont et al., 2022b) (donain-aggostic) and SDF-StyleGAN (Zheng et al., 2022) (domain-speciffe The result demonstrates that our method generates more sophisticated details while maintaining smooth surfaces compared to the baselines. Moreover, Fig. 11 shows that our DDMI generates diverse and high-fidelity 3D shapes across various object categories: airplane, car, chair, desk, gun, lamp, ship,
19

---

Published as a conference paper at ICLR 2024
Video generation. Fig. 14 presents video generation results the SkyTimelapse dataset. The sults demonstrate the capability of our model to generate visually appealing and coherent sequences of images.
Neural radiance fields generation. Fig. 15 displays the NeRF generation results of our model on the SRN car dataset in comparison to Functa.
E BROADER IMPACTS
The development of generative models with controllability across diverse data domains has been long-standing objective in the field. Our proposed generative model. DDMI, represents significant advancement in the realm of INR generative models, offering high-fidelity generation and effective applicabilityy to wide range of data domains. Moreover, one key strength of DDMI is its capability to generate samples at arbitrary scales, providing enhanced controllability that can be leveraged in conjunction with existing conditional generation techniques Rombach et al. (2022). This opens up exciting possibilities for tasks like text-guided super-resolution video generation, where the fine- grained control offered by DDMI can yield compelling results. By enabling controllable generation across different data modalities, DDMI has the potential to drive further advancements in creative applications, and other domains that rely on generative modeling.
However, is important to acknowledge potential concerns that arise with the deployment of gen- erative models Tinsley et al. (2021), including DDMI. Like other generative models, there is risk of revealing private or sensitive information present in the data. Additionally, generative models may exhibit biases Esser et al. (2020) that present in the dataset used for training. Addressing these concerns and ensuring the ethical and responsible deployment of generative models is crucial mitigate potential negative impacts and promote fair and inclusive use of the technology.
LIMITATIONS
Domain aggnosicc INR generation has shown promising results in a wide range of applications with great flexibility. The proposed method enhances the expressive power of INR generation via asym- metric variational autoencoder (VAF) connecting the discrete data space and the continuous function space. One caveat of the proposed method is the relatively long generation time. This is one well- known common disadvantage of diffusion models. Sec. D.I, we studied advanced solvers to speed up the generation process but it is not satisfactory. With small number of function evalu- ations (NFEs), the generation suffers from quality degradation. More efficient generation schemes leveraging compact latent spaces will be interesting future direction.
20

---

Published as a conference paper at ICLR 2024
Figure 8: Uncurated samples on AFHQv2 Cat for various resolutions generated from single DDMI model.
Figure 9: Uncurated samples on AFHQv2 Dog for various resolutions generated from single DDMI model.
21

---

Published as a conference paper at ICLR 2024
G
Figure 10: Uncurated samples on CelebA-HQ for various resolutions generated from single DDMI model.
Figure 11: Uncurated samples on ShapeNet generated from DDMI model.
22

---

Published as a conference paper at ICLR 2024
(a) GASP Dupont (2022b)
(b) OIF-StyleGAN Zheng et al. (2022)
(c) Ours
Figure 12: Qualitative comparison on 3D INR generation. We compare GASP (domain-agnostic), SDF-StyleGAN (domain-specific)) and DDMI 3D shapes.
Figure 13: Nearest neighbors of our CelebA-HQ model, computed in the features of VGG16. The leftmost sample is from our model, and the remaining samples in each row are its nearest neigh- bors.
23

---

Published as a conference paper at ICLR 2024
Figure 14: Uncurated samples SkyTimelapse generated from DDMI.
24

---

Published as a conference paper at ICLR 2024
(a) Functa Dupont et al. (2022a)
(b) Ours
Figure 15: Qualitative comparison on NeRF generation. Our model generates sophisticated de- tails and vivid color texture while maintaining smooth surfaces. In contrast, Functa exhibits limita- tion in capturing intricate details and tends to produce blurry texture.
25

---

Published as a conference paper at ICLR 2024
Figure 16: Uncurated samples on AFHQv2 Cat generated from DDMI model using ODE solver
with 10-3 tolerance (Avg. NFB=25).
Figure 17: Uncurated samples on AFHQv2 Dog generated from DDMI model using ODE solver with 10 tolerance (Avg. NFE=50).
26


Reviews:
[{"review_id": "Egmwy6hd7K", "reviewer": "Reviewer_6uL5", "summary": "This work presents an improved framework for designing a domain agnostic latent diffusion architecture that can right away model the distributions over any class of continuous implicit neural functions (INR) without any architectural modifications. \nTo that extent, the existing works attempts to generates the weights of MLP that parameterize INRs while keeping the input positional embeddings (PEs) fixed.\nHowever, the recent works on INRs demonstrates that, careful designs of PEs instead of MLPs achieves better efficiency and effectiveness in representing INRs.\nMotivated by this development, this work proposes DDMI that generates sample-specific adaptive PEs while keeping the weights of MLP fixed.\nSpecifically, they develop D2C-VAE whose encoder maps explicits discrete space signals to latent space and its decoder maps latent space to continuous basis fields.\nApart from LDM for D2C mapping, DDMI hierarchically decomposes basis fields (HDBFs) into multiple scales while the MLP progressively performs coarse to fine conditioning (CFC) using these multi-scale PE from HDBFs.\nExtensive experiments across different input modalities shows DDMI outperforming recent SoA on INR generative models.", "questions": "Please provide clarity on points raised under weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Quality and Clarity\nThe paper is well-written and easy to follow. Many quantitative results, ablation studies, and implementation details with hyper-parameters are discussed in the supplementary. These served as helpful guide. \n\nOriginality and Significance\n1. It is the first work to combine the power of latent diffusion model with input adaptive positional embeddings for generating implicit neural representations.\n2. Same architecture is used for generating arbitrary continuous INRs using discrete inputs from various domains that includes images, videos, 3D shapes and NeRFs.\n3. Overall, better qualitative and quantitative results when compared to SoA discrete generative models, domain-specific INR generative models and domain-agnostic generative models.\n4. Due to explicit tuning of multiple scales, DDMI is capable of generating signals of multiple resolutions without any loss of signal fidelity.", "weaknesses": "1. Limited novelty. I see this work as a natural extension to other works on domain-agnostic LDMs for INRs such as Functa (Dupont et. al, 2022a) and DPF (Zhuang et. al, 2023). The only difference between DDMI and Functa is replacing dynamic MLP weights with dynamic positional embeddings. \n2. DDMI made three changes w.r.t. Functa -- dynamic MLP weights -> dynamic PEs, HDBFs, and CFC. It is still unclear how much improvement was achieved by making a shift from dynamic MLP to dynamic PEs. The ablation only covers the relative effects of HDFCs and CFC.\n3. Scalability -- The work claims that the prior works are not scalable as dynamically modifying MLP weights limit its expressive power. However, DDMI has not demonstrated any scalability. Most of the results are either on single or few class dataset, while the number of samples are in the order of 10K. Currently, it seems to me that semantic class specific INR generative model is being fit. True scalability to larger community is model's ability to generate high quality samples from diverse set of classes. Consider training on Objaverse scale of dataset.\n4. Significance -- Given that almost all high-fidelity LDM models are obtained by domain-specific LDMs, I fail to understand the real value behind creating domain-agnostic architecture. It make sense to anyone, if priors from multiple domains are reused which is possible only if DDMI was trained by sampling input discrete data across domains.\n5. Domain Agnostic ? It is not clear how the presented DDMI is domain-agnostic. I presume the input and output of LDMs will remain the same when trained for signals from each domain. However, the implementation details in appendix suggests that dataset-specific architecture was deployed with LDM input/output also changing likewise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kh7coGlN9O", "reviewer": "Reviewer_qBmp", "summary": "This paper proposes a domain-agnostic latent diffusion models for generating implicit neural representations across three domains: images, videos, and 3D shapes. The main novelty is that they let the latent variables generating adaptive basis fields for different examples, and learn a single rendering function afterwards. Multi-resolution basis fields have been used to further boost the performance. Extensive experiments have been performed on those three domains to showcase that the method works better than other domain-agnostic INR approaches.", "questions": "Please see the comments above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper proposed to instead of learning to generate different rendering functions, learn to generate different basis fields, which seems to be a more principled way. The effectiveness has been justifed by the extensive experiments and comparison with baseline approaches. The multiscale structure of generating basis fields seems to help generating fine-grained details for high-resolution rendering.", "weaknesses": "- I'm a bit concerned about the soundness of D2C-VAE the paper proposed. Specifically, how to justify eq. (2) is a valid training objective, does it correspond to an ELBO of a log-likelihood function? It'd be better to formulate the problem and training objective in a more principled and statistically sound way, e.g., by expressing everything in the function space. \n\n- It'd be nicer to have some intuitive explanation on the reason why modeling PE works better than modeling the the rendering function $\\omega$.\n\n- All experimental results are shown on relatively easy datasets on images, 3d and videos.Would be curious to see how this approach work with more challenging and scaled up datasets on various domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tam0ShrO1o", "reviewer": "Reviewer_KC2J", "summary": "This work proposes a new INR generative model called Domain-agnostic Latent Diffusion Model (DDMI) that can generate continuous signals in various domains. DDMI consists of the D2C-VAE and a diffusion model in its latent space. The D2C-VAE maps discrete data into a latent space and then generates hierarchically decomposed basis fields for adaptive PEs, shifting the primary expressive capacity from MLP to PE. Coarse-to-Fine Conditioning is applied to return the signal value for the queried coordinate, further improving the performance of generating INR.", "questions": "1. **Why use the occupancy function to represent a 3D shape instead of the SDF?** Many compared baselines learn a generative model on 3D INR by modeling the distribution of SDF, but this work adopts the occupancy function. How will using SDF as INR affect the proposed method's performance? Can the difference in 3D shape representation cause unfair comparisons?\n\n2. **How to project sparse point clouds to be processed by a 2D CNN?** It is mentioned in Appendix A that an orthographic projection is performed on the PointNet local pooling features to three canonical 2D planes, and a shared 2D CNN encoder is used to obtain tri-plane latent features. However, the output of a PointNet encoder with a local pooling operation is still a sparse \"point cloud\" in the shape of `BxNxD`, and the adjoined arrays in the tensor are not actually adjoined spatially. How does a 2D CNN perform on the sparse input?\n\n3. **Will the code be publicly available?** Releasing the code is not mentioned in the paper, but it would be helpful to facilitate future related research and provide much more details about the method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **The proposed method is straightforward and effective.** The proposed position embedding modeling and hierarchical decoding are straightforward, yet they are versatile and effective across seven benchmark datasets.\n\n2. **Experiments are thorough.** Experiments across four modalities and ablation studies including the proposed two modules are conducted, indicating the superior performance and versatility of DDMI.\n\n3. **Analysis of the decomposition of HDBFs is intriguing.** The investigation strategy and visualizations of results in spatial and spectral domains provide insightful information.", "weaknesses": "**Some expressions are confusing and unclear.**\nFor instance, when it comes to the tri-plane grid features shown in Figure 2, the input discrete data includes images, 3D shapes and videos. However, the tri-plane is only applicable to 3D shapes and videos, and not to 2D images, which use a 2D plane as their latent space type.\nAdditionally, there should be more information provided about D2C-VAE in the main paper, such as the representation and dimensionality of latent variables and basis fields, rather than including it only in the appendix. It becomes difficult to follow the information without having to read the appendix.\nLastly, It's also unclear how to project sparse point clouds to be processed by a 2D CNN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F7rKiI3JpN", "reviewer": "Reviewer_9btm", "summary": "The paper presents a probabilistic model for functions using INRs. The main features are:\n\n- A VAE projects the discrete data input X into a latent space.\n- An adaptive PE module replaces hyper network that generates weights. The VAE decoder generates a hierarchy of basis fields that account for the multi-scale properties of the data to be generated.\n- This hierarchy of PE is introduced sequentially into the reconstructing MLP.\n\nThe DDMI model presents robust performance across a variety of experiments.", "questions": "Did you try to train a more powerful VAE latent space instead of training a latent diffusion model? I'm just wondering how important is to incorporate the diffusion model. It seems to me that the DDMI gains mostly come from using the adaptive hierarchical PE.\n\nI did not see many details about the posterior distribution used in the training of the D2C-VAE. Can you please introduce more details about this point? \n\nWhat is the dimension of the latent space z? Have you studied the model's performance as this dimension varies?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and clear\n- Experimental setting demonstrates a robust performance compared to other methods\n- Methodology is rigorous and conclusions are supported by evidence.", "weaknesses": "To my understanding, the present work presents a well-designed combination of features and methods already present in the state-of-the-art:\n\n- The use of signal fields and adaptive PE in implicit models has been reported by Zhuang et al. (ICLR 2023). \n- A VAE approach for probabilistic INRs has been recently published by Koyuncu et al. (ICML 2023): \"Variational Mixture of HyperGenerators for Learning Distributions Over Functions.\" This paper is not even mentioned in the present work, and it is a must for the authors to include it, not only in the literature review but also in the experimental setting. Authors also consider using various hypergenerators to capture different data scales and modalities in that work.\n- I do not think the paper also compares with \"Implicit Diffusion Models for Continuous Super-Resolution\" (CVPR 2023), in terms of tools and results obtained. Comparisons with this paper are important too.\n- Using a generative latent diffusion model directly follows the idea of Rombach et al. (CVPR 2022).\n- Hierarchies of multi-scale PE have already been developed in the literature. The paper \"Composite Slice Transformer: An Efficient Transformer with Composition of Multi-Scale Multi-Range Attentions\" (ICLR 2023) is a good reference. \n\nWhile combining existing ideas into a robust model is not a demerit of the paper per se, the paper's contributions are limited. \n\nAlso, when explaining the interpolation and the basis functions, more details are needed since these concepts are hard to understand the way they are written.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)