Hyper-Transforming Latent Diffusion Models
Authors: Ignacio Peis
Batuhan Koyuncu
Isabel Valera
Jes Frellsen
Venue: ICML 2025 poster (2025)
Keywords: Latent Diffusion Models
Transformers
INRs
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a generative framework for INRs that integrates a Transformer-based hypernetwork decoder into latent diffusion models, enabling scalable INR generation and efficient adaptation via hyper-transforming, which fine-tunes only the decoder.

Abstract: We introduce a novel generative framework for functions by integrating Implicit Neural Representations (INRs) and Transformer-based hypernetworks into latent variable models. Unlike prior approaches that rely on MLP-based hypernetworks with scalability limitations, our method employs a Transformer-based decoder to generate INR parameters from latent variables, addressing both representation capacity and computational efficiency. Our framework extends latent diffusion models (LDMs) to INR generation by replacing standard decoders with a Transformer-based hypernetwork, which can be trained either from scratch or via hyper-transforming—a strategy that fine-tunes only the decoder while freezing the pre-trained latent space. This enables efficient adaptation of existing generative models to INR-based representations without requiring full retraining. We validate our approach across multiple modalities, demonstrating improved scalability, expressiveness, and generalization over existing INR-based generative models. Our findings establish a unified and flexible framework for learning structured function representations.

Hyper- Latent Diffusion Models
Ignacio Peis
Batuhan Koyuneu 34 Isabel Valera Jes Frellsen 12
Abstract
We introduce a novel generative framework for functions by integrating Implicit Neural Represen- tations (INRs) and Transformer-bass hypernet- works into latent variable models. Unlike prior ap- proaches that rely MLP-based hypernetworks with scalability limitations, our method employs Transformer- based decoder to generate INR pa- rameters from latent variables, addressing both representation capacity and computational effi- ciency. Our framework extends latent diffusion models (LDMs) to INR generation by replacing standard decoders with Transformer- hy- pernetwork, which can be trained either from scratch or via hyper-transforming a strategy that fine-tunes only the decoder while freezing the pre- trained latent space. This enables efficient adapta- tion of existing generative models to INR-based representations without requiring full retraining. We validate our approach across multiple modali- ties, demonstrating improved scalability, expres- siveness, and generalization over existing INR- based generative models. Our findings establish a unified and flexible framework for learning struc- tured function representations.
x0.125
0..25
x0.5
x1
x2
x4
Figure 1: Samples from LDMI at multiple resolutions.
ities. In contrast, Implicit Neural Representations (INRs) have emerged as a powerful alternative that parametrises sig- nals as continuous functions (Sitzmann et al., 2020; Milden- hall et al., 2021). By leveraging INRs, generative models can represent complex data distributions at arbitrary resolu- tions, yet learning expressive distributions over INR param- eters remains a fundamental challenge.
A common approach to modelling distributions over INRs is use hypernetworks (Ha et al., 2017), which generate the weights and biases of an INR conditioned on latent code (Dupont et al., 2022b; Koyuncu et al., 2023). How- ever, MLP-based hypernetworks suffer from scalability bot- tlenecks, particularly when generating high-dimensional INRs, as direct parameter regression constrains flexibility and expressiveness. More recently, Transformer based hy- pernetworks have been proposed alleviate these issues, introducing attention mechanisms to efficiently predict INR parameters (Chen & Wang, 2022; Zhmoginov et al., 2022). Nonetheless. existing approaches such as Trans-INR (Chen & Wang, 2022) remain deterministic, limiting their applica- bility in probabilistic frameworks.
1. Introduction
Generative modelling has seen remarkable advances in re- cent years, with diffusion models achieving state-of-the-art performance across multiple domains, including images, videos. and 3D synthesis (Ho et al., 2020; Dhariwal & Nichol, 2021; Rombach et al., 2022). A key limitation of ex- isting generative frameworks, however, is their reliance on structured output representations, such as pixel grids, which constrain resolution and generalisation across data modal-
In this work, we introduce a novel framework for INR gen- eration, named Latent Diffusion Models of INRs (LDMI), that incorporates our proposed Hyper-Transformer Transformer Decoder (HD), probabilistic Transformer-based decoder for learn- ing distributions over INR parameters. Our approach com- bines the strengths of hypernetworks with recent advances in meta-learning for INRs while integrating into latent diffusion- based generative frameworks. Unlike previous
DDepartment of Applied Mathematics and Computer Science, Technical University of Demmark. Copenhagen, Demmark "Pioneer Centre for Artificial Intelligence, Copenhagen, Denmark 'Saarland University, Saarbrücken, Germany "Zuse School ELIZA. Corre- spondence to: Ignacio Peis <ipeaz@ @ dtu.dk>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Hlyer--Transorne Latent Difflusion Models
works, our HD decoder employs full Transformer archi- tecture, where Transformer Encoder processes latent vari- ables, and Transformer Decoder generates INR parameters via cross- attentioon.. This design enables flexible, probabilis- tic generation of neural representations across diverse data modalities, surpassing the deterministic limitations of prior Transformer-based hypernetworks (Chen & Wang, 2022: Zhmoginov al.. 2022).
LDMs first encode data into lower-dimensional latent space Rd, where d < D, using a probabilistic encoder (-)-) to produce rich latent representations qu (z(y))) These are decoded via (x)-) to accurately recover data samples by modeling the conditional distribution p> (y)z). A B- V-V (Higgins et al., 2017) with low B. a Vector-Quantizer VAE (VQ-VAE; VAE; van den Oord et al., 2017), or VQGAN (Esser et al., 2021) are suitable choices for learning the structured latent space. Additionally, for image data, perceptual losses can be incorporated to further enhance generation quality (Larsen et al., 201 Johnson et al., 2016; Hou et al., 2017; Dosovitskiy & Brox, 2016; Hou et al., 2019).
Our proposed framework, LDMI, supports two training paradigms: (i) full training, where the model is trained from scratch alongside a latent diffusion model (LDM; Rom- bach et al., 2022), and (ii) hyper-transforming, where a pre-trained LDM is adapted by replacing its decoder with the HD, allowing for efficient transfer learning without re- training the entire generative pipeline. This flexibility en- ables LDMI to scale effectively while leveraging existing pre-trained diffusion models.
In the second stage, LDMs approximate the aggregate poste- rior (e((2) f qo ( Pama (y) dy with diffusion model pe (z), shifting the generative modeling task from data space to latent space and significantly reducing computational cost. While both continuous-time SDE-based approaches (Song et al., 2021b) and discrete Markov chain formulations (Ho et al., 2020; Song et al., 2021a) are viable, we follow the DDPM- based approach as adopted in the original LDM implementation (Rombach et al., 2022).
Contributions Our key contributions can be summarized as follows:
We introduce the HD decoder, a full Transformer-based probabilistic decoder for learning distributions over INR parameters.
DDPMs (Ho et al., 2020) train using an objective de- rived from variational lower bound on the negative log- likelihood, reparameterized for efficiency as a denoising task. This results in simplified loss that closely resembles denoising score matching (Song et al., 2021b), and is widely used in statc-of-the-ar models (Saharia et al., 2022; Dhari- wal & Nichol, 2021). For latent diffusion, the loss applied to z (z(g)) yielding:
We integrate our method into Latent Diffusion Models (Rombach et al., 2022), with support both full training and hyper-transforming enabling efficient adaptation of pre-trained models. We refer to this approach as LDMT.
The HD decoder achieves scalability and generalisation across data modalities, overcoming the bottlenecks of MLP-based hypernetworks and extending beyond deter- ministic Transformer-based methods.
LDDPM R.y, [||e [lc-colf co., [)1]
(I)
We demonstrate the effectiveness of our approach on var- ious generative tasks, showcasing its superiority in mod- elling high-resolution, structured data.
where y ~ Palta ~ qo (= y). (1, and c ~ NO, I). This objective corresponds to predicting the added noise in a fixed forward process, effectively denoising z, toward the original latent z.
By integrating probabilistic INR modelling within diffusion- based generative frameworks and efficient latent-to- parameters generation, our work establishes a new direc- tion for flexible and scalable generative modelling with unconstrained resolution. The following sections provide a detailed description of the background (Section 2), the proposed methodology (Section 3), and empirical results (Section 4).
Sampling In DDPM, the reverse posterior density is no longer Markovian and coincides with the inference model proposed later in DDIM (Song et al., 2021a). In DDIM, it is demonstrated that faster sampling can be achieved without retraining, simply by using the posterior approximation for the estimation I, which re-defines the generative process as:
[N (2, ( (z
if = otherwise,
Po Pa(z.-1lz))
(2)
2. Background and Related Work
2.1. Latent Diffusion Models
and uses that an estimation of é can be computed by
Latent Diffusion Models (LDM; Rombach et al., 2022) are a class of generative models that learn efficient representations by applying diffusion processes in compressed latent space. Unlike traditional diffusion models that operate directly in high-dimensional data spaces E IRP to approximate p(y),
2 fo(o,t Var (1 - -I-VI-ar co(zz, co(zi,t)).
1
(3)
where a; = -I's Cg This formulation leads to improved efficiency using fewer steps. More details are provided in Appendix A.2.
2

---

Hlyper-Transforinng Latent Difflusion Models
2.2. Probabilistic Implicit Neural Representations
LoRA adapters for personalizing diffusion models from few examples. While their focus is efficient adaptation, our framework (LDMI) builds full generative model over INR parameter space. Our Hlype-Transsoome resolution-agnostic generation across modalities (images, 3D fields, climate data), representing a new direction in function-level generative modeling.
Implicit Neural Representations (INRs; Sitzmann et al., 2020) model continuous functions by mapping coordi- nates æ to output features y via neural network fa (z)xz (Mildenhall et al., 2021; Mescheder et al., 2019; Tancik et al., 2020). Recent generative approaches adapt this idea for flexible conditional generation at arbitrary resolutions (Dupont et al., 2022a; Koyuncu et al., 2023). Unlike mod- els that fit p(y) over structured data, INRs approximate the conditional p(yl allowing data sampling and uncer- tainty estimation at any socation-especinng beneficial for spatially-varying data.
Decoder enables
Although prior methods succeed in deterministic reconstruc- tion, they do not support probabilistic generation serve as decoders for latent variable models. Our approach addresses this by using full Transformer architectures conditioned on tensor- sensor-shaped latent variables.
Probabilistic INR generation typically involves meta- learning hidden representations that are mapped to INR parameters 4 The main challenge lies in learning these representations under uncertainty, with two goals: (i) em- bedding observed data into a posterior for conditional gener- ation, and (ii) sampling from prior for synthesis. Koyuncu et al. (2023) address this via a latent variable z and varia- tional framework combining a flow-based prior ps(z) (z) with an encoder qu (z xx y). using an MLP-based hypernetookk to map z to & (Ha et al., 2017). However, scaling to high- resolution images is limited by the complexity of the INR parameter space. Similarly, Dupont et al. (2022b) use ad- versarially trained hypemetworks, which improve stability but lack conditional generation and face the same scalability bottlenecks.
2.4. Function-valued Stochastic Processes
Alternative approaches like Neural Diffusion Processes (NDPs) (Dutordoir et al., 2023) and Simformer (Gloeck- ler et al., 2024) define distributions over function values, not over network parameters. NDPs denoise function val- ues directly using permutation-invarian attention, mimick- ing Neural Processes, while Simformer learns joint diffu- sion model over data and simulator parameters. In contrast, our method directly generates INR parameters via a hyper- transformer, offering continuous function generation with resolution independence.
3. Methodology
We introduce Latent Diffusion Models of Implicit Neural Representations (LDMI), novel family of generative mod- els operating in function space. While previous probabilistic INR approaches have explored latent variable models for INRs (Koyuncu et al., 2023; Dupont et al., 2022b) and two-stage training schemes incorporating diffusion- based priors in the second stage (Park et al., 2023; Dupont et al., 2022a; Bauer et al., 2023), these methods face key limita- tions. Notably, MLP-based hypernetworks suffer from ca- pacity bottlenecks, while existing frameworks lack a unified and compact approach for handling diverse data modalities.
Two recent strategies aim to address these issues. Functa (Dupont et al., 2022a; Bauer et al., 2023) learns neural fields per datapoint via optimization of modulation codes (referred as funclas), then trains a generative model on the resulting functaset. Alternativly, Chen et al. (2024) leverage LDMs by pretraining an autoencoder (e.g., 3-VAE or VQ-VAE) with an image renderer incorporated to the decoder, and applying a diffusion model on its latent space.
2.3. Transforrans
Hypernetworks (Ha et al., 2017) map latent representations # to INR parameters Φ- = (z), where Jo is a learned generator and (g(y)a) (a)) defines the INR. While the hyper- network is shared across the dataset, the INR is instance- specific, generated dynamically by the hypernetwork. This design, however, struggles with large INRs, as generating full parameter vectors introduces capacity and optimiza- tion bottlenecks. To overcome this, recent work has pro- posed Transforersss (Chen & Wang, 2022; Zhmogi- nov et al., 2022), which use attention to generate INR pa- rameters in modular chunks. This design avoids flattening and allows scalable, targeted weight generation.
To address these challenges, we propose the Hyper- Transformer Decoder (HD), which extends the flexibility of hypemetworks with the scalability of Transformer-basee architectures. Unlike prior deterministic frameworks (Chen & Wang, 2022), HD introduces a probabilistic formulation, enabling uncertainty modelling and improved generative capacity for INRs.
3.1. Notation
We define F= (f: + y} as the space of continu- ous signals, where X' represents the domain of coordinates, and y the codomain of signal values, or features. Let D be a dataset consisting of N pairs D = {(X,.. Y,) W Here, X2, and Y, conform a signal as a collection of D,,
Building on this paradigm, recent work, (HyperDream- Booth; Ruiz et al., 2024) extends this idea by generating
3

---

Epper-Transformin Latent Difflssin Models
W
b
Ep
[0(=|XX (Y Y)
9
Wi
b
W
by
(x)
(a) Encoder.
(b) INR Generation.
Figure 2: Overview of the VAE.. (a) The inference model, trained in the first stage, where encoder maps data into the variational parameters of the approximate posterior; (b) The full decoder: the HD module transforms the latent variable into the weights and biases of an INR, enabling continuous signal representation.
coordinates X, = (o))))
(n) Jim x af") E X' and signal values
auxiliary neural network known hypernetwork (Ha et al., 2017), typically MLP-based, which modulates the INR parameters based on a lower-dimensional latent repre- sentation:
Y, = (v) J Vi e.
3.2. Generative Model
go(z);
(6)
We aim to model the stochastic process that generates con- tinuous signals (X, Y) using neural networks p(y|a: +) = pp (z) known as Implicit Neural Representations (INRs). Unlike discrete representations such as pixel grids, INRs provide compact and differentiable function approximations, making them well-suited for wide range of data modalities, including images (Chen al., 2021), 3D shapes (Mescheder al., 2019), and audio (Sitzmann al., 2020), while inher- ently supporting unconstrained resolution.
where E =W,,,, xd, is a tensor-shaped latent code with spatial dimensions Hs x W, and channel dimension de. This latent space serves as a compressed representation of the continuous signal. However, MI.P-baed hypernet- works introduce a fundamental bottleneck: the final layer must output all parameters of the target INR, which leads to scalability issues when modulating high- high-capacity INRs. As a result, prior works employing MLPP-based hypernet- works typically use small INR architectures, often restricted to three-layer MLPs (Koyuncu et al., 2023; Dupont et al., 2022b). Additionally, these works rely on standard MLPs with ReLU or similar activations and often preprocess co- ordinates using Random Fourier Features (RFF) (Tancik et al., 2020) to capture high- freh-frryenncy details. However, RFF-based embeddings struggle generalize to unseen coordinates without careful validation. In contrast, SIREN (Sitzmann et al., 2020) inherently captures the full frequency spectrum by incorporating sinusoidal activations, making it more effective for super-resolution tasks.
Implicit Neural Representation Previous works have demonstrated that even simple MLP-based architectures ex- hibit remarkable flexibility, accurately approximating com- plex signals. In this work, we define our INR as an MLP with T. hidden layers:
ho = (xx),
h; = o (Wih--1+ + bi), 1= I=1,...,4, L,
(4)
(x) = hい = W -L.
where y denotes optional coordinate encoding, and is non-linearity. The INR parameters are given by & (W1, bi) The network parametrise a likelihood distribu- tion, A, over the target signal, that is,
However, using SIREN as the INR module introduces fur- ther challenges. As commented by their authors, optimizing SIRENs with not carefully chosen uniformly distributed weights yields poor performance both in accuracy and in convergence speed. This issue worsens when the weights are not optimized, but generated by a hypernetwork.
po (y(z) M(y: (x)).
(5)
Challenges in Modeling INR Parameter Distributions Learning generative model over the INR parameter space is highly non-trivial due to two fundamental challenges. First, small perturbations in parameter space can result in drastic variations in the data space, making direct modelling difficult. Second, the dimensionality of the flattened pa- rameter vector & scales poorly with the INR's width and depth, leading to significant computational and optimization challenges.
To overcome these limitations, we introduce the Hyper- Transformer Decoder (HD), a Transformer-bast based hypernet- work designed scale effectively while preserving the flex- ibility of larger INRs and ensuring proper modulation of SIREN weights.
3.2.1. HYPER- T NSFORMER DECODER
The Hyper- Transformer Decoder (HD) is a central compo- nent of our autoencoding framework for generating INRs, referred as I-VAE and illustrated in Figure 2. inter-
A common approach in prior works (Koyuneu et al., 2023; Dupont al., 2022b;a) is model & implicitly using an

---

Siper-Tranffrming Latent Diffusion Models
di
W
W?
W2
W
Latent tokens
Weight tokens
Wi b
Transformer Encoder
Transformer Decoder
W
Weight Reconst.
Wi [ b
2
WE
Wz | by
Figure Diagram of the Hypper-rranffommr Decoder (HD). The latent variable is tokenized and processed by Transformer Encoder. A Transformer Decoder, initialized with learnable grouped weights Wi, cross-attends the latent tokens to generate the of grouped weights Wy. The full weight matrices W, are then reconstructed by combining the grouped weights with learnable template weights WP. Biases by are learned as global parameters.
nal architecture is detailed in Figure 3. Mathematically, it is defined as a function of the latent tensor that produces the parameters & = (z) of the INR. We elaborate on previous work that proposed Vision Transformers (Doso- vitskiy al., 2021) for meta-learning of INR parameters from images (Chen& & Wang, 2022), to design an efficient full Transformer architecture (Vaswani et al., 2017) that processes latent variables into INR parameters.
Let W E R represent a weight matrix of an INR layer (omitting the layer index / for simplicity), where each column is denoted as w. for E (l,)...dm)). Directly mapping a Transformer token to every column is compu- tationally expensive, it would require processing long sequences. Instead, define G groups per weight matrix, where each group represents k du divisibility). This results in Transformer working with grouped weight matrices Wi, w E RR since Wi (input tokens) and W (output tokens) have the same di- mensions. The full weight matrix is reconstructed as:
& columns (assuming
Tokenizer The HD decoder begins by splitting tensor- shaped latent variables into N patches 2p E IR P of fixed size (P, P), where N = H W=/p², Each patch is then flattened and projected into lower-dimensional em- bedding space using shared linear transformation. The embedding for each patch serves as an input token for the Transformer.
w. = R(wf wh))
where R is the reconstruction operator, w, denotes the eth column of W, and wp is the corresponding column of Wh Rduxdd set of learnable base parameters that serve as weight template.
Transformer Encoder Following tokenization, the first half of the HD decoder, Transformer Encoder, applies multi-head self-attention mechanism, repeated for several layers, outputting tokenized embeddings that we referred to as latent tokens.
In Trans-INR (Chen & Wang. 2022), the Transformer-based hypernetwork generates normalized weights using the fol- lowing reconstruction method:
R (norm)
(wje/A], ma))
w
lc/k1
+
u
(8)
1c/17
Transformer Decoder Unlike prior work (Chen & Wang, 2022) that only employs Transformer Encoders, we intro- duce a Transformer Decoder that cross-attends to latent tokens to generate output tokens, which are then mapped to the column weights of the INR weight matrices. The input to the Transformer Decoder consists of globally shared, learnable set of initial weight tokens, denoted as W which serve as queries. The decoder processes these queries via cross-attention with the latent tokens, which act as keys and values to obtain the output set of column weights, W
However, we found this approach unsuitable for generating weights in INRs with periodic activations. We hypothesize that the normalization in Equation (8) causes the resulting weights and their gradients to vanish when l/wl ( || leading training instability.
To address this, we introduce ator that removes the normalization constraint, yielding significantly more stable training, particularly for INRs with periodic activations:
reconstruction oper-
Weight Grouping Following Chen et al. (2021), we adopt a similar weight grouping strategy to balance precision and computational efficiency in INR parameter generation. How- ever, our approach differs in the reconstruction strategy ap- plied to the ceeefforme--geeeratte weights.
(9) This formulation ensures that when l!wl [/wB//a]] = 0, the base weights remain unchanged, preventing instability in the generated INR parameters.
R (scale)
(w) W mb) = (1 + w. @ w.b
5

---

Hyper- Slyper-Trnntorming Latent Difflusion Models
is essential to distinguish between the two sets of globally shared learnable parameters. The sequence W i: (of length G' x L) serves the initialization for the grouped weight tokens and is provided as input to the Transformer Decoder. In contrast, the sequence Wh 1:L (of length dán x L) acts as a global reference for the full set of INR weights and is used exclusively for the reconstruction Equation (9).
g( Z1: 2)
po(z)
a(zt-1lee (z (2,,)) ())
Figure 4: Schematic of the DDIM-based latent diffusion.
This weight grouping mechanism allows the HD decoder to dynamically adjust the trade-off between precision and efficiency by tuning G. As result, our method scales ef- fectively across different model sizes while preserving the benefits of Transformer- based hypernetwork parameteriza- tion.
choice facilitates smooth interpolations and improves the quality of inferred representations.
In addition, following Rombach et al. (2022), for the case of images, incorporate perception losses (Zhang et al., 2018), and patch-based (Isola et al., 2017) adversarial ob- jectives ( (Dosovitskiy & Brox, 2016; Esser et al., 2021; Yu et al., 2022).
3.3. Variational Inference
Given our implicit modelling of INR parameters, the objec- tive is to learn a structured and compact probabilistic latent space po (=) ) that enables meaningful posterior inference via Bayesian principles. Specifically, we seek approximate the true posterior x(=|X, Y))), facilitating accurate and flex- ible inference. To achieve this, we extend the Variational Autoencoder (VAE) framework (Kingma & Welling, 2013) to the efficient generation of INRs, which we refer to as I-VAE (depicted in Figure 2).
However, while this training strategy ensures effective infer- ence, direct generation remains poor due to the discrepancy between the expressive encoder distribution and the simplis- tic Gaussian prior. To address this mismatch, introduce second training stage, where fitted to the marginal posterior distribution, enhancing the generative capacity of the learned latent space.
reffuusionbbasee model is
The complexity of the likelihood funetion makes the true posterior Xx X, Y) intractable. To address this, VAEs in- troduce encoder network that approximates the posterior through a learned variational distribution. Our encoder, de- noted as (X, Y), processes both coordinate inputs X and signal values Y, outputting the parameters of a Gaus- sian approximation:
3.4. Latent Diffusion
In this stage, we leverage the pre-trained I-VAE and the highly structured latent space encoding INRs to fit a DDPM (Ho et al., 2020) to the aggregate posterior:
qo (z) - Ex (X,)) (2) Y)
(12)
(= = N' (z; YX,
(10)
In other words, we fit learnable prior over the latent space to approximate the structured posterior induced by the en- coder. Specifically, we minimize the Kullback-Leiblee di- vergence
This formulation enables training via amortized variational inference, optimising lower bound on the fog-margina likelihood log p(Y Y X), known as the Evidence Lower Bound (ELBO):
DKL. (1) (l
(13)
where po(x) represents the learned diffusion-basee prior over the latent space. We instantiate the following variation of the Denoising Score Matching (DSM; Vincent, 2011) objective presented in Equation (1):
LVAB vb- (x)(x [logps
(11)
- DKL
Y)
where we omit the explicit dependence of d on o and for clarity. The hyperparameter 33, introduced by Higgins et al. (2017), controls the trade-off between reconstruction fidelity and latent space regularization, regulating the amount of information compression in the latent space. The case 3 = corresponds to the standard definition of the ELBO.
LDDPM = Ex,Y [AcO]l= -o(=,, [
(14)
which approximated via Monte Carlo sampling. Specifi- cally, we draw (X, Y) Pdota <~ -,(O0, I), and ~ U(1, where T denotes the num- ber of diffusion steps. The weighting function X(I) mod- ulates the training objective; in our case, we set X(t) = 1, corresponding to an unweighted variational bound that en- hances sample quality. The complete training procedure for LDMI outlined in Appendix B.
Paata(X, (X, Y), z {= Y).
During the first stage of training, following LDMs (Rom- bach et al., 2022), we impose simplistic standard Gaussian prior p(z) = (0, I), and we set a low 3 value to en- courage high reconstruction accuracy while promoting a structured latent space that preserves local continuity. This
6

---

Hlyper-Transoore Latent Diffusion Models
GASP
Funeta
Ouns
Ours
(a) CelebA HQ 64 x 64
(b) CelebA-HQ 256 x 256
Figure 5: Uncurated samples from GASP, Functa (diffusion-based and our LDMI trained with CelebA-HQ at 64x64 resolution (a). In (b), our model was trained on CelebA-HQ at x56 using the hyper-transforming approach.
3.5. Jyype-Trrasffrming Latent Diffusion Models
Our evaluations' span multiple domains: (1) natural image datasets, including CelebA-HQ (Liu et al., 2015) at sev- eral resolutions and ImageNet (Russakovsky et al., 2015); (2) 3D objects, specifically the Chairs subelass from the ShapeNet repository (Chang et al., 2015). which provides approximately 6,778 chair models for 3D reconstruction and shape analysis; and (3) polar climate data, using the ERA5 temperature dataset (Hersbach et al., 2019) to ana- lyze global climate dynamics. In our hyper-transforming setup, we leverage publicly available pre-trained Latent Dif- fusion Models from Rombach et al. (2022), specifically the LDM-VQ-4 variants trained on CelebA-HQ at 256 256 resolution and ImageNet, respectively.
As an alternative to full training, we introduce highly efficient approach for training LDMI when a pre-trained LDM for discretized data is available. In this setting, we eliminate the need for two-stage training by leveraging the structured latent space learned by the pre-trained LDM. Specifically, we freeze the VAE encoder and the LDM and train only the HD decoder to maximize the log- ligeiiidd of the decoded outputs, using the objective
(o) = Eg (=|x,t) [log pa ( x
(15)
optionally combined with perceptual and adversarial objec- tives. The HD decoder sufficiently expressive to extract meaningful information from the pre-trained latent space.
4.2. Generation
We refer to this strategy hyper-transformiing as it enables efficient adaptation of standard LDM to an INR-based framework without retraining the latent encoder or diffusion prior. Notably, this approach is compatible with LDMs based on VAEs, VQVAEs, or VQGANs-Ste since that is required access pre-trained encoder and diffusion model. The full procedure is outlined Algorithm 1.
Figure 1 shows samples generated by LDMI across diverse data modalities and resolutions. To obtain these, we ple from the latent diffusion model, decode the latents into INRs using our HD decoder, and evaluate the resulting func- tions different coordinate grids. Notably, CelebA-HO at (256 x 256) a challenging dataset not addressed by prior baselines.
In Figure 5, we provide samples from LDMI trained on CelebA at 64x64 and 256x256, demonstrating its ability to generate high-quality and diverse images across different scales, and the qualitative superiority against the baselines. Additionally, in Table 1, we report FID scores of our sam-
4. Experiments
4.1. Training Setup
Depending on the nature of the data, we utilize different ar- chitectures for the encoder. For image and climate data, we employ ResNets (He et al., 2016), while for 3D occupancy data, we employ 3DDcconvoltiional networks.
'The code for reproducing experiments can be found ht https: / /ttp::///gihuuu httpbb.tbbubbgbe/// com/ ipeispiis///eeee LDMI.

---

HlyperTraansfrming Latent Diffusion Models
GT
Funeta
VAMol!
Ours
(a) CelebA-HQ
(b) ERA5
(c) ShapeNet Chairs
GT
EHE
Ours
(d) ImageNet
(e) CelebA-HQ (256 256)
Figure 6: Reconstructions from Functa (diffusion-based) VAMoH, and LDMI compared to ground truth (GT) across three datasets (a)- In (d) and (e) LDMI was trained by hyper-transforming pre-trained LDMs.
Model
PSNR (dB)
FID
HN Params
construction performance across diverse object categories and scenes, including textures, object boundaries and colour distributions.
CelebA-HQ (64 64)
GASP (Dupont et al., 20226)
7.42
25.7M
Functa (Dupont al., 2022a)
30.7
40.40
VAMoH (Koyuncu et al., 2023)
23.17
66.27
Quantitative evaluations Table indicate competitive performance compared to existing INR-based generative models. For CelebA-HQ 64 64, we achieve PSNR value of 24.80 dB, which compares competitively to the previous methods. The high PSNR score suggests there is mini- mal information loss during the encoding-decoding process while maintaining perceptual quality. It is important to note several key distinctions when interpreting the comparative results in this Table. While Functa exhibits higher PSNR val- ues, this advantage stems from its test-time optimization pro- cedure-fitting separate modulation vector per test image using ground truth information -rathe than the amortized inference approach employed by our model. This funda- mental methodological difference undermines a direct com- parison, though we include these results for completeness.
LDMI
24.80
18.06
8.06M
Image (256 256)
Spatial Functa (Bauer 2023)
38.4
8.5
LDMI
20.69
6.94
102.78M
Table 1: Metrics on CelebA-Ho and ImageNet.
ples for CelebA and ImageNet, highlighting the model's performance in terms of image quality and reconstruction accuracy. Importantly, while GASP achieves a lower FID score CelebA-HQ. cannot perform reconstructions due to its adversarial design. In contrast, our LDMI supports both high-quality sampling and accurate reconstructions, as demonstrated in the following experiment.
As for generation, our model enables seamless reconstruc- tion arbitrary resolutions. Figures 9 and 10 show re- sults from LDMI trained on CelebA-HQ (256 x 256) and (64 x 64), respectively. To achieve this, we encode the observations, sample from the posterior distribution, and transform these latent samples into INRs using our HD de-
4.3. Reconstruction
Figure 6 presents a qualitative comparison between origi- nal images from the test split and their reconstructions by our model. As shown in the figure, our model successfully preserves fine details and global structures across different images. On ImageNet, our model demonstrates robust re-
8

---

Hyper- Hlyee--Transorming Latent Difflusion Models
coder. The resulting INRs then evaluated on grids of varying resolutions produce reconstructions ranging from x x1/8 to x4. Importantl, such flexible resolution control is not supported by methods like GASP, which cannot perform reconstructions due to their adversarial design.
Input
INT
TIV
To further validate our claims regarding cross- modslito generalization, we evaluate LDMI on two additional, di- verse datasets: ShapeNet Chairs and ERA5 climate data. As shown in Table 2, while Functa achieves better per- formance on Chairs- possibly due to being optimization- based-LDMI attains higher reconstruction quality than VA- MoH (inference-
Samples
INT
across both datasets, outperforming it by 0.5% on Chairs and 5.6 dB on ERA5. The substantial performance gain on the ERA5 climate dataset highlights LDMI'S exceptional capability representing complex, con- tinuous spatio-temporal signals beyond standard visual data, further supporting our argument for truly general-purpose INR-based generative framework.
TIW
Figure 7: Conditional inpainting results on CelebA-HQ at 256 x 256. The second and third rows present two con- ditional samples generated by our LDMI for the missing regions at the centre.
Model
Chairs (%) 1
ERA5 (dB)
f
Functa (Dupont et al., 2022a) VAMoH (Koyuncu al., 2023)
99.51
34.9
96.75
39.0
4.5. Data Completion
LDMI
97.25
44.6
We evaluate our model on structured completion tasks, where parts of the input are masked with predefined pat- terns. For each case, show the masked input and two samples drawn from the posterior. As shown in Figure 7, the model uses visible context to produce realistic, coherent outputs. On CelebA-HQ 256 x 256, it reconstructs missing features- such as eyes, mouth, or hairline while preserv- ing consistency with the unmasked regions. These results highlight its ability generalize from partial observations.
Table Reconstruction quality (accuracy in % and PSNR in dB) on ShapeNet Chairs and ERA5 climate data, demon- strating LDMI'S ss strong generalization capabilities across modalities. Note that GASP is omitted as it is not applicable to INR reconstruction tasks.
The dual capability of high-quality generation and accurate reconstruction positions LDMI as a versatile foundation for various downstream tasks requiring both generative and reconstructive abilities, unique advantage over specialized alternatives optimized for only one of these objectives.
5. Conclusion
We introduced Latent Diffusion Models of Implicit Neural Representations (LDMT), a framework that combines the expressiveness of INRs with the generative power of Latent Diffusion Models. Our typer-transforming approach al- lows efficient adaptation of pre-trained LDMs to INR-based generation without retraining the full model. Experiments CelebA-HQ, ImageNet, ShapeNet Chairs, and ERA5 demonstrate strong performance in both quantitative and qualitative evaluations.
4.4. Jyper-Transforming
On ImageNet and CelebA-HQ at 256 256, our LDMI was trained using the hyper-transforming strategy. As discussed in Section 3.5, instead of training the model from scratch, we reuse the encoder and diffusion model from pre-trained LDM, training only the Hyper- Transformer Decoder. This approach leverages both the structured latent space learned by the autoencoder and the diffusion model already fitted to it. Training then focuses solely on transforming latent codes into the function space of INR parameters, enabling efficient specialization without retraining the generative backbone.
Importantly, LDMI overcomes limitations of prior hypernet- works through modular, probabilistic Transformer decoder that supports scalable, expressive INR generation. While MLP- based methods struggle with dense parameter outputs and Transformer- based ones are typically deterministic, our design enables probabilistic modelling and efficient synthe- sis. Integrating INRs with latent diffusion yields a flexible, resolution- assouutio-agnoostc framework caaneworr-ddvancinn generative tasks requiring structural consistency, fidelity, and scalability.
Our method is agnostic to the latent encoder-VAR, VQ- the flexibility of the hyper-
VAE, or JoGAN-highligee transforming framework. These results demonstrate its scal- ability for based synthesis and adaptation of pre-trained LDMs to resolution -solution-agnosticc generation.

---

Hyper- Trppe-rrannfrrming Latent Diffusson Models
Acknowledgements
Dhariwal, and Nichol, A. Diffusion Models Beat GANs
Image Synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
This research was supported by the Villum Foundation through the Synergy project number 50091, by the Novo Nordisk Foundation through the Center for Basic Ma- chine Learning Research in Life Science (MLLS, grant NNF200C0062066) and by the European Union (ERC- 2021-STG, SAML, 101040177).
Dosovitskiy, A. and Brox, T. Generating Images with Per- ceptual Similarity Metrics Based on Deep Networks. In Advances in Neural Information Processing Systems, vol- ume 29, 2016.
Ignacio Peis acknowledger support by Danish Data Science Academy, which is funded by the Novo Nordisk Foundation (NNF21SA0009429) Batuhan Koyuneu is supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education. The views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority be held responsible.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An Image is Worth 16x Words: Transformers for Image Recognition at Scale. In International Conference Learning Representations, 2021.
Dupont, E., Kim, H., Eslami, S. A., Rezende, D. J., and Rosenbaum, D. From data functa: Your data point is a function and you can treat it like one. In Interna- tional Conference on Machine Learning, pp. 5694-5725. PMLR, 2022a.
Impact Statement
Dupont, E., Teh, Y. W., and Doucet, A. Generative Models as Distributions of Functions. In International Confer- ence on Artificial Intelligence and Statistics, pp. 2989- 3015. PMLR, 2022b.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work. none which we feel must be specifically highlighted here.
Dutordoir, V., Saul, A., Ghahramani, Z., and Simpson, F. Neural Diffusion Processes. In International Conference on Machine Learning, pp. 8990-9011 PMLR, 2023.
References
Bauer, M., Dupont, E., Brock, A., Rosenbaum, D., Schwarz, J. R., and Kim, H. Spatial Functa: Scaling Functa to ImageNet Classification and Generation. arXiv preprinl arXiv:2302.03130 2023.
Esser, P., Rombach, R., and Ommer, B. Taming Transform- ers for High- ReghuRessoution Image Synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12873-12883, 2021.
Chang. A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang. Q. Li, Z., Savarese, S., Savva, M.. Song, S., Su, H., Xiao, J., Yi, L., and Yu, F. ShapeNet: An Information -ichatiioo-ichhhh-- 3D Model Repository. Technical Re- port arXiv: www.:1112.000 512.03012 [cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
Gloeckler, M., Deistler, M., Weilbach, C. D., Wood, F., and Macke, J. H. All-in-one Simulation-based Inference. International Conference on Machine Learning, pp. 15735-15766. PMLR, 2024.
Ha, D., Dai, A. M., and Le, Q. V. Hypernetworss. In International Conference on Learning Representations, 2017.
Chen, Y. and Wang, X. Transformers Meta-Learners for Implicit Neural Representations. In European Conference Computer Vision, pp. 170-187. Springer, 2022.
He, K., Zhang, X., Ren, S., and Sun, Deep Residual Learn- ing for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Chen, Y., Liu, S., and Wang, X. Learning Continuous Image Representation with Local Implicit Image Function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628-8638, 2021.
Hersbach, H., Bell, B., Berrisford, P., Biavati, G., Horányi, A., Muñoz Sabater, J., Nicolas, J., Peubey, C., Radu, R., Rozum, I., et al. ERA5 Monthly Averaged Data on Single Levels from 1979 to Present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS), 10:252-266 2019.
Chen, Y., Wang, O., Zhang, R., Shechtman, E., Wang, X., and Gharbi, M. Image Neural Field Diffusion Models. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8007-8017, 2024.
10

---

Hyper-Transforming Latent Difflusion Models
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (-VAE: Learning Basic Visual Concepts with Constrained Varia- tional Framework. In International Conference on Lear- ing Representations, 2017.
Park, D., Kim, S., Lee, S., and Kim, J. DDMI: Domain- Agnostic Latent Diffusion Models for Synthesizing High- Quality Implicit Neural Representations. In The Twelfth International Conference on Learning Representations, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Proba- bilistic Models. Advances in Neural Information Process- ing Systems, 33:6840-6851, 2020.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Om- mer, B. High-Resolutee Image Synthesis with Latent Diffussonn Models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Hou, X., Shen, I.., Sun, K., and Qiu, G. Deep Feature Con- sistent Variational Autoencoder. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1133-1141. IEEE, 2017.
Ruiz, N., Li, Y., Jampani, V., Wei, W., Hou, T., Pritch, Y., Wadhwa, N., Rubinstein, M., and Aberman, K. Hyper- HyperNetworks for Fast Personalization of Text-to-lmage Models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pp. 6527-6536, 2024.
DrcamBooth:
Hou, X., Sun, K., Shen, L., and Qiu, G. Improving Vari- ational Autoencoder with Deep Feature Consistent and Generative Adversarial Training. Neurocomputing, 341: 183-194, 2019.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115:211-222, 2015.
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image- to-Image Translation with Conditional Adversarial Net- works. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 1125-1134, 2017.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image Super-Resolution via Iterative Refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713-4722 2022.
Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Computer Vision ECCV 2016, pp. 694-711. Springer, 2016.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wet- zstein, G. Implicit Neural Representations with Periodic Activation Functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Kingma, D. P. and Welling, M. Auto-encoding Variational Bayes. arXiv preprint arXiv:1312.6114 2013.
Koyuncu, B., Martin, P. S., Peis, I., Olmos, P.M and Valera, I. Variational mixture of hypergenerators for learning distributions over functions. In International Conference on Machine Learning, pp. 17660-17683, PMLR, 2023.
Song, J., Meng, C., and Ermon, S. Denoising Diffusion Im- plicit Models. In International Conference on Learning Representations, 2021a
Larsen, A. B. L., Sønderby, S. K., Larochelle, H., and Winther, O. Autoencoding Beyond Pixels Using a Learned Similarity Metric. In International Conference on Machine Learning, pp. 1558-1566 PMLR, 2016.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-Based Generative Modeling through Stochastic Differrntial Equations. In Interna- tional Conference Learning Representations, 2021b.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep Learning Face Attributes the Wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730- 3738, 2015.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Ken Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng, R. Fourier Features Let Networks Learn High Frequency Functions Low Dimensional Domains. In Advances in Neural Information Processing Systems, volume 33, pp. 7537-7547, 2020.
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., and Geiger, A. Occupaney Networks: Learning 3D Re- construction in Function Space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4460-4470, 2019.
van den Oord, A., Vinyals, O., and kavukcuoglu, k. Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng. R. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Commu- nications of the ACM, 65(1):99-106, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Atten- tion Is All You Need. In Advances in Neural Information Processing Systems, volume 30, 2017.

---

Hyper-Transforming Latent Diffusion Models
Vincent, P. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7): 1661-1674, 2011.
Yu, J., Li, X., Koh, Y., Zhang. H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, and Wu, Y. Vector-Quantized Im- age Modeling with Improved VQGAN. In International Conference on Learning Representations, 2022.
Zhang. R., Isola, P., Efros, A. A., Shechtman, E.,, and Wang, O. The Unreasonable Effectiveness of Deep Features Perceptual Metric. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2018.
Zhmoginov, A., Sandler, M., and Vladymyrov, M. Hyper- Transformer: Model Generation for Supervised and Semi- Supervised Few-Shot Learning. In International Confer- ence Machine Learning, pp. 27075-27098 PMLR, 2022.
12

---

Latent Diffusion Models
A. Background Extension
A.I. Autoencoding-Iass Models
When opting by 3-VAE (Higgins et al., 2017) for the first stage of LDMs (Rombach et al., 2022), a weak regularization encourages accurate likelihoods px(y) =) parameterized by the decoder Dx(-) while preserving latent expressiveness. This achieved by assuming p(z) (O, I) and maximizing the Evidence Lower Bound (ELBO):
VAV = E (=lv) [x)g)) B DEL (gv(z)g) || p(=))
(16)
and setting is set small value to encourage accurate reconstructions.
In contrast, VQVAE replaces the continuous latent representation with discrete codebook. During training, the encoder output is quantized using the closest embedding from the codebook, enabling more structured representation. The objective becomes:
LVQAVE = yy - (x()(( + ||sg|E (v) =11² + [|sg[z] (
(17)
where sg[:] indicates stop-gradient operation that prevents the encoder from directly updating the codebook embeddings.
A.2. Diffusion Models
In DDPM, the forward diffusion process incrementally corrupts the observed variable 2g using a Markovian noise process with latent variables 21:7
T
(ZITT|20) = II (2x1(==1),
(18)
where g(zx|z=-1) = - 8,1), and Be is a noise schedule controlling the variance at each time step.
The reverse generative process is defined by the following Markov chain
Po Po(=o:T)= == p(zT)
II
(o(2
(19)
t=1
where pz(z)-1lz))= == N (Ha(Zz, t), oî) is defined as a Gaussian whose mean
3e
no [ 1)
2:
VI - EE((2,)))
(20)
is obtained by neural network that predicts the added noise using neural network co(22,t). t).
A more suitable inference distribution for ending with compact objective can be expressed via the reverse posterior conditioning on the observation Zy:
12, N
-N (z (Z1, (2), 20), 8.).
(21)
The model is learned by minimizing the variational bound on negative log-likelihood, which can be expressed as a sum of terms,
const
Lo
CDDPM = DrI. (a(2T|z0) || p(2T)) + - p(2)p(2))))) +
(22)
DKL (g(z)- Ize, 20) (0(z--1/z1)
Efficient training is achieved by uniformly sampling ~ U(1,7) T) and optimizing the corresponding L,-1, which, by deriving Equation (22), can be further simplified to a denoising score-matching loss:
LDUPM = Ezo.cie ||c (z (2(
(23)
where 6 (O. I).
13

---

Hyper- Typpe-Transformin Latent Diffusion Models
1. INFERENCE
The reverse posterior density defined in Equation (21) is no longer Markovian and coincides with the inference model proposed in DDIM (Song et al., 2021a), where it is demonstrated that faster sampling be achieved without retraining, simply by redefining the generative process as:
N (18) (z1), of1 g z1-1|ze, f/) (zi)
ift
Po (2(--1122)) =
(24)
otherwise,
and considering that an estimation of 20 fo(z t) can be computed as:
1 Vos 22) vI -i (0 t))
fo(z),t)
(25)
where 0r II:m ag This formulation enables deterministic sampling with improved efficiency using fewer steps.
B. Training LDMI
Algorithm
Hyper-Trrasformingg LDM
Input: Dataset ((X; Y1) pre-trained LDM (frozen encoder &g, frozen diffusion model po), Transforerae decoder pa
repeat
Sample batch (X, Y ~ Paam (X, Y)
Sample latent using frozen encoder: gv Xm, Ym)
Compute likelihood loss: (i) using Equation (15) if image data then
Add perceptual loss: Cpenopn Add adversarial loss: Ladv
end if
Update decoder parameters minimize total loss until convergence
Algorithm Training LDMT
Input: Dataset (X,, Yi)i = 1 encoder Ev, decoder pe, diffusion model Po
Stage 1: Training I-VAE
repeat
Sample batch (X, Y) ~ Pauaa (X, Y) Sample latent: z ~ Lu (2 X Y)
Compute ELBO loss: LvAu 16) using Equation (11)
if image data then
Add perceptual loss: Cperept Add adversarial loss: Ladv end if
Update parameters o, v minimize total loss until convergence
Stage 2: Training DDPM repeat
Sample batch (X, Y) Paam (X, Y)
Sample latent: qu (x X,
Sample noise: e ~~ (0,
Sample timestep: & ~ (1.
Compute DDPM loss: LDDPM using Equation (14)
Update parameters minimize LOOPM
until convergence
14

---

Sypper-Trnsforming Latent Diffusson Models
In addition to the hyper-transforming approach described in the main text, LDMI also be trained from scratch using two-stage process. This follows the standard Latent Diffusion Model (LDM) training pipeline but incorporates Hyper-Transformer Decoder (HD) for INR generation. The training procedure consists of:
1. Stage 1: Learning the Latent Space with I-VAE - We train a Variational Autoencoder for INRs (I-VAE) to encode continuous signals into a structured latent space. The encoder learns an approximate posterior l6 (2 X, Y), while the decoder reconstructs signals from the latent variables. The training objective follows the Evidence Lower Bound (ELBO) defined in Equation (11). To enhance reconstruction quality, additional perceptual or adversarial losses may be applied for certain data types.
2. Stage 2: Training the Latent Diffusion Model (LDM) Given the structured latent space obtained Stage we fit diffusion- generative model (=( the aggregate posterior qol (z). This stage follows the standard DDPM framework, minimizing the objective in Equation (14). The learned diffusion prior enables generative sampling in the latent space, from which the HD decoder infers INR parameters.
The full training procedure is summarized in Algorithm 2.
C. Additional Experiments
C.1. Scalability of LDMI
A key strength of our approach lies in its parameter efficiency and scalability when compared alternative methods. As shown in Table 3, while previous approaches such as GASP (Dupont et al., 2022b) or VAMoH (Koyuncu et al., 2023) require substantial hypernetwork parameters (25.7M) to generate relatively small INR weights (50K), LDMI achieves superior performance with only 8.06M hypernetwork parameters while generating 330K INR weights for a 5-layer network.
Method
HN Params
INR Weights
Ratio (INR/HN)
GASP/AMMH
25.7M
50K
0.0019
LDMI
8.06M
330K
0.0409
Table 3: Parameter efficiency of hypernetworks (HN) in GASP/VaMoH and LDMI.
We confirm this efficiency effect through ablation study comparing transformer-based HD decoder against standard MLP design, as MLPs are commonly used in hypernetwork implementations despite their limitations in modeling complex dependencies. Table 4 shows that on CelebA-HO -HQ. our HD architecture not only achieves superior reconstruction quality (with a 2.79 dB improvement in PSNR) but does so with significantly fewer parameters- less than half compared to the MLP architecture.
Method
HN Params
PSNR (dB)
LDMT-MLP LDMT-AD
17.53M
24.93
8.06M
27.72
Table 4: Ablation study comparing MLP and hyper-transformer HD decoders on CelebA-HQ.
This parameter efficiency highlights a crucial advantage of our approach: hyper-transformer's ability to capture complex inter-dimensional dependencies translates to more effective weight generation with a more compact architecture. The superior scaling properties of LDMI suggest that it can handle larger and more complex INR architectures without prohibitive parameter growth, making it particularly suitable for high- tigh-resolution generation tasks that demand larger INRs.
C.2. Samples at Multiple Resolutions
To further demonstrate the flexibility of our model, we present additional qualitative results showcasing unconditional samples generated at varying resolutions. As described the main text, we sample from the latent diffusion prior, decode the latents into INRs using our HD decoder, and evaluate them on coordinate grids of increasing resolution.
15

---

Typee-Transfommig Latent Diffusion Models
CelebA- HQ (236. 256)
ColebA-11Q (64 64)
ERAS
3D
Figure 8: Additional uncurated samples from LDMI at multiple resolutions.
Figure 8 displays generations from LDMI trained on data across different modalities, rendered at scales ranging from x0.125 x4. These results highlight the resolution resolution-agmostic nature our approach and its ability to produce coherent, high-quality outputs across wide range of spatial resolutions.
C.3. Reconstructions at Multiple Resolutions
We provide additional reconstruction results further illustrate LDMI's ability operate seamlessly across wide range of output resolutions. Each input image is encoded into latent representation, sampled from the posterior, and decoded into an INR using HD decoder. The resulting INR is then evaluated over coordinate grids of increasing density to generate reconstructions at progressively higher resolutions.
Figures 9 and 10 present reconstructions from LDMT trained on CelebA-HQ at (256 x 256) and (64 x 64), respectively. Reconstructions are rendered at resolutions ranging from x0.125 to x4, showcasing the model's ability to maintain spatial coherence and fine details even under extreme upsampling. Notably, such flexible reconstruction not supported by baseline methods like GASP due architectural limitations. Furthermore, datasets this complexity are not even addressed by other reconstructionncappanee Functa (Bauer et al., 2023).
baselines such VAMoH (Koyuneu al., 2023), Functa (Dupont et al., 2022b), Spatial
D. Experimental Details
This section outlines the hyperparameter settings used to train LDMI across datasets. We describe key components, including the encoder, decoder, diffusion model, and dataset settings. Table summarizes the hyperparameters used in our experiments,
16

---

Hypee-Transforming Latent Difflusion Models
covering both stages of the generative framework: (i) the first-stage autoencoder--either VQVAE 3-VAE, depending the dat (ii) the second-stage latent diffusion model. It lists architectural choices such latent dimensionality, diffusion steps, attention resolutions, and optimization settings (e.g., batch size, learning rate), along with details of the HD decoder, tokenizer, Transformer modules, and INR architecture. For image- based tasks, we use perceptual and adversarial losses to improve sample quality, following Esser et al. (2021). ). All models were trained on NVIDIA H1 00 GPUs.
17

---

Typee-Transfrmmin Latent Diffusion Models
GT
ke
x0. 125
x0.25
x0.5
22
Figure 9: Reconstructions test CelebA-HQ (256 x 256) images by LDMI at multiple resolutions. The ground truth image first passed through the encoder, which produces the parameters of the posterior distribution. latent code is then sampled and transformed into the parameters of the INR using our HD decoder. By simply evaluating the INRs at denser coordinate grids, can generate images at increasingly higher resolutions.
18

---

Trpee-Transfrmmin Latent Diffusion Models
GT
OT
x0. 125
x0.25
x0.5
X2
Figure 10: Reconstructions of test CelebA-HQ (64 x 64) images by LDMI at multiple resolutions. The ground truth image first passed through the encoder, which produces the parameters of the posterior distribution. A latent code is then sampled and transformed into the parameters of the INR using our HD decoder. By simply evaluating the INRs at denser coordinate grids, can generate images at increasingly higher resolutions.
19

---

Hypee--rrnnsorming Latent Diffusion Models
CelebA
CelebA-HO 64
CelebA-HQ 256
ImageNet
ERAS
Chairs
resolution
64 x 64 image 3-VAE
64 x 64
256 x 256
256 x 256
46 x 90
32 x 32 x 32
modality
image 3-VAR
image VQVAE
image VQVAR
polar 3-VAR
occupancy
first stage model
(-VAE
Encoder
codebook size
8192
8192
lehnne_s
3
3
3
3
3
base channelss
64
64
128
128
32
32
chamult
1,2,4
1,2,4
1,2,4
1,2,4
1,2,4
1,2,4
num blocks
2
2
2
2
2
dropout
0.1
0.2
kl kl-weight
Ic-05
1c-04
1.0c-6
1.0c-6
pere. weight
1.
1.
Discriminator
layers
2
2
3
3
n filters
32
64
64
64
dropout
0.2
disc_weight
0.75
0.75
0.75
0.6
decoder
Tokenizer
latent_size patch size heads head dim
16 x 16
16 x 16
64 x 64
64 x 64
11 x 22
x 4
2
2
4
4
1
4
4
4
4
32
32
32
32
32
32
Transformer
token_dim
192
384
768
136
128
encoder_laycrs
4
decoder layers
6
5
6
4
heads
6
6
12
4
head dim
48
48
64
64
32
32
feedforward.dig dim
768
768
1536
3072
512
512
groups
64
64
128
128
64
64
dropout
0.1
0.1
0.2
INR
Type
SIREN
STREN
STREN
MLP
SIREN
layers
5
5
hidden_dim
256
256
256
256
256
256
point_ence enc. een..encccmmm dim
256
W
30.
30.
30.
30.
30.
Latent Diffusion
shape.z
16
3 64 x 64
768
768
12288
12288
726
1024
diffusion _tteps
1000
1000
1000
1000
1000
1000
noise_schedule
linear
linear
linear
linear
linear
linear
base channels
64
32
224
192
64
128
ch_mult
1,2,3,4
1,2,3,4
1,2.3,4
1,2,3.5
1,2,3,4
1,2
attn resolutions
2. 4.
2. 4.
8, 16. 32
8. 16. 32 192
head.cct channels
32
32
32
32
64
blocks
2
2
2
2
2
class, class_cond cooodd
crossattn
context.dim dim
512
transformers.o depth
Training
batch size
64
64 300k, 200k
32
32
64
128
iterations
300k, 400k
1M
1.4M
500k
300k, 60k
Ir
Ic-06, 2c-06
lc-06, 2c-06
4.5c-6
1.0c-6
Ic-06, 2c-06
Ic-06, 2c-06
hypee-transfomming
LDM version
VQ-F4 (no-attn)
VQ-F4 (no-attn)
Table 5: Architecture details.
20


Reviews:
[{"review_id": "5szj3BrJg8", "reviewer": "Reviewer_ER44", "summary": "This paper introduces a new generative framework called Latent Diffusion Models of Implicit Neural Representations (LDMI), which integrates Implicit Neural Representations (INRs) into transformer-based latent diffusion models. The key component is to use a Hyper-Transformer Decoder (HD) to replace traditional MLP-based hypernetworks and address their limitations of scalability and efficiency. This module generates INR parameters from latent variables. Experiments have shown the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and thoughtful feedback, as well as for recognizing the novelty and clarity of our work. Below, we address the key concerns raised.\n\n## On the scope and nature of the contribution\n$\\texttt{LDMI}$ is not intended to compete with standard diffusion models that operate in pixel space. These models approximate $p(\\boldsymbol{y})$ over discrete grids. In contrast, we model $p(\\boldsymbol{y}|\\boldsymbol{x})$ as a stochastic process, where $\\boldsymbol{y}$ is a function represented via INR parameters $\\theta$. This defines a substantially richer and more complex generative space—orthogonal to that of the suggested references. To our knowledge, $\\texttt{LDMI}$ is the first to use Transformers as decoders to generate INR weights from latent samples in this setting.\n\nThat said, we appreciate the reviewer’s suggestion and agree that including and discussing these works helps contextualize our contributions. We have added the suggested pixel-based diffusion models [A, B, C] to the Related Work section and explicitly discussed how our problem setting fundamentally differs.\n\n## On baseline comparisons\nTo provide fair comparisons in the INR setting, we benchmark against the closest functional baselines. In addition, to emphasize the differences with respect to the provided references, we have added super-resolution experiments (e.g., [samples](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_samples.png?v=485b880b) and [reconstructions](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_recs_celebahq256.png?v=5cc0265)) to show that LDMI generalizes across resolutions without retraining—a key property enabled by its function-based design.\n\n## On scalability of the $\\texttt{HD}$ Decoder\nWhile our original submission already compared $\\texttt{LDMI}$ against MLP-based hypernetworks (used in all baselines), we have now further strengthened our scalability analysis. The updated Table 1 includes hypernetwork sizes:\n\n| **Model**                         | **PSNR (dB) ↑** | **FID ↓** | **HN Params ↓** |\n|----------------------------------|-----------------|-----------|-----------------|\n| **CelebA-HQ (64 × 64)**          |                 |           |                 |\n| GASP       | N/A               | **7.42**  | 25.7M           |\n| Functa     | $\\mathbf{\\leq 30.7}$      | 40.40     | N/A               |\n| VAMoH      | 23.17           | 66.27     | 25.7M           |\n| **$\\texttt{LDMI}$**                         | 27.72           | 11.08      | **8.06M**       |\n|                                  |                 |           |                 |\n| **ImageNet (256 × 256)**         |                 |           |                 |\n| Spatial Functa  | $  \\mathbf{\\leq 38.4} $   | $ \\geq 8.5  $| N/A               |\n| **$\\texttt{LDMI}$**                         | 20.69           | **6.94**  |  102.78M\n\nGASP and VAMoH use ~25.7M parameters to generate 50K weights for shallow 3-layer INRs. In contrast, our $\\texttt{HD}$ decoder uses **only 8.06M parameters** to generate 330K weights for a deeper 5-layer INR. Despite using 70% fewer parameters, $\\texttt{LDMI}$ performs better, scaling to complex signals like [CelebA-HQ $(256 \\times 256)$](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_recs_celebahq256.png?v=5cc0265)—not tackled by any baseline.\n\nTo show superiority against latent diffusion and MLPs, as suggested, the table below compares $\\texttt{LDMI}$ using either an MLP or our $\\texttt{HD}$ on CelebA-HQ:\n\n| Method                          | HN Params               | PSNR (dB)  |\n| ------------------------------- | ----------------------- | ---------- |\n| $\\texttt{LDMI}$-MLP             | 17.53                   | 24.93      |\n| $\\texttt{LDMI}$-$\\texttt{HD}$   | **8.06M**               | **27.72**  |\n\n\n## On the competitiveness of our method\nWe believe a few clarifications are helpful for contextualizing the results in Table 1.\n\nFuncta’s high PSNR arises from **test-time optimization**: it fits a separate modulation vector per test image using ground truth, unlike our amortized inference approach. This undermines a fair comparison, but we include it for completeness and clarify this in the revised manuscript.\n\nWhile GASP reports lower FID on CelebA, it **cannot perform reconstructions** due to its GAN-based design, limiting its use in conditional tasks.\n\nConsidering these factors, LDMI provides a balanced solution—competitive in both sampling and reconstruction—while being more scalable and efficient.\n\n## Discussion on Diffusion Transformers\nWe appreciate the suggested references and have added them to the Related Work section. These approaches apply Transformers as denoising backbones in pixel-space diffusion. Our use is fundamentally different: we apply Transformers as hypernetworks to generate INR weights, enabling function-level generation beyond the limitations of discrete grids.", "claims_and_evidence": "About the claim of scalability of hyper-transformers. The current version of this paper lacks clear experimental evidence to show that there is scalability in hyper-transformers.", "methods_and_evaluation_criteria": "Yes. The methods and evaluation criteria make sense.", "theoretical_claims": "Yes. I have checked the theory of diffusion models and hyper-nets discussed in this paper.", "experimental_designs_or_analyses": "Yes, I checked the experimental design. A key problem with this paper is the lack of comparison to the advanced diffusion models [A, B, C] or a baseline of diffusion models with MLP-based hyper-networks. This comparison is essential to the claim of scalability of hyper-transformers.\n\n[A] Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis[J]. Advances in neural information processing systems, 2021, 34: 8780-8794.\n\n[B] Bao F, Nie S, Xue K, et al. All are worth words: A vit backbone for diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 22669-22679.\n\n[C] Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2023: 4195-4205.", "relation_to_broader_scientific_literature": "This paper is a new method of introducing the hyper-nets into diffusion models for image generation and upgrading the MLP-based hyper-nets into hyper-transformers.", "essential_references_not_discussed": "The paper lacks a discussion on the following works about diffusion transformers:\n\n[A] Bao F, Nie S, Xue K, et al. All are worth words: A vit backbone for diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 22669-22679.\n\n[B] Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2023: 4195-4205.", "other_strengths_and_weaknesses": "Strengths:\n\n- The motivation of using hyper transformers to address the scalability limitation of traditional hypernets is clear.\n- This paper is well-written and easy to follow.\n\nWeaknesses:\n- The claim of scalability has not been verified. In line 60, the authors claim that the proposed hyper-transformer decoder can solve the scalability of hyper-networks. However, the authors only verify that the hyper-transformer can work on ImageNet in Table 1. There is no evidence of whether the larger hyper-transformer achieves better performance than the MLP-based hyper-nets of a similar size.\n- Table 1 only compares the method, Spatial Functa. The authors have not compared their method to the advanced diffusion models [A, B, C] or a baseline of diffusion models with MLP-based hyper-networks.\n\n[A] Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis[J]. Advances in neural information processing systems, 2021, 34: 8780-8794.\n\n[B] Bao F, Nie S, Xue K, et al. All are worth words: A vit backbone for diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 22669-22679.\n\n[C] Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2023: 4195-4205.", "other_comments_or_suggestions": "Please refer to my questions in the “weakness” section.", "questions_for_authors": "Please refer to my questions in the “weakness” section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4kWcd5xnIv", "reviewer": "Reviewer_LB9r", "summary": "The authors propose a novel method for generating the parameters of implicit neural representations (INRs) representing real data. They use a latent diffusion framework, which first trains a VAE to learn a rich latent representation of data, then trains a diffusion generative model on the learned representations to generate new samples. The main innovation is the introduction of a hyper-transformer decoder, which uses an encoder-decoder transformer network to translate from encoded latent representations to INR parameters. Besides training the VAE and diffusion from scratch, the authors also introduce a hyper-transforming paradigm, in which a pre-trained encoder and diffusion model are frozen and only the hyper-decoder weights are updated. Experiments demonstrate that the proposed method performs comparably to baselines.\n\n#### Updates after rebuttal period\nThe authors have made sincere efforts to address the points I raised, and the new changes and additions to the manuscript strengthen the writing and clarify the mathematical details. In light of these revisions, I will raise my original review score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the clarity, motivation, and contributions of our work. Your comments helped improve the paper significantly.  Below, we address each of the concerns raised in your review.\n\n## On the notation used for weights\nWe appreciate your comments on the novelty of our weight generation strategy and agree that clearer notation helps with readability. In the revised version, we have clarified the notation as follows:\n\n- The bar in $\\bar{\\mathbf{W}}$ denotes globally shared, learnable parameters (e.g., $\\bar{\\mathbf{W}}^\\text{i}$ and $\\bar{\\mathbf{W}}^\\text{b}$).\n- The superscript $i$ in $\\bar{\\mathbf{W}}^\\text{i}$ refers to “*input*” to the Transformer Decoder. The superscript $\\text{o}$ in $\\mathbf{W}^\\text{o}$ refers to its “*output*”, which transforms these global input weights by attending to the latent tokens.\n- The subscript $l$ in $\\mathbf{W}_l$ identifies the target INR layer. We omit this when not needed for clarity.\n- The superscript $\\text{b}$ in $\\bar{\\mathbf{W}}^\\text{b}$ stands for the “*base*” weights, which serve as the template modulated by $\\mathbf{W}^\\text{o}$ via our grouping and reconstruction strategy.\n\nThese clarifications are now reflected in the manuscript and Figure 2.\n\n## On the correctness Equation 7 and the weight dimensions \nWe highly appreciate that you pointed this out, you are completely right. We have corrected the following two issues: \n- We corrected the typo in line 272 to have $\\bar{\\mathbf{W}}^\\text{b} \\in \\mathbb{R}^{d_{\\text{out}}\\times d_{\\text{in}}}$.\n- Now we use the Hadamard or element-wise product $\\odot$ for the weight reconstruction.\n\nIn Equation (7), each $w_{\\lfloor c / k\\rfloor}^{\\mathrm{o}}$ (we're skipping bold notation here due to rendering issues) is an output token of the Transformer, and also the $\\lfloor c / k\\rfloor$-th column of the grouped weight matrix $\\mathbf{W}^{\\text{o}} \\in \\mathbb{R}^{d_{\\text{out}}\\times G}$. The length of the Transformer decoder sequences are thus $L\\times G$, which is the total number of grouped columns. We can set the embedding dimension of the Transformer to $d_{out}$ or project the tokens using one feed-forward layer, depending on the case. \n\nAdditionally, we corrected the typo in line 238 to have $d_\\text{in}$. We hope after these corrections, the concern is fully addressed. \n\n## On the competitiveness of our method\nOur key strength is scalability: we outperform baselines while using fewer hypernetwork parameters. For instance:\n- GASP/VAMoH: 25.7M params → 50K INR weights\n- $\\texttt{LDMI}$: 8.06M params → 330K INR weights (5-layer network)\n\nWe also extended Table 1 and added experiments on resolution generalization at [generation](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_samples.png?v=485b880b) and reconstruction, including [CelebA-HQ $(256 \\times 256)$](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_recs_celebahq256.png?v=5cc02656), which are not tackled by prior work. We refer to our [response to Reviewer ER44](https://openreview.net/forum?id=yhgcRwJ9Dn&noteId=ldvx3foEnj), where we include the updated Table. To clarify the context behind other baselines:\n- Functa uses test-time optimization, tuning each test sample with ground-truth.\n- GASP cannot perform reconstructions due to its GAN-based design.\n\nIn sum, $\\texttt{LDMI}$ offers a strong balance of quality, scalability, and generalization.\n\n## On the choice of the weight reconstruction method\nThank you for this insightful question. Initially, we adopted a normalization-based reconstruction (inspired by Trans-INR), which worked well for MLP-based INRs. However, this approach led to instability when generating SIREN weights, particularly due to vanishing gradients—hindering generalization in tasks like super-resolution.\n\nTo address this, we introduced a novel scaling-based reconstruction:\n$$\n\\left(1+w_{\\lfloor c / k\\rfloor}^{\\text{o}}\\right) \\odot \\bar{w}_c^{\\text{b}}.\n$$\n\nThis avoids collapse when $\\boldsymbol{w}_{\\lfloor c / k\\rfloor}^{\\mathrm{o}} \\approx 0$ and provides stable training for high-frequency signals—leading to the strong results in our revision.\n\n## On coordinate input handling in the hyper-transforming setup\nThank you for raising this question. Our goal is to model data as samples from a stochastic process, using our $\\texttt{HD}$ decoder to map latents into functions. In the hyper-transforming setup, we use convolutional encoders (often pre-trained) that implicitly capture coordinate information from structured grids:\n- For image and ERA5 data: ResNet encoders\n- For 3D occupancy fields: 3D convolutional encoders\n\nThe decoder then produces INR weights, enabling evaluation at arbitrary coordinates and decoupling resolution from representation.", "claims_and_evidence": "The claims are backed up.", "methods_and_evaluation_criteria": "The methods appear sound.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "Experiments appear sound.", "relation_to_broader_scientific_literature": "The field of diffusion models, and LDMs in particular, has been growing as the de facto most popular method of generative modeling. While normally restricted to fixed grids of pixels, the introduction of implicit neural representations to this modeling paradigm represents an important shift toward a more flexible representation of the inputs and outputs.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "## Strengths\n\n- The paper is well-written. There is a clear flow of ideas from background to motivation which sets the stage for the explanation of the main innovations and the results. In addition, the authors take care to explain most of their design choices and the mathematical objects that comprise these designs to clarify their purpose (with a few mistakes and/or errors, as explained in the weaknesses section).\n\n- The hyper-transformer decoder is a well-motivated way to convert latent representations to INR parameters. Specifically, the use of learnable template weights and biases is a clever way to reduce computational and memory complexity.\n\n- The hyper-transforming training paradigm allows the re-use of powerful, existing models trained on large datasets. This can drastically reduce the computational requirements of the method compared to training from scratch.\n\n\n## Weaknesses\n\n- Since this is a novel idea, the authors should make the notation as easily understandable as possible - especially for the hyper decoder (HD). However, the repeated usage of, e.g., $\\mathbf{W}$ for weight matrices with only subscripts and superscripts to differentiate between the types of weights makes understanding the HD more difficult. This is not a large weakness, as I believe the authors present their method quite clearly otherwise, but may be something to consider for future revisions. \n\n- Similar to the point above, the dimensions of the weights are inconsistent. For example, in line 264 the authors state that the dimensions of $\\bar{W}^i$ are $d_{out} \\times G$ and Figure 2 shows each $\\bar{W}^i$ having sequence length $G$, implying that sequence length corresponds to the number of columns. However, in line 272, $\\bar{W}^b$ are stated to have dimensions $d_{in} \\times d_{out}$ but the sequence length given in Figure 2 is $d_{in}$ which corresponds to the number of rows instead of the columns. This makes it difficult to reconcile the interactions between the weights and hidden states. \n\n- The right hand side of Eq (7) is a (normalized) dot product of two vectors, which should result in a scalar value and not in a column vector as desired. If the authors mean to use the dot operator ($\\cdot$) as an element-wise product between the two vectors, they should clarify this to avoid confusion. \n\n- The results seem to show that the proposed method is not much better, if at all, than competing baselines. Table 1 indicates that other methods achieve either superior reconstruction accruracy or superior generative ability. The ImageNet samples in Figure 3b appear unnatural and unconverged.", "other_comments_or_suggestions": "- Line 238, right column: \"$d_{i}n$\" should be \"$d_{in}$\"", "questions_for_authors": "- How did you choose the specific normalization of the output weight matrix as shown in Eq (7)? I.e., why did you choose to normalize the output by the L2 norm of the product of the two input columns?\n\n- In the hyper-transforming training setup, how do you pass the coordinate inputs to the encoder? Pre-trained VAE models used in latent diffusion generally do not take this information as part of their inputs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fcgTVxpw7x", "reviewer": "Reviewer_WKA5", "summary": "This paper proposes a new framework for INR generation (LDMI) which combines latent diffusion models and a transformer based hyper network for learning the distributions over INR parameters. The hyper network transforms the latent variables through a transformer encoder and decoder and generates the INR parameters. \nBeyond the vanilla end-to-end training pipeline of a LDMI, it enables also using a pre-trained LDM and a hyper transformer for efficient transfer learning without full retraining.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for highlighting relevant connections to related work. Below, we address each of the raised concerns in detail.\n\n## On the relation to HyperDreamBooth\nWe appreciate your suggestion to consider HyperDreamBooth, which we have now cited and discussed in the revised manuscript. While the goals of our work differ, there are indeed interesting architectural parallels. Both approaches leverage transformer-based hypernetworks to generate weights, motivated by scalability and efficiency.\n\nHyperDreamBooth focuses on personalizing pre-trained text-to-image diffusion models by generating low-rank residuals (via LoRA) from a single image—enabling rapid adaptation to new subjects. In contrast, $\\texttt{LDMI}$ introduces a generative model over the space of implicit neural representations (INRs), which represent continuous functions across diverse modalities (e.g., images, 3D occupancy fields, and climate data).\n\nRather than modulating a pre-trained model, our Hyper-Transformer Decoder generates the full parameter set of an INR from latent samples, enabling resolution-agnostic, function-level generation. While both methods rely on Transformer-based hypernetworks, $\\texttt{LDMI}$ operates in a distinct regime of generative modeling over functions.\n\n## On the competitiveness of our method and strengthened evaluation\nThank you for raising this point. In response, we have significantly strengthened our empirical results to more clearly demonstrate the advantages of our approach.\n\nFirst, we added results on higher-complexity datasets such as CelebA-HQ $(256 \\times 256)$, which fall outside the scope of prior baselines. Our model demonstrates strong performance in both [generation](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_samples.png?v=485b880b) and [reconstruction](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_recs_celebahq256.png?v=5cc02656) at multiple resolutions—without retraining—showcasing the benefits of INR-based generation.\n\nSecond, we updated Table 1 with new results and included the number of hypernetwork parameters to highlight $\\texttt{LDMI}$’s key strength: scalability. For example:\n- GASP/VAMoH require 25.7M parameters to generate ~50K INR weights.\n- $\\texttt{LDMI}$ uses 8.06M parameters to generate ~330K INR weights for a deeper, 5-layer INR.\n\nDespite having significantly fewer parameters, $\\texttt{LDMI}$ delivers superior performance.  For details, please refer to our [response to Reviewer ER44](https://openreview.net/forum?id=yhgcRwJ9Dn&noteId=ldvx3foEnj), where the updated table is provided. \n\nTo further contextualize this comparison, we highlight some key aspects of the baselines:\n- Functa relies on **test-time optimization**, fitting each test INR with access to ground-truth data. While this explains its high PSNR, it departs from our amortized inference setting and limits fair comparison. We include it for completeness given the task similarity.\n- GASP **cannot perform reconstructions** due to its GAN-based design, limiting its use in conditional tasks.\n\nIn summary, $\\texttt{LDMI}$ offers a compelling trade-off between quality, scalability, and generalization, supported by stronger experiments in the revised version.\n\n## On the hyper-training setting and training efficiency\n\nThank you for this question. In our hyper-transforming setup, we use the publicly available pre-trained LDM from [Rombach et al., 2022], specifically the LDM-VQ-4 variant, trained on CelebA-HQ at $(256 \\times 256)$ resolution, and the LDM-VQ-8, trained on ImageNet. We freeze both the encoder and the diffusion-based prior, replacing only the decoder with our $\\texttt{HD}$ module. While Section 3.5 of the original submission specified the frozen components, we have now revised the text to make this distinction more explicit.\n\nThis setup provides a key advantage: significantly faster training. By freezing most of the architecture and training only the decoder, we reduce the number of learnable parameters and avoid the need for a full two-stage training pipeline—common in diffusion models—where the autoencoder and the prior must be learned sequentially. Our hyper-transforming approach thus offers a lightweight, modular alternative that efficiently adapts pre-trained models to new functional decoding regimes.", "claims_and_evidence": "The authors claim that their method is effective in generation tasks as well as hyper-transforming tasks. However, the results from Table 1 show that on CelebA LDMI doesn't achieve comparable PSNR score compared to Functa and the FID score is much higher than GASP. Also on ImageNet the PSNR is not comparable with Spatial Functa although the FID score does outperform its rival.", "methods_and_evaluation_criteria": "The datasets they evaluate on are reasonable, including multiple standard image datasets.\nThe metrics they use are standard. (PSNR, FID) \nHowever, they do not provide evidence that the model can achieve comparable performance with other baselines with either less parameters or less training samples etc.", "theoretical_claims": "I have not spotted error in the proofs of any theoretical claims.", "experimental_designs_or_analyses": "They show qualitative results of their method on generation, reconstruction, hyper-transforming and data completion tasks. However, the experimental results are not sufficient to show the effectiveness of their approach.", "relation_to_broader_scientific_literature": "This paper is connected to latent diffusion models, transformer-based hypernetworks also the implicit neural representation (INR) literature.", "essential_references_not_discussed": "Ruiz, Nataniel, et al. \"Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024. \n\nThe above mentioned paper uses a hyper network approach to personalize text-to-image models which should be relevant to this paper.", "other_strengths_and_weaknesses": "Strength:\nThe idea is well explained.\nWeakness:\nThe quantitive results of the paper are not strong enough to show the effectiveness of the proposed approach.", "other_comments_or_suggestions": "No", "questions_for_authors": "In the hyper-training setting, which frozen LDM do you use, I don't think it's specified in the paper. \nYou also mentioned that it is only been trained for a limited number of iterations and achieve the qualitative results shown in Figure 3b, do you have results regarding metrics vs training time to show the effectiveness of your approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ssAYTsxHdS", "reviewer": "Reviewer_XroW", "summary": "This work introduces a novel \"LDMI\" framework which empowers latent diffusion models to generate Implicit Neural Representations (INRs). The proposed  Hyper-Transformer Decoder enables the space of INR parameters to be learned in a flexible and probabilistic manner. Empirical tests are conducted on a range of image and shape reconstruction tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and encouraging remarks. Your comments helped us significantly improve the manuscript. Below, we address all the concerns raised.\n\n## On Our Claims Regarding Unconstrained Resolution\nWe agree that validating our model’s ability to generalize to unseen coordinates is essential. Following your suggestion, we added new experiments on super-resolution. During [reconstruction](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_recs_celebahq256.png?v=5cc02656), test images at the training resolution are encoded, decoded into INRs, and evaluated on denser grids. For [sampling](https://anonymous.4open.science/api/repo/LDMI_pre-7F42/file/experiments/figures/super_samples.png?v=485b880b), latent codes are drawn from the diffusion prior and decoded similarly. These experiments confirm that $\\texttt{LDMI}$ captures continuous signals and generalizes to higher resolutions—highlighting a key advantage of INR-based generation.\n\n\n## New quantitative results for ERA5 and ShapeNet\nTo better support our claims of modality generalization, we now report the PSNR measured on the ERA5 and Chairs datasets:\n\n| Method    | Chairs   | ERA5    |\n| --------  | -------- |-------- |\n| Functa     |    29.2     |    34.9     |\n| VAMoH     |    38.4      |    39.0     |\n| $\\texttt{LDMI}$     |     **38.8**      |     **44.6**     |\n\nWe omit GASP (not applicable to reconstruction due to their GAN-based framework). $\\texttt{LDMI}$ generalizes without adaptation and outperforms comparable models.\n\n## On CelebA and CelebA-HQ\nYou are correct—the original submission trained on CelebA, while baselines used CelebA-HQ. We now explicitly distinguish them and retrain $\\texttt{LDMI}$ on CelebA-HQ at $64 \\times 64$ for a fair comparison. To further demonstrate scalability, we train $\\texttt{LDMI}$ on CelebA-HQ at $256\\times 256$, achieving strong results not addressed by prior work.\n\n\n## On related work: NDPs and SimFormer\nThank you for pointing out these complementary works. We agree that approaches such as Neural Diffusion Processes, and SimFormer, are interesting and relevant within the broader context of function-space modeling with diffusion processes. In the following, we highlight the main difference. \n\n**NDPs** leverage a diffusion model to define distributions over function values at given coordinates. Their architecture explicitly enforces exchangeability and permutation invariance via a bi-dimensional attention mechanism, and their sampling mechanism mimics Gaussian processes and related meta-learning methods such as Neural Processes. **Importantly**, the function itself is not represented via a neural network whose parameters are generated or learned—rather, the model learns to denoise function values directly, conditioned on inputs.\n\n**Simformer** is designed for simulation-based inference (SBI), where the goal is to infer unknown parameters of stochastic simulators from observations. It treats both data and parameters as random variables and learns a diffusion model over the joint distribution $p(\\boldsymbol{x}, \\boldsymbol{\\theta})$, allowing for flexible sampling of any conditional (e.g., posterior, likelihood, marginals). Parameters may include function-valued (infinite-dimensional) components, but they are not represented as INRs—rather, they are input variables within the inference pipeline. Simformer excels at amortized Bayesian inference with unstructured or missing data and flexible conditioning.\n\nWe added these references and discussion to the Related Work section of our revised manuscript.\n\n## On the interpretation of pixel values as integrals\nWe agree that pixels approximate integrals over regions. Following [SIREN, NeRF, Functa], we adopt the standard Dirac delta approximation by treating pixel values as samples at center coordinates. We’ve made this assumption explicit in the revised paper.\n\n\n## On the core challenge we address\nThank you for raising this point. We have clarified the motivation in the revised manuscript.\n\nOur goal is to overcome the scalability bottlenecks of MLP-based hypernetworks in generative modeling of functions via INRs. While INRs excel at representing continuous signals, prior methods (e.g., Functa, GASP, VAMoH) require hypernetworks with tens of millions of parameters—even for small 3-layer INRs—making scaling impractical.\n\nTo address this, we propose the $\\texttt{HD}$ decoder, a Transformer-based hypernetwork that maps latent samples to INR weights. Its parameter count remains fixed as INR size grows, with only sequence length increasing. We further optimize efficiency with a grouping strategy, enabling generation of large INRs at low cost.\n\nIntegrating this design into latent diffusion models allows $\\texttt{LDMI}$ to model more complex signals and generalize across resolutions efficiently.", "claims_and_evidence": "Most claims within the paper are well supported by the experimental evidence. Perhaps the statement on \"our work establishes [....] generative modelling with unconstrained resolution\" is a bit lacking in evidence since output resolutions seem to match training resolutions in the experiments.", "methods_and_evaluation_criteria": "A good variety of datasets are explored, and for the case of ImageNet and CelebA, standard metrics are appropriately used such as PSNR and FID. It would certainly be preferable to also see some quantitative metrics for the ERA5 and ShapeNet chairs however - few robust scientific conclusions can be drawn by eye.", "theoretical_claims": "No, while some theoretical background is given, the key results here are empirical.", "experimental_designs_or_analyses": "In the CelebA experiment in FIgure 4, the baselines are trained on a different dataset to the LDMI. While the claim is made that this makes the task easier for the baselines, this lack of a controlled experiment prevents rigorous conclusions to be drawn from the differences in performance.", "relation_to_broader_scientific_literature": "This work sits in the popular field of diffusion models and hypernetworks, and provides a good review of relevant prior works in the area.", "essential_references_not_discussed": "To the best of my knowledge, no essential references are missing.\nSomewhat more tangentially, there are some complementary works on performing functional inference with diffusion models which may be of interest such as:\nNeural Diffusion Processes, Dutordoir et al, ICML 2023\nAll-in-one simulation-based inference, Gloecker et al, ICML 2024", "other_strengths_and_weaknesses": "It ought to be viable to perform inference at a higher resolution than what was used during training, as it's one of the key advantages of possessing a INR, but I don't see explicit examples, perhaps some examples could be added to the supplementary material?", "other_comments_or_suggestions": "I would recommend making it explicitly clear that pixel values correspond to integrals over a finite region, so are not ideally suited for this application, but they serve more as a useful testbed.", "questions_for_authors": "Relating to the above point, I feel this paper would be stronger if first it spends a bit of time clarifying the key problem/challenge they wish to tackle, before then outlining the solution.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes the use of Implicit Neural Representations within a Transformer-based hypernetwork, framework for improving the scalability of the latent variable models. The key idea is to replace decoders with transformer hypernetworks. 

The paper received four reviews with mixed opinions. Issues were raised regarding the validity of the key claim of scalability vis a vis the competing method (GASP), notational inconsistency and writing. 

While writing can be improved in the revision, I do believe that the following comment by the Reviewer ER44, on paper's key claim on scalability warrants a deeper examination - 

"This suggests that GASP leverages its larger parameter count to achieve superior performance, showing the scaling ability of GASP rather than LDMI. This experiment fails to demonstrate that LDMI can similarly enhance performance by scaling parameters or weights" 

I read through the paper and liked the idea  and encourage the Authors to take a relook at that comment and thoroughly back their key claims with more experiments.
Decision: Accept (poster)