ICML 2025

Hyper-Transforming Latent Diffusion Models
Ignacio Peis 1 2 Batuhan Koyuncu 3 4 Isabel Valera 3 Jes Frellsen 1 2
Abstract
We introduce a novel generative framework for
functions by integrating Implicit Neural Represen-
tations (INRs) and Transformer-based hypernet-
works into latent variable models. Unlike prior ap-
proaches that rely on MLP-based hypernetworks
with scalability limitations, our method employs
a Transformer-based decoder to generate INR pa-
rameters from latent variables, addressing both
representation capacity and computational effi-
ciency. Our framework extends latent diffusion
models (LDMs) to INR generation by replacing
standard decoders with a Transformer-based hy-
pernetwork, which can be trained either from
scratch or via hyper-transforming—a strategy that
fine-tunes only the decoder while freezing the pre-
trained latent space. This enables efficient adapta-
tion of existing generative models to INR-based
representations without requiring full retraining.
We validate our approach across multiple modali-
ties, demonstrating improved scalability, expres-
siveness, and generalization over existing INR-
based generative models. Our findings establish a
unified and flexible framework for learning struc-
tured function representations.
1. Introduction
Generative modelling has seen remarkable advances in re-
cent years, with diffusion models achieving state-of-the-art
performance across multiple domains, including images,
videos, and 3D synthesis (Ho et al., 2020; Dhariwal &
Nichol, 2021; Rombach et al., 2022). A key limitation of ex-
isting generative frameworks, however, is their reliance on
structured output representations, such as pixel grids, which
constrain resolution and generalisation across data modal-
1Department of Applied Mathematics and Computer Science,
Technical University of Denmark, Copenhagen, Denmark 2Pioneer
Centre for Artificial Intelligence, Copenhagen, Denmark 3Saarland
University, Saarbr¨ucken, Germany 4Zuse School ELIZA. C...