Controllable Data Generation with Hierarchical Neural Representations
Authors: Sheyang Tang
Xiaoyu Xu
Jiayan Qiu
Zhou Wang
Venue: ICML 2025 poster (2025)
Keywords: implicit neural representations
diffusion model
Primary area: deep_learning->other_representation_learning

Abstract: Implicit Neural Representations (INRs) represent data as continuous functions using the parameters of a neural network, where data information is encoded in the parameter space. Therefore, modeling the distribution of such parameters is crucial for building generalizable INRs. Existing approaches learn a joint distribution of these parameters via a latent vector to generate new data, but such a flat latent often fails to capture the inherent hierarchical structure of the parameter space, leading to entangled data semantics and limited control over the generation process. Here, we propose a **C**ontrollable **H**ierarchical **I**mplicit **N**eural **R**epresentation (**CHINR**) framework, which explicitly models conditional dependencies across layers in the parameter space. Our method consists of two stages: In Stage-1, we construct a Layers-of-Experts (LoE) network, where each layer modulates distinct semantics through a unique latent vector, enabling disentangled and expressive representations. In Stage-2, we introduce a Hierarchical Conditional Diffusion Model (HCDM) to capture conditional dependencies across layers, allowing for controllable and hierarchical data generation at various semantic granularities. Extensive experiments across different modalities demonstrate that CHINR improves generalizability and offers flexible hierarchical control over the generated content.

Controllable Data Generation with Hierarchical Neural Representations
Sheyang Tang' Xiaoyu Xu X
Jiayan Qiu 2 Zhou Wang'
Abstract
You al., 2024), offering compact and universal representa- tions across diverse data modalities such audio (Su et al., 2022), images (Sitzmann et al., 2020; Dupont et al., 2021), videos (Chen et al., 202 2023), and 3D volumes (Milden- hall et al., 2021; Zhao et al., 2022; Michalkiewicz. et al., 2019). By modeling data as functions fo : X + F, with X' and F being the input (e.g., pixel coordinates) and output (e.g., RGB values), INRs implicitly encode data hidden manifold within the parameter space of 0, capturing the underlying structure of the data. By modeling the distribu- tion of parameters p(0), generative INRs present promis- ing potentials for universal data generation (Dupont et al., 2022c;a; Bauer et al., 2023; You et al., 2024). Nevertheless, two fundamental questions have long been overlooked:
Implicit Neural Representations (INRs) represent data as continuous functions using the parame- ters of a neural network, where data information is encoded in the parameter space. Therefore. modeling the distribution of such parameters is crucial for building generalizable INRs. Exist- ing approaches learn joint distribution of these parameters via latent vector to generate new data, but such a flat latent often fails to capture the inherent hierarchical structure of the param- eter space, leading to entangled data semantics and limited control over the generation process. Here, propose Controllable Hierarchical Implicit Neural Representation (CHINR) frame- work, which explicitly models conditional de- pendencies across layers in the parameter space. Our method consists of two stages: In Stage- we construct Layers-of- offexxstts (LoE) network, where each layer modulates distinet semanties through unique latent vector, enabling disentan- gled and expressive representations. In Stage-2, we introduce Hierarchical Conditional Diffusion Model (HCDM) to capture conditional dependen- cies across layers, allowing for controllable and hierarchical data generation at various semantic granularities. Extensive experiments across differ- ent modalities demonstrate that CHINR improves generalizability and offers flexible hierarchical control over the generated content.
How are these parameters related to data semantics, and hence how to control the parameters to generate expected semantics?
Addressing this gap is critical for advancing the controlla- bility of INR-based generative frameworks. Notably, we observe that INRs (e.g., SIREN (Sizzmann et al., 2020)) naturally exhibit a hierarchical structure, where each layer progressively expands the representational capacity of the model (Yüce et al., 2022). This expansion is intrinsically linked to the frequency basis of INRs: earlier layers cap- ture coarse-grained features, while later layers progressively refine fine- aineggraneed details (Section 2.2 provides detailed analysis). As shown in Fig. 1, this progression aligns with the semantic abstraction hierarchy in data. For example, in facial images, hierarchical semantics are reflected in progres- sively detailed facial attributes, such as overall facial shape, expression, and shape of eyes. This connection between INR's 's hierarchical structure and data semantic abstraction offers a natural pathway to achieve hierarchical control.
1. Introduction
Implicit Neural Representations (INRs) are powerful tools to represent complex data as continuous functions with neu- ral networks (Tancik et al., 2020; Mildenhall et al., 2021;
However, existing generative INR approaches (Dupont et al., 2022a: Bauer et al., 2023; You al., 2024) overlook this hierarchy, instead modeling the joint distribution of flat- tened INR parameters, either directly as p(0) or through the distribution of condensed latents. It ignores the cor- respondence between the INRs' expanded representation capacity and the hierarchical semantics in the data. This misalignment brings two challenges: (1) Control over the generated content is limited. While generative models, such as latent diffusion, are employed with INRs to generate new
'Department of Electrical and Computer Engineering, Uni- versity of Waterloo, Waterloo, Canada "School of Comput- ing and Mathematical Sciences, University of Leicester, Le- icester, United Kindom. Correspondence to: Xiaoyu Xu <x423xu uwaterloo.ca>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Controllable Data Generation with Hierarchical Neural Representations
SpenL
Lien
3ipert.3
Lipent
Lipest
Figure 1. Hicrarchical generation universal data modality. Top: expanded representational capabilities of a Srvers-of-Experre INR model. Bottom: aligning these capabilities with hierarchical data patterns enables precise control over the generation process. Each column presents generated samples resulting from a divergence in routing at specific layer, with arrows indicating the shared routing in the preceding layers.
data, they cannot link the sampled noise to the expected semantics in the output. (2) The generalizability of INRs to unseen data is impaired. The joint distribution learning encodes the entangled semantics together into one flat latent, where the co-occurrence of certain semantics is inevitable. Therefore, the diversity of the generated data is limited.
results in minor differences in details.
The contributions of our paper summarized follows:
We are the first to achieve hierarchical control in gen- erative INRs by modeling hierarchical and conditional dependencies in INR parameters. This enables layer-wise, precise control over data semantics during generation.
To address this gap, we propose a Controllable Hierarchical Implicit Neural Representation (CHINR) framework that investigates the hierarchical structure of INRs, promoting layer-wise control in the generation process. Our method starts by training collection of INRs on dataset. Each layer of an INR is parameterized as a Mixture-of-Exxper (MoE) layer to increase expressivity, where set of expert weights and latents are learned. The experts at each layer are shared across the dataset, while the latents are dtta-specific, routing the data flow and modulating the contribution of experts. Layers of MoE are cascaded to form an INR, which we call a Layers-oofExpeere (LoE) network. Consequently, a LoE with L layers will have L latents adapted to the fitted data, effectively capturing and relating its complex patterns with layers latents. By modeling the conditional dependencies of these layer-wise latents with a hierarchi- cal conditional diffusion model (HCDM), we maintain the hierarchical structure of INRs. This unlocks controllable generation process, aligning the layers in INRs with the hi- erarchies of data semantics for the first time. As illustrated in Fig. 1, the data flow for a generation process resembles a tree-like structure: the routing in the next layer con- strained by the paths in previous layers, allowing full con- trol of where different routing strategy should be explored. Early deviations in routing lead to significant semantic dif- ferences in the generated content, while later deviation
We model the INR as LoE framework to enhance expres- sivity. This design aligns the inherent hierarchy of data semanties with INR's expanded representation capacity, enabling the generation of diverse data.
The proposed CHINR shows broad versatility across modalities, highlighting the broad applicability hierar- chical control and conditional dependency modeling to data semanties with inherent hierarchies.
We conduct extensive experiments across various modal- ities, demonstrating CHINR's superior performance in reconstruction and generation metrics.
2. Background
In this section, we introduce INR and generative INRs, while highlighting their connections to our proposed approach. We also analyze the inherent hierarchy in INR's parameters.
2.1. Implicit Neural Representation & Generative INRs
Implicit Neural Representations (INRs) parameterize data such as audio, images, video, and 3D voxels as mappings from coordinates to signals, enabling a unified framework for various data modalities (Genova al., 201 9a;b; Xie
2

---

Controllable Data Generation with Hierarchical Neural Representations
et al., 2022). Remarkable progress has been made to en- hance the representation quality, efficiency and compact- ness for audio (Zuiderveld et al., 2021; Luo et al., 2022; Su et al., 2022; Lanzendörfer & Wattenhofer), images (Sitz- mann et al., 2020; Fathony et al., 2020; Chen et al., 2021b; Xu et al., 2022a; Saragadam et al., 2023; Yu et al., 2024; Xu et al., 2024b; 2022b; Qiu et al., 2021), 3D contents (Milden-
Fourier Transform, the input frequency domain Q is com- posed of 01 frequency basis, 52 =2= [ Gz, 2a--1 Ac- cording to the Tancik et al. (2020) and Yüce et al. (2022), an MLP layer with periodic activation sin(-) only expands the input frequency basis in sparse and limited bandwidth. The equation can be reformulated as:
hall et al., 2021; Barron et al., 2021: Tiwari al.. 2022:
fo(x)= = Σ w'ER(())
Oww sin(w' xx),
Ortiz et al., 2022; Zhao et al., 2022; Ruan et al., 2024; Qiu
et al, 2020; 2019; Yang et al., 2020), and videos (Chen et al., a:21a; Li et al., 2022; Yan et al., 2024). Despite performing well on different modalities, INRs struggle to generalize to multiple and unseen data, as each instance is typically overfitted with a separate MLP. To address this, two key strategies have emerged: (1) learning content-specific in- put features (Yu et al., 2021; Hu et al., 2023; Lazova et al., 2023) and (2) modulating customizing network param- eters with latents or hypernetworks (Mehta et al., 2021;
Ow X W2
Jo. (W (i(.o)
(2)
c=0
4-1
H(2) S Σ c=0
B. E & Be S K}.
Σ
where Js. denotes a Bessel function, W 1i denotes the column e of W1. The Eq. (2) reveals the properties of each sin(-) activated INR layer in two aspects. First, the output spectrum of layer 2, i.e. Oug', is dependent the spectrum of layer 1, determined by W Second, the output frequency domain H(22) is sparse since 3. is an integer, so (S2) only covers sparse frequency space spanned by the basis {. These suggest that INR layers' spectrum and frequency ba- sis inherently exhibit a sparse and hierarchical structure, encoded by 0, which extends their representation ability, Latent-modulation approaches like mNIF (You et al., 2024), which model the parameter distribution p(0) with the surro- gate task of modeling the latent distribution p(h). overlook the hierarchy in h transferred from 0. Ignoring this hierar- chy leads to reduced expressivity and generalizability, and limited control over the generation process.
Wang et al., 2022; Dupont et al., 2022b; Kim et al., 2023; Xu et al., 2024a). Generative models (Goodfellow et al.,
2014; Ho et al., 2020) further extend INR's capability to generate new data. GRAF (Schwarz et al., 2020) and GI- RAFFE (Niemeyer & Geiger, 2021) generate shape and appearance codes from noise, which are combined with coordinates to construct scenes. Erkoç et al. (2023) use a diffusion model to generate INR weights. Dupont et al. (2022c); Du et al. (2021); Koyuncu et al. (2023) train hyper- networks to generate INR parameters. Dupont et al. (2022a); Bauer et al. (2023); Park et al. (2024) employ a two-stage framework to learn the distribution of latents that map to or modulate INRs, and generate new content by sampling in the latent space. mNIF (You et al., 2024) further en- hances the expressivity of INR via model averaging. These methods essentially model the distribution of INR parame- ters p()) by learning the latent distributions p(h), but fail to capture the layer-wise hierarchical structure of INR pa- rameters (Section 2.2), limiting their ability to accurately model distributions and control generation. Building on the latent modulation approach, we introduce a hierarchical con- ditional diffusion model, capturing dependencies between layer-wise latents for improved generalization and control.
3. Proposed Method
Our method uses a two-stage framework to align the hier- archy of data semantic and INR's ' expanded representation ability, as shown in Fig. 2. In Stage-1, we train individual INRs to fit a target dataset. In Stage-2, we use generative models to learn weight distributions to generate new data.
Directly modeling INR's weight distribution brings three challenges: (1) independently trained INRs make it hard to extract shared information for distribution learning, (2) the high dimensionality of raw weights makes distribution mod- eling highly challenging, and (3) it ignores the hierarchical structure of INRs. To address them, we configure INR as (LoE) network, where each layer con- tains set of shared expert weights and an instance-specific latent. As shown in Fig. 2 left, the inference process builds each INR layer by layer, combining experts with corre- sponding latents. This structure captures shared information through experts, simplifies distribution learning by focusing on latents, and explicitly models conditional dependencies within the hierarchical structure of INRs. In the following
2.2. Hierarchy Analysis of INR
In this section, we review the INR architecture and ana- lyze its inherent hierarchical representation ability. Using SIREN (Sitzmann et al., 2020) as an example, a two-layer SIREN is generally formulated as:
(1) fx(x) = W2 sin(Wn 7xx)), 0 WW, Wa],
where 7(x)= = (i x) Ω E Ra Xci denotes po- sitional embedding of coordinates x, W₂ E Rout X02,
W1 E X denote the parameters of each layer. The bias is omitted for simplification. From the perspective of
3

---

Controllable Data Generation with Hierarchical Neural Representations
Stage
Latent
0
1r02 latents
Collection
Latent
Forward
Stage
Noise
Expert
O0Oo
GooO
oooo
Backward
Latent
Latent to condition
O 0OOO GOOD
Input
Output
Expert
Figure 2. CHINR consists of two stages. In Stage-1, a Layer-of-Expee (LoE) model represents data with instance-specifie latents and shared experts. The latent at each layer (shaded differently) modulates the mixture of experts at that layer. Stage-2 introduces a Hierarchical Conditional Diffusion Model (HCDM) to learn layer-wise conditional distributions of latents. At inference, we sample latents according to the conditional chain to achieve hicrarchical control.
sections, we first define the LoE structure and learning task, followed by detailed explanations of the two stages.
learning, we adopt an interleaved training procedure in- spired by CAVIA (Zintgraf et al., 2019), where the experts and latents are updated alternately in separate training loops. In the inner loop, we fix 0 and adapt the latents h to data samples. Within each inner loop, h is first randomly ini- tialized around zero and then updated for few steps. In the outer loop, θ is optimized based on the updated h. This ensures each data data-specific latent can be effectively learned within a few iterations, encouraging faster convergence and adaptation to new data, which essential for distribution modeling and generalization in Stage-2. In the case of auto- decoding, we jointly optimize all parameters, maintaining latent bank for the dataset and updating the sampled batch of latents in each iteration. Unlike meta-learning, auto- decoding does not require second-order derivatives, making it more computationally efficient. Due to this efficiency, we apply auto-decoding specifically for NeRF training.
3.1. Problem Statement
Suppose an INR So has L layers. For layer I, we learn a collection of K cross-data shared expert weights Ø' = (0), 03 ................................ 0xx (fully connected layers) and a unique la- tent h' E RD for each data instance. At inference, the operation at layer / is y sin(wn (0' yy)),, where y represents each layer's output and wo is constant factorr 0' 8'=Σ, n= =x 8% aaf denotes instance- -seenccc parameters at layer 1, modulated by a gating vector a²', which is computed by a gating module (h') = a' = [ai, ob,,.., .......,,,. a'x]]] Compared with directly learning the gating vectors, (-) allows for more compact latent h' that benefits distribu- tion learning. By modulating the contribution of experts via latents, each layer gains the flexibility to adapt to individual data samples with a shared basis. As L layers are cascaded to form the final INR, its expressive capacity is significantly enhanced through the integrated contributions across layers.
In both approaches, each data-specific latent h consists of L components separately modulating L layers in LoE INR. This setup facilitates conditional distribution modeling in Stage-2, as opposed to joint distribution learning like mNIF and Functa, enabling hierarchical and controllable generation- a crucial capacity lacking in prior works.
In 1tage-1, we optimize the shared network parameters {0), 0², .... (6,6} and fully characterize each cettance-specifee INR by layer-wise stacked latents h h h². .n...... h+] E # This layer-wise structure ables hierarchical modeling of INR parameters, which aligns with data semantic hierarchy, allowing for layer-wise depen- dency modeling in Stage-2, and controllable data generation.
3.3. Stage-2: Conditional Distribution Learning
Given a collection of latents H = {h,,-.. bw lh E R obtained from data instances, where L denotes number of layers, and H the dimension of each layer's latent, Stage-2 aims at learning latent distribution p(h). Instead of blindly modeling joint distribution, we reformulate p(h) as:
3.2. Stage-1: Learning a Dataset of LoE INRs
Similar to Functa (Dupont et al., 2022a) and mNIF (You et al., 2024), we use meta-learning and auto-decoding to train both the data-specific latents h and the shared pa- rameters 0 for the LoE INR during Stage-1. For meta-
p(h) p(h h2. ... .h² = -p(h') I
p(h (ehhnn>) 2')), (3)
1
4

---

Controllable Data Generation with Hierarchical Neural Representations
with concatenated tensors, we its unused portion to zero. (2) Given compressed v we obtain binary condition c²: c (v)(())), where Q(-) denotes binarization operation, ar denotes Sigmoid. We set c' as zero tensor since h has no condition. Now we get the condition c², ,",""",""
Hinary
conditions
Quantization
Sigmoid
Compressor
Latent
Hierarchical generation. With condition c = [c', cd), noise sample hy [h...... .h4]. time step t. we are ready to generate a complete latent ho- To generate a component hy. the backward process is formulated as: l(h6,r(c') = p(hl)-) p
Condition formation. Layers of latent h are first concate- nated and then compressed. The compressed tensors are binarized generate low- wiw-dimesional binary conditions.
Figure 3.
c²). ). Note that a com- plete backward process, generating a sample from p(hh) c'), is exactly the implementation of p ((h' ( in Eq. (3), where ef corresponds to hat We adopt UNet (Ronneberger et al., 2015) Pe as in Song et al. (2021); Ho et al. (2020) to obtain p(h/_1lh', d¹) - N(hh : Eo (b L, c²), (()), where
where p(h=1) ppphp ..... h 1) denotes the joint prob-
ability of the first - 1 layers. We design a hierarchical conditional diffusion model (HCDM) that learns the con- ditional dependency p(h') h in Eq. (3). Figure 2 right illustrates HCDM with forward and a backward process.
c co (()),,,c')= Va.(1-a; 19b4 vor 1(1 al1-ai)uo
3.3. 1. FORWARD PROCESS
1-a:
We initialize h at step 0 as ho = [bb. ho with a condi- tional chain of length L. The forward process for each layer h' is formulated as:
at = Be, at II ai, E(t) (1-+))(----11))
I.
1-0.
(5)
Each backward process starts from a noise h2 and gener- ates a latent component h with condition e'. By iteratively sampling from p(h') and p p(h ph'[hh with / = 2,-.. L, a complete latent ha [h h11 is generated. The fi- Ex [Nlle - eg co(h,, cf )l%),
T ((h)rlbi) == II qhi/[
(4)
q(h/lh/_ := NN /I 1..
nal objective is: Lream where H. denotes all latents, e denotes Gaussian sample from ./ I). and À is a constant coefficient. For inference, we first generate h' from Gaussian noise, then perform a chain of conditional sampling from p(h([h<t)
where (((/(h)_1) denotes the posterior distribution of h2 given h1-11 T denotes the number of diffusion steps. ",,... ,, ,,r denote the variance schedule of added Gaus- sian noise A((-). By the forward process, the noise sample hy Tby. h1] is generated from hg [hg.... hh/l-
<!) until we get the complete latent h used by the LoE generate new data.
3.3.2. BACKWARD PROCESS
4. Experiments
The backward process models the prior distribution as in Eq. (3). To model the hierarchical structure, this process should express the conditional dependency p(h') (1)) for all components. Therefore, we take h as condition, and generate h' for 1 1.... I iteratively. Next, we explain the details of how to process h h<l condition.
In this section, we first introduce the datasets and evalua- tion criteria. We then show CHINR's 's controllability of the generation process across various modalities. A quantitative evaluation is provided to highlight its hierarchical control over semantic attributes. Next, we evaluate CHINR's recon- struction and generation quality. Furthermore, we analyze the latent space, showing how data semantics are embedded within the INR's weight space, granting it compositional property across layers. Finally, we present ablation stud- ies highlighting the importance of conditional dependency modeling and the functionality of binary conditions.
Condition formation. To generate h = [h',...,iil ,h'], we prepare for each h' a condition vector c' encapsulating h<t Since h lives in a low-dimensional manifold (e.g.. 64), c' should contain less information to prevent HCDM from memorizing all one-to-one mappings (h + h'), where p(h) inevitably degenerates into p(ht). Therefore, embed the conditions h into lower-dimensional binary vector. Fig. 3 shows this process with two steps. (1) All h h are concatenated and compressed with a com- pressor W to get a compressed tensor v E RC: v' = W concat(h's (h', ........ hh)., W E RC h E R with C being the compressed dimension. To match W's dimension
In our experiments, both Stage-1 and Stage-2 are trained and evaluated on the CelebA-HQ 642 (Karras et al., 2018), ShapeNet 643 (Chang et al., 2015), SRN-Cars (Sitzmann et al., 2019), and AMASS (Mahmood et al., 2019) datasets. All experiments are implemented in Pytorch and run on a single Nvidia RTX3090 GPU. For evaluation metrics, we use peak signal-to-noll noise ratio (PSNR), structural sim-
5

---

Controllable Data Generation with Hierarchical Neural Representations
Sexple
Figure 4. Controllable hierarchical generation by progressively fixing the layer-wise latent three modalities.
ilarity (SSIM) (Wang et al., 2004), and accuracy to as- sess Stage-1 reconstruction, and Fréchet inception distance (FID) (Heusel et al., 2017), and coverage evaluate Stage-2 generation performance. Further details about implementa- tion can be found in the Appendix.
Table I. Quantitative evaluation of hierarchical control on CelebA- HQ (tep), ShapeNet (middle), and SRN-Car (bottom).
Attributes
Random
Lg
Lg
Oval face
30.3%
73.2%
85.7%
90.2%
Blonde hair
39.6%
40.5%
85.8%
91.4%
96.8%
4.1. Hierarchical Controllable Generation
Smiling
30.8%
31.2%
33.6%
83.0%
92.5%
Red lips
10.5%
11.3%
13.7%
12.5%
95.3%
In this section, we show hierarchical controllable genera- tion enabled by conditional sampling of HCDM, crucial capability missing in existing works. To sample a latent [h'.... ,. we begin by sampling h with HCDM taking a Gaussian noise as input. Next, we sample h², con- ditioned on the binary vector generated by h' This process continues layer by layer until hd is generated, forming complete latent h. This layer-wisc latent then modulates the LoE to produce images, voxels, or NeRF renderings.
Chair
12.8%
93.6%
98 9%
100.0%
100.0%
w/ Arms (Chair)
26.3%
29.3%
78.8% %
97.0%
100.0%
Cushioned (Chair)
19.2%
21.2%
32.3%
87.9 %
100.0%
SUV 19.4%
76.9 %
94.8%
97.8%
100.0%
Boxy
32.8%
46.3%
88,8 %
96.3%
97.0%
w/ Spoiler
14.9%
19.4%
24.6%
68.7 %
89.6%
ther quantitatively evaluate controllability in Table 1. We select representative attributes and start by randomly gen- erating (w/o control) samples, then compute the ratio of samples where each attribute is present (column Random). For each attribute, we take the positive samples and progres- sively fix the latents of initial layers while sampling the rest (Columns L1-1). At each step, we compute the positive ratio of the newly generated samples to verify whether an attribute is preserved. This experiment assesses the model's s ability to maintain control over fixed attributes while varying others. The results show the effectiveness of coarse-to-fine layer-specific control enabled by the latent hierarchy. Bold numbers denote significant increase preservation rate compared the previous layer, showing that the correspond- ing attributes effectively controlled that layer.
Qualitative results. Fig. 4 shows hierarchical control on different modalities. For each, we show three samples generated through the full chain of conditional sampling (h'...h h²) column 1. In column we fix their h' from the first column and sample the remaining 2h2...) .)). In column 3, we fix both ht and h² from the second column and conditionally sample the rest. This progressive fixation allows us to control the finer details of the generated output.
In Fig. 4, each run starts from a different h' resulting in highly different semanties (column in each modality). For face images, when h' is fixed, generations exhibit similar overall contours but different facial features and hairstyles (columın 2). Fixing h' and h causes variations in facial details, like eye shape and hair color (column 3). Fixing first three layers (column 4) results in changes limited to skin tone. Lastly, h affects global properties like fore- ground/backkround color, shown by heat maps. For voxels, hierarchy control is manifested in category (e.g. chair and plane), object parts (e.g. armrest and cushion), and finer details (c.g. textures). For NeRF, hierarchy occurs at car type (e.g. SUV), shape (e.g. boxy), parts (e.g. spoiler), etc. Additional examples are provided in the Appendix.
4.2. Reconstruction Generation Metrics
Table 2 presents reconstruction and generation results on different modalities. Our model outperforms existing meth- ods most datasets. It achieves the highest reconstruction PSNR, thanks to the expanded representation capacity of LoE and layer-wise latent learning. It also shows superior generation performance, highlighting the effectiveness of hierarchical conditional modeling in capturing diverse data semantics. On CelebA-HQ. FID score slightly behind
Quantitative evaluation of hierarchisal control. We fur-
6

---

Controllable Data Generation with Hierarchical Neural Representations
Table 2. Quantitative results on different datascts.
Reconstruction PSNR
Generation FID
CelebA-HO
SSIM
Functa (Dupont et al., 2022a)
26.2:10.3
0.795:200015
41.0=0.2
GEM (Du et al., 2021)
26.5+0.4
0.814+000018
30.8
GASP (Dupont et al., 2022c)
31.6++0.8
0,90240.021
13.6=0.3
mNIF (You al., 2024)
34.5:0.2
0.957:10.005
13.2=0.1
CHINR
34,9+1.3
0.964=0.006
13.4=0.1
ShapeNet
PSNRT
Accuracy"
Coveraget
Functa (Dupont et al., 2022a)
22.1=:0.3
0,983-0.005
0.437:10.005
Figure 6. Latent composition. top: 9 randomly generated images. row 2-4: replace h² to h of the first sample (green boxes) with top images. Red hoxes highlight representative examples.
GEM (Du et al., 2021)
21.4+0.4
0,977=0.007
0,408=00.003
GASP (Dupont et al., 2022c)
16.7++0.8
0.928+00.011
0.343+00.009
mNIF (You et al., 20224
21.4+00.3
0.975+000008
0.435 000000
CHINR
22.3:10.2
0.988:000005
0.441:0.002
SRN-Cars
PSNRT
SSIM
FID1
Functa (Dupont et al., 2022a)
24.3:10.2
0.738:10.009
80.1=0022
the composed semantics are compatible across layers.
mNIF (You al., 2024)
26.0:10.33
0.763100013
79.3=0.33
CHINR
26.3:10.2
0.780::0.011
77.8=002
1.3.1. LATENT COMPOSITION
We explore CHINR's controllability through latent com- position. We find that the learned latents are aayer-wise compositional. Given two latents h = [h], .... h2] h2 = [h hyj. exchanging a specific part, e.g. h and h2 re- sults in the corresponding semantic changes in the generated content. As shown in Fig. 6, we randomly sample latents from HCDM, trained on CelehA-HQ, and render the images in the first row. These faces exhibit diverse characteristics including expressions, hairstyles, facial orientations, skin tones, etc. In the second row, we replace h2 of the first sample with that of the other 8 samples, while keeping the rest latents unchanged. The facial orientation and hairstyles change accordingly, while facial features remain the same. This indicates that the second layer encodes these specific semanties disentangled from others, laying the foundation for controllable hierarchical generation. In the third row, when h2 is replaced, we observe that only facial features are swapped, while other characteristics such as orientation remain unchanged. In the last row, only skin tone changes. More examples can be found in the Appendix.
mNIF. However, we observe that mNIF tends to "memo- rize" the training set. To verify this, we generate 1, 000 samples and compute the average of their minimum L2 dis- tances to training images. lower value indicates higher degree of memorization. mNIF obtains a value of 6.243 whereas CHINR obtains 15.971, indicating that our model generalizes better by generating new images that differ more noticeably from training data. Examples of retrieval results are provided in the Appendix.
INI
DINE
Q7
It's noted that latent composition can disrupt the conditional chain, leading to incompatible latents and thus image arti- facts. Nevertheless, this experiment illustrates how and why our method works. It shows that image semantics can be embedded and disentangled in parameter space, offering a new perspective on image generation.
Figure 5. Uncurated generations for CelebA -QQ (lefi), ShapeNet (middle), and SRN-Cars (right) datasets.
Fig. 5 displays uncurated samples generated by our model compared to mNIF. With HCDM, our model generates high- quality samples with rich details. More results including the AMASS dataset are provided in the Appendix.
4.3.2. LAYER-WISE HIERARCHY ANALYSIS
We visualize hierarchical dependencies of latents see how data semantics are encoded in INR's weight space. Fig. 7 shows conditional distributions of latents from adjacent lay- ers, trained on CelebA-HQ. Specifically, we apply PCA to Stage- latents and plot their distributions in grayscale (e.g.. Fig. 7a shows distributions for layers and 2). We then run the generation process five times to obtain five latents at
4.3. Analysis
We uncover the success of hierarchical controllable genera- tion. We first show that each layer modulates disentangled semantics, giving our LoE model a compositional property. Moreover, laser-wise conditional dependency modeling suc- cessfully captures the hierarchy of data semantics, ensuring
7

---

Controllable Data Generation with Hierarchical Neural Representations
we evaluate the importance of conditional dependency by progressive ablation where excluded layers trained with unconditional diffusion models. Results show that strength- ening conditional modeling improves generation quality, emphasizing its necessity for high-quality generations.
(8)
b
Figure 7. Visualization layer-wise hierarchical dependencies. Gray regions show the distribution of latents from Stage- I, while colored regions represent the sampled latents from Stage-2.
Table 4. Ablation on different binary condition lengths (8, 12, 15, and 20) when training Stage-2 on CelebA-HQ.
Model
std;
stdd2
std22
std34
std44
each layer, as depicted in color Figs. 7a and 7b left. Based on HCDM, we plot the resulting conditional distributions of latents at subsequent layers, shown by colored regions on the right. Additionally, we show generated images cor- responding to different samples. We can see clear patterns of a hierarchical structure where semantics vary at different granularities. For example, layer determines the overall contours of faces. When layer 2 is determined, variations in layer 3 affect facial expressions without changing orienta- tions. Furthermore, the latent sampling space at each layer is constrained by its preceding layer, ensuring compatibil- ity between layers when representing data. Therefore, the generated data semantics are hierarchically controllable.
I
0.7766
0.9153
0.9363
0.8772
0.8631
HCDM
0.7766
0.9092
0.9292
0.8663
0.8550
IICMMs
0.7766
0.5344
0.5478
0.3257
0.2570
MCDM20
0.7766
0.1032
0.1121
0.0853
0.0766
Ablation on binary condition. We show that the length of binary conditions impacts the effectiveness of learning con- ditional dependencies, as in Table 4. We set binary lengths to 8, 12, 15, 20 and train HCDM on CelebA-HQ. Initially, we sample 5, 000 latents and compute the standard devia- tion (std,) of h'. Since h' is sampled w/o condition, it shows high values irrelevant to binary lengths. Then, we select 10 random samples from h as conditions and get 5, 000 samples for h2 The standard deviation (st de- creases as the binary condition length increases since longer conditions contain more information from preceding layers. Once it reaches certain threshold, the standard deviation approaches zero, turning the conditional chain into direct one-to-one mapping, thus diminishing controllability. How- ever, the length is too small, e.g. 0, all parts will be independent hence losing conditional dependency. There- fore, we empirically set the length as 12. We repeat this procedure for all layers and observe similar results.
4.4. A Allaion Studies
wlo condition
condition
Figure& 8. Ablations on conditional modeling.
5. Conclusion
Table 3. Ablation number of layers in conditional chain.
We proposed the Controllable Hierarchical Implicit Neural Representation (CHINR) framework, addressing the limita- tions of existing generative INRs which ignore the hierarchy in parameters and data semantics and lack control. By struc- turing INR as a Layers-of-Expers network and leveraging hierarchical conditional diffusion model, our approach captures conditional dependencies across layers, enabling hierarchical controllable data generation.
Layers chain
3,2,3,455
1,2,3,4
1,2,3
1,2
None
Independent layers
None
3,4,5
1,2,3,4.5
FID
13.4
13.6
15.5
52.8
112.7
Ablation on condition modeling, To show the importance of conditional dependency modeling, we train an uncondi- tional diffusion model that directly maps noise to layer-wise latents in Stage-2. We then sample p(h') independently for 1= 1..... L to generate full latents. The resulting images for CelebA-HQ are shown in Fig. top, while images gen- erated with conditional modeling are shown in bottom for comparison. Although human faces are recognizable in top, the noticeable artifacts highlight that independently sam- pled layer-wise latents cannot ensure consistent semantic composition across layers. In contrast, conditional mod- eling successfully achieves this compatibility. In Table 3,
One limitation is scalability to larger datasets, as the shared network may struggle to capture complex and diverse data patterns. A possible solution is to incorporate local infor- mation using patch-wise modulation (Mehta et al., 2021; Bauer et al., 2023) INRs with localized nonlinearity (e.g.. WIRE (Saragadam et al., 2023)). Future directions include exploring sparse gatings, as in Mixture of Experts (Wang et al., 2022), to promote expert diversity and specializa- tion. Additionally, learning layer- wise semantic hierarchy in
8

---

Controllable Data Generation with Hierarchical Neural Representations
Stage-1 can be guided through predefined attributes or deep clustering. This would allow the model to develop more interpretable and distinct semantics across layers, improving control over fine-gre ane-grained details and desired characteristics.
Dupont, E., Kim, H., Eslami, S. A., Rezende, D. J., and Rosenbaum, D. From data to functa: Your data point is a function and you can treat it like one. In Interna- tional Conference on Machine Learning, pp. PMLR, 2022a.
Impact Statement
Dupont, E., Loya, H., Alizadeh, M., Golinski, A., Teh, Y. W., and Doucet, A. Coin++: Neural compression across modalities. Transactions on Machine Learning Research, 2022b.
This paper advances generative INRs by introducing a novel approach for controllable data generation. Our experiments include datasets that contain human-related data (e.g., facial images), but we do not foresee any immediate societal con- cerns requiring specific attention. We encourage responsible use of our method in applications involving human data.
Dupont, E., Teh, Y. W., and Doucet, A. Generative models distributions of functions. In International Conference on Artificial Intelligence and Statistics, pp. 2989-3015. PMLR, 2022c.
References
Erkoç, Z., Ma, F., Shan, Q.. NieBner, M., and Dai, A. Hyper- diffusion: Generating implicit neural fields with weight- space diffusion. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 14300-14310) 2023.
Barron, J. T., Mildenhall, B., Tancik, M., Hedman, P., Martin- Bruaala, R., and Srinivasan, P. P. Mip-nerf: A multiscale representation for anti- neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5855-58644 2021.
Fathony, R., Sahu, A. K., Willmott, D., and Kolter, J. Mul- tiplicative filter networks. In International Conference on Learning Representations, 2020.
Bauer, M., Dupont, E., Brock, A., Rosenbaum, D., Schwarz, R., and Kim, H. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.031 00000003333...0 2023.
Genova, K., Cole, F., Sud, A., Sarna, A., and Funkhouser, T. Deep structured implicit functions. arXiv preprint arXiv:1912.066 1912.0612. 2:2, 2019a.
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q. Li, Z.,, Savarese, S., Savva, M., Song, S., Su, H., et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rXiv:1512.03012, 2015.
Genova, K., Cole, F., Vlasic, D., Sarna, A., Freeman, W. T., and Funkhouser, T. Learning shape templates with struc- tured implicit functions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7154- 7164, 2019b.
Chen, H., He, B., Wang, H., Ren, Y., Lim, S. N., and Shri- vastava, A. Nerv: Neural representations for videos. Advances in Neural Information Processing Systems, 34: 21557-21568, 2021a.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Chen, H., Gwilliam, M., Lim, S.-N., and Shrivastava, A. Hnerv: A hybrid neural representation for videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10270-10279, 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Chen, Y., Liu, S., and Wang. X. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628-8638, 2021b.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Du, Y., Collins, K., Tenenbaum, J., and Sitzmann, V. Learning signal-agnostic manifolds of neural fields. Ad- vances in Neural Informatonn Processing Systems, 34: 8320-8331, 2021.
Hu, S., Hong, F., Pan, L., Mei, H., Yang, L., and Liu, Z. Sherf: Generalizable human nerf from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9352-9364, October 2023.
Dupont, E., Goliński, A., Alizadeh, M., Teh, Y. W., and Doucet, A. Coin: Compression with implicit neural rep- resentations. arXiv preprint 1rXiv:2/03.000 00.003133..33 03133, 2021.
Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progres- sive growing of gans for improved quality, stability, and
9

---

Controllable Data Generation with Hierarchical Neural Representations
variation. In International Conference Learning Rep- resentations, 2018
neural radiance fields for view synthesis. Communica- tions of the ACM, 65(1):99-106. 2021. 20211
Kim, C., Lee, D., Kim, S., Cho, M., and Han, W.-S. General- izable implicit neural representations via instance pattern composers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp- 11808-11817, June 2023.
Niemeyer, M. and Geiger, A. Giraffe: Representing scenes compositional generative neural feature fields. In Pro- ceedings of the IEFF/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 11453-11464, 2021.
Ortiz, J., Clegg, A., Dong, J., Sucar, E., Novotny, D., Zoll- hoefer, M., and Mukadam, M. isdf: Real-time neural signed distance fields for robot perception. arXiv preprint arXiv:2204 022966, 2022.
Koyuneu, B., Martin, P. S., Peis, I., Olmos, M., and Valera, Variational mixture of hypergenerators for learning distributions over functions. In International Conference on Machine Learning, pp. 77660-17683, PMLR, 2023.
Park, D., Kim, S., Lee, S., and Kim, J. Ddmi: Domain- agnostic latent diffusion models for synthesizing high- quality implicit neural representations. In The Twelfth International Conference on Learning Representations, 2024.
Lanzendörfer, L. A. and Wattenhofer, R. Siamese siren: Audio compression with implicit neural representations. In ICML 2023 Workshop Neural Compression: From Information Theory to Applications.
Lazova, V., Guzov, V., Olszewski, K., Tulyakov, S., and Pons-Moll, G. Control-nerf: Editable feature volumes for scene rendering and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4340-4350 January 2023.
Qiu, J., Wang, X., Maybank, S. J., and Tao, D. World from blur. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8493 8504, 2019.
Li, Z., Wang, M., Pi, H., Xu, K., Mei, J., and Liu, Y. E-nerv: Expedite neural video representation with disentangled spatial-temporal context. In European Conference Computer Vision, pp. 267-284. Springer, 2022.
Qiu, J., Yang, Y., Wang. X., and Tao, D. Hallucinating visual instances in total absentia. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 264-262222 282. Springer. 2020.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Qiu, J., Yang, Y.. Wang. X., and Tao, D. Scene essence. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8322 8333, 2021.
Luo, A., Du, Y., Tarr, M., Tenenbaum, J., Torralba, A., and Gan, C. Learning neural acoustic fields. Advances in Neural Information Processing Systems, 35:3165-3177, 2022.
Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., and Black, M. J. Amass: Archive of motion capture as surface shapes. Proceedings of the IEEE/CVF inter- national conference on computer vision, pp- 2019.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assise intervention 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pp. 234 241. Springer, 2015.
Mehta, I., Gharbi, M., Barnes, C., Shechtman, E., Ra- mamoorthi, R., and Chandraker, M. Modulated periodic activations for generalizable local functional represen- tations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14214-14223, 2021.
Ruan, H., Shao, Y., Yang. Q.. Zhao, L., and Niyato, D. Point cloud compression with implicit neural representations: A unified framework. arXiv preprint 24551 2024.
Michalkiewicz, M., Pontes, J. K., Jack, D., Baktashmot- lagh, M., and Eriksson, A. Implicit surface representa- tions as layers in neural networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision,
Saragadam, V., LeJeune, D., Tan, J., Balakrishnan, G., Veer- araghavan, A., and Baraniuk, R. G. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18507-18516, 2023.
pp. 4743 4752, 2019.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes
10

---

Controllable Data Generation with Hierarchical Neural Representations
Schwarz, K., Liao, Y., Niemeyer, M., and Geiger, A. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33: 2020.
Xu, S., Wang, Y., Fan, M., Yoon, B.-J., and Qian, X. Uncertainty-awant continuous implicit neural representa- tions for remote sensing object counting. In International Conference on Artificial Intelligence and Statistics, pp. 4105-4113, PMLR, 2024a.
Sitzmann, V., Zollhöfer, M., and Wetzstein, G. Scene repre- sentation networks: Continuous 3d-structure-aware neu- ral scene representations. Advances in Neural Information Processing Systems, 32, 2019.
Xu, X., Qiu, J., Wang, X., and Wang, Z. Relationship spa- tialization for depth estimation. In European Conference on Computer Vision, pp. 615-637. Springer, 2022b.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit neural representations with peri- odic activation functions. Advances in neural information processing systems, 33:7462-7477, 2020.
Xu, X., Qiu, J., Yu, B., and Wang, Z. Visual relationship transformation. In European Conference on Computer Vision, pp. 251-272. Springer, 2024b.
Yan, H., Ke, Z.,, Zhou, X., Qiu, T., Shi, X., and Jiang, D. Ds-nerv: Implicit neural video representation with decomposed static and dynamic codes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23019-23029, 2024.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
Su, K., Chen, M., and Shlizerman, E. Inras: Implicit neural representation for audio scenes. Advances in Neural Information Processing Systems, 35:8144 1188884-158,, 2022.
Yang, Y., Qiu, J., Song, M., Tao, D., and Wang, X. Learning propagation rules for attribution map generation. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pp. 672-688. Springer, 2020.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Ken S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng, R. Fourier features let networks learn high fre- queney functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
You, T., Kim, M., Kim, J., and Han, B. Generative neural fields by mixtures of neural implicit functions. Advances in Neural Information P Processing Systems, 36, 2024.
Yu, A., Ye, V., Tancik, M., and Kanazawa, A. pixelnerf: Neural radiance fields from one or few images. In Pro- ceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 4578-4587 2021.
Tiwari, G., Antić, D., Lenssen, J. E.,, Sarafianos, N., Tung, T., and Pons-Moll, G. Pose-ndf: Modeling human pose manifolds with neural distance fields. European Con- ference on Computer Vision, pp. 572-589. Springer, 2022.
Yu, H., Li, R., Xie. S., and Qiu, J. Shadow bhadht-enlightened image outpainting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7850-7860, 2024.
Wang, P., Fan, Z., Chen, T., and Wang. Z. Neural im- plicit dictionary learning via mixture-of ofeeet training. In International Conference on Machine Learning, pp. PMLR, 2022.
Yüce,, G., Ortiz-liménez, G., Besbinar, B., and Frossard, P. A structured dictionary perspective on implicit neural representations. Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 19228 19238, 2022.
Wang. Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F,, Tompkin, J., Sitzmann, V., and Sridhar, S. Neural fields in visual computing and beyond. Com- puter Graphics Forum, volume 41, pp. 641-676. Wiley Online Library, 2022.
Zhao, W., Liu, X., Zhong, Z., Jiang, J., Gao, W., Li, G., and Ji, X. Self-supervised arbitrary -scale point clouds upsam- pling via implicit neural representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1999-2007, 2022.
Xu, D., Wang, P., Jiang, Y., Fan, Z., and Wang, Z. Signal processing for implicit neural representations. Advances in Neural Information Processing Systems, 35:13404- 13418, 2022a.
Zintgraf, L., Shiarli, K., Kurin, V., Hofmann, K., and White- son, S. Fast context adaptation via meta-learning In International Conference on Machine Learning, pp. 7693- 7702. PMLR, 2019.

---

Controllable Data Generation with Hierarchical Neural Representations
Zuiderveld, J., Federici, M., and Bekkers, E. J. Towards lightweight controllable audio synthesis with conditional implicit neural representations. In NeurlPS 2021 Work- shop on Deep Generative Models and Downstream Ap- plications, 2021.
12

---

Controllable Data Generation with Hierarchical Neural Representations
A. Details on Experimental Setup
Implementation details. The LoE structure can be configured with the number of layers L, the number of experts at each layer K, the channel dimension of each expert C, and the dimension of the latent at each layer H. denoted tuple (L, K, C, H). We train LoEs of (5, 128, 256, 128), (5, 256, 64, 256), (6, 256, 64, 64), and (5, 64, 64, 64) in CelebA- HQ (Karras al., 2018), ShapeNet (Chang et al., 2015), SRN-Cars (Sitzmann et al., 2019), and AMASS (Mahmood et al., 2019) datasets, respectively. We follow mNIF (You et al., 2024) the data processing protocols for CelebA-HQ, ShapeNet, and SRN-Cars datasets. Details about the AMASS dataset are provided in Appendix B.3.
Training details. In Stage-1, we train LoEs via meta-learning on CelebA- -HQ ShapeNet, and AMASS, and with auto- decoding on SRN-Cars. We use a batch size of 32, an outer learning rate of le-4, an inner learning rate of 1 with 3 steps, and train the LoE for 800 epochs in the meta-learning setting. For auto-decoding experiments on SRN-Cars, we use a batch size of 8, a learning rate 1e-4, and train the LoE for 1000 epochs. In both settings, we use the AdamW (Loshchilov, 2017) optimizer without weight decay. In Stage-2, we set the training batch size to be 32, learning rate 1e-4, and cosine scheduler with minimum learning rate 0.0. We train the HCDM for 1000 epochs with the AdamW optimizer.
B. Additional Experimental Results
B.1. Generalizability Analysis Through Retrieval
Generated
Retrieved from training set
MNIF
CHINR
Figure 9. Retrieval on CelebA-HQ: mNIF retrieves images closely resembling those from the training set, while CHINR demonstrates hetter generalization by producing distinct new images.
We use retrieval to compare the generalizability of CHINR and mNIF on the CelebA-HQ dataset. Specifically, we generate samples and retrieve the closest images from the training set. As shown Fig. 9, mNIF generates samples that are very similar to the training images, suggesting a higher chance of "memorization". In contrast, CHINR demonstrates better generalization by producing "new" samples that differ more noticeably from the training data.
B.2. More Generated Samples
Figure 10 shows more generated samples on CelebA-Net, ShapeNet, and SRN-Car datasets.
13

---

Controllable Data Generation with Hierarchical Neural Representations
Figure 10. More generated samples of CelebA-HQ, ShapeNet, and SRN-CARNN Cars data.
Table 5. Quantitative results AMASS.
Model
MSF.
mNIF (You al., 2024)
0.015
CHINR
0.011
1111111
1
Figure 11. Generated motions with HCDM, each denotes one sampled data.
B.3. AMASS Experiments
We apply our proposed CHINR model to the AMASS dataset 3D human motions. For each motion sequence, we use 200 frames, with each frame represented by 165 values corresponding to the locations and rotations of body joints. As result, each data instance is formatted grid with size 200 165. In 1tage-1, the LoE is employed to fit the motion instances. In Stage-2, we set the binary lengths to 8 to avoid memorizing conditions.
Reconstruction and generation results. The reconstruction performance is shown in Table 5, The randomly generated motions are shown in Fig. 11.
Semantic-level Interpolation. Since the LoE successfully learns the consistent latent space, we can perform semantic-level
14

---

Controllable Data Generation with Hierarchical Neural Representations
Figure 12. Semantic interpolation for AMASS data. Anchor sequential motions (indicated by the red and green dashed boxes) are first fitted with LoE obtain latents. Then semantic-level interpolation is performed by interpolating the latents, The dashed box denotes the start motion, and the green dashed box denotes the end motion.
interpolation for motions. As shown in Fig. 12, given two fitted sequential motions with LoE, each corresponds latent, interpolate the latent from the start motion (indicated by the red dashed box) to the end motion (indicated by the green dashed box) linearly with ratio (0.2, 0.4, 0.6, 0.8). We can see that the interpolated motions change smoothly from the start to the end. Semantic-level interpolation can be useful in the gaming industry, and 3D-digital content generation.
Temporal-level interpolation. Since the INR can generate data instances in any resolution, we can easily enlarge the input coordinates' resolution in the time dimension to achieve temporal-level interpolation. We set the length the time dimension be 200 and 400, then get motions with LoE, The interpolated results are submitted as videos named "motion_shon short.mp4" and "motion "motionloog.mmpee long.mp4".
B.4. Hierarchical Controllable Generation
More examples of hierarchical controllable data generation are presented Fig. 13.
B.5. Latent-based Retrieval
We show an application of data retrieval by latents, since they already embed rich semantic meanings. We first obtain the latents for the target data by fitting it the LoE through a few gradient steps. Once the latents are optimized, they can be used retrieve similar data by comparing their latent representations the searched set, allowing us search for semantically similar examples within the latent space. Figure 14 shows this process by using images from the test-split of CelebA-HQ as the targets, and train-split images as the searched set. We demonstrate two approaches for retrieval: (1) using the flattened h for all layers, and (2) layer-wise retrieval using each layer's 's latent h', As shown in Fig. 14, retrieval by the flattened h will retrieve samples that are broadly similar, while layer-wise retrieval retrieves samples with specific semantic similarities. For example, h² retrieves faces with similar orientations, while h retrieves faces with similar facial features such eye shape and expressions.
15

---

Controllable Data Generation with Hierarchical Neural Representations
Simpled
Fixed
(a) CelebA-HQ
(b) ShapeNet
(c) SRN-Cars
Figure 13. More examples of hierarchical controllable generation CelebA-HQ. ShapeNet, and SRN-Cars data.
16

---

Controllable Data Generation with Hierarchical Neural Representations
Target
Closer
Target
Closer
Retrival by all layers
b
h
h
h
Retrival by each layer
ha
h
h
Figure 14. Latent-base based retrieval via two approaches: retrieval by all layers and retrieval by each layer.
C. Analysis
In this section, we provide more analysis of the latent space and the functionalities of binary conditions.
C.1. Latent Space Analysis
Here, we analyze the latent space further, focusing its interpolation capabilities and providing more results of hierarchy analysis.
Figure 15. Latent space interpolation performed for LoE, with four corner points representing the anchor examples rendered stage 1. The intermediary points are generated through the bilinear interpolation of the latents associated with these four anchors. The interpolation evaluated on datasets CelebA-IIQ, ShapeNet, and SRN-Cars.
I.I. LATENT INTERPOLATION
To illustrate that our model learns a consistent and metric latent space, following definitions in Du et al. (2021), we perform latent space interpolation in two ways: complete interpolation, and layer-wise interpolation.
17

---

Controllable Data Generation with Hierarchical Neural Representations
Figure 16. Layerwise interpolation. The red boxes denote the start and the green boxes denote the end. For the CelebA-HQ. the layers 2 + 4 are interpolated respectively while other layers are fixed. For the ShapeNet, the layers + 3 are interpolated respectively.
Complete Interpolation is shown in Fig. 15. Four corners present the signals with latent generated from Stage-1. The intermediary signals are bilinearly interpolated from four corners latent space. The results demonstrate that the learned latent metric and consistent with human perception.
Layer-wise Interpolation. Since LoE embeds semantics hierarchically in different parts of the latent, can interpolate each part to control specific semantics. As shown in Fig. 16, we interpolate the second, third, and fourth parts of the latent associated with red-boxed signals, with the corresponding parts of the right side latent. For CelebA-HQ samples, we find that the facial orientation, facial features, and skin tone can be interpolated independently. This demonstrates that each part of the latent also constructs a metric and consistent manifold.
C.1.2. LAYER- WISE HIERARCHY ANALYSIS
LAYER-WISE
Layer-wise correlation analysis. We perform layer-wise correlation analysis latents show the necessity of conditional dependency modeling. We compute cross-layer correlation between latents using Singular Vector Canonical Correlation Analysis (SVCCA) (Raghu et al., 2017). Fig. 17a displays the pairwise correlations between h across layers, trained on Celeba-HQ in Stage-1, showing the non-negligible correlation between layers. Similar findings in other datasets are shown in Figs. 17b and 17c. This underscores the importance of modeling conditional distributions p(h' <t). rather than independent marginal distributions p(h') in Stage-2.
0.54
0.36
0.29
0.22
0.64
0.45
0.68
0.28
0.11
0.21
0.6
0.29
0.31
0.27
0.36
0.44
0.42
0.27
0.45
0.51
0:48
0.28
0.27
0.31
0.31
0.3
027
0.29
0.48
0.23
0.25
0.27
0.5
0.42
0.22
0.27
0.27
0.37
0.28
0.31
0.25
0:34
(a) CelehA-HQ.
(b) ShapeNet.
(c) SRN-Cars.
Figure Correlation between learned latents across layers, trained on CelebA-IIQ (Karras al., 2018) ShapeNet (Chang et al., 2015) and SRN-Cars (Sitzmann al., 2019). The non-negligible correlation between adjacent layers (e.g., h' and h²) reveals the necessity of conditional distribution learning.
Layer-wise dependency visualization. We provide more layer-wise dependency visualizations in Fig. 18. This highlights the learned layer-wise dependencies in alignment with data semantic hierarchies.
18

---

Controllable Data Generation with Hierarchical Neural Representations
layer 4
byyer 4
layer
0.06
00/ plos
00
032
0.00
-0.02
-0.2
000
4034
-006
-208
1002
2.00
9.82
0.0
9MUUP 0980 10222 1000 8965 0083
2019
-0006
-1.66
00020 1.02 0029
0.90
0076-22
02 -0.1
00
Figure 18. Visualization of conditional distributions across layers 3, 4, 5. The gray regions present the distribution of latents from Stage-1. while the colored regions represent the sampled latents from Stage-2.
Clusters fist
Clusters of fire
Claises layer
Clusters of thind layer
Clastors sacond laye
Chosters layer
Clsstes of fourth layer
fourth
Outee
Figure 19, Clusters of each part of latent and binary conditions. The dotted plot presents clusters of each part of latents trained on CelebA-HQ. The gray distribution plot presents the distribution each part latents, starred scatter plot presents clusters of latents with similar binary conditions.
19

---

Controllable Data Generation with Hierarchical Neural Representations
C.2. Binary condition Analysis
We analyze the clustering of latents and binary conditions on CelebA-HQ dataset, as shown in Fig. 19. Firstly, we use the KMeans algorithm to get 10 clusters of latents, shown as the dots in the figure. Then select three anchor latents, generate three binary conditions with HCDM, and search the nearest binary-corresponder latents. The nearest neighbors represented by the stars. We can observe that the binary conditions embed the latents' information and form a consistent binary condition space. This binary condition space corresponds to the latent space.
20


Reviews:
[{"review_id": "OCNoG7wGZe", "reviewer": "Reviewer_XvvS", "summary": "The paper introduces a novel framework to capture hierarchical data semantics with implicit neural representations, enabling improved control over data generations. The framework is structured in two stages. For the first stage, a layer-of-expert (LOE) architecture is employed to capture general semantics with shared experts and distinct semantics with latent vectors. For the second stage, a hierarchical conditional diffusion model (HCDM) is employed to learn the distribution of latent vectors. The HCDM models inherent hierarchical structure as conditional dependencies between layer l and layers <l. The framework is evaluated on four different domain data, i.e. images, point clouds, neural radiance fields, and motions, and presents superior performance in controllability and reconstruction ability.\n\n## update after rebuttal\nI will keep my ratings since most of my concerns are solved.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer XvvS for the valuable feedback.\n\n### **W1. Scalability to larger datasets**\n\nThanks for raising this scalability concern. The CHINR framework focuses on establishing the connection between INR parameters and data semantics for controllable generation. While our experiments use smaller datasets, the core framework is generalizable to larger-scale data as long as this fundamental connection holds. To handle increased complexity, more efficient learning methods should be employed to handle more complex data patterns. \n\nBeyond simply adding more layers or experts, we can adopt localized solutions such as patch-wise or spatial-adaptive modulation, as explored successfully in [1, 2]. This reduces the burden of a single set of INR parameters to represent the whole data and allows the model to capture local structures, improving scalability and convergence without significantly increasing inference cost.\n\n### **W2. Reliance on meta-learning**\n\nThanks for pointing out this thoughtful concern. We agree that meta-learning requires a shared initialization for fast adaption, which works well for data with consistent structure (e.g., faces) but may struggle with more complex data. This is an intrinsic issue of the meta-learning method. To address this, one can allow more inner-loop updates at the cost of increased training time. It is also possible to incorporate more adaptive initialization or weight update strategies [3, 4] to better handle data variability. Alternatively, we can use auto-decoding as done in the NeRF experiments, which avoids the need for fast adaptation and may offer better generalization in such cases.\n\n### **W3. Lack of targeted modification mechanism**\n\nThank you for the insightful comment. CHINR is designed to leverage the hierarchical structure of INR parameters for layer-wise semantic control, rather than direct attribute editing. Although it doesn't currently support direct modifications like changing lip color, it can be extended with attribute supervision (in Stage 1) or latent manipulation to enable targeted edits—an exciting direction for future work.\n\n### **Reference**\n[1] Bauer, Matthias, et al. Spatial functa: Scaling functa to imagenet classification and generation. 2023.\n\n[2] Park, Dogyun, et al. DDMI: Domain-Agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations. ICLR, 2024.\n\n[3] Wang, Ruohan, et al. Structured prediction for conditional meta-learning. Neurips, 2020.\n\n[4] Baik, Sungyong, et al. Meta-learning with adaptive hyperparameters. Neurips, 2020.", "claims_and_evidence": "Each claimed contribution is supported by clear evidence.", "methods_and_evaluation_criteria": "The proposed hierarchical approach makes sense. It aligns INR structure with data semantic for layer-wise control. The evaluation criteria (e.g., Table 1) are reasonable in showing the controllability of CHINR.", "theoretical_claims": "The theoretical claims seem correct. Equation 2 explains the hierarchical structure of INR, which serves as the foundation of the hierarchical approach.", "experimental_designs_or_analyses": "The experimental designs are valid in proving the controllability achieved through the hierarchical approach. The analyses in section 4.3 are convincing and show evidence of disentangled semantic. The only issue would be the scale of data as CHINR is currently evaluated in small-scale datasets.", "relation_to_broader_scientific_literature": "CHINR aims at bridging the weight space structure and data semantics for controlled generation, which is new to the INR literature. The idea of hierarchical/progressive data generation has been widely explored in a broader literature (e.g., GANs, VAEs). The difference is that CHINR models this hierarchy in network parameter space instead of data feature space.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\n1. The idea of bridging INR weight hierarchy and semantic hierarchy is novel, and the layer-wise generation effectively models this hierarchy.\n\n2. The experiments show CHINR’s controllability in different datasets, validating its hierarchical approach across multiple domains.\n\n3. The paper is well written. The motivation, methodology, and experiments are clearly presented.\n\nWeaknesses:\n\n1. Despite performing well on small datasets like CelebA-HQ, CHINR’s scalability to larger and more complex data (e.g., higher-resolution images or shapes) is unclear. Adding more experts or layers may hinder training convergence and raise inference costs. In addition, longer conditional chains are more prone to error accumulation, affecting generation quality.\n\n2. The reliance on meta-learning may limit generalization to more diverse data patterns. Since latents are initialized from a small distribution for fast adaptation, they struggle to capture broader variations.\n\n3. CHINR allows attribute variation at different levels but lacks a mechanism for targeted modification (e.g., changing pink lips to red). This limits its use in real-world applications such as image and shape editing, where targeted adjustments are essential.", "other_comments_or_suggestions": "See the weaknesses part.", "questions_for_authors": "See the weaknesses part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TyvzE4x36m", "reviewer": "Reviewer_T7ed", "summary": "The paper presents a novel method, CHINR, for controllable generative INR by exploiting the hierarchy structure in parameters. The authors employ a Layers-of-Experts (LoE) network to encode data with layer-wise latents and propose a Hierarchical Conditional Diffusion Model (HCDM) to learn conditional dependency in layers. Experiments show that CHINR enables precise control at different granularities during generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer T7ed for the thoughtful comments.\n\n### **W1. Out-of-distribution samples in conditional chain.**\n\nThank you for your insightful feedback. We address error accumulation during both the training and inference phases.\n\nIn the training phase, we begin by teaching the model to capture inter-layer dependencies, using ground truth layer-wise latents (obtained from Stage 1) as conditions. Subsequently, we fine-tune the model to construct a coherent conditioning chain by progressively incorporating generated latents as conditions.\n\nDuring inference, we further mitigate potential degradation by adjusting the standard deviation of the input noise in the diffusion process. This reduces the likelihood of generating outlier latents that could compromise output quality.\n\n### **W2. Binary condition.**\n\nWe choose to use binary condition for the following reasons:\n\n1. With fewer bits to represent data, the model is less likely to memorize every detail of the training data. Instead, it must extract the core features that are most useful/general across many examples. This acts as a form of regularization.\n\n2. As the model processes the input data, it learns to assign similar items to the same quantized value. Over time, these discrete codes become representative of certain semantic concepts or clusters, which correspond to the inherent structure of the data.\n\n### **W3. Generalize to data without inherent frequency hierarchy.**\n\nThank you for your interesting question. CHINR leverages the frequency-based representational hierarchy inherent in INR structure, which aligns well with data like images and 3D shapes with spatial frequency hierarchy. The current formulation may not be directly applicable for modalities like text that don't follow such a frequency hierarchy. However, CHINR’s core idea—modeling hierarchical latent dependencies across network layers—may still be adapted by identifying alternative structures (e.g., syntactic or semantic hierarchies) relevant to the modality.\n\n### **W.4 Reconstruction quality affects Stage 2**\n\nThe sensitivity of hierarchical control to quality variations is evident in two key aspects:\n\n1. Reconstruction Quality in Stage 1: The effectiveness of hierarchical control in Stage 2 is bounded by the reconstruction quality achieved in Stage 1. If the proposed LoE fails to learn hierarchically structured latents during Stage 1, it limits the controllability during the generation in Stage 2. Our findings show that when the PSNR on CelebA-HQ exceeds 25, no visually noticeable distortions are observed, and hierarchical control performs reliably.\n\n2. Quality of Ground-Truth Data: The controllability is also constrained by the quality of the ground-truth data used in Stage 1. Higher noise levels hinder the learning of conditional dependencies. In the extreme case where the data is entirely noise, the resulting latents lack meaningful structure. As the noise in the ground-truth data increases, both reconstruction and generation quality progressively degrade.", "claims_and_evidence": "The claims are supported by clear evidence.", "methods_and_evaluation_criteria": "The proposed CHINR makes sense in solving controllable data generation. The evaluation criteria such as success rate and FID are convincing.", "theoretical_claims": "The theoretical claims are correct and clearly explained. We observed that Equation (2) in main paper is grounded in explaining the representation ability of INR, while Equation (3) models corresponding hierarchical representations.", "experimental_designs_or_analyses": "The experimental designs and analyses are convincing. Figure 4 clearly presents controllable data generation, and Table 1 presents accordingly quantitative evidence. Figure 6 intuitively explains why the CHINR works.", "relation_to_broader_scientific_literature": "The paper explores an orthogonal direction to existing INR literature by focusing on the controllability of data generation. It argues that methods such as Functa, mNIF, and GASP process the latent modulation vector indiscriminately, and instead proposes modeling this vector as a conditional chain to enable hierarchical control.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strength:\n- The observation of connection between hierarchy of semantics and model parameters is interesting and inspiring, and the idea is well-motivated.\n- The proposed diffusion model is a novel method to model the hierarchical dependency within parameters.\n- The paper is well-written and structure is straightforward. The mathematical formulations appear correct.\n- The empirical results show strong evidence of successful controllability across various modalities. The paper presents thorough analysis that semantics are disentangled in parameter space.\n\nWeakness:\n- Since the parameters are generated with a conditional chain, it has chances getting out-of-distribution samples. The out-of-distribution causes erroneous parameters, which will accumulate if occurred in early layers. How do you solve this problem?\n- The binary condition length must be carefully tuned on each dataset to balance controllability and generalization. Can the authors explain more about the choice of binary condition?\n- How does the model generalize to data without inherent frequency hierarchy, e.g. text?\n- The stage-2 is trained with reconstructed data of stage-1, how does the reconstruction quality affect the conditional modeling in stage-2? Can stage 2 learn compatible conditional chain if the reconstruction quality is low?", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mv20J9Gxlz", "reviewer": "Reviewer_N8d9", "summary": "The paper introduces CHINR, a framework for controllable data generation using hierarchical neural representations. It addresses limitations of existing generative INR approaches that fail to capture hierarchical structures in data, leading to limited control over generation. The method consists of two stages: Stage-1 constructs a Layers-of-Experts network where each layer has its own latent vector for disentangled representations, and Stage-2 introduces a Hierarchical Conditional Diffusion Model to capture dependencies across layers for controllable generation. The framework enables hierarchical control over generated content at various semantic granularities. Experiments across different modalities show improved generalization and controllability compared to existing methods.\n\n## update after rebuttal\nI hope the authors could refine the larger dataset problem in future revisions. As a concensus is reached among reviewers, I will keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer N8d9 for the valuable comments.\n\n### **W1. Scalability to larger datasets.**\n\nThank you for pointing this out. As discussed in the conclusion, scaling CHINR to larger datasets is a known challenge. While CHINR demonstrates the core idea of hierarchical control through INR parameter modeling, extending it to more complex data can be achieved by enhancing the INR representation capacity and employing more efficient training approaches. For example, localized solutions such as patch-wise or spatial-adaptive modulation methods [1,2,3] can help INRs better capture local structures and improve scalability.\n\nThanks for pointing out the layout problem of Fig. 4 and 7. We will increase the font size of those figures.\n\n### **Reference**\n[1] Wang, Peihao, et al. Neural implicit dictionary learning via mixture-of-expert training. ICML, 2022.\n[2] Bauer, Matthias, et al. Spatial functa: Scaling functa to imagenet classification and generation. 2023.\n[3] Park, Dogyun, et al. DDMI: Domain-Agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations. ICLR, 2024.", "claims_and_evidence": "The proposed CHINR achieves outstanding performance. This is verified by experiments.", "methods_and_evaluation_criteria": "The paper proposes LCDM and LoE as methods. The benchmarks are CelebA-HQ, ShapeNet, and SRN-Cars, which are commonly-used ones.", "theoretical_claims": "No theoretical claims are involved in this paper.", "experimental_designs_or_analyses": "I found no issues regarding experimental designs or analysis.", "relation_to_broader_scientific_literature": "While existing INR methods fail to leverage the hierarchy of semantic abstraction, this paper introduces hierachical control in generative INRs.", "essential_references_not_discussed": "No from my knowledge.", "other_strengths_and_weaknesses": "Strengths:\n1. Novelty: hierarchical control is achieved, different from previous works.\n2. Comprehensive evaluation: the multi-modal experiments have been conducted to validate the method in various scenarios.\n\nWeaknesses:\nScalability to larger datasets, as is mentioned in \"Conclusions\".", "other_comments_or_suggestions": "Layout problem: the font size of legend and ticks in Fig. 4 and 7 are too small. The authors should consider revising these figures.", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a271F0q2RV", "reviewer": "Reviewer_aYYY", "summary": "The paper proposes a framework for controllable data generation using hierarchical implicit neural representations (INRs). \nIt models conditional dependencies across layers in the parameter space to improve control over the generation process.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer aYYY for the constructive feedback.\n\n### **W1. Lack of predefined interpretability for each layer.**\n\nThank you for the thoughtful comment. CHINR is designed to align the hierarchical structure of INRs with semantic abstraction, allowing each layer to control different levels of detail. While the semantics of each layer are not predefined (they are learned freely in Stage 1), the progressive control in Figure 4 and latent composition in Figure 6 demonstrate consistent layer-wise influence, suggesting that meaningful hierarchical semantics naturally emerge through training. Incorporating attribute supervision into Stage 1 to enforce predefined semantics at each layer is a promising direction to improve interpretability and fine-grained control in practical applications.\n\n### **W2. Generalize to more complex data.**\n\n1. Thank you for your insightful comments. Compared with other state-of-the-art methods such as StyleGAN, the data instance size represented by CHINR is significantly smaller (e.g., $5 \\times 64$ latent vector). Additionally, the LoE utilizes a five-layer neural network, which inherently limits its generation quality. The primary advantage of using implicit neural representations (INRs) lies in their ability to represent data at arbitrary resolutions, rather than in achieving high reconstruction fidelity.\n\n2. To generalize to more complex data, two main directions can be pursued: (1) enhancing the capacity of the INR, and (2) refining the meta-learning pipeline. For the first approach, dividing each data instance into smaller patches can significantly reduce the burden on the INR, allowing it to better capture local structures.\nFor the second, existing meta-learning pipelines typically initialize the latent vector from a Gaussian distribution and optimize it over just three steps. This process can be improved by enabling more efficient optimization over a greater number of steps, thereby enhancing performance and adaptability.\n\n### **W3. Training/inference cost**\n\nThanks for your suggestion. We analyze the resource consumption of CelebA experiments in the following. All experiments were conducted on an RTX 3090 GPU with a batch size of 8 for both training and inference.\n\n*Training cost for Stage 1 and 2*:\n|         | Time | Memory |\n|---------|------|--------|\n| Stage 1 | 50h  | 4.3GB  |\n| Stage 2 | 28h  | 18.9GB |\n\n*Inference cost of CHINR, generating 1000 samples following HCDM*:\n | Time | Memory |\n|------|--------|\n| 2h  | 8.4GB |", "claims_and_evidence": "The paper presents clear evidence to support the claims.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are reasonable.", "theoretical_claims": "The theoretical claims are correct and consistent. Equations (1), (2), and (3) discuss the foundation of hierarchical property of INR parameters. Equations (4) and (5) presents how to formulate the hierarchy modeling with diffusion models.", "experimental_designs_or_analyses": "The paper presents valid quantitative and qualitative experiments. Table 1 and 2 show the superior performance over existing methods, i.e. Functa, mNIF, GEM, and GASP. The analyses in Section 4.3 are intuitive. The ablation study demonstrates the effectiveness of conditional modeling.", "relation_to_broader_scientific_literature": "The key contribution of the paper is the controllability of data generation with INRs. The proposed CHINR uses hierarchical latent vector and layer-of-experts to represent data instance while prior methods, e.g. Functa, and mNIF, use flat latent vector and MOE to represent data. The CHINR proposed hierarchical conditional diffusion model to model the conditional distribution, while prior methods model the joint distribution of flat latents.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strength:\n- The paper introduces a first-of-its-kind, hierarchical way to model INRs by incorporating \nlayer-wise conditional dependencies, allowing fine-grained control over semantic generation.\n- The introduction of LOE greatly enhances the model’s ability to generate diverse semantics. \nThe expert-sharing mechanism also improves parameter efficiency.\n- The framework is tested comprehensively across multiple modalities, including images, 3D \nobjects, and motion sequences.\n\nWeakness:\n- While the model enforces a hierarchical structure, the exact meaning of each layer’s latent representation is not explicitly defined. Users must manually inspect outputs to understand how different latents influence semantic factors. This lack of predefined interpretability might make fine-grained control difficult in real-world applications.\n- Compared to state-of-the-art generative models such as StyleGAN or diffusion models, \nCHINR appears to be restricted to relatively simple image contents and object representations. How does the model generalize to more complex data?\n- The proposed framework introduces a multi-stage training pipeline involving meta-learning, auto-decoding, and hierarchical diffusion modeling, which is computationally intensive. A breakdown of training/inference time and resource requirements would be useful.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ZXNeLQbDm", "reviewer": "Reviewer_ZbmH", "summary": "This paper proposes a hierarchical Implicit Neural Representations (INR) framework that aims to provide better control over hierarchical representations during the generation process. In the first stage of the framework, a Layer-of-Experts (LoE) model is trained, and a latent variable is learned for each layer. In the second stage, the hierarchical relationship between different layers is modeled by a conditional diffusion process. Experiments are conducted on several image datasets, and evaluations and comparisons are made with baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ZbmH for the valuable feedback.\n\n### **W1. Qualitative results.**\n\nThank you for the valuable feedback. We agree that earlier layers exhibit more visible influence in Figure 4. This is expected in our hierarchical design—deeper layers naturally introduce finer, subtler variations since prior layers are fixed. Regarding vehicle color in the NeRF dataset: it is not controlled by a specific layer but emerges from all layers jointly. This is because NeRF INRs jointly regress RGB and density at the output, causing color to be decoded at the final stage and entangled with spatial attributes, rather than controlled by a specific layer.\n\nWe appreciate your suggestion to adopt Figure 1’s visualization style. However, Figure 4 was designed to showcase different conditional chains per row, and we cannot revise it due to rebuttal constraints. Nevertheless, please note that some layer effects are visible and aligned with Table 1—for example, in the NeRF samples, a spoiler appears in the 3rd and 4th samples of the first row, with layer 3 fixed in the latter, supporting its role in controlling that attribute.\n\n### **W2. Marginal improvements.**\nThank you for the suggestion. Our primary goal is not to optimize reconstruction/generation quality alone, but to introduce a framework that enables hierarchical control—a key capability missing in prior generative INR methods such as mNIF and Functa.\n\nAs shown in Table 2, CHINR achieves notable improvements in evaluation metrics (e.g., PSNR/FID/SSIM) over most baselines except mNIF. However, CHINR exhibits significantly less memorization than mNIF, indicating better generalization. Furthermore, we aim to demonstrate that the conditional chain enables controllable generation without sacrificing quality—rather than to show it outperforms all models on every metric.\n\n### **TW1. How is $h^l$ computed.**\n$h^l$ is an instance-specific latent at layer $l$, which is mapped to a gating vector that weighted averages the experts at layer $l$. To compute $h^l$, we use meta-learning or auto-decoding (Section 3.2) by initializing it around zero and optimizing it for each data instance.\n\n### **TW2. Figure 2 meaning.**\nThanks for pointing this out. We will make the captions clearer to explain this figure. Specifically, the transparency of noise refers to the noise schedule in the diffusion process, where the noise gradually dominates in the forward process. We shade the color of latents to distinguish latents from different layers. The latent at each layer is mapped to a gating vector that weighted averages the mixture of experts at that layer.\n\n### **TW3. $||\\epsilon,\\epsilon_\\theta()||^2$ in Line 242.**\nThanks for pointing this out. It should be $||\\epsilon - \\epsilon_\\theta()||^2$ as used in diffusion models. We will correct this in the final version.", "claims_and_evidence": "I found two major issues with the claims and evidence.\n\nFirst, the paper claims that the proposed framework achieves hierarchical controllable generation. However, this claim remains questionable to me, given the qualitative results shown in Figure 4. While it is clear that the first layer has significant effects on the generation by controlling the fixed “type” of the data, such as the general appearance of a face, the object category in 3D shapes, and vehicle types, the effects of the rest of the layers seem very weak without any clear patterns. For example, which layer or layers control the color of the vehicle, as an important semantic feature for that dataset? The authors are highly encouraged to replace the visualization methods in Figure 4 with those in Figure 1 to clearly demonstrate the effects of each layer through multiple branches of each generated image, allowing readers to better understand which representations are controlled by previous fixed layers and which are affected by the remaining layers. Additionally, it would be very helpful if the authors could relate the observations from the qualitative results to the quantitative results in Table 1 and demonstrate whether they align with each other.\n\nSecond, the authors claim that the proposed framework outperforms existing methods on most datasets. However, it seems that many improvements are marginal. I appreciate the authors’ efforts in reporting the variance of performance in Table 2, but it is necessary to conduct a statistical test and report the p-value to demonstrate whether these marginal improvements are statistically significant.", "methods_and_evaluation_criteria": "Yes, I believe most of the methods make sense and are suitable, while the qualitative results in Figure 4 can be improved by adapting the methods in Figure 1, as mentioned above.", "theoretical_claims": "TW (1): In line 188, how is $h^l$ computed? Is it obtained by sampling from a component of the mixture or by computing the average? \n\nTW (2): Figure 2, as the main figure for the proposed framework, lacks sufficient captions to explain the details in the figure. For example, what does the transparency of noise indicate? What does the shade of color indicate? What is the relationship between each “latent” and the mixture of experts? \n\nTW (3): In line 242, what does $||ε, ε_{θ}()||^2$ mean? Did the authors mean to write $||ε - ε_{θ}()||^2$, as used in diffusion models?", "experimental_designs_or_analyses": "Yes, I checked all the qualitative and quantitative results. I believe the experiments are well-designed, but the analysis is not sufficient to support the key claims of this paper, as mentioned above in \"Claims And Evidence\".", "relation_to_broader_scientific_literature": "The proposed hierarchical model is novel and promising for INR works, which is a solid research direction worth investing in. However, the significant advantages and improvements of the current version of the proposed framework remain somewhat questionable compared to the baselines, as evidenced by the results.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "# after rebuttal\nI thank the authors for the response. Most of my concerns have been addressed, and I have increased my score accordingly.", "questions_for_authors": "Q(1): I look forward to the authors’ response to my concerns about the qualitative results, as described in the “Claims and Evidence” section, which will have the highest impact on my future decision. \n\nQ(2): It would be very helpful if the authors could clarify my confusion listed in the “Theoretical Claims” section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a genuinely novel and well-motivated approach to controllable generation within the INR framework. The core contribution – linking parameter hierarchy to semantic control via conditional modeling – is significant and opens up new possibilities for generative INRs. The method is technically sound and supported by extensive experiments demonstrating the key claim of hierarchical controllability across diverse data types. While limitations regarding scalability, interpretability, and targeted editing exist, they are reasonably acknowledged as scope limitations or avenues for future work. The strong consensus among reviewers, bolstered by effective author rebuttals, supports acceptance. This work is a valuable addition to the literature on generative modeling and implicit neural representations.
Decision: Accept (poster)