Controllable Data Generation with Hierarchical Neural Representations

Authors: Sheyang Tang, Xiaoyu Xu, Jiayan Qiu, Zhou Wang
Primary area: generative models
Keywords: Implicit Neural Representations, Generative INR, Generative Models, Diffusion Models

Abstract:
Implicit Neural Representations (INRs) represent data as continuous functions using the parameters of a neural network, where data information is encoded in the parameter space. Therefore, modeling the distribution of such parameters is crucial for building generalizable INRs. Existing approaches learn a joint distribution of these parameters via a latent vector to generate new data, but such a flat latent often fails to capture the inherent hierarchical structure of the parameter space, leading to entangled data semantics and limited control over the generation process. Here, we propose a $\textbf{C}$ontrollable $\textbf{H}$ierarchical $\textbf{I}$mplicit $\textbf{N}$eural $\textbf{R}$epresentation ($\textbf{CHINR}$) framework, which explicitly models conditional dependencies across layers in the parameter space. Our method consists of two stages: In Stage-1, we construct a Layers-of-Experts (LoE) network, where each layer modulates distinct semantics through a unique latent vector, enabling disentangled and expressive representations. In Stage-2, we introduce a Hierarchical Controllable Diffusion Model (HCDM) to capture conditional dependencies across layers, allowing for controllable and hierarchical data generation at various semantic granularities. Extensive experiments on CelebA-HQ, ShapeNet, SRN-Cars, and AMASS datasets demonstrate that CHINR improves generalizability and offers flexible hierarchical control over the generated content.

Decision: reject
Reviewer scores: 5, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730417064484

thread_type:
official_review

<thread_object>
creator: reviewer_ww4m

<content>
summary:
The paper introduces a novel framework called Controllable Hierarchical Implicit Neural Representation (CHINR), aimed at enhancing data generation and control through hierarchical neural representations. CHINR leverages implicit neural representations (INRs) to represent complex data as continuous functions and incorporates hierarchical dependencies across layers to improve generalizability and control. The framework operates in two stages: first, it builds a Layers-of-Experts (LoE) network where each layer uses specific latent vectors to manage distinct data features, allowing for a disentangled representation. In the second stage, a Hierarchical Controllable Diffusion Model (HCDM) is used to capture conditional dependencies across layers, enabling nuanced control of generated content. Experiments across various datasets (e.g., CelebA-HQ, ShapeNet, and AMASS) show that CHINR offers improved generalizability and hierarchical control over generated data.

strengths:
1. CHINR’s two-stage framework with LoE and HCDM effectively captures hierarchical structures in data, enhancing representation control and generalisability.
2. The model is rigorously evaluated across multiple modalities, showing consistent improvement over baseline methods, especially in quality and generalisation.
3. The model’s ability to control data generation at multiple granular levels (e.g., overall structure, fine details) is a substantial advancement for applications requiring precision.

weaknesses:
see questions.

questions:
1. Could the proposed binary conditions be replaced or optimised further to improve interpretability or training efficiency?
2. How does CHINR handle data types with less hierarchical structure (e.g., text), and is its control mechanism adaptable to such cases?
3. How sensitive is the hierarchical control to variations in input data quality, and could this affect generalisation in real-world applications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730428185356

thread_type:
official_review

<thread_object>
creator: reviewer_lwtn

<content>
summary:
The authors exploit a hierarchical and sparse structure prevalent in INRs to conceive a data generation method that is more controllable and modular than contemporary approaches.

I am far from knowledgable in this field. Take my review with a heap of salt.

strengths:
- The compositionality aspect of this method is enticing
- The controllability and clear hierarchical order (Fig 1) is fascinating
- The premise of utilizing the hierarchical structure of the SIREN INR seems intuitive and reasonable to me.

weaknesses:
- Table 1 needs uncertainties to gauge superiority or consistency. The method  does not seem super strong on the performance front.
- Given the discrepancy in the conditional chain (and thus worse image quality) when controlling the latents as done in Fig. 6 (see L456ff), is the controllability aspect still a valid bonus? Is this a drawback that can be retroactively fixed by somehow iterating multiple times over the hierarchies?

questions:
- How does it happen that, for instance in the middle of Figure 1, the hierarchies so nicely align with human abstractions over the data? It seems somewhat coincidental to me that the first layer corresponds to the type of object (chair, jet, table) and the next layer corresponds to the type of chair, and so on.
- It would be nice to see an equation for the LoE. Is each expert a SIREN model? Are the experts fully connected, and then controlled by a layer-specific gating function?
- In eq.3, isn’t $p(h^l|h^{<l}) = p(h^l|h^{l-1})$ (i.e. markov)?
- The design of the condition seems very ad-hoc, specifically the quantization. Can you motivate this further beyond “removing enough information to not overtrain”? Since the very coarse quantization causes a discontinuity in the condition, I expect this to have weird side effects, albeit maybe not super prominent in this domain.
- Nit: citation needed in L32 for INRs
</content>
</thread_object>

</thread>
<thread>
creation date:
1730473285025

thread_type:
official_review

<thread_object>
creator: reviewer_wk7n

<content>
summary:
This paper introduces a novel deep generative model for Implicit Neural Representations (INRs), designed to capture hierarchical dependencies between INR layers through a structured latent space. The proposed model, CHINR, employs a Layers of Experts (LoE) framework, where weights are globally shared, and the expert weights are obtained through data-specific learnable latent vectors mapped into gates for the LoE. After learning these latent vectors, the second stage leverages a hierarchical conditional diffusion model to approximate $p(\mathbf{h})$ in a layered manner. Extensive empirical evaluation supports the design choices, showing that CHINR outperforms most baselines while offering improved interpretability and controllable generation capabilities.

strengths:
- The contributions are both relevant and well-supported by empirical results.
- The paper is clearly structured and easy to follow.
- The proposed method demonstrates robustness across evaluations.
- Limitations are thoughtfully discussed, with suggestions for future work to address them.

weaknesses:
- The background section omits a few relevant recent works.

questions:
### Background

- Recent generative models of INRs are not referenced in the background section. While [1-4] are included as methods that learn the latent distributions that map to or modulate INR parameters, it’s important to note that not all of these methods use meta-learning or a two-stage training approach. Specifically, [1,5,6] introduced end-to-end models where both the latent model and the latent-to-INR model are learned jointly, using GANs and VAEs. Also, there are other recent two-stage works that trained diffusion models over the latent vectors mapped to INRs, such as [7]. I suggest the authors include all the references to these additional works.

### Minor Comments

- In Section 3, the text states that "the high dimensionality of raw weights is intractable." However, in [1,5,6] the full set of INR weights is efficiently generated via hypernetworks, achieving competitive results.

- Typo on Line 465: "dependecy" should be corrected to "dependency."

### References

[1]: Dupont, Emilien, Yee Whye Teh, and Arnaud Doucet. "Generative Models as Distributions of Functions." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

[2]: Du, Y., Collins, K., Tenenbaum, J., & Sitzmann, V. (2021). Learning signal-agnostic manifolds of neural fields. Advances in Neural Information Processing Systems, 34, 8320-8331.

[3]: Dupont, Emilien, et al. "From data to functa: Your data point is a function and you can treat it like one." International Conference on Machine Learning. PMLR, 2022.

[4]: Bauer, Matthias, et al. "Spatial functa: Scaling functa to ImageNet classification and generation." arXiv preprint arXiv:2302.03130 (2023).

[5]: Koyuncu, Batuhan, et al. "Variational Mixture of HyperGenerators for Learning Distributions over Functions." International Conference on Machine Learning. PMLR, 2023.

[6]: Xu, Siyuan, et al. "Uncertainty-aware Continuous Implicit Neural Representations for Remote Sensing Object Counting." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

[7]: Park, Dogyun, et al. "DDMI: Domain-agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations." The Twelfth International Conference on Learning Representations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650484387

thread_type:
official_review

<thread_object>
creator: reviewer_1ywi

<content>
summary:
In this paper, authors first posit that current generative Implicit Neural Representations (INRs) ignore hierahical structure in latent parameter distributions. To address this issues, authors propose Controllable Hierarchical Implicit Neural Representation (CHINR), consisting of two innovations: (1) Layers-of-Experts (LoE) Network and (2) Hierarchical Controllable Diffusion Model (HCDM).

strengths:
1. The paper addresses a core limitation in Implicit Neural Representations (INRs): the use of a "flat" latent distribution $h$ which overlooks the inherent hierarchical structure of the data. To resolve this, the authors propose a diffusion model that leverages conditional distributions to capture this hierarchy. The motivation is clearly articulated, and the solution is well-suited to tackle the challenge by directly modeling hierarchical dependencies in the latent space.

2. The paper is well-written, presenting the proposed method with clarity and precision. Despite the complexity of the approach, the authors have structured the content effectively, making it accessible and straightforward to follow.

weaknesses:
1.  The improvement over the most comparable method, mNIF, appears incremental. This paper builds upon the mNIF framework, adopting a similar two-stage process with latent modulation and a mixture of networks/layers, targeting the same challenge of capturing layer-wise hierarchical structure. However, despite the added complexity and potential inference cost, the performance gains relative to mNIF methods seem incremental. I am not sure the increased model complexity justifies the marginal improvements — since the method seems quite complicated. 

2. Although the analysis in Section 4.3 effectively demonstrates how different layers control distinct semantic aspects of the generated content and highlights the importance of conditional modeling, the ablation study in Section 4.4 is less convincing. The authors primarily provide anecdotal examples without sufficient quantitative evidence. Even though the examples shown in Figure 8 indicate the importance of conditioning, expanding the experimental details and including quantitative metrics would be helpful — since it is so central to the main contribution of the paper.

questions:
1. I am not an expert on this, but I noticed that mNIF reports the number of parameters and inference cost across different generation models, which seems relevant for a fair comparison. Would it be important to consider the size of the data generator as a factor here? How does CHINR's inference cost and model size compare to its competitors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730664102899

thread_type:
official_review

<thread_object>
creator: reviewer_gfwf

<content>
summary:
This paper proposes a novel method for Implicit Neural Representations (INRs) called Controllable Hierarchical Implicit Neural Representation (CHINR). The method introduces a hierarchical approach to sampling and constructing latents that captures the layer-wise dependencies in the weights of a network and allows for controllable generation. The method is benchmarked on data generation against previous INR models, and qualitative results show the efficacy of the hierarchical approach.

strengths:
-The hierarchical approach is interesting and worth exploring in more detail.

-The diffusion approach is novel and seems like a good approach to modeling the causal structure between the latents across different layers.

weaknesses:
Overall, the paper seems incomplete to me. Table 1 is missing a lot of values, and there is a lack of benchmarking. The idea is not wholly uninteresting—in fact, I think with more benchmarking on the novel aspects of their method, the authors can put together a compelling manuscript. Ultimately, however, this manuscript is not ready for ICLR. I suggest the authors use this feedback to prepare for a submission to the next available proceedings.

-The glaring weakness in this paper is the lack of benchmarking against alternative methods. Even the only benchmarking table, Table 1, is incomplete. The only method that was fully benchmarked is mNIF, which seems to be the predecessor upon which this paper is based. Furthermore, the generation results show only marginal improvement, and it is hard to determine whether these are significant without repeats and standard errors. E.g. line 354: “Our model outperforms existing methods on most datasets.”—this is not sufficiently justified by the half-completed table.

-Related to the previous point, the hierarchical component of the model is the heart of the method, yet the only analyses I see are the qualitative figures and the correlation matrix between layers. The authors should think of a benchmark that can exploit the hierarchical structure of their model to demonstrate its utility.

-Without some kind of benchmark of the utility of the hierarchical construction of latents, the method presents itself as only a slight generalization of mNIF with much more baggage, owing to the diffusion component needed to generate the hierarchical latents. Additionally, the generation benchmark shows CHINR is at best comparable to mNIF.

-The paper is a bit hard to read—for example, I recommend clearly outlining the training procedure in stage-1, as I don’t think the mNIF method is a well-known standard method. This is less of a concern than the previous points.

questions:
See the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732308886346

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We sincerely  thank all reviewers for their constructive comments and insightful questions. We are happy to see all reviewers recognize our novelty in improving controllability of generative INRs by conditional dependency modeling. We have provided detailed responses to each comment to clarify ambiguities and highlight our contributions. The revised parts are marked in blue in the main paper.

Before answering each comment, we would like to clarify a few keypoints:

## Contributions 

The main contributions of our work are demonstrated across four aspects: **controllability, generalizability, versatility**, and **quality**, which are beyond a slight generalization of existing works.
Specifically, our main contributions are:

- **Controllability**: Our work distinguishes from previous methods in the controllability of generating fine-grain details, which has never been explored in the implicit neural representation (INR) community.

- **Generalizability**: Our work can avoid trivial memorization thus generating more diverse data compared with the SOTA methods, such as mNIF. Please find the discussion and visual samples from Line 357-363 in the main paper and Figure 9 in the supplementary material, respectively.

- **Versatility**: The CHINR's performance is validated across four different data domains, i.e. facial images, point clouds, NeRFs, and motions. This demonstrates the proposed hierarchical control and conditional dependency modeling is broadly applicable to data semantics with an inherent hierarchy structure.

- **Quality**: Our work outperforms the SOTA methods on both the reconstruction and generation metrics, which can be seen from Table 1 in the main paper.

## Conditional depency modeling

We thank Reviwers gfwF and 1yWi for insghtful questions about evaluating the hierarchy structure of the model, which we demonstrate through qualitative results in main paper. We recognize their suggestions on quantitative results as a valuable supplement for clarification of our contributions, and include the detailed discussions in "Response to Reviwer gfwF (2/4)" and "Response to Reviewer 1yWi (2/3)".

We thank Reviwer LwTN for the insightful questions about discrepency in conditional chain. We provide detailed discussions in "Response to Reviewer LwTN (2/6)" on how we address this concern by conditonal sampling and our designed training and inference strategies.

We hope our responses address your concerns. We are happy to engage in further discussions. Please let us know if you have any other questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a framework for capturing hierarchical data semantics in implicit neural representation (INR) models, aiming to generate images with improved control over expressivity and diversity. The framework consists of two main components. First, multilayer INRs are trained on a dataset, with each layer parameterized by a mixture of experts and a latent vector learned through auto-decoding. This approach results in a flexible model. Second, a distribution over the latent representations is learned, factorized in a hierarchical structure where the distribution at layer k is conditioned on the latent distributions of layers <k. This distribution is modeled as a hierarchical diffusion process. The motivation is that each layer captures specific semantics (e.g., frequency features) of an image. Once trained, the model generates images by sampling from the latent distribution space. The model is evaluated on four image datasets and compared quantitatively to baselines in terms of reconstruction ability and qualitatively in its capacity to generate realistic and diverse data by leveraging the learned hierarchical distribution.

The main novelty of the paper lies in its hierarchical component and the conditional latent distribution. While quantitative evaluations demonstrate state-of-the-art (SOTA) performance in reconstruction and generation, the improvements are incremental compared to simpler methods that do not model hierarchical INR dependencies, as highlighted by the reviewers. Thus, the primary contribution of the paper lies in its potential for better control over pattern generation. Reviewers acknowledged this novelty and recognized the significance of generating diverse, high-quality images with controlled attributes. They also appreciated the authors' efforts during the rebuttal process. However, reviewers noted that the qualitative analysis did not fully demonstrate the model's ability to capture different levels of abstraction effectively. While it is understandable that this aspect is more challenging to evaluate than plain reconstruction, the authors are encouraged to strengthen this part of their contribution in future work.