Title: Generative Neural Fields by Mixtures of Neural Implicit Functions
Venue: NeurIPS 2023
Accepted: yes
Keywords: generative neural fields; implicit neural representation; model averaging
Abstract: We propose a novel approach to learning the generative neural fields represented by linear combinations of implicit basis networks. Our algorithm learns basis networks in the form of implicit neural representations and their coefficients in a latent space by either conducting meta-learning or adopting auto-decoding paradigms. The proposed method easily enlarges the capacity of generative neural fields by increasing the number of basis networks while maintaining the size of a network for inference to be small through their weighted model averaging. Consequently, sampling instances using the model is efficient in terms of latency and memory footprint. Moreover, we customize denoising diffusion probabilistic model for a target task to sample latent mixture coefficients, which allows our final model to generate unseen data effectively. Experiments show that our approach achieves competitive generation performance on diverse benchmarks for images, voxel data, and NeRF scenes without sophisticated designs for specific modalities and domains.

Generative Neural Fields

Abstract

We propose a novel approach to learning the generative neural fields represented by linear combinations of implicit basis networks. Our algorithm learns basis networks in the form of implicit neural representations and their coefficients in a latent space by either conducting meta-learning or adopting auto-decoding paradigms. The proposed method easily enlarges the capacity of generative neural fields by increasing the number of basis networks while maintaining the size of a network for inference to be small through their weighted model averaging. Consequently, sampling instances using the model is efficient in terms of latency and memory footprint. Moreover, we customize denoising diffusion probabilistic model for a target task to sample latent mixture coefficients, which allows our final model to generate unseen data effectively. Experiments show that our approach achieves competitive generation performance on diverse benchmarks for images, voxel data, and NeRF scenes without sophisticated designs for specific modalities and domains.

1 Introduction

Implicit neural representation (INR) is a powerful and versatile tool for modeling complex and diverse data signals in various modalities and domains, including audio [12], images [35], videos [40], 3D objects [28, 6], and natural scenes. INR expresses a data instance as a function mapping from a continuous coordinate space to a signal magnitude space rather than using a conventional representation on a discrete structured space. In particular, a representation with a continuous coordinate allows us to query arbitrary points and retrieve their values, which is desirable for many applications that only have accessibility to a subset of the target data in a limited resolution. INRs replace the representations based on high-dimensional regular grids, such as videos [9] and 3D scenes [25], with multi-layer perceptrons (MLPs) with a relatively small number of parameters, which effectively memorize scenes.

Generative neural fields aim to learn distributions of functions that represent data instances as a form of neural field. They typically rely on INRs to sample their instances in various data modalities and domains. The crux of generative neural fields lies in how to effectively identify and model shared and instance-specific information. To this end, feature-wise linear modulation (FiLM) [29] and hyper-network (HyperNet) [16] are common methods for modulating representations. FiLM

latent space

coefficients

input coordinates

outputs

=

=

=

=

INR

implicit bases

mNIF

image

voxel

radiance field

-y

(C1,2）→(yR,yG,yB)

(1,∞2,∞3）→(y)

Φ;~ Platent

(x1,22,3)→(yR,yG,yB,ya)

X

{α}

{w@,b@}

adjusts hidden features using an affine transform, which consists of element-wise multiplication and bias addition. Due to the ease of optimization, the effectiveness of FiLM has already been validated in several tasks including 3D-aware generative modeling [6, 7]. However, the operation of FiLM is too restrictive while introducing additional parameters for modulation. On the other hand, HyperNet is more flexible because it directly predicts network parameters. Unfortunately, the direct prediction of model parameters is prone to unstable training and limited coverage of the data distribution generated by the predicted network. Therefore, to reduce solution spaces, HyperNet often employs dimensionality reduction techniques such as low-rank decompositions of weight matrices [35, 12] and combinations of multiple network blocks [16]. However, all these works give limited consideration to the inference efficiency.

Inspired by the flexibility of HyperNet and the stability of FiLM while considering efficiency for inference, we propose a mixture of neural implicit functions (mNIF) to represent generative neural fields. Our approach employs mNIFs to construct generative networks via model averaging of INRs. The mixture components in an mNIF serve as shared implicit basis networks while their mixands define relative weights of the bases to construct a single instance. Our modulation step corresponds to computing a weighted average of the neural implicit functions. The mixture coefficients are optimized by either meta-learning or auto-decoding procedures with signal reconstruction loss. Such a design is effective for maximizing the expressibility of INRs while maintaining the compactness of inference networks. The proposed approach is versatile for diverse modalities and domains and is easy to implement.

Figure 1 demonstrates the overview of the proposed approach, and we summarize the contributions of our work as follows:

We introduce a generative neural field based on mNIF, a modulation technique by a linear combination of NIFs. The proposed approach effectively extracts instance-specific information through conducting meta-learning or applying auto-decoding procedures.

Our modulation allows us to easily extend the model capacity by increasing the number of basis networks without affecting the structure and size of the final model for inference. Such a property greatly improves the efficiency of the sampling network in terms of inference time and memory requirements.

Our model achieves competitive performance on several data generation benchmarks without sophisticated domain-specific designs of algorithm and architecture configuration.

The rest of this paper is organized as follows. We first discuss related works about generative neural fields in Section 2. The main idea of the proposed approach and the training and inference procedures are discussed in Sections 3 and 4, respectively. Section 5 presents empirical quantitative and qualitative results with discussions, and Section 6 concludes this paper.

2 Related Work

Implicit neural representations (INR) are often optimized for representing a single instance in a dataset, and have been extended in several ways to changing activation functions [33, 30], utilizing input coordinate embeddings [36, 26] or taking advantage of a mixture of experts [37, 23, 24]. Since the INR framework is difficult to generalize a whole dataset, generative neural fields have been proposed to learn a distribution of functions based on INRs, each of which corresponds to an example in the dataset.

Generative manifold learning (GEM) [12] and generative adversarial stochastic process (GASP) [14] are early approaches to realizing generative neural fields. GEM implements the generative neural field using the concept of hyper-network (HyperNet) [16], which predicts a neural network representing an instance. Training GEM is given by the auto-decoding paradigm with manifold learning. However, GEM involves extra computational cost because the manifold constraint requires accessing training examples. GASP also employs a HyperNet for sampling instances, and its training utilizes a discriminator taking an input as a set of coordinate and feature pairs. Since the output of the discriminator should be permutation invariant, it follows the design of the network for the classification of point cloud data. However, training GASP is unstable and the discriminator design is sometimes tricky, especially in NeRF. Functa [13] proposes the generative neural fields with SIREN [33] modulated by FiLM [29] and introduces a training strategy based on meta-learning. However, meta-learning is computationally expensive due to the Hessian computation. On the other hand, diffusion probabilistic fields (DPF) [41] proposes a single-stage diffusion process with the explicit field parametrization [19]. However, its high computational cost hampers the applicability to complex neural fields such as NeRF. HyperDiffusion [15], which is concurrent to our work, presents a framework directly sampling the entire INR weight from the learned diffusion process instead of exploiting latent vectors for modulating INR weights. However, this work demonstrates instance generation capability only on voxel domains, not on images.

Model averaging [38] is often used for enhancing performance in discriminative tasks without increasing an inference cost. Our approach applies model averaging to generative models and verifies its effectiveness in generative neural fields based on INRs. Note that we are interested in efficient prediction by learning a compact representation through a weighted mixture; our method jointly trains basis models and their mixture coefficients.

3 Generative Neural Fields with a Mixture of Neural Implicit Functions

This section describes how to define generative neural fields. The formulation with a mixture of neural implicit functions (mNIF) enforces implicit basis networks to represent shared information across examples and allows latent mixture coefficients to encode instance-specific information through a linear combination of basis networks. We show that generative neural field via mNIF is effectively optimized to predict the latent vector for the construction of models generating high-quality samples.

3.1 Implicit Neural Representations

Implicit neural representation (INR) expresses a data instance using a function from an input query, x ∈ N d , to its corresponding target value, y ∈ R k . Since each input coordinate is independent of the others, a mapping function parametrized by θ , f θ ( · ) , is typically represented by a multi-layer perceptron (MLP) with L fully connected (FC) layers, which is given by

where f ( i ) for ∀ i ∈ { 1 , 2 , . . . , L } denote FC layers while f (0) and f ( L +1) indicate the input and output layers, respectively. Each layer f ( i ) performs a linear transform on its input hidden state h ( i ) and then applies a non-linear activation function to yield the output state h ( i +1) , expressed as

where σ ( · ) is an activation function and W ( i ) ∈ R W × W and b ( i ) ∈ R W are learnable parameters. The operations of the input and output layers are defined as h 1 = f (0) ( x ) and ˆ y = f ( L +1) ( h ( L ) ) ,

respectively. Consequently, a collection of learnable parameters θ in all layers is given by

where W (0) ∈ R W × d , b (0) ∈ R W , W ( L +1) ∈ R k × W , and b ( L +1) ∈ R k . For INR, the mean squared error between a prediction ˆ y and a target y is typically adopted as a loss function, L ( · , · ) , which is given by

Among the variations of INRs, we employ SIREN [33], which adopts the sinusoidal function as an activation function and introduces a sophisticated weight initialization scheme for MLPs. Supposing that SIREN is defined with an MLP with L hidden layers, the i th layer of SIREN is given by

where w 0 is a scale hyperparameter to control the frequency characteristics of the network. The initialization of SIREN encourages the distribution of hidden sine activations to follow a normal distribution with a standard deviation of 1.

3.2 Mixtures of Neural Implicit Functions

We propose generative neural fields based on mNIF, which is an extension of the standard INR for modulating its model weight. We define a set of NIFs, which is used as basis networks, and construct a generative neural field using a mixture of the NIFs. This is motivated by our observation that a generative neural field is successfully interpolated using multiple basis networks and model averaging works well for model generalization [38].

The operation in each layer of our mixture model is given by

where g ( i ) m ( h ( i ) ) = W ( i ) m h ( i ) + b ( i ) m is the i th -layer operation of the m th neural implicit function, and α ( i ) m is a mixture coefficient of the same layer of the same neural implicit function. Note that modulating the network is achieved by setting the mixand values, { α ( i ) m } . Similar to the definition of θ described in Eq. (3), the parameter of each mixture is given by

The operation in the resulting INR obtained from model averaging is given by

where

All the learnable parameters in the mixture of NIFs is defined by { θ 1 , . . . , θ M } . The remaining parameters are mixture coefficients, and we have the following two options to define them: (i) sharing mixands for all layers α ( i ) m = α m and (ii) setting mixands to different values across layers. The first option is too restrictive for the construction of INRs because all layers share mixture coefficients while the second is more flexible but less stable because it involves more parameters and fails to consider the dependency between the coefficients in different layers. Hence, we choose the second option but estimate the mixture coefficients in a latent space, where our method sets the dimensionality of the latent space to a constant and enjoys the high degree-of-freedom of layerwise coefficient setting. To this end, we introduce a projection matrix T to determine the mixture coefficients efficiently and effectively, which is given by

where φ ∈ R H denotes a latent mixture coefficient vector. This choice compromises the pros and cons of the two methods. Eventually, we need to optimize a set of parameters, θ all glyph[defines] { θ 1 , . . . , θ M , T } for a generative neural field, and a latent vector φ to obtain an instance-specific INR.

3.3 Model Efficiency

The proposed modulation technique is based on model averaging and has great advantages in the efficiency during training and inference. First, the computational complexity for each input coordinate is small despite a large model capacity given by potentially many basis networks. Note that, even though the number of neural implicit basis function in a mixture increases, the computational cost per query is invariant by Eq. (8), which allows us to handle more coordinates queries than the models relying on FiLM [29]. Second, the model size of our generative neural fields remains small compared to other methods due to our model averaging strategy, which is effective for saving memory to store the neural fields.

When analyzing the efficiency of generative neural fields, we consider two aspects; one is the model size in the context adaptation stage, in other words, the number of all the learnable parameters, and the other is the model size of the modulated network used for inference. By taking advantage of the modulation via model averaging, our approach reduces memory requirement significantly both in training and inference.

4 Learning Generative Neural Fields

The training procedure of the proposed approach is composed of two stages: context adaptation followed by task-specific generalization. In the context adaptation stage, we optimize all the learnable parameters for implicit basis functions and their coefficients that minimize reconstruction error on examples in each dataset. Then, we further learn to enhance the generalization performance of the task-specific generative model, where a more advanced sampling method is employed to estimate a context vector, i.e. , a latent mixture coefficient vector. This two-stage approach enables us to acquire the compact representation for neural fields and saves computational costs during training by decoupling the learning neural fields and the training generative model. We describe the details of the two-stage model below.

4.1 Stage 1: Training for Context Adaptation

The context adaptation stage aims to learn the basis networks and the latent mixture coefficient vector, where the mixture model minimizes the reconstruction loss. This optimization is achieved by one of two approaches: meta-learning or auto-decoding paradigm.

With meta-learning, we train models in a similar way as the fast context adaptation via meta-learning (CAVIA) [42]; after a random initialization of a latent vector φ j for the j th instance in a batch, we update φ j in the inner-loop and then adapt the shared parameters θ based on the updated φ j . Since this procedure requires estimating the second-order information for the gradient computation of θ , the computational cost of this algorithm is high. Algorithm 1 presents this meta-learning procedure.

We observe that random initialization of a latent vector during meta-learning is more stable than the constant initialization originally proposed in CA VIA when training mNIF with a large number of mixture components or the high dimensionality of latent vectors. We intend to diversify INR weights at the start phase of meta-training, ensuring that not all samples are represented with identical INR weights.

Auto-decoding paradigm [12, 28] is a variant of multi-task learning, where each instance is considered a task. Contrary to the meta-learning, we do not reset and initialize the latent vector φ j for each instance but continue to update over multiple instances. The optimization procedure via auto-decoding is presented in Algorithm 2.

4.2 Stage 2: Optimizing for Task-Specific Generalization

The aforementioned context adaptation procedure focuses only on the minimization of reconstruction error for training samples and may not be sufficient for the generation of unseen examples. Hence,

Algorithm 1 Meta-learning with mNIF

Algorithm 2 Auto-decoding with mNIF

we introduce the sampling strategy for the context vectors φ and customize it to a specific target task. To this end, we adopt the denoising diffusion probabilistic model (DDPM) [18], which employs the residual MLP architecture introduced in Functa [13].

5 Experiments

This section demonstrates the effectiveness of the proposed approach, referred to as mINF, and discusses the characteristics of our algorithm based on the results. We run all experiments on the Vessl environment [2], and describe the detailed experiment setup for each benchmark in the appendix.

5.1 Datasets and Evaluation Protocols

We adopt CelebA-HQ 64 2 [21], ShapeNet 64 3 [8] and SRN Cars [34] dataset for image, voxel and neural radiance field (NeRF) generation, respectively, where 64 2 and 64 3 denotes the resolution of samples in the dataset. We follow the protocol from Functa [13] for image and NeRF scene and generative manifold learning (GEM) [12] for voxel.

We adopt the following metrics for performance evaluation. To measure the reconstruction quality, we use mean-squared error (MSE), peak signal-to-noise ratio (PSNR), reconstruction Fréchet inception distance (rFID), reconstruction precision (rPrecision) and reconstruction recall (rRecall). In image generation, we use Fréchet inception distance (FID) score [17], precision, recall [32, 27] and F1 score between sampled images and images in a train split. Voxel generation performance is evaluated by coverage and maximum mean discrepancy (MMD) metrics [1] on a test split. In NeRF scene generation, we use FID score between rendered images and images in a test split for all predefined views for evaluation. To compare model size of algorithms, we count the number of parameters for training and inference separately; the parameters for training contain all the weights required in the training procedure, e.g. , the parameters in the mixture components and project matrix for our algorithm, while the model size for inference is determined by the parameters used for sampling instances. For the evaluation of efficiency, we measure the number of floating point operations per second (FLOPS), latency in terms of frames per second (fps), and the amount of memory consumption for a single sample.

5.2 Main Results

We present quantitative and qualitative results, and also analyze the effectiveness of the proposed approach in comparison to the previous works including Functa [13], GEM [12], GASP [14], and DPF [41]. Note that we adopt the model configurations with the best generation performance for the other baselines.

5.2.1 Quantitative Performance

We compare results from our approach, mNIF, with existing methods in terms of four aspects: reconstruction accuracy, generation quality, model size, and inference efficiency. We present quantitative results from two configurations of our model, mNIF (S) and mNIF (L), which correspond to small and large networks, respectively.

As shown in Tables 1, 2, and 3, our approach consistently achieves state-of-the-art reconstruction and generation quality on the image, voxel, and NeRF scene datasets except the reconstruction on the voxel dataset, in which our model is ranked second. Moreover, mNIF is significantly more efficient than other methods in terms of model size and inference speed in all cases. Considering the model size and inference speed, the overall performance of mNIF is outstanding.

Note that the light configuration of our model denoted as mNIF (S) is most efficient in all benchmarks and also outperforms the baselines in voxel and NeRF. In SRN Cars, mNIF (S) demonstrates huge benefit in terms of efficiency compared to Functa; 199 times less FLOPS, 49 times more fps, and 22 times less memory in single view inference. In the image domain, the performance of mNIF (S) is comparable to GASP in terms of F1 but worse in FID. We conjecture that FID metric overly penalizes blurry images, which is discussed in several works [12, 13, 31, 20].

5.2.2 Qualitative Generation Performance

Figure 2 illustrates generated samples for qualitative evaluation. We visualize the results from our models on each benchmark together with GASP on CelebA-HQ 64 2 and Functa on SRN Cars to compare results in the image and NeRF scene tasks, respectively. In the image domain, our model, mNIF (L), generates perceptually consistent yet slightly blurry samples compared to ground-truths, while the examples from GASP with a similar FID to ours have more artifacts. In the NeRF scene, rendered views from our model, mNIF (S), and Functa exhibit a similar level of image blur compared to ground-truths. Since both methods use vanilla volumetric rendering, adopting advanced rendering techniques, such as hierarchical sampling [25, 6] and anti-aliasing methods [4, 3] for NeRF scenes, would improve rendering quality.

mNIF (L)

GASP [14]

Ground Truth

(a) CelebA-HQ

64

2

mNIF (L)

Ground Truth

(b) ShapeNet

64

3

mNIF (S)

Functa [13]

Ground Truth

(c) SRN Cars

5.3 Analysis

We analyze the characteristics of our trained model for better understanding via latent space exploration and various ablation studies.

5.3.1 Latent Space Interpolation

Figure 3 illustrates the results of interpolation in the latent space; the corner images are samples in the training dataset and the rest of the images are constructed by bilinear interpolations in the latent space. These images demonstrate the smooth transition of sampled data, confirming that the latent vectors learned at the first stage manage to capture the realistic and smooth manifold of each dataset.

5.3.2 Configurations of Mixture Coefficients in mNIF

Table 4(a) examines the performance by varying the types of mixture coefficients: i) shared mixture coefficients across all layers, i.e. , α ( i ) m = α m , ii) layer-specific mixture coefficients, and iii) layerspecific mixture coefficients projected from a latent vector as shown in Eq. (10). Among the three mixture settings, the last option yields the best performance and is used for our default setting.

Table 4(b) presents the effect of the number of mixtures, denoted by M . Reconstruction performance generally improves as the number of mixtures increases, but such a tendency saturates at around M = 256 . We also measure the precision and recall between the ground-truth and reconstructed samples denoted by rPrecision and rRecall, respectively. According to our observation, increasing the size of the mixture improves the recall more than it does the precision.

Table 4(c) shows the effect of the latent dimension on the reconstruction performance of mNIF. Increasing the latent dimension H leads to improving the reconstruction performance in both the train and test splits due to the degree-of-freedom issue.

(a) CelebA-HQ

64

2

(b) ShapeNet

64

3

(c) SRN Car

5.3.3 Diversity in Learned Neural Bases

To investigate the diversity of the basis functions per layer, Figure 4 visualizes the absolute value of the pairwise cosine similarity between the weight matrices of the basis models. In the cosine similarity matrices, an element with a small value indicates that the corresponding two neural basis functions are nearly orthogonal to each other, which serves as evidence of diversity in the learned basis functions. The visualization also reveals the following two characteristics. First, the lower layers are more correlated than the upper ones partly because low-level features inherently have less diversity. Second, a small subset of neural bases have high correlations with others. We hypothesize that the concentration of high correlation among the small subset of neural bases could be mitigated by introducing a loss function enforcing orthogonality.

5.3.4 Context Adaptation Strategies

We perform the ablation study with respect to diverse optimization strategies and demonstrate the resulting reconstruction performance in Table 5. We train small mNIFs with ( L, W, M, H ) = (2 , 64 , 256 , 256) on the CelebA-HQ 64 2 dataset. It is clear that the meta-learning approach using second-order gradient computation yields favorable results compared to other options. Running more inner-loop iterations marginally improves performance, but at the cost of additional computational complexity in time and space. Therefore, we set the number of inner-loop iterations to 3 for all meta-learning experiments.

Interestingly, the auto-decoding strategy surpasses meta-learning with first-order gradient computation. Note that auto-decoding is more efficient than other methods in terms of both speed and memory usage when evaluated under the same optimization environment. This efficiency of auto-decoding is

layer 1

layer 2

layer 3

layer 4

layer 5

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

particularly beneficial for NeRF scene modeling, which requires a large number of input coordinate queries. Our model runs on a single GPU for training on 8 NeRF scenes per batch while Functa needs 8 GPUs for the same purpose.

5.3.5 Longer Training of Context Adaptation

Table 6 shows the performance of our approach, mNIF, with respect to the number of epochs for the first stage training under the identical stage 2 training environment. When we train mNIF (L) with ( L, W, M, H ) = (5 , 128 , 256 , 512) on the CelebA-HQ 64 2 dataset, reconstruction performance is positively correlated with training length but longer training may be prone to overfitting considering the trend of generalization performance.

6 Conclusion

We presented a novel approach for generative neural fields based on a mixture of neural implicit functions (mNIF). The proposed modulation scheme allows us to easily increase the model capacity by learning more neural basis functions. Despite the additional basis functions, mNIF keeps the network size for inference compact by taking advantage of a simple weighted model averaging, leading to better optimization in terms of latency and memory footprint. The trained model can sample unseen data in a specific target task, where we adopt the denoising diffusion probabilistic model [18, 13] for sampling context vectors φ . We have shown that our method achieves competitive reconstruction and generation performance on various domains including image, voxel data, and NeRF scene with superior inference speed.

The proposed method currently exhibits limited scalability beyond fine-grained datasets. We hypothesize that the latent space in mNIF trained on a dataset with diverse examples, such as CIFAR-10, is not sufficiently smooth compared to other datasets to be generalized by adopting a diffusion process. A possible solution to these challenges is to incorporate local information into the architecture based on neural implicit representations, as proposed in recent research [39, 5]. We reserve this problem for future work.

Acknowledgments and Disclosure of Funding

This work was partly supported by the National Research Foundation of Korea (NRF) grant [No. 2022R1A2C3012210, No.2022R1A5A708390811] and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant [NO.2021-0-0268, No. 2021-0-01343], funded by the Korea government (MSIT).

References

P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and generative models for 3d point clouds. arXiv preprint arXiv:1707.02392 , 2017.

J. An. Model development with vessl, 2023. URL https://www.vessl.ai/ . Software available from vessl.ai.

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV , 2021.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-NeRF: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 , 2021.

M. Bauer, E. Dupont, A. Brock, D. Rosenbaum, J. R. Schwarz, and H. Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130 , 2023.

E. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR , 2021.

E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. D. Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, T. Karras, and G. Wetzstein. Efficient geometry-aware 3d generative adversarial networks. In CVPR , 2022.

A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015.

H. Chen, B. He, H. Wang, Y. Ren, S.-N. Lim, and A. Shrivastava. NeRV: Neural representations for videos. In NeurIPS , 2021.

Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In CVPR , 2019.

D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In ICLR , 2016.

Y. Du, K. M. Collins, J. B. Tenenbaum, and V. Sitzmann. Learning signal-agnostic manifolds of neural fields. In NeurIPS , 2021.

E. Dupont, H. Kim, S. M. A. Eslami, D. J. Rezende, and D. Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In ICML , 2022.

E. Dupont, Y. W. Teh, and A. Doucet. Generative models as distributions of functions. In AISTATS , 2022.

Z. Erkoç, F. Ma, Q. Shan, M. Nießner, and A. Dai. HyperDiffusion: Generating implicit neural fields with weight-space diffusion. In ICCV , 2023.

D. Ha, A. Dai, and Q. V. Le. Hypernetworks. In ICLR , 2016.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In ICLR , 2022.

S. Jung and M. Keuper. Internalized biases in fréchet inception distance. In Workshop on Distribution Shifts, NeurIPS 2021 , 2021.

T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Progressive growing of gans for improved quality, stability, and variation. In ICLR , 2018.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Z.-H. Lin, W.-C. Ma, H.-Y. Hsu, Y.-C. F. Wang, and S. Wang. NeurMiPs: Neural mixture of planar experts for view synthesis. In CVPR , 2022.

Z. MI and D. Xu. Switch-NeRF: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In ICLR , 2023.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. In SIGGRAPH , 2022.

M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, and J. Yoo. Reliable fidelity and diversity metrics for generative models. In ICML , 2020.

J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR , 2019.

E. Perez, F. Strub, H. de Vries, V . Dumoulin, and A. Courville. FiLM: Visual reasoning with a general conditioning layer. In AAAI , 2018.

S. Ramasinghe and S. Lucey. Beyond Periodicity: Towards a unifying framework for activations in coordinate-mlps. In ECCV , 2022.

A. Razavi, A. van den Oord, and O. Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NeurIPS , 2019.

M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. In NeurIPS , 2018.

V. Sitzmann, J. N. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In NeurIPS , 2019.

V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene Representation Networks: Continuous 3d-structure-aware neural scene representations. In CVPR , 2019.

I. Skorokhodov, S. Ignatyev, and M. Elhoseiny. Adversarial generation of continuous images. In CVPR , 2021.

M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS , 2020.

P. Wang, Z. Fan, T. Chen, and Z. Wang. Neural implicit dictionary via mixture-of-expert training. In ICML , 2022.

M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML , 2022.

Z. Wu, Y. Jin, and K. M. Yi. Neural fourier filter bank. In CVPR , 2023.

S. Yu, J. Tack, S. Mo, H. Kim, J. Kim, J.-W. Ha, and J. Shin. Generating videos with dynamicsaware implicit generative adversarial networks. In ICLR , 2022.

P. Zhuang, S. Abnar, J. Gu, A. Schwing, J. M. Susskind, and M. A. Bautista. Diffusion probabilistic fields. In ICLR , 2023.

L. M. Zintgraf, K. Shiarlis, V . Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta-learning. In ICML , 2019.

A Implementation Details

A.1 Benchmark

A.1.1 Images

We use CelebA-HQ dataset [21] for our work. We divide entire images into 27,000 images for train and 3,000 images for test split which is provided by Functa [13]. We use the pre-process dataset from the link 1 . To measure the quality of generated images, we compute Fréchet inception distance (FID) score [17], precision and recall [32, 27] between sampled images and images in a train split.

A.1.2 Voxels

We utilize the ShapeNet dataset [8] for 3D voxel generation from IM-Net [10]. It has 35,019 samples in the train split and 8,762 samples in the test split. The dataset contains a voxel with 64 3 and 16,384 points extracted near the surface. To evaluate the generative model, we sample 8,764 shapes, which is the size of the test set, and generate all points in the voxel. Then, we compute 2,048 dimension mesh features from the voxel following the protocol in [10]. Generation performance with coverage and maximum mean discrepancy (MMD) metrics [1] is evaluated using Chamfer distance on test split. We follow the evaluation procotol from generative manifold learning (GEM) [12].

A.1.3 Neural Radiance Field (NeRF)

We adopt SRN Cars dataset [34] for NeRF modeling. SRN cars dataset has a train, validation, and test split. Train split has 2,458 scenes with 128 × 128 resolution images from 50 random views. Test split has 704 scenes with 128 × 128 images from 251 fixed views in the upper hemisphere.

We adopt the evaluation setting of NeRF scene from functa which utilizes 251 fixed views from the test split. Consequently, we can compute FID score based on rendered images and images in a test split with equivalent view statistics.

A.2 Implicit Neural Representation with Mixture of Neural Implicit Functions

A.2.1 Images

We train our mixtures of Neural Implicit Functions (mNIF) on images from train split of CelebA-HQ 64 2 dataset. Generative implicit neural representation (INR) for image takes 2D input coordinates ( x 1 , x 2 ) and returns 3D RGB values ( y R , y G , y B ) . During optimization, we use dense sampling, which queries all input coordinates ( 4096 = 64 × 64 ) to the network inputs, for image benchmark. We use mNIF configuration with the number of hidden layers L = 5 , the channel dimension of hidden layers W = 128 , the number of mixtures M = 384 and the latent vector dimension H = 512 . throughout our experiments. We use meta learning for training the mNIF on CelebA-HQ 64 2 . We use Adam [22] optimizer with the outer learning rate glyph[epsilon1] outer = 1 . 0 × 10 -4 and batch size 32. We use a cosine annealing learning rate schedule without a warm-up learning rate. We take the best mNIF after optimizing the model over 800 epochs with inner learning rates glyph[epsilon1] inner ∈ { 10 . 0 , 1 . 0 , 0 . 1 } . The configuration of mNIF (S) is ( L, W, M, H ) = (4 , 64 , 256 , 1024) and mNIF (L) (5 , 128 , 384 , 512) . The best performed model uses glyph[epsilon1] inner = 1 . 0 .

A.2.2 Voxels

We train our model on ShapeNet 64 3 dataset. Our generative neural field for voxel takes 3D input coordinates ( x 1 , x 2 , x 3 ) and returns y σ indicating whether queried coordinate are inside or outside. For model optimization, we use sub-sampling with 4,096 points in a voxel with 64 3 from the points. Then, we use all 16,384 points for constructing latent vectors. Note that despite relying on sub-sampling, the 16,384 points sampled from the near surface of a voxel are dense enough to effectively represent its content within the voxel. We use meta learning for training the mNIF on ShapeNet 64 3 . We use mNIF (S) and (L) configuration with ( L, W, M, H ) = (4 , 64 , 256 , 512) and ( L, W, M, H ) = (5 , 128 , 512 , 1024) . We take the best mNIF after optimizing the model over 400 epochs with inner learning rates glyph[epsilon1] inner ∈ { 10 . 0 , 1 . 0 , 0 . 1 } .

A.2.3 NeRF Scenes

We train our model on SRN Cars dataset. We use a simplified NeRF protocol without a view dependency on neural fields. So, generative neural field returns 4D outputs, RGB values and density ( y R , y G , y B , y σ ) , with given 3D input coordinates ( x 1 , x 2 , x 3 ) . We use no activation function for RGB output and exponential linear unit (ELU) [11] for density output as suggested in Functa. During optimization, we use sparse sampling with 32 views per scene, 512 pixels per view and 32 points per ray for each scene. We use auto-decoding procedure for efficient optimization instead of meta-learning procedure to avoid the cost of second-order gradient computation. In auto-decoding procedure, we simply harvest latent vectors for the dataset jointly trained during the optimization procedure. We use mNIF (S) configuration with ( L, W, M, H ) = (4 , 64 , 256 , 128) . We use Adam [22] optimizer with outer learning rate glyph[epsilon1] outer = 1 . 0 × 10 -4 and batch size 8. We use cosine annealing without a warm-up learning rate for the learning rate schedule. We use inner learning rate glyph[epsilon1] inner = 1 . 0 × 10 -4 and use weight decay on latent vector φ j to regularize the latent space as in auto-decoding framework. We take the best mNIF after optimizing the model over 2000 epochs.

A.2.4 Weight Initialization of mNIF

Weight initialization is crucial for optimizing INR, such as SIREN [33]. We enforce mixture coefficients at the start as the inverse of the number of mixtures α ( i ) m = 1 /M to give the equivalent importance for all implicit bases in mixtures. For latent mixture coefficients, we set bias of projection matrix into 1 /M to satisfy above condition.

A.3 Denoising Diffusion Process on Latent Space

We train unconditional generation models for latent space for image, voxel and NeRF scene for generating latent coefficients vector. We use the equivalent setting for the training diffusion process throughout all experiments and benchmarks. We implement DDPM [18] on latent vector space based on residual MLP [13] from Functa as a denoising model. The channel dimension of residual MLP is 4096 and the number of block is 4. We use Adam optimizer and set learning rate 1 . 0 × 10 -4 and batch size 32. We train model 1000 epochs on each dataset and schedule the learning rate with cosine annealing without a warm-up learning rate. We use T = 1000 time steps for diffusion process.

A.4 Libraries and Code Repository

Our implementation is based on PyTorch 1.9 2 with CUDA 11.3 and PyTorch Lightning 1.5.7. 3 Our major routines are based on the code repositories for Functa, 4 SIREN, 5 latent diffusion model, 6 guided diffusion 7 and HQ-Transformer. 8 For inference efficiency, we count FLOPS with the fvcore library 9 and check inference time and memory consumption on GPU by PyTorch Profiler. 10 The efficiency of mNIF are evaluated on NVIDIA Quadro RTX 8000.

B Analysis

B.1 Visualization

We provide additional generation examples in Figures 5 and 6 and additional interpolation results of the proposed algorithm in Figures 7 and 8.

C Limitations

As discussed in concolusion, we observe that our model has limited scalability beyond fine-grain dataset; our model works well on reconstruction while not on generation on CIFAR10. To analyze the generation on CIFAR10, we introduce an alternative sampling scheme to replace the latent diffusion model described in our main paper. Drawing inspiration from the sampling strategy in generative manifold learning (GEM), we employed latent space interpolation between two samples: one random sample from the training set and another random sample from its neighborhood. The resulting context vector, is formed by a simple interpolation:

where φ i denotes context vector for i -th sample in training set, N ( i ) is a set of indices of the neighborhood of sample and is sampled from the uniform distribution ranged between 0 to 1.

This sampling by interpolation scheme achieves 59.67 (FID) while sampling from diffusion model does 84.64 (FID). From the results, two major observations can be made: Sampling through interpolation yields a superior FID score than the latent diffusion model, suggesting issues with the latter's interaction with context vectors. The FID score for interpolation sampling, while better than diffusion, remains suboptimal. The observation that the context vector interpolation is not fully effective for the CIFAR10 model indicates a less smooth latent space in CIFAR10 compared to other datasets. This disparity could be impacting the generation quality.

We hypothesize that the limitations of the SIREN [33] architecture might be at the root of these issues. An INR work [39] categorizes SIREN as an INR with predominantly frequency encodings and a lack of space resolution, leading to compromised content quality. A separate study [5] underlines this, stating SIREN's unsatisfactory generation performance (FID 78.2) when applied to CIFAR10. This research introduces a new SIREN architecture where biases in hidden layers of INR are varying with respect to spatial resolution. We anticipate that adopting an INR with spatial information as an alternative to SIREN will better accommodate diverse datasets, given that our approach-a linear combination of INR weights-is broadly compatible with various INR methods.

D Broader Impacts

The proposed image model is trained on the CelebA-HQ 64 2 dataset, which raises the possibility of unintended bias in the generated human face images due to the biases present in the dataset. Additionally, when extending the generative neural field model to large-scale datasets, there is a potential for generating unintended biased contents, although the current version of the work primarily focuses on small-scale datasets. We acknowledge that these concerns are important themes that should be thoroughly discussed in the future.

(a) CelebA-HQ

64

2

3

(b) ShapeNet

64

(a) View 0

(b) View 32

(c) View 64

(d) View 96

(e) View 128

(f) View 160

(g) View 192

(h) View 224

(i) View 250

(a) CelebA-HQ

64

2

(b) ShapeNet

64

3

(intra-class)

3

137421 947 127

(c) ShapeNet

64

(inter-class)

(a) View 0

(b) View 32

(c) View 64

(d) View 96

(e) View 128

(f) View 160

(g) View 192

(h) View 224

(i) View 250

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new representation of neural fields based on a mixture of neural fields. Neural fields is another name for implicit neural representation (INR) which is a representation of data points (like images or 3d scenes), that maps an input of 2d or 3d position into the output of the RGB color and/or shape information.  \n\nThe paper builds on previous work like GASP [12], Functa [13] and GEM [1] and proposes a new representation on which the generative model is trained. As in [13] the generative model is a diffusion model and the novelty here is the representation which oi based on a mixture model.The mixture model representation proposed is implemented as a shared set of INRs which weights are averaged with a vector of mixture weights that are optimized per data-point.\n\nAfter describing the model, the paper demonstrates its effectiveness in different data modalities like images (Celeba-HQ), 3D shapes (ShapeNet) and 3D scenes (SRN Cars), where the model performs well compared to previous approaches.  Then some additional ablation and analysis are presented.\n\nSTRENGTHS:\nThe main strength of this paper is the novel INR representation proposed. This representation follows a recent line of work on the topic, that aims to leverage the advantages of INRs into the setup of training generative models on different data modalities. A main challenge there is capturing 3D data in an efficient way, which is still not acheivable in standard direct methods.  \nThe proposed model is an interesting direction with a potential to contribute to the research of this topic.\n\nWEAKNESSES:\nThe main weaknesses of this paper are the following:\n1. The description of the model and the comparisons to previous work is lacking. Although some of the aspects are simple and clearly presented and the comparison to previous models is extensive and contains many different metrics, there still remains some important aspects that are not sufficiently described (see questions below).\n2. After describing the model, the authors \"jump\" to final metrics like PSNR and FIDs. Given that the main contribution of this work is the mixture representation itself, it is essential to show what this representation captures. Figure 5 is a good start in this direction, but what is still missing is some understanding of what the different components in the mixture are doing, and how they are combined. The authors could, for example, show the reconstructed images of different components in isolation as the number of components grow. Another component which is directly used in the model without investigation is the hierarchical mixture (mixture of every layer in the network). How does this affect the components? How are the differences in lower layer components manifested compared to differences in higher level components?\n\nQUESTIONS:\n1. The meta learning algorithm presented in algorithm 1:  why is $\\phi$ randomaly initialized at every iteration? This is different than MAML that learns an initialization, and CAVIA [17] (which the authors state as reference for this) that keeps the initialization constant. This is an interesting difference that should be discussed. \n2. Algorithm 2: The pseudo code is not clear about the indexes of $\\phi$ how are they being updated through the iterations?\n3. One aspect that is missing from all tables and comparisons is the size of the representation. For the proposed model this would be the number of mixture components times the number of layers. This is an important aspect of the model since it defines the rate-distortion tradeoff. The way the authors count parameters (learnable vs. inference) is interesting and of significant value, but adding the size of the representation is essential.\n4. How are the previous models chosen for comparison? These models usually contain several options with different sizes of the network and the representation, where the reconstruction quality usually grows with the size of the network.\n5. Line 115: What exactly is invariant to what?\n6. The ablation in the first section of 5.3 is not very useful. As stated above, it would be better to try to understand  what every component in the mixture captures. \n6. In figure 5, I'm not sure I understand what the plots show. Are the matrices sorted somehow?\n\nLIMITATIONS:\nThe main limitation that is missing from the discussion in the paper is the relation of the quality of representation and its size (see weakness #2 and question #3).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with the problem of learning the latent space of data through the concept of meta-learning INRs (or neural fields). In contrast to prior works that consider either the adaptation of whole neural field parameters or some linear modulation parameters, the proposed method considers some basis neural field parameters as the shared parameters and adapts to coefficients for this basis to be adapted. The proposed method better encodes the given data of diverse modality, compared with prior INR-based data encoding scheme.\n\nSTRENGTHS:\n- The observation that the linear combination of neural fields as basis function parameters is quite interesting and novel.\n- The quantitative results are better than prior works.\n- The paper is generally well-written and easy to follow, while some important details are missing in the current status (see the weakness below)\n\nWEAKNESSES:\n- For generating new data from the proposed framework, specification of the latent sampling strategy (e.g., what is a prior distribution of $\\phi$) is crucial, while the paper only states \"off-the-shelf sampling method\" in L179. Explaining how the sampling is performed would make the paper much stronger. \n- I guess this method requires much more GPU memory, as it uses multiple basis neural field parameters with the (second-order) gradient-based meta-learning method. In particular, this method requires about 10 times more parameters than Functa.\n- The experiments are mostly conducted under fine-grained datasets; I wonder if the method can be worked well with more complex datasets (e.g., CIFAR-10 or ImageNet-64x64) compared with other methods.\n\nQUESTIONS:\n- Is there any constraints (or ranges) for the coefficients $\\alpha_i$s?\n\nLIMITATIONS:\nThe paper addresses the limitations and negative social impacts in the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to increase the learning capacity of implicit neural networks (INRs). The idea is to define a set of basis implicit functions that can be mixed to produce an instance-specific INR for each instance in a dataset. Notably, the approach does not affect the inference performance as once an instance-specific INR is produced, the cost of coordinate sampling is the same as with any other INR. This method allows implicit neural networks to be generalized on entire datasets rather than overfitting to a single instance while still demonstrating high fidelity of samples. The training procedure is split into two steps, the first step being a meta learning or auto decoding problem to fit the basis INRs with a reconstruction loss to the training data, and the second step is training a latent model to generate samples that determine the mixing coefficients. The paper demonstrates solid results and evaluation on different datasets of varying modalities including images, voxels and NeRFs.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The methodology is well explained and is simple but effective. The inference performance is significantly better than three state-of-the-art methods considered, and so are the extensive generation metrics like FID, precision, recall, coverage etc. The methodology and design choices are comprehensively evaluated on multiple datasets, and modalities with ablation studies.\n\nWEAKNESSES:\nThe paper makes a solid contribution and I do not see any major flaws.\n\nQUESTIONS:\nTypos:\nLine 108: resepctively -> respectively\nLine 177: requires -> require\n\nI was wondering if the authors could add more interpolation results as in Figure 4, perhaps in the supplementary material. In particular, it would be interesting to see how interpolation works across ShapeNet categories.\n\nLIMITATIONS:\nYes, limitations and societal impact are sufficiently discussed in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

comments: SUMMARY:
The paper proposes a mixture of neural implicit functions for generating neural fields via model averaging of implicit representations trained through context adaptation (meta-learning/auto-decoding paradigm) and subsequently task-specific generalization (additional sampling strategy with respect to the latent mixture coefficient vector).

STRENGTHS:
This well-written paper clearly elucidates its contributions with convincing results.

WEAKNESSES:
- The authors are requested to discuss limitations within the bounds of the main paper.
- Minor Corrections
    - Line 108, resepctively → respectively
    - Line 226, the both → both the
    - Line 278, effective learn → effectively learns
    - Caption of Figure 1 ideally should elucidate the concept of neural radiance fields.

QUESTIONS:
Experiments lack quantitative evaluation/baseline. The authors are requested to clarify/highlight baseline models in proposed evaluation protocols.

LIMITATIONS:
- The proposed approach does not scale.
- The current model exhibits aliasing artifacts in the neural radiance field scene. 
- There’s potential for bias within the system.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an approach for learning generative neural fields using linear combinations of implicit basis networks, called Mixture of Neural Implicit Functions (mNIF). The basis networks and their coefficients in mNIF can be learned through meta-learning or auto-decoding techniques. By this means, the capacity of generative neural fields can be increased by adding more basis networks while maintaining the size of each instance. The optimized network is further fine-tuned to make generalized predictions by optimizing with respect to latent vectors, enabling the representation of unseen data. Experimental results demonstrate performance across diverse benchmarks.\n\nSTRENGTHS:\n+ The paper demonstrates strong writing, with clear notation and intuitively-rendered figures. The proposed method is well-motivated and easy to understand.\n\n+ The experiments, although conducted on simple datasets like CelebA and Shapenet, effectively validate the method's effectiveness and parameter efficiency in both 2D and 3D domains. The proposed method achieves competitive performance in generating Implicit Neural Representations (INRs), surpassing the performance of all compared baselines according to metrics. The interpolation results show promise.\n\nWEAKNESSES:\n- Some important related work and baselines are missing, including three papers (references [1][2][3]) that explore the idea of using Mixtures of Experts (MoE) in INRs, as well as reference [4] as an additional image generation baseline and reference [5] as a strong baseline for shape generation. As a result, the proposed method does not appear entirely novel in the field and resembles existing latent code-based implicit representations, differing primarily in the incorporation of latent information through different computation paradigms. The use of MoE is also common in the community. It is recommended to exhaustedly discuss the difference between the aforementioned works and highlight key contributions.\n\n- The motivation for using the mixture of Neural Implicit Fields (NIFs) is mentioned as better memory efficiency, but it is recommended to directly compare run-time memory with the baselines. Additionally, it would be beneficial to include inference time comparisons with the presented methods.\n\n\n[1] NeurMiPs: Neural Mixture of Planar Experts for View Synthesis\n\n[2] Neural Implicit Dictionary via Mixture-of-Expert Training\n\n[3] Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields \n\n[4] Adversarial Generation of Continuous Images\n\n[5] HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion\n\nQUESTIONS:\n1. What training method is used throughout the experiments? \n\n2. Why Functa is not compared in the voxel reconstruction/generation task on ShapeNet?\n\nLIMITATIONS:\nThe main text of the paper does not address the limitations of the proposed method. From my understanding, scaling up the method to larger and more complex datasets could still present challenges.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}