Generative Neural Fields by Mixtures of Neural Implicit Functions
Authors: Tackgeun You
Mijeong Kim
Jungtaek Kim
Bohyung Han
Venue: NeurIPS 2023 poster (2023)
Keywords: generative neural fields; implicit neural representation; model averaging

Abstract: We propose a novel approach to learning the generative neural fields represented by linear combinations of implicit basis networks. Our algorithm learns basis networks in the form of implicit neural representations and their coefficients in a latent space by either conducting meta-learning or adopting auto-decoding paradigms. The proposed method easily enlarges the capacity of generative neural fields by increasing the number of basis networks while maintaining the size of a network for inference to be small through their weighted model averaging. Consequently, sampling instances using the model is efficient in terms of latency and memory footprint. Moreover, we customize denoising diffusion probabilistic model for a target task to sample latent mixture coefficients, which allows our final model to generate unseen data effectively. Experiments show that our approach achieves competitive generation performance on diverse benchmarks for images, voxel data, and NeRF scenes without sophisticated designs for specific modalities and domains.

Generative Neural Fields by Mixtures of Neural Implicit Functions
Tackgeun You tackgeun. youposste. ac. kr
Mijeong Kim mijsong- kim@anu ac kr
Jungtack Kim j jungtaek. kimēpitt edu edu
Bohyung Han
bhhan@snu ac kr
ECE & 'IPAL, Seoul National University, South Korea $CSE, POSTECH, South Korea *University of Pittsburgh, USA
Abstract
We propose novel approach to learning the generative neural fields represented by linear combinations of implicit basis networks. Our algorithm learns basis networks in the form of implicit neural representations and their coefficients in latent space by either conducting meta-learning or adopting auto-decoding paradigms. The proposed method easily enlarges the capacity of generative neural fields by increasing the number of basis networks while maintaining the size of a network for inference to be small through their weighted model averaging. Consequently, sampling instances using the model is efficient in terms of latency and memory footprint. Moreover, we customize denoising diffusion probabilistic model for a target task to sample latent mixture coefficients, which allows our final model to generate unseen data effectively. Experiments show that our approach achieves competitive generation performance on diverse benchmarks for images, voxel data, and NeRF scenes without sophisticated designs for specific modalities and domains.
Introduction
Implicit neural representation (INR) is powerful and versatile tool for modeling complex and diverse data signals in various modalities and domains, including audio [12], images (35], videos (40) 3D objects (28] 6. and natural scenes. INR expresses data instance
function mapping
from continuous coordinate space to signal magnitude space rather than using a conventional representation on discrete structured space. In particular, representation with a continuous coordinate allows us to query arbitrary points and retrieve their values, which is desirable for many applications that only have accessibility subset of the target data in a limited resolution. INRs replace the representations based on digh--imensional regular grids, such as videos [9] and 3D scenes (25). with multi-layer perceptrons (MLPs) with relatively small number of parameters, which effectively memorize scenes.
Generative neural fields aim learn distributions of functions that represent data instances form of neural field. They typically rely on INRs to sample their instances in various data modalities and domains. The crux of generative neural fields lies how to effectively identify and model shared and instance-specific information. To this end, fcature-wise linear modulation (FiLM) [29] and heper-network (HyperNet) [16] are common methods for modulating representations. FiLM
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

coordinates
outputs
image
voxel
X
INR
fi bll)
(2) m (Btas Mar)
(2t H (36)
radiance field
coefi coeffiiiett dieent
latent space
im
(1T) Pa at) me (ans Uc: Ui (t.36)
Figure I: Overview of the generation procedure using the proposed generative neural field based on mixtures of neural implicit function (mNIF). Our model is applicable to various types of data such as images, voxels, and radiance fields. To generate an instance, we first sample a context vector Φ from prior distribution (Piate)) estimated by denoising diffusion probabilistic model. We then perform a weighted model averaging on implicit bases [W!!) bin using mixture coefficients (am)) derived from the context vector.
adjusts hidden features using an affine transform, which consists of element-wise multiplication and bias addition. Due to the ease of optimization, the effectiveness of FiLM has already been validated in several tasks including 3D-aware generative modeling 6 77. However, the operation of FiLM is too restrictive while introducing additional parameters for modulation. On the other hand, HyperNet more flexible because it directly predicts network parameters. Unfortunately. the direct prediction of model parameters is prone to unstable training and limited coverage of the data distribution generated by the predicted network. Therefore, reduce solution spaces, HyperNet often employs dimensionality reduction techniques such as low-rank decompositions of weight matrices [35 12] and combinations of multiple network blocks [66]. However, all these works give limited consideration to the inference efficiency.
Inspired by the flexibility of HyperNet and the stability of FiLM while considering efficiency for inference, we propose a mixture of neural implicit functions (mNIF) to represent generative neural fields. Our approach employs mNIFs to construct generative networks via model averaging of INRs. The mixture components in an mNIF serve as shared implicit basis networks while their mixands define relative weights of the bases construct single instance. Our modulation step corresponds to computing weighted average of the neural implicit functions. The mixture coefficients are optimized by either meta-learning or auto-decoding procedures with signal reconstruction loss. Such a design is effective for maximizing the expressibility of INRs while maintaining the compactness of inference networks. The proposed approach is versatile for diverse modalities and domains and is easy to implement.
Figure I demonstrates the overview of the proposed approach, and we summarize the contributions of our work as follows:
We introduce generative neural field based on mNIF, modulation technique by lin- ear combination of NIFs. The proposed approach effectively extracts instance-specific information through conducting meta-learning or applying auto-dccoding procedures.
Our modulation allows us to easily extend the model capacity by increasing the number of basis networks without affecting the structure and size of the final model for inference. Such a property greatly improves the efficiency of the sampling network in terms of inference time and memory requirements.
Our model achieves competitive performance on several data generation benchmarks without sophisticated domain-speciff designs of algorithm and architecture configuration.
The rest of this paper is organized as follows. We first discuss related works about generative neural fields in Section 2 The main idea of the proposed approach and the training and inference procedures are discussed Sections and respectively. Section presents empirical quantitative and qualitative results with discussions, and Section concludes this paper.

---

2 Related Work
Implicit neural representations (INR) are often optimized for representing a single instance in dataset, and have been extended in several ways to changing activation functions [33 30) utilizing
input coordinate embeddings [36 [6] or taking advantage of a mixture experts 37 23 24]. Since
the INR framework is difficult generalize whole dataset, generative neural fields have been
proposed to learn a distribution of functions based on INRs, each of which corresponds to an example
in the dataset.
Generative manifold learning (GEM) 12 and generative adversarial stochastic process (GASP) 14 are early approaches to realizing generative neural fields. GEM implements the generative neural field using the concept of hyper-network (HyperNet) [16], which predicts a neural network representing an instance. Training GEM is given by the auto-decoding paradigm with manifold learning. However, GEM involves extra computational cost because the manifold constraint requires accessing training examples. GASP also employs HyperNet for sampling instances, and its training utilizes discriminator taking an input as a set of coordinate and feature pairs. Since the output of the discriminator should be permutation invariant, it follows the design of the network for the classification of point cloud data. However, training GASP is unstable and the discriminator design is sometimes tricky, especially in NeRF, Functa [13] proposes the generative neural fields with SIREN (33) modulated by FiLM [29] and introduces a training strategy based on meta-learning. However, meta-learning is computationally expensive due to the Hessian computation. On the other hand, diffusion probabilistic fields (DPF) [41] proposes a single-stage diffusion process with the explicit field parametrization [9]. However, its high computational cost hampers the applicability complex neural fields such as NeRF HyperDiffusion (15), which is concurrent to our work, presents a framework directly sampling the entire INR weight from the learned diffusion process instead of exploiting latent vectors for modulating INR weights. However, this work demonstrates instance generation capability only on voxel domains, not on images.
Model averaging [38] is often used for enhancing performance in discriminative tasks without increasing inference cost. Our approach applies model averaging to generative models and verifies its effectiveness in generative neural fields based on INRs. Note that we are interested in efficient prediction by learning compact representation through weighted mixture; our method jointly trains basis models and their mixture coefficients.
3 Generative Neural Fields with a Mixture of Neural Implicit Functions
This section describes how to define generative neural fields. The formulation with a mixture of neural implicit functions (mNIF) enforces implicit basis networks to represent shared information across examples and allows latent mixture coefficients to encode instance-specific information through a linear combination of basis networks. We show that generative neural field via mNIF is effectively optimized to predict the latent vector for the construction of models generating high-quality samples.
3.1 Implicit Neural Representations
Implicit neural representation (INR) expresses a data instance using a function from an input query, Nd. to its corresponding target value, y E TR& Since each input coordinate is independent of the others, a mapping function parametrized by 0, (-).), is typically represented by a multi-layer perceptron (MLP) with L fully connected (FC) layers, which is given by
y = fx(x) = p(L+1) ..... f(1) o p(0) (x),
(1)
where f() for Vi E (1,2...., T.} denote FC layers while ((0) and f(L+1) indicate the input and output layers, respectively. Each layer p() performs linear transform its input hidden state h and then applies a non-lineat activation function yield the output state h (1+1), expressed as
h (++1) (h)) = a (w by)) i (1,2....,
where o(-) is an activation function and w E Wx and (() E RWW are learnable parameters. The operations of the input and output layers are defined as h (x) and ý p((+1)(h())
3

---

respectively, Consequently, a collection of learnable parameters @ in all layers is given by
θ
w b w(), b W b((t1)
(3)
where W E R (()) E RW W(L+1) R*xW and b E R For INR. the mean squared error between a prediction y and a target y is typically adopted as a loss function, ((.-). which is given by
((g.y) = 118 yl1
(4)
Among the variations of INRs, we employ SIREN [33.. which adopts the sinusoidal function as an activation function and introduces sophisticated weight initialization scheme for MLPs. Supposing that SIREN is defined with an MLP with L hidden layers, the 1th layer of SIREN is given by
(5) where wo is a scale hyperparameter to control the frequency characteristies of the network. The initialization of SIREN encourages the distribution of hidden sine activations to follow a normal distribution with standard deviation of 1.
h (i+1) sin
We
(w + +b()) E L).
3.2 Mixtures of Neural Implicit Functions
We propose generative neural fields based on mNIF, which is an extension the standard INR for modulating its model weight. We define set of NIFs, which is used as basis networks, and construct a generative neural field using a mixture of the NIFs. This is motivated by our observation that generative neural field is successfully interpolated using multiple basis networks and model averaging works well for model generalization [38].
The operation in each layer of our mixture model is given by
h (++1) sin W/o Σ
a gm (h(t))
(6)
where (+) (h()) w ( is the tt layer operation of the mhth neural implicit function, (1) and Ohn is mixture coefficient of the same layer of the neural implicit function. Note that modulating the network is achieved by setting the mixand values, {am ()) ). Similar to the definition of 0 described in Eq. 33, the parameter of each mixture is given by
0m
(D)) blo)
((L-1) ((L+1)
The operation in the resulting INR obtained from model averaging is given by
M
M
M
gm (h()==
=
Σ
w m
(t)
h
(
a
bb m
=
w
h0 h
(
(8)
m=11
m=1
m=1
where
W
(6
and b (1)
O
b m
All the learnable parameters in the mixture of NIFs is defined by [ 0 )- The remaining parameters are mixture coefficients, and have the following two options define them: (i) sharing mixands for all layers (i) am and (ii) setting mixands to different values across layers. The first option is too restrictive for the construction of INRs because all layers share mixture coefficients while the second is more flexible but less stable because it involves more parameters and fails to consider the dependency between the coefficients in different layers. Hence, estimate the mixture coefficients in latent space, where our method sets the dimensionality of the latent space constant and enjoys the high degree-of-freedon of layerwise coefficient setting. To this end, we introduce a projection matrix T to determine the mixture coefficients efficiently and effectively, which is given by
choose the second option but
ai 5 ... a (0)
(0)
(1)
00 ... 0i
(1)
(L+1)
... a
((+1)
(10)

---

where Φ E R denotes latent mixture coefficient vector. This choice compromises the pros and cons of the two methods. Eventually, we need to optimize set of parameters, Oau _ {0)..... a generative neural field, and a latent vector to obtain an instance-specific INR.
. M, T3 for
3.3 Model Efficiency
The proposed modulation technique is based on model averaging and has great advantages in the efficiency during training and inference. First, the computational complexity for each input coordinate is small despite large model capacity given by potentially many basis networks. Note that, even though the number of neural implicit basis function in a mixture increases, the computational cost handle more coordinates queries than the models relying on FiLM (29). Second, the model size of our generative neural fields remains small compared other methods due our model averaging strategy, which is effective for saving memory to store the neural fields.
per query is invariant by Eq. (8). which allows
When analyzing the efficiency of generative neural fields, consider two aspects; one is the model size the context adaptation stage, in other words, the number of all the learnable parameters, and the other is the model size of the modulated network used for inference. By taking advantage of the modulation via model averaging, our approach reduces memory requirement significantly both in training and inference.
Learning Generative Neural Fields
The training procedure of the proposed approach is composed of two stages: context adaptation followed by task-specific generalization. In the context adaptation stage, we optimize all the learnable parameters for implicit basis functions and their coefficients that minimize reconstruction error on examples in each dataset. Then, we further learn to enhance the generalization performance of the task-specific generative model, where more advanced sampling method is employed estimate a context vector, i.e., a latent mixture coefficient vector. This two-stage approach enables us to acquire the compact representation for neural fields and saves computational costs during training by decoupling the learning neural fields and the training generative model. We describe the details of the two-stage model below.
4.1 Stage 1: Training for Context Adaptation
The context adaptation stage aims to learn the basis networks and the latent mixture coefficient vector, where the mixture model minimizes the reconstruction loss. This optimization is achieved by one of two approaches: meta-learning auto-decoding paradigm.
With meta-learnin, we train models in similar way as the fast context adaptation via meta-learning (CAVIA) [42] after a random initialization of a latent vector Φ; for the j"h instance in batch, we update bb in the inner-loop and then adapt the shared parameters 0 based on the updated Φy. Since this procedure requires estimating the second-order information for the gradient computation of O, the computational cost of this algorithm is high. Algorithm presents this meta-lcarning procedure.
We observe that random initialization of latent vector during meta-learning is more stable than the constant initialization originally proposed CAVIA when training mNIF with a large number of mixture components or the high dimensionality of latent vectors. We intend to diversify INR weights the start phase of training, ensuring that not all samples are represented with identical INR weights.
Auto-decoding paradigm 12 28 is variant of multi-task learning, where each instance is considered a task. Contrary to the meta-learning, we do not reset and initialize the latent vector Φb for each instance but continue update over multiple instances. The optimization procedure via auto-decoding is presented in Algorithm 2
4.2 Stage 2: Optimizing for Task-Specific Generalization
The aforementioned context adaptation procedure focuses only on the minimization of reconstruction error for training samples and may not be sufficient for the generation of unseen examples. Hence,
5

---

Algorithm Meta-learning with mNIF
1: Randomly initialize the shared parameter 0 of mNIF.
2: while training do
3: Sample a mini-batch B = ((c,,,
4: for j = 1 to [B] do
5: Initialize latent vector Φ () .V,00 E R
6: for 0 to Nineer do
7:
Update the latent vector: of f(tt1) t- (m)) 8: end for
élatent V V /=0fm)
9: Update the shared parameters: t- Eshared VoL(fo /
10: end for
11:
end while
Algorithm 2 Auto-decoding with mNIF
1: Randomly initialize the shared parameter O of mNIF
2: Initialize a latent vector Φ ~ (N E RR for all samples.
3: while training do
4: Sample a mini-batch
5: Define a joint parameter: ΦB [Ø; j=11B
6: Update the parameters:
{0, 0g) + (8. Vo. E, (l
7: end while
we introduce the sampling strategy for the context vectors & and customize to a specific target task. To this end, we adopt the denoising diffusion probabilistic model (DDPM) [88] which employs the residual MLP architecture introduced in Functa (13].
Experiments
This section demonstrates the effectiveness of the proposed approach, referred to as mINF, and discusses the characteristics our algorithm based the results. We run all experiments on the Vessi environment (2), and describe the detailed experiment setup for each benchmark in the appendix.
5.1 Datasets and Evaluation Protocols
We adopt CelebA-HQ 6 (2]). ShapeNet 64 (8 and SRN Cars 34] dataset for image, voxel and neural radiance field (NeRF) generation, respectively, where 64 and 64 denotes the resolution of samples in the dataset. We follow the protocol from Functa for image and NeRF scene and generative manifold learning (GEM) [12 for voxel.
We adopt the following metrics for performance evaluation. To measure the reconstruction quality, use mean-squared error (MSE), peak signal-to-noise ratio (PSNR), reconstruction Fréchet inception distance (rFID), reconstruction precision (rPrecision) and reconstruction recall (rRecall). In image generation, we use Fréchet inception distance (FID) score [17]. precision, recall [32 27] and F1 score between sampled images and images train split. Voxel generation performance evaluated by coverage and maximum mean discrepancy (MMD) metrics [1] on test split. In NeRF scene generation, we use FID score between rendered images and images in test split for all predefined views for evaluation. To compare model size of algorithms, we count the number of parameters for training and inference separately; the parameters for training contain all the weights required in the training procedure, e.g., the parameters in the mixture components and project matrix for our algorithm, while the model size for inference determined by the parameters used for sampling instances. For the evaluation of efficiency, we measure the number of floating point operations per second (FLOPS), latency in terms of frames per second (fps), and the amount of memory consumption for single sample.

---

Table Image reconstruction and generation performance on CelebA-HQ 642. GEM, and Functa are brought from the corresponding papers.
The results of DPF;
Model
Params
Reconstruction
Generation
Inference Fffficiency
Learnable
Inference
PSNR t rFID
FID | Precision
Rocall + F1 1
GFLOPS
fps 1
Memory (MB).
Functa
13
2,629.6
26.6
28.4
40.4
0.577
0.397
0.470
8.602
332.9
GEM
12
99.0 M
921.3 K
30.4
0.642
0.502
0.563
3.299
559.6
70.3
GASP
14
34,2
83.1
13.5
0.836
0,312
0.454
0.305
1949.3
16.4
DPF [4]
62.4 M
13.2
0.866
0.347
0.495
mR/F (8)
466M
17IK
"1.3
10.9
2TT
"6787
0.324
"0.439"
0.066
29.58.
"10.
mNIF
33.4 M
83.3
34.5
5.8
13.2
0.902
0.544
0.679
0.340
891.3
24.4
Table 2: Voxel reconstruction and generation performance on ShapeNet 64 The results of DPF, and GEM are brought from the corresponding papers.
Model
Params
Reconstruction
Generation
Inference Efficiency
Learnable
Inference
MSE
PSNR
Coverage f
MMD
GFLOPS
fps f
Memory (MB) |
GASP
14
34.2 M
83.1 K 921.3 K
0,0296
16.5
0.341
0.0021
8.7
180.9
763.1
GEM
12
99.0 M
0.0153
21.3
0.409
0,0014
207.0
16.7
4010.0
DPF
62.4
0.419
0.0016
(S)
4.6
17.2K
20.9
0.430
0.00
3.4
T91.3
642.
mNIF (L)
46.3
83.3
0.0166
21.4
0.437
0.0013
21.6
69.6
1513.3
5.2 Main Results
We present quantitative and qualitative results, and also analyze the effectiveness of the proposed approach in comparison to the previous works including Functa [3] GEM [12), GASP [14], and DPF [41). Note that adopt the model configurations with the best generation performance for the other baselines.
5.2.1 Quantitative Performance
We compare results from our approach, mNIF, with existing methods in terms of four aspects: recon- struction accuracy, generation quality, model size, and inference efficiency. We present quantitative results from two configurations of our model, mNIF (S) and mNIF (L), which correspond to small and large networks, respectively.
As shown in Tables 2 and our approach consistently achieves state-of-tt of-the-art art reconstruction and generation quality on the image, voxel, and NeRF scene datasets except the reconstruction on the voxel dataset, in which our model is ranked second. Moreover, mNIF is significantly more efficient than other methods in terms of model size and inference speed in all cases. Considering the model size and inference speed, the overall performance of mNIF is outstanding.
Note that the light configuration of our model denoted mNIF (S) is most efficient in all benchmarks and also outperforms the baselines in voxel and NeRF, In SRN Cars, mNIF (S) demonstrates huge benefit in terms of efficiency compared to Functa; 199 times less FLOPS, 49 times more fps,, and 22 times less memory in single view inference. In the image domain, the performance of mNIF (S) is comparable to GASP in terms of F1 but worse in FID. We conjecture that FID metric overly penalizes blurry images, which is discussed in several works 12 31
5.2.2 Qualitative Generation Performance
Figure iillustrates generated samples for qualitative evaluation. We visualize the results from our models on each benchmark together with GASP on CelebA-HQ 642 and Functa on SRN Cars to compare results in the image and NeRF scene tasks, respectively. In the image domain, our model, mNIF (L), generates perceptually consistent yet slightly blurry samples compared to ground-truths. while the examples from GASP with a similar FID to ours have more artifacts. In the NeRF scene, rendered views from our model, mNIF (S), and Functa exhibit similar level of image blur compared to ground-truths. Since both methods use vanilla volumetric rendering, adopting advanced rendering techniques, such as hierarchical sampling 25 6 and anti-aliasing methods 3 for NeRF scenes, would improve rendering quality.
7

---

Table NeRF scene reconstruction and generation performance on SRN Cars, The results of Functa are brought from the corresponding papers.
Model
# Params
Reconstructoon
Generation
Inference Efficiency
Learnable
Inference
f
FID4
TFLOPS 4
Eps
Memory (GB) J
Functa 1+
3.9
3,41866
24.2
80.3
1.789
2.0
28.0
mNNF (S)
4.6 M
17.2
25.9
79.5
0.009
97.7
1.3
mNIF (L)
mNIF (S)
mNIF (L)
GASP 14
Functa i
Ground Truth
Ground Truth
Ground Truth
(a) CelebA-HQ 64
(b) ShapeNet 6-1
(c) SRN Cars
Figure 2: Comparison of generated samples from our models, mNIF (S) and mNIF (L., with the ground-truth and the bascline methods such as GASP [14] and Functa [3], on the CelebA-HO 64
(2a). ShapeNet 64 2b) and SRN Cars (2c) datasets.
5.3
Analysis
We analyze the characteristics of our trained model for better understanding via latent space explo- ration and various ablation studies.
5.3.1 Latent Space Interpolation
Figure iillustrate the results of interpolation in the latent space: the corner images are samples in the training dataset and the rest of the images constructed by bilinear interpolations in the latent space. These images demonstrate the smooth transition of sampled data, confirming that the latent vectors learned the first stage manage capture the realistic and smooth manifold of each dataset.
5.3.2 Configurations of Mixture Coefficients in mNIF
Table a) examines the performance by varying the types of mixture coefficients: i) shared mixture coefficients across all layers, i.e., Ohn Omn ii) layer-specific mixture coefficients, and iii) layer- specific mixture coefficients projected from a latent vector as shown in Eq. (TO). Among the three mixture settings, the last option yields the best performance and used for our default setting.
(8)
Table presents the effect of the number of mixtures, denoted by M. Reconstruction performance generally improves as the number of mixtures increases, but such a tendency saturates at around M = 256 We also measure the precision and recall between the ground-trute and reconstructed samples denoted by rPrecision and rRecall, respectively. According to observation, increasing the size of the mixture improves the recall more than it does the precision.
Table C) shows the effect of the latent dimension on the reconstruction performance of mNIR Increasing the latent dimension H leads to improving the reconstruction performance both the train and test splits due to the degree-of-freedom issue.

---

CelebA-HQ 64
(b) ShapeNet 64
(c) SRN Car
Figure 3: Reconstructed images via latent vector interpolations. The examples the four corners surrounded by green boxes are reconstructed instances using the latent vectors on CelebA-HQ 642, ShapeNet 64 and SRN Cars. The rest of samples are generated via bilinear interpolations of latent vectors corresponding to the images at the corners.
Table 4: Ablation study on the mixture coefficient configuration of mNIF on CelebA-Ho HHlebA-HQuen 642. Note that L, W. M,, and H denote hidden layer depth, hidden layer width, the number of mixtures, and the dimensionality of a latent coefficient vector, respectively.
# Params
Train
Test
Exp
Mixture
(I, H.
Learnable
Inference
PSNR f rFID
rPrecision
rRecall
PSNR
rFID
Shared
(2,64, 64, 64)
557.2
8.7
22.20
50.70
0.497
0.003
22.11
55.87
(a) Layer-specitic
(2. 64, 64, 256)
557.2 K
8.7 K
24.45
38.23
0.461
0.013
24.35
42.65
Latent
64,
623.0 K
"8.7
25.27
31.67
0.334
0.000
25.09
36.83
(2, 64, 16, 256)
155.8 K
22.02
53.01
0.433
0.001
21.84
57.23
(2, 64, 64, (4,25620, 256)
623.0 K
25.27
31.67
0.534
0.040
25.09
36.85
(b)
Latent
8.7
(2, 64, 256, 256)
2.5
26.64
24.62
0.640
0.134
25.74
32.17
(2, 1024, 256)
26.84
23.71
0.642
0.155
25.85
32.14
(5, 128, 256, 256)
21.8
31.17
10.65
0.890
0.750
25.35
31.58
(c)
Latent
(5, 128, 256,
22.3
83.3
32.11
8.96
0.918
0.845
24.79
128, 256, 1024)
23.2 M
32.71
8.09
0.935
0.893
29.45
22.05
5.3.3 Diversity in Learned Neural Bases
To investigate the diversity of the basis functions per layer, Figure visualizes the absolute value of the pairwise cosine similarity between the weight matrices the basis models. the cosine similarity matrices, an element with small value indicates that the corresponding two neural basis functions are nearly orthogonal each other, which serves as evidence of diversity in the learned basis functions. The visualization also reveals the following two characteristics. First, the lower layers are more correlated than the upper ones partly because low-level features inherently have less diversity. Second, a small subset of neural bases have high correlations with others. We hypothesize that the concentration high correlation among the small subset of neural bases could be mitigated by introducing loss function enforcing orthogonality.
5.3.4 Context Adaptation Strategies
We perform the ablation study with respect to diverse optimization strategies and demonstrate the resulting reconstruction performance in Table 5 We train small mNIFs with (L, W. M, H) (2, 64, 256, 256) on the CelebA-HQ 642 dataset. It is clear that the meta-learning approach using second-order gradient computation yields favorable results compared to other options. Running more inner-loop iterations marginally improves performance, but at the cost of additional computational complexity in time and space. Therefore, set the number of inner-loop iterations to for all meta-learning experiments.
Interestingly, the nuto-decoding strategy surpasses meta-learning with first-order gradient computation. Note that auto-decoding more efficient than other methods in terms of both speed and memory usage when evaluated under the same optimization environment. This efficiency of auto-decoding is

---

layer 1
layer
layer
layer 4
layer
25
51
Figure 4: Visualization the absolute value of cosine similarity between neural basis functions, observed in five different layers. We compute the similarity of the hidden layers in mNIF (L) trained on CelebA-HQ 64². Diagonal values in these matrices are set to zero for visualization.
Table 5: Study on context adaptation strategy.
Table Study on longer training of mNIF.
Strategy
Second-order
Nuner
PSNR 1
rFI L
Epoch
Reconstruction
Generation
Auto-decoding
24.68
34.31
PSNR
rFID 4
FID
22.6%
37.27
400
32.11
8.96
14.96
Meta-lcaming
3
26.64
24.64
800
33.10
7.31
14.30
26.77
24.37
1200
33.61
6,91
15.25
particularly beneficial for NeRF scene modeling, which requires large number of input coordinate queries. Our model runs on single GPU for training on 8 NeRF scenes per batch while Functa needs 8 GPUs for the same purpose.
5.3.5 Longer Training of Context Adaptation
TJshoss the performance of our approach, mNIF, with respect to the number of epochs for the first stage training under the identical stage 2 training environment. When train mNIF (L) with (I, W, M, H) (5, 128, 256, 512) on the CelebA-HQ 64 dataset, reconstruction performance is positively correlated with training length but longer training may be prone overfitting considering the trend of generalization performance.
6 Conclusion
We presented a novel approach for generative neural fields based on mixture of neural implicit functions (mNIF). The proposed modulation scheme allows to easily increase the model capacity by learning more neural basis functions. Despite the additional basis functions, mNIF keeps the network size for inference compact by taking advantage of simple weighted model averaging, leading to better optimization in terms of latency and memory footprint. The trained model can sample unseen data in specific target task, where adopt the denoising diffusion probabilistic model I8 13 for sampling context vectors Φ. We have shown that our method achieves competitive reconstruction and generation performance various domains including image, voxel data, and NeRF scene with superior inference speed.
The proposed method currently exhibits limited scalability beyond fine-grained datasets. We hypothe- size that the latent space in mNIF trained on a dataset with diverse examples, such as CIFAR- 10, is not sufficiently smooth compared to other datasets to be generalized by adopting diffusion process. possible solution to these challenges to incorporate local information into the architecture based neural implicit representations, proposed in recent research [39 5]. We reserve this problem for future work.
Acknowledgments and Disclosure of Funding
This work was partly supported by National Research Foundation of Korea (NRF) grant [No. 2022R1 1 A2C3012220, No.2022R AASA70839081 11] and the Institute of Information & communica- tions Technology Planning Evaluation (ITTP) grant INO.2021-0-022 No. 2021-0-01343), funded by the Korea government (MSIT).
10

---

References
[1] Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and genera- tive models for 3d point clouds. arXiv preprint 170.029707.02292... 2017.
[2] An. Model development with vessl, 2023. URL https: wwsss//wwwwweesll..........l ai/ Software available from vesslai.
[3] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-NeRF: multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
[4] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-NeRF: Anti-aliased neural radiance fields. arXiv preprint arXiv:2304.06706 2021.
[5] M. Bauer, E. Dupont, A. Brock, D. Rosenbaum, J. Schwarz, and H. Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130, 2023.
[6] E. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
[7] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. D. Mello, O. Gallo, L. J. Guibas,
J. Tremblay, S. Khamis, T. Karras, and G. Wetzstein. Efficient geometry-aware 3d generative adversarial networks. In CVPR, 2022.
[8] A. X. Chang, Funkhouser, L. Guibas, P. Hanrahan. Q. Huang. Li, Savarese, M. Savva, S. Song. H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An -ifommttion-iich 3d model repository. arXiv preprint arXiv: 1111.0001122.. 2015.
[9] H. Chen, B. He, H. Wang, Y. Ren, S.-N. Lim, and A. Shrivastava. NeRV: Neural representations for videos. In NeurlPS, 2021.
[10] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
[11] D.-A. Clevert, T. Unterthiner, and Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In ICLR, 2016.
[12] Y. Du, K. M. Collins, J. B. Tenenbaum, and V. Sitzmann. Learning signal-agnostic manifolds of neural fields. In NeurIPS, 2021.
[13] E. Dupont, H. Kim, S. M. A. Eslami, D. J. Rezende, and D. Rosenbaum. From data to functa: Your data point is function and you can treat like one. In ICML., 2022.
[14] E. Dupont, Y. W. Teh, and Doucet. Generative models as distributions of functions. In AISTATS, 2022.
[15] Z. Erkoç. F. Ma. Q. Shan, M. Nießner, and A. Dai. HyperDiifusion: Generating implicit neural fields with weight-space diffusion. In ICCV, 2023.
[16] D. Ha, A. Dai, and Q V. Le. Hypernetworks. In ICLR, 2016.
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. In NeurIPS, 2017.
[18] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[19] A. Jaegle, S. Borgeaud, J.B.. Alayrac, C. Doersch, C. lonescu, D. Ding, S. Koppula, D. Zoran,
A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira.
Perceiver 10: general architecture for structured inputs & outputs. In ICLR, 2022.
[20] S. Jung and M. Keuper. Internalized biases in fréchet inception distance. In Workshop on Distribution Shifts, NeurIPS 2021, 2021.
[21] Karras, Laine, M. Aittala, J. Hellsten, Lehtinen, and T. Aila. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018.

---

[22] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[23] Z.-H. Lin, W.-C. Ma, H.-Y. Hsu, Y.-.. F. Wang, and S. Wang. NeurMiPs: Neural mixture of planar experts for view synthesis. In CVPR, 2022.
[24] Z. MI and D. Xu. Switch-NeRF: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In ICLR, 2023.
[25] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[26] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. In SIGGRAPH, 2022.
[27] M. F. Naeem, S. Oh, Uh, Y. Choi, and Yoo. Reliable fidelity and diversity metrics for generative models. In ICML, 2020.
[28] J. J. Park. P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[29] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. FiLM: Visual reasoning with a general conditioning layer. In AAAI, 2018.
[30] Ramasinghe and Lucey. Beyond Periodicity: Towards a unifying framework for activations in coordinate-mlps. In ECCV, 2022.
[31] A. Razavi, A. van den Oord, and O. Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NeurlPS, 2019.
[32] M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. In NeurlPS, 2018.
[33] V. Sitzmann, N. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural represen- tations with periodic activation functions. In NeurlPS, 2019.
[34] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene Representation Networks: Continuous di-strrcttur--waae neural scene representations, In CVPR, 2019.
[35] I. Skorokhodov, S. Ignatyev, and M. Elhoseiny. Adversarial generation of continuous images. In CVPR, 2021.
[36] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keii N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. T. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurlPS, 2020.
[37] P. Wang, 7. Fan, T. Chen, and Z. Wang. Neural implicit dictionary via mixture-of-expert training. In ICML, 2022.
[38] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lppes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML,, 2022.
[39] Z. Wu, Jin, and K. M. Yi. Neural fourier filter bank. CVPR, 2023.
[40] S. Yu, J. Tack, S. Mo, H. Kim, J. Kim, J.-W. Ha, and J. Shin. Generating videos with dynamics- aware implicit generative adversarial networks. In ICLR, 2022.
[41] Zhuang, S. Abnar, J. Gu, A. Schwing, M. Susskind, and M. A. Bautista. Diffusion probabilistic fields. In ICLR, 2023.
[42] L. M. Zintgraf, K. Shiarlis, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta-learning. ICMI, 2019.
12

---

A Implementation Details
A.1 Benchmark
A.1.1
Images
We use CelebA-HQ dataset [21] for our work. We divide entire images into 27,000 images for train and 3,000 images for test split which is provided by Functa [33. We use the pre-process dataset from
the link
To measure the quality of generated images, we compute Fréchet inception distance (FID)
score 17 precision and recall 32 27 between sampled images and images in a train split.
A.1.2 Voxels
We utilize the ShapeNet dataset [8] for 3D voxel generation from IM-Net 10]. It has 35,019 samples in the train split and 8,762 samples in the test split. The dataset contains a voxel with 64 and 16,384 points extracted near the surface. To evaluate the generative model, we sample 8,764 shapes, which is the size of the test set, and generate all points in the voxel. Then, we compute 2,048 dimension mesh features from the voxel following the protocol in [TO]. Generation performance with coverage and maximum mean discrepancy (MMD) metrics [T] is evaluated using Chamfer distance on test split. We follow the evaluation procotol from generative manifold learning (GEM) [[2]
A.1.3 Neural Radiance Field (NeRF)
We adopt SRN Cars dataset 34 for NeRF modeling. SRN cars dataset has a train, validation, and test split. Train split has 2,458 scenes with 128x 128 resolution images from 50 random views. Test split has 704 scenes with 128x 128 images from 251 fixed views in the upper hemisphere.
We adopt the evaluation setting of NeRF scene from functa which utilizes 251 fixed views from the test split. Consequently, we can compute FID score based on rendered images and images in test split with equivalent view statistics.
A.2 Implicit Neural Representation with Mixture of Neural Implicit Functions
A.2.1
Images
We train our mixtures of Neural Implicit Functions (mNIF) on images from train split of CelebA-HQ 642 dataset. Generative implicit neural representation (INR) for image takes 2D input coordinates (2) and returns 3D RGB values (yR, PG, yB). During optimization, we use dense sampling, which queries all input coordinates (4096 64 x 64) to the network inputs, for image benchmark. We use mNIF configuration with the number of hidden layers L 5. the channel dimension of hidden layers W = 128, the number of mixtures M = 384 and the latent vector dimension H = 512. throughout our experiments. We use meta learning for training the mNIF CelebA-HQ 642. We use Adam (22 optimizer with the outer learning rate Coueer 1.0 x 10 and batch size 32. We use cosine annealing learning rate schedule without warm-up learning rate. We take the best mNIF after optimizing the model over 800 epochs with inner learning rates Cimer E (10.0, 1.0, 0.1). The configuration of mNIF (S) is (T,, W, M, H) (4, 64, 256, 1024) and mNIF (L.) (5, 128, 384, 512). The best performed model uses Ginner = 1.0.
A.2.2 Voxels
We train our model on ShapeNet 643 dataset. Our generative neural field for voxel takes 3D input coordinates (21,I2, and returns yo indicating whether queried coordinate are inside or outside. For model optimization, we use sub-sampling with 4,096 points in voxel with 643 from the points. Then, we use all 16,384 points for constructing latent vectors. Note that despite relying on sub-sampling, the 16,384 points sampled from the near surface of voxel are dense enough to effectively represent its content within the voxel. We use meta learning for training the mNIF on ShapeNet 643. We use mNIF (S) and (I.) configuration with (T., W, M, H) = (4, 64, 256, 512) and (L, W, M, H) (5, 128, 512, 1024). We take the best mNIF after optimizing the model over 400 epochs with inner learning rates einer E (10.0, 0.13.
13

---

A.2.3 NeRF Scenes
We train our model SRN Cars dataset. We use a simplified NeRF protocol without view dependency on neural fields. So, generative neural field returns 4D outputs, RGB values and density (yR. yG. VB: Vo), with given 3D imput coordinates (I1, 43)) We use no activation function for RGB output and exponential linear unit (ELU) [T] for density output suggested in Functa. During optimization, we use sparse sampling with 32 views per scene, 512 pixels per view and points per ray for each scene. We use auto-decoding procedure for efficient optimization instead of meta-learning procedure avoid the cost of second-order gradient computation. simply harvest latent vectors for the dataset jointly trained during the optimization procedure. We use mNIF (S) configuration with (L, W, M, H) = (4, 64, 256, 128). We use Adam [22] optimizer with outer learning rate couter 1.0 x 10-4 and batch size 8. We use cosine annealing without warm-up learning rate for the learning rate schedule. We use inner learning rate éinner = 1.0 x 10-4 and use weight decay on latent vector Φ, to regularize the latent space in auto-decoding framework. We take the best mNIF after optimizing the model over 2000 epochs.
auto-decoding procedure, we
A.2.4 Weight Initialization of mNIF
Weight initialization is crucial for optimizing INR, such as SIREN coefficients at the start as the inverse of the number of mixtures a'm 1/M to give the equivalent importance for all implicit bases in mixtures. For latent mixture coefficients, we set bias of projection matrix into 1/M to satisfy above condition.
33
We enforce mixture
A.3 Denoising Diffusion Process on Latent Space
We train unconditional generation models for latent space for image, voxel and NeRF scene for generating latent coefficients vector. We use the equivalent setting for the training diffusion process throughout all experiments and benchmarks. We implement DDPM [18] on latent vector space based residual MLP [13 from Funeta
denoising model. The channel dimension of residual MLP is 4096 and the number of block is 4. We use Adam optimizer and set learning rate 1.0 x 10 and batch size 32. We train model 1000 epochs on each dataset and schedule the learning rate with cosine annealing without a warm-up learning rate. We use T 1000 time steps for diffusion process.
A.4 Libraries and Code Repository
Our implementation based on PyTorch 1.07 with CUDA 11.3 and PyTorch Lightning 1.5.7 Our major routines are based on the code repositories for Functa SIREN latent diffusion model! guided diffusion and HQ-Transformer For inference efficiency, we count FLOPS with the fycore library
@ and check inference time and memory consumption on GPU by PyTorch Profiler TO The efficiency of mNIF are evaluated on NVIDIA Quadro RTX 8000.
B Analysis
B.1
Visualization
We provide additional generation examples in Figures 5 and 6 and additional interpolation results of the proposed algorithm in Figures] and
PPyTh htttps://pytorch.orgg
3 PPyTorc
"Functa httt:://ggechbbbccmm/eesss/////p
*SIREN httppp///ggthhbboooo////aaaaaaar
'Latent Diffusion
"fvcore http://:whhchcooeeeeeeeeeee/e//r
10pyTom Torch Profiler httpa//www.......coo//////////aa html)
14

---

C Limitations
As discussed concolusion, we observe that our model has limited scalability beyond fine-grain dataset; our model works well on reconstruction while not on generation on CIFARIO. To analyze the generation on CIFARIO, we introduce an alternative sampling scheme to replace the latent diffusion model described in our main paper. Drawing inspiration from the sampling strategy in generative manifold learning (GEM), we employed latent space interpolation between two samples: one random sample from the training set and another random sample from its neighborhood. The resulting context vector, is formed by simple interpolation:
@sample Φ: (1-a) -xx(s))
(11)
where di denotes context vector for i-th sample in training set, .ii)) set of indices of the neighborhood of sample and is sampled from the uniform distribution ranged between 0 to 1.
This sampling by interpolation scheme achieves 59.67 (FID) while sampling from diffusion model does 84,64 (FID). From the results, two major observations can be made: Sampling through interpo- lation yields superior FID score than the latent diffusion model, suggesting issues with the latter's interaction with context vectors. The FID score for interpolation sampling, while better than diffusion, remains suboptimal. The observation that the context vector interpolation is not fully effective for the CIFAR10 model indicates a less smooth latent space in CIFARIO compared to other datasets. This disparity could be impacting the generation quality.
We hypothesize that the limitations of the SIREN 33 architecture might be at the root of these issues. An INR work (39) categorizes SIREN lack of space resolution, leading to compromised content quality. separate study [5] underlines this, stating SIREN's unsatisfactory generation performance (FID 78.2) when applied CIFARIO. This research introduces a new SIREN architecture where biases in hidden layers of INR are varying with respect to spatial resolution. We anticipate that adopting an INR with spatial information as an alternative to SIREN will better accommodate diverse datasets, given that our approach linear combination of INR weights- broadly compatible with various INR methods.
INR with predominantly frequency encodings and
D Broader Impacts
The proposed image model is trained on the CelebA-HQ 642 dataset, which raises the possibility of unintended bias the generated human face images due to the biases present the dataset. Additionally, when extending the generative neural field model to large-scale datasets, there is potential for generating unintended biased contents, although the current version of the work primarily focuses on small-scale datasets. We acknowledge that these concerns important themes that should be thoroughly discussed in the future.
15

---

(a) CelebA-HQ 64
(b) ShapeNet 64
Figure 5: Generated samples from our models trained on CelebA-HQ 64 5a) and ShapeNet 64 (5b).
16

---

(a) View 0
(b) View 32
(c) View
(d) View 96
(e) View 128
(f) View 160
(g) View 192
(h) View 224
(i) View 250
Figure 6: Generated 25 samples with diverse views from our model trained on SRN Cars.
17

---

(a) CelebA-IIQ 64
(b) ShapeNet 64 (intra-class)
(c) ShapeNet 64 (inter-class)
Figure 7: Reconstructed samples from latent vectors for images 7ap and voxels 77 7c). Samples with green boxes reconstructed instances from latent vectors estimated from train split for each dataset. The other samples are generated with bilinear interpolation of latent vectors from corner instances. In the third row 7c] plot three interpolations with four different classes correspondingly, (left) chair (top left), cabinet (top right), display (bottom left), sofa (bottom right) (middle) loudspeaker, sofa, airplane, bench (right) airplane, bench, car, rifle. We use Mitsuba3 renderer for drawing voxel image.
18

---

(a) View 0
(b) View 32
(c) View
(d) View 96
(e) View 128
(f) View 160
(g) View 192
(h) View 224
(i) View 250
Figure 8: Reconstructed view images from latent vector for NeRF scene. Image annotated by green
boxes are synthesized instances from latent vectors estimated from SRN Cars train split. We select 9 views from the pre-detined 251 views in split. The other samples are generated by bilinear interpolation of samples at the corners.
19


Reviews:
[{"review_id": "TytBqKsXkK", "reviewer": "Reviewer_fjUK", "summary": "This paper introduces an approach for learning generative neural fields using linear combinations of implicit basis networks, called Mixture of Neural Implicit Functions (mNIF). The basis networks and their coefficients in mNIF can be learned through meta-learning or auto-decoding techniques. By this means, the capacity of generative neural fields can be increased by adding more basis networks while maintaining the size of each instance. The optimized network is further fine-tuned to make generalized predictions by optimizing with respect to latent vectors, enabling the representation of unseen data. Experimental results demonstrate performance across diverse benchmarks.", "questions": "1. What training method is used throughout the experiments? \n\n2. Why Functa is not compared in the voxel reconstruction/generation task on ShapeNet?", "limitations": "The main text of the paper does not address the limitations of the proposed method. From my understanding, scaling up the method to larger and more complex datasets could still present challenges.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "+ The paper demonstrates strong writing, with clear notation and intuitively-rendered figures. The proposed method is well-motivated and easy to understand.\n\n+ The experiments, although conducted on simple datasets like CelebA and Shapenet, effectively validate the method's effectiveness and parameter efficiency in both 2D and 3D domains. The proposed method achieves competitive performance in generating Implicit Neural Representations (INRs), surpassing the performance of all compared baselines according to metrics. The interpolation results show promise.", "weaknesses": "- Some important related work and baselines are missing, including three papers (references [1][2][3]) that explore the idea of using Mixtures of Experts (MoE) in INRs, as well as reference [4] as an additional image generation baseline and reference [5] as a strong baseline for shape generation. As a result, the proposed method does not appear entirely novel in the field and resembles existing latent code-based implicit representations, differing primarily in the incorporation of latent information through different computation paradigms. The use of MoE is also common in the community. It is recommended to exhaustedly discuss the difference between the aforementioned works and highlight key contributions.\n\n- The motivation for using the mixture of Neural Implicit Fields (NIFs) is mentioned as better memory efficiency, but it is recommended to directly compare run-time memory with the baselines. Additionally, it would be beneficial to include inference time comparisons with the presented methods.\n\n\n[1] NeurMiPs: Neural Mixture of Planar Experts for View Synthesis\n\n[2] Neural Implicit Dictionary via Mixture-of-Expert Training\n\n[3] Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields \n\n[4] Adversarial Generation of Continuous Images\n\n[5] HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments on improving the writing and analysis of the proposed method. We answered your two concerns on missing baselines and analysis on inference cost as below.\n\n## W1. missing related works and baselines\n\n### INR with mixture of experts\nThank you for suggesting the related research. The research you've mentioned is closely related in that it constitutes mixtures of experts. However, there are two main differences between our research and the one you've suggested. Firstly, our research dealt with generative neural fields or code-based implicit representation, whereas the cited study only tackled single scene fitting. The second difference is whether INR is used as an expert or not.\n\nResearch on mixtures of experts has two design points: 1) which experts to use and 2) how to combine the results of these experts. The cited studies employ INR as an expert and structure the model by either ensembling or selecting from the outputs of the experts. Our research significantly differs in that we construct the mixture using INR’s weight. This seems like a minor difference, but it showcases a major difference in terms of INR's efficiency.\n\nIn the case of [1, 2], calculations must be performed for all experts, so there's no computational advantage. In the case of [3], only the selected expert is used, and the rest are ignored, which offers a computational advantage. However, the full capacity of model isn't utilized at all.\n\nOur study avoids such trade-offs of mixtures of experts by structuring mixtures based on weights of INR. By doing so, the entire model capacity is utilized, and the INR operates with averaged-out weight, thereby promoting inference efficiency. This feature is especially efficient in 3D voxels or NeRF scenes with numerous input coordinate queries. To the best of our knowledge, there is no INR work on aggregating entire model weights within weight space. We will revise the paper to cover discussion referred works.\n\n### Additional baselines for generative neural fields\nIn our study, we focus on algorithms for generating neural fields; especially the algorithms that can handle **every modality such as 2D image, 3D voxel, and NeRF**. That’s why we take GEM, GASP, Functa as the baseline methods for our work. \n\nContrary to our baselines, INR-GAN [4] is only capable of image generation. INR-GAN relies on an image-specific convolutional discriminator while generator is based on INR. So, this work can not fully take the benefits of continuous representation of INR and. We will add this work as baseline of image-specific INR work in our manuscript with discussion.\n\nHyperDiffusion [5] is concurrent to our work, which will appear in ICCV23. The proposed framework is generic to modality, however, this work demonstrates generation on voxel and voxel video, not on images. We will be able to make a comparison after the release of the detailed implementation and introduced dataset.\n\n## W2. analysis on inference costs - speed and memory\nOur algorithm is efficient in terms of inference cost compared to our baselines. We measure the memory consumption and inference cost of a single sample (generated INR). Please check the Table A of the attached PDF for detailed information.\n\nGEM / Functa: Our algorithm shows efficient inference in all modalities (image, voxel, NeRF) with better performance. Notably in SRN Cars, our model demonstrates efficient computation and memory over Functa; **199 times less FLOPS, 49 times faster fps, and 22 times less memory** in single view inference. It reinforces our motivation for the proposed method.\n\nGASP: GASP shows more efficient inference compared to our models reported in the main paper. We additionally report the light configuration for our model on image and voxel. Light configuration of our model is the most efficient method in image and voxel and also outperforms the baselines in voxel. In image domain, our light model performs worse than GASP. We conjecture that FID metric over-penalizes blurriness, which is discussed in several works including Functa.\n\nWe count FLOPS with the fvcore library [6] and check inference time and memory usage with PyTorch Profiler [7].\n\n## Q1. training method throughout experiments\nWe use two steps for training our generative neural field.\nIn the first step, we conduct a context adaptation by meta-learning (algorithm1) or auto-decoding (algorithm2) to training our generative neural field. This procedure train neural implicit functions, or experts, in all mixtures to represent each sample (an image, a voxel, or a NeRF) in the training set with a context vector. As a result, the context vector modulates the neural field to represent a specific sample.\nIn the second step, the generative model is trained with the discovered context vectors (of training samples); Denoising diffusion probabilistic process is selected for the generative model. Trained generative model can generate a novel context vector and we can build a neural field with this context vector. \nThe advantage of this two-stage methodology is 1) acquiring compact representation for neural fields and 2) saving computational cost during training by decoupling the learning neural fields and the training generative model.\n\n## Q2. Missing voxel performance of Functa.\nWe can not reproduce the performance of Functa due to a longer training time of Functa on voxel generation. Note that Functa requires 25.5 times more computational costs in FLOPS than our model as reported in Table A of the attached PDF. We adopt reported performances from the baselines while Functa does not report reconstruction and generation performances on ShapeNet $64^3$ with evaluation metric introduced in GEM. We will report voxel performance of reproduced Functa with our training setting.\n\n[6] fvcore, https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md\n[7] PyTorch Profiler, https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html"}, {"review_id": "PMRq1LueLp", "reviewer": "Reviewer_uPSW", "summary": "The paper proposes a mixture of neural implicit functions for generating neural fields via model averaging of implicit representations trained through context adaptation (meta-learning/auto-decoding paradigm) and subsequently task-specific generalization (additional sampling strategy with respect to the latent mixture coefficient vector).", "questions": "Experiments lack quantitative evaluation/baseline. The authors are requested to clarify/highlight baseline models in proposed evaluation protocols.", "limitations": "- The proposed approach does not scale.\n- The current model exhibits aliasing artifacts in the neural radiance field scene. \n- There’s potential for bias within the system.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This well-written paper clearly elucidates its contributions with convincing results.", "weaknesses": "- The authors are requested to discuss limitations within the bounds of the main paper.\n- Minor Corrections\n    - Line 108, resepctively → respectively\n    - Line 226, the both → both the\n    - Line 278, effective learn → effectively learns\n    - Caption of Figure 1 ideally should elucidate the concept of neural radiance fields.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad to hear your positive comment. We will reflect the minor corrections you mentioned in the final version. The other concerns will be answered as below.\n\n> Experiments lack quantitative evaluation/baseline. The authors are requested to clarify/highlight baseline models in proposed evaluation protocols.\n\nSince this research area has actively developed recently, we were struggling to compare our method to the earlier methods. \n\nWe adopted the models (i.e., GEM, GASP, and Functa) that simultaneously deal with 2D images and 3D scenes as baselines These methods can be easily trained just by changing the dataset with equivalent architecture for all benchmarks with different modalities.\n\nWe follow the protocol from Functa for images and the protocol from GEM for voxels.  because the two models are two-stage generative INR and exploit context adaptation procedures. On evaluation, we used an evaluation protocol from Functa for images and NeRF and GEM for voxels. We will revise the paper to clarify the baselines."}, {"review_id": "UqguwSj9xh", "reviewer": "Reviewer_2vix", "summary": "This paper proposes an approach to increase the learning capacity of implicit neural networks (INRs). The idea is to define a set of basis implicit functions that can be mixed to produce an instance-specific INR for each instance in a dataset. Notably, the approach does not affect the inference performance as once an instance-specific INR is produced, the cost of coordinate sampling is the same as with any other INR. This method allows implicit neural networks to be generalized on entire datasets rather than overfitting to a single instance while still demonstrating high fidelity of samples. The training procedure is split into two steps, the first step being a meta learning or auto decoding problem to fit the basis INRs with a reconstruction loss to the training data, and the second step is training a latent model to generate samples that determine the mixing coefficients. The paper demonstrates solid results and evaluation on different datasets of varying modalities including images, voxels and NeRFs.", "questions": "Typos:\nLine 108: resepctively -> respectively\nLine 177: requires -> require\n\nI was wondering if the authors could add more interpolation results as in Figure 4, perhaps in the supplementary material. In particular, it would be interesting to see how interpolation works across ShapeNet categories.", "limitations": "Yes, limitations and societal impact are sufficiently discussed in the supplementary material.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is well-written and easy to follow. The methodology is well explained and is simple but effective. The inference performance is significantly better than three state-of-the-art methods considered, and so are the extensive generation metrics like FID, precision, recall, coverage etc. The methodology and design choices are comprehensively evaluated on multiple datasets, and modalities with ablation studies.", "weaknesses": "The paper makes a solid contribution and I do not see any major flaws.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for emphasizing the strengths of the proposed method and complimenting the current version of the work. \n\n## Q1. additional interpolation plots\nWe visualize additional inter-class interpolation results in Figure B of attached one-page PDF. We conduct bilinear interpolation with samples from four distinct classes in ShapeNet $64^3$.\nWe can observe that inter-class interpolation is still working as in intra-class interpolation although there exist some clutters with interpolation of dissimilar objects. We will update those visualizations in the future version of the submission."}, {"review_id": "suBFjYwIMQ", "reviewer": "Reviewer_ZMK5", "summary": "This paper deals with the problem of learning the latent space of data through the concept of meta-learning INRs (or neural fields). In contrast to prior works that consider either the adaptation of whole neural field parameters or some linear modulation parameters, the proposed method considers some basis neural field parameters as the shared parameters and adapts to coefficients for this basis to be adapted. The proposed method better encodes the given data of diverse modality, compared with prior INR-based data encoding scheme. ", "questions": "- Is there any constraints (or ranges) for the coefficients $\\alpha_i$s? ", "limitations": "The paper addresses the limitations and negative social impacts in the supplementary materials. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The observation that the linear combination of neural fields as basis function parameters is quite interesting and novel.\n- The quantitative results are better than prior works.\n- The paper is generally well-written and easy to follow, while some important details are missing in the current status (see the weakness below)\n", "weaknesses": "- For generating new data from the proposed framework, specification of the latent sampling strategy (e.g., what is a prior distribution of $\\phi$) is crucial, while the paper only states \"off-the-shelf sampling method\" in L179. Explaining how the sampling is performed would make the paper much stronger. \n- I guess this method requires much more GPU memory, as it uses multiple basis neural field parameters with the (second-order) gradient-based meta-learning method. In particular, this method requires about 10 times more parameters than Functa.\n- The experiments are mostly conducted under fine-grained datasets; I wonder if the method can be worked well with more complex datasets (e.g., CIFAR-10 or ImageNet-64x64) compared with other methods.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for complimenting the proposed model, experiments and writing quality. Our model can achieve both flexibility on model capacity and inference efficiency based on linear combination of model weights of neural fields, which is related to question 2. We answered your questions as below. We will revise our paper by reflecting your comments.\n\n## W1. explanation on latent diffusion\nWe adopt denoising diffusion probabilistic process (DDPM) [1] framework for modeling latent space for context vector. DDPM progressively denoises a random vector into a context vector during sampling from latent space. This sample process is done by the reverse process of DDPM. It is formulated as\n\n$p( \\mathbf{x} _T ) = \\mathcal{N} ( \\mathbf{x} _T ; \\mathbf{0} , \\mathbf{I}), p _{\\theta} ( \\mathbf{x} _{0:T}) = p (\\mathbf{x} _T \\prod _{t=1}^{T} p _\\theta (\\mathbf{x} _{t-1} | \\mathbf{x} _t ), p _\\theta (\\mathbf{x} _{t-1} | \\mathbf{x} _t ) = \\mathcal{N} (\\mathbf{x} _{t-1} ; \\mathbf{\\mu} _\\theta (\\mathbf{x} _t , t), \\mathbf{\\Sigma} _\\theta (\\mathbf{x} _t , t))\n$\n\nWe train a denoising model to denoise data by using the relationship between data and its corrupted one with Gaussian noise. This corrupting process is called forward process and formulated as follow.\n\n$q (\\mathbf{x} _t | \\mathbf{x} _0 ) = \\mathcal{N} ( \\mathbf{x} _t ; \\sqrt{\\overline{\\alpha} _t } \\mathbf{x} _0 , (1-\\overline{\\alpha} _t)\\mathbf{I}, q(\\mathbf{x} _t | \\mathbf{x} _{t-1} ) = \\mathcal{N} (\\mathbf{x} _{t-1} ; \\sqrt{1-\\beta _t} \\mathbf{x} _{t-1} , \\beta _t \\mathbf{I})$.\n\nwhere $\\alpha _t = 1 -\\beta _t$ and $\\overline{\\alpha} _t = \\prod _{s=1}^{t} \\alpha _s$.\n\nMean and variance of the reverse conditionals $\\mathbf{\\mu} _\\theta (\\mathbf{x} _t , t), \\mathbf{\\Sigma} _\\theta (\\mathbf{x} _t , t)$ by neural network. We adopt residual MLP introduced in Functa for our denoising model in latent space.\nThe parameter of denoising model can be optimized by minimizing\n\n$\\mathbb{E} _q [ KL [ q(\\mathbf{x} _{t-1} | \\mathbf{x} _t , \\mathbf{x} _0 || p _\\theta (\\mathbf{x} _{t-1} | \\mathbf{x} _t ]]$\n\nWe expect that tightly coupling two steps training procedure. A set of context vectors in our model is acquired by context adaptation procedure by Algorithm 1 or 2 in our paper. It is intialized as random vector and resulted into represent a single sample after the procedure. This process can be considered as a forward process in our model substituting fixed forward process by Gaussian noise. \n\n\n\n[1] Denoising Diffusion Probabilistic Models, NeurIPS 2020\n\n## W2. comparison to Functa in terms of memory\n\nOur model consumes much less GPU memory than Functa during **both training and inference time**. \n\n\nThe below table exhibits the GPU memory consumptions of our model and Functa; both models are trained with the meta-learning procedure on CelebA-HQ $64^2$ with batch size 32. Our method requires 4.3 and 5.9 times less GPU memory for training and inference, respectively, although it is true that our method requires more learnable parameters (Ours: 33.4M, Functa: 3.3M).\n\n\nMemory    | Ours\t\t| Functa\n| ------------- | ------------- | ------ |\nTraining (batch 32)\t| 9.3 GB \t| 40.2 GB \nInference \t\t| 24.4 MB\t| 144.1 MB\n\n\nIt is because that memory requirement for generative neural field is highly dependent on the size of inference parameter than that of learnable parameter due to the large memory consumption on tremendous input coordinate queries. This property is our main motivation for efficient design; aggregation INR weights within weight space. Note that our inference parameter is much smaller than those of Functa. \n\nAs the reviewer mentioned, a second-order gradient method requires more memory if the learnable parameter becomes larger. However, due to the **tremendous input coordinate queries** of this generative neural field, the impact of inference parameters exceeds the impact of the learnable parameters. This reasoning can be validated by comparing memory consumption during training and inference. Although training requires more memory than inference due to the second-order gradient method, the gap between the two methods is still maintained. So, this observation reinforces our motivation for the proposed method; reducing inference parameters.\n\n\n\n## W3. Beyond fine-grain dataset\nTraining on a more complex dataset, such as CIFAR10, is a good suggestion, but unfortunately, we need more time to analyze a model on a new dataset. Currently, we observe that our model works well on reconstruction while not on generation. We will discuss this experiment during the author-discussion period.\n\n\n## Q1. constraints for the coefficient\nWe have no constraint on the coefficient except for the initialization. As explained in section B in the supplementary material, we set alphas as 1/M at initialization to satisfy SIREN initialization scheme for shared and layerwise mixture coefficients, where M is the number of mixtures.\n"}, {"review_id": "FLA7FOCV0T", "reviewer": "Reviewer_YZcJ", "summary": "This paper proposes a new representation of neural fields based on a mixture of neural fields. Neural fields is another name for implicit neural representation (INR) which is a representation of data points (like images or 3d scenes), that maps an input of 2d or 3d position into the output of the RGB color and/or shape information.  \n\nThe paper builds on previous work like GASP [12], Functa [13] and GEM [1] and proposes a new representation on which the generative model is trained. As in [13] the generative model is a diffusion model and the novelty here is the representation which oi based on a mixture model.The mixture model representation proposed is implemented as a shared set of INRs which weights are averaged with a vector of mixture weights that are optimized per data-point.\n\nAfter describing the model, the paper demonstrates its effectiveness in different data modalities like images (Celeba-HQ), 3D shapes (ShapeNet) and 3D scenes (SRN Cars), where the model performs well compared to previous approaches.  Then some additional ablation and analysis are presented.", "questions": "1. The meta learning algorithm presented in algorithm 1:  why is $\\phi$ randomaly initialized at every iteration? This is different than MAML that learns an initialization, and CAVIA [17] (which the authors state as reference for this) that keeps the initialization constant. This is an interesting difference that should be discussed. \n2. Algorithm 2: The pseudo code is not clear about the indexes of $\\phi$ how are they being updated through the iterations?\n3. One aspect that is missing from all tables and comparisons is the size of the representation. For the proposed model this would be the number of mixture components times the number of layers. This is an important aspect of the model since it defines the rate-distortion tradeoff. The way the authors count parameters (learnable vs. inference) is interesting and of significant value, but adding the size of the representation is essential.\n4. How are the previous models chosen for comparison? These models usually contain several options with different sizes of the network and the representation, where the reconstruction quality usually grows with the size of the network.\n5. Line 115: What exactly is invariant to what?\n6. The ablation in the first section of 5.3 is not very useful. As stated above, it would be better to try to understand  what every component in the mixture captures. \n6. In figure 5, I'm not sure I understand what the plots show. Are the matrices sorted somehow?", "limitations": "The main limitation that is missing from the discussion in the paper is the relation of the quality of representation and its size (see weakness #2 and question #3).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The main strength of this paper is the novel INR representation proposed. This representation follows a recent line of work on the topic, that aims to leverage the advantages of INRs into the setup of training generative models on different data modalities. A main challenge there is capturing 3D data in an efficient way, which is still not acheivable in standard direct methods.  \nThe proposed model is an interesting direction with a potential to contribute to the research of this topic.", "weaknesses": "The main weaknesses of this paper are the following:\n1. The description of the model and the comparisons to previous work is lacking. Although some of the aspects are simple and clearly presented and the comparison to previous models is extensive and contains many different metrics, there still remains some important aspects that are not sufficiently described (see questions below).\n2. After describing the model, the authors \"jump\" to final metrics like PSNR and FIDs. Given that the main contribution of this work is the mixture representation itself, it is essential to show what this representation captures. Figure 5 is a good start in this direction, but what is still missing is some understanding of what the different components in the mixture are doing, and how they are combined. The authors could, for example, show the reconstructed images of different components in isolation as the number of components grow. Another component which is directly used in the model without investigation is the hierarchical mixture (mixture of every layer in the network). How does this affect the components? How are the differences in lower layer components manifested compared to differences in higher level components?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for complimenting our modulation scheme, mixtures of neural implicit functions for neural fields. Following part, we will describe the properties of our model to solve the raised concerns by the reviewer.\n\n\n\n## W1 Comparison to previous work\n\nIn our study, we focus on algorithms for generating neural fields; especially the algorithms that can handle **every modality such as 2D image, 3D voxel, and NeRF**. That’s why we take GEM, GASP, Functa as the baseline methods for our work, and we describe the comparison to the previous works on question 3 and 4 in below.\n\n## W2 lack of analysis on the proposing method\n\n\n### Ablations on mixture type, such as mentioned as hierarchical mixture\nIn Section C.2 of our supplementary material, we investigated the reason for selecting current architecture among three types of mixture coefficient: shared, layerwise (referred to as hierarchical mixture by the reviewer), and latent. \nBuilding upon the observation, we adopt the latent mixture coefficient for all our models in the experiments. We will reference this ablation study in the main paper to ensure clarity.\n\n### Deeper analysis on mixtures\nWe have observed that the information for each sample is distributed quite uniformly across all mixtures within the hidden layers.\nIn Figure A (a), we illustrate the images reconstructed by incrementally increasing the number of mixtures in each layer.\nTo visualize this figure, we need an order for the neural implicit functions (NIF) within each mixture. We begin by projecting a context vector into layerwise coefficients, as our model leverages latent mixture coefficients to merge these coefficients. Subsequently, we calculate the mean values of the layerwise coefficients across all training samples and sort for computing significance of each NIF in mixtures for every layer.\nIn Figure A (a), $M=0$ displays the reconstructed image using the mean vector of layerwise coefficients. $M=48$ presents the image reconstructed with layerwise coefficients from the 1st to the 48th important coefficients, with the remaining coefficients set to zero.\nFigure A (a) displays a gradual transformation of the reconstructed image. We hypothesize that this reflects the nature of the distributed representation in our methodology. This observation aligns with the visualization of diversity presented in Figure 5 of the main paper. This suggests that numerous NIFs contribute to the representation of an image.\n\n\nFigure A (b) shows reconstructed images relative to layer depth. In the top row, we illustrate the reconstructed image by progressively adding layerwise coefficients for the higher layers to the mean layerwise coefficients. In contrast, the bottom row demonstrates the inverse by systematically omitting layerwise coefficients from the lower to the higher layers. Notably, both the addition of the final layer and the removal of the initial layer have minimal impact on the reconstruction quality. Unlike the input and output layers, the transitions among the hidden layers are more pronounced. From these observations, we infer that the mixtures within all hidden layers encapsulate overarching information from the datasets. In contrast, the mixtures in the input and output layers seem to capture comparatively less information than the hidden layers.\n\n## Q1 initialization of context vector\nWe observe that random initialization is more stable than constant initialization in CAVIA with a large number of mixtures ($M >= 256$) in our model. \nIf the training process with CAVIA is done without collapsing, however, random initialization and constant initialization of context vectors.\n\n## Q2 how to update $\\phi$ in algorithm 2\nFor the auto-decoding with mNIF, $\\phi$ are parameters shared along all samples and $\\psi_j$ is a latent vector of $j$-th sample, where both of them are learnable parameters. \nAt every iteration, the shared parameter $\\phi$ and the latent vectors of samples in the current batch $\\psi_B$ are updated.\n\n## Q3 dimension of context vector\nWe use latent mixture coefficients with fixed hidden representation size denoted $H$. This hidden representation size is the dimension of context vector, because the projection matrix reduces the dimension of all alphas into the projected dimension as written in Eq 10.\n\nWe use 512 in image projected from 7 $\\times$ 384. and 1024 in voxel projected from 7 $\\times$ 384. In 128 in NeRF projected from 6 \\times 128.\nWe set the size of context vector as the best performed configuration from each model. Functa uses 256 for image, 64 for NeRF. GEM uses 1024 for image and voxel. GASP uses 64 for image and voxel.\n\n## Q4 How are the previous models chosen for comparison?\nWe adopt the model configurations with **the best generation performance** from the previous works. While the configuration of the existing model was fixed, the configuration of the proposed model was changed.\n\n## Q5 initialization of SIREN\nInvariant to output distribution at initialization.\n\n## Q6 analysis on mixtures\nPlease check the response to W2 in the above paragraph.\n\n## Q7 how to visualize diversity in Figure 5\nTo show the diversity of learned neural implicit functions, we compute the absolute value of cosine distance between two neural implicit functions in mixtures. This value becomes zero when two NIFs are orthogonal and becomes one if they are correlated. \n\nThe initial visualization of these values appears as random noise, due to the absence of a pre-defined order for the NIFs in the mixtures. To order each mixture, we follow the procedure outlined below:\nFirst, we compute this value for all mixtures and acquire pairwise value matrices for all mixtures.\nSecond, we take the largest value from pairwise values and take the pair indices (i, j) for each mixture.\nThird, we add pair indices i and j into the ordered list if an index is not in the list.\nFourth, iterate the second and the third step until consuming all indices for each mixture."}]

Author rebuttal:
We sincerely appreciate your valuable comments to improve our work. We presume that there are no strong negative comments and we earnestly responded to your concerns; please see the respective comments. Before answering your questions and concerns, we would like to highlight our contributions by quoting your comments.

Firstly, all the reviewers consented that our proposed method is novel and interesting; **Reviewer fjUK** commented *The proposed method is well-motivated and easy to understand,* **Reviewer uPSW** commented *This well-written paper clearly elucidates its contributions with convincing results,* **Reviewer 2vix** commented *The methodology is well explained and is simple but effective,* **Reviewer ZMK5** commented *The observation that the linear combination of neural fields as basis function parameters is quite interesting and novel,* and **Reviewer YZcJ** commented *The proposed model is an interesting direction with a potential to contribute to the research of this topic.*

Secondly, many reviewers left positive comments on the experimental results; **Reviewer fjUK** commented *The experiments, although conducted on simple datasets like CelebA and Shapenet, effectively validate the method's effectiveness and parameter efficiency in both 2D and 3D domains. (skipped) The interpolation results show promise,* **Reviewer uPSW** commented *This well-written paper clearly elucidates its contributions with convincing results,* **Reviewer 2vix** commented *The inference performance is significantly better than three state-of-the-art methods considered, and so are the extensive generation metrics like FID, precision, recall, coverage etc,* and **Reviewer ZMK5** commented *The quantitative results are better than prior works.*

Lastly, most of the reviewers agreed that our paper is well-written and well-organized; **Reviewer fjUK** commented *The paper demonstrates strong writing, with clear notation and intuitively-rendered figures,* **Reviewer uPSW** commented *This well-written paper clearly elucidates its contributions with convincing results,* **Reviewer 2vix** commented *The paper is well-written and easy to follow,* and **Reviewer ZMK5** commented *The paper is generally well-written and easy to follow, while some important details are missing in the current status.*


Meta-review: Author rebuttal addressed most of the reviewer concerns and all reviewers are leaning accept after final discussion.
Decision: Accept (poster)