Neural Experts: Mixture of Experts for Implicit Neural Representations

Authors: yizhak  ben-shabat
hewa  chamin
  koneputugodage
sameera  ramasinghe
stephen  gould
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 02:26:30

Abstract: Implicit neural representations (INRs) have proven effective in various tasks including image, shape, audio, and video reconstruction. These INRs typically learn the implicit field from sampled input points. This is often done using a single network for the entire domain, imposing many global constraints on a single function. 
In this paper, we propose a mixture of experts (MoE) implicit neural representation approach that enables learning local piece-wise continuous functions that simultaneously learns to subdivide the domain and fit it locally. 
We show that incorporating a mixture of experts architecture into existing INR formulations provides a boost in speed, accuracy, and memory requirements. Additionally, we introduce novel conditioning and pretraining methods for the gating network that improves convergence to the desired solution. 
We evaluate the effectiveness of our approach on multiple reconstruction tasks, including surface reconstruction, image reconstruction, and audio signal reconstruction and show improved performance compared to non-MoE methods. Code is available at our project page https://sitzikbs.github.io/neural-experts-projectpage/ .

Main paper: 
 introduction: 
 Implicit neural representations (INRs) are a type of neural network that can encode a wide range of signal types, including images, videos, 3D models, and audio [31, 27, 5, 39, 22] . Instead of storing discretely sampled signals (e.g., on a uniform grid), INRs approximate signals using a continuous function defined by a neural network. Given an input coordinate, the network is optimized to estimate the signal value at that coordinate. These neural representation networks have gained popularity because they can effectively fit even complex or high-dimensional signals for various tasks, including 3D reconstruction [31, 27, 2, 3, 14, 5] , novel-view synthesis [28, 4] , neural rendering [43] , and pixel alignment [34] .Current neural representation models, while powerful, are often limited by their architecture, which typically involves multi-layer perceptrons (MLPs). This design has two inherent drawbacks. The first is a parallelization and scale limitation: since INRs represented by a MLP process each coordinate by the whole network, all parameters have to contribute to the output of every point in the domain, making parameter optimization difficult. The second is a locality limitation: a desirable property of INRs is for features to change rapidly (to allow modelling of sharp boundaries) which does not arise naturally in optimized MLPs due to spectral bias [41] . Consequently, pure-MLP networks inhibit the ability to scale, fit the signal, and perform local operations on signals represented by the network.Recent advances in large language models (LLMs) have shown that facilitating scaling is crucial for performance [17, 45, 42] . In LLMs, this is often done using a mixture of experts (MoE) architecture [11, 16, 30, 38, 21] . MoEs provide inherent parallelizability, enabling efficient computation across different computation hardware. Additionally, MoEs inherently subdivide the input data, allowing the network to focus on distinct regions or features for more effective and specialized learning. While MoEs have proven their merit in various tasks (e.g., normal estimation [6] , multiclass classification [8] , and LLMs), they have so far been overlooked for INRs.In this paper we introduce Neural Experts, a novel mixture of experts architecture for implicit neural representations (MoE INRs) . This architecture can be applied to any existing MLP-based INR with enhanced levels of control over the network. Specifically, MoE INRs offer the flexibility to either explicitly control locality or allow it to be learned implicitly while also enabling new levels of parallelization. One intuitive explanation for the effectiveness of MoEs for INRs is that while traditional INRs approximate using continuous functions, MoEs can naturally represent using piecewise continuous functions, facilitated by the gating function of the manager network which has a global perspective of the signal throughout training. This effectively partitions the input space into regions of expertise for individual experts.We demonstrate the effectiveness of our proposed Neural Experts approach on a variety of applications including the representation of images, audio, and 3D shapes. Our work takes an important step towards making neural representations more flexible and effective for a wide range of tasks. Specifically, the key contributions of this work are:• Introducing Mixture of Experts for Implicit Neural Representations, which simultaneously subdivides the domain, reconstructs signals, and enables parallelization.• Deriving a novel manager architecture and initialization that enable domain subdivision without ground truth.• Demonstrating the effectiveness of our approach on a diverse set of applications in image, audio, and 3D surface reconstruction, showing improved reconstruction performance with fewer parameters compared to non-MoE methods. 
related work: 
 Implicit Neural Representations. Implicit Neural Representations (INRs) are continuous function approximators often based on multilayer perceptrons (MLPs). Their continuous nature benefits many reconstruction tasks but is particularly advantageous when dealing with irregularly sampled signals, such as point clouds [31, 2, 14] . However, standard MLPs have been shown to perform poorly when fitting signals with high frequencies [41, 39] . A common method to address this is to apply frequency encoding in the MLP, where frequencies are explicitly provided to the network. This can either be done by Fourier feature encoding [28, 41] or by using activation functions such as sinusoidal [39] , Gaussian [32] , wavelet [35] and sinc [37] functions. However these approaches are sensitive to the frequency bandwidth at initialization, and lack control of the bandwidth [5, 22] . Thus improvements include supplying a large bandwidth and reducing it over optimization using regularization [5] or explicit frequency decomposition [48, 22] .Another approach is to specialise subsets of the parameters for certain regions of the domain. This can be done by specialising parameters for each cell in a grid of the domain [7] , which can be improved by making it hierarchical [40] , adaptive [26] or operate on Laplacian pyramids [36] . Another method is to have a set of weights that get chosen based on the input's spatial position with a predetermined pattern [15, 23] , or by hashing [29] . However the former is inefficient parameter-wise for general use, where the complexity of the signal could be localized to specific regions, and the latter does not specialise the parameters to important regions naturally.Our method, on the other hand, allows the locally operating parameters (expert subnetworks), to change their local region during training. This is done by their joint optimization with the manager network that subdivides the domain.The mixture of experts framework [16, 18, 30] has shown to be very effective for different tasks. Its strengths stems from its architectural design that subdivides a network into multiple experts and a manager network (also referred to as gating or routing network), and its loss that encourages the manager to give higher weighting to better performing experts. This allows the network to subdivide the input space based on the task's loss and encourages different experts to specialize in reconstructing specific signals over that space.Many works have focused on the MoE framework including formulating the experts as Gaussian processes [44] , adding experts sequentially [1] , introducing hierarchy [47] , and ensembles-like formulation [13] . Recent advancements have shown that it is crucial for scaling up large language models [17, 45, 42] . In particular, since the capacity of a network is limited by the number of parameters it has, Shazeer et al. [38] propose MoE layers as a general purpose neural component to drastically scale parameters without a proportional increase in computation. This allows for improved parallelization and scaling [21] , but cannot use the original MoE loss and requires a balancing loss to ensure load balancing and sparsity. Various works propose to use this layer within INRs [49, 46] , however their use of the layers do not allow for sharp boundaries like the original MoE formulation.Given the advantages of MoEs, we propose to apply similar principals to INRs. Unlike a straightforward extension, we will show that training an INR-based manager requires careful initialization and conditioning for improved performance.It is important to note that the MoE framework can only work for INRs within a task that gives a loss per instance (which the MoE framework then changes to a weighted sum of losses over experts).Thus while the MoE framework can be used for many INR tasks (such as signal representation and reconstruction), it cannot be used for NeRF. Rebain et al. [33] show a NeRF-specific solution that significantly departs from the MoE-framework.The manager module is also composed of two components, both also modeled using MLPs. The manager encoder, parametrized by θ E m , receives the input samples x and outputs an intermediate representation Φ E m (x). A key component of our approach is to condition the manager using the signal. To do that, we concatenate the expert encoder Two key elements of our approach include the conditioning and pretraining of the manager that improve signal reconstruction with fewer parameters.output with the manager encoder output and feed that into the next MLP. This results in the manager's final output q = {q i | i = 1, . . . N experts }, which provides the selection criteria of each expert. More formally, let Φ m E (x) ⊕ Φ e E (x) be the concatenation of outputs from the manager encoder and expert encoder, respectively. ThenEQUATIONis the selection probability vector for all experts.Finally, let j = argmax{q 1 , . . . , q N } denote the index of the expert with largest selection probability.Then the reconstructed signal is given byEQUATIONIt is important to note that the expert outputs are chosen based on the manager's prediction before expert computation is executed. This approach facilitates parallelization of expert computations (at inference) by routing samples exclusively to the most suitable expert.Manager pretraining. For INRs, the initialization is crucial for performance [39, 2, 5] . Since the manager network is essentially another INR we propose to pretrain it to output a random uniform expert assignment. For that, we generate the random uniform assignment y seg as ground truth and train the manager using a cross entropy loss for each input coordinate,EQUATIONThe segmentation loss is then averaged over all input samples.This loss provides two main benefits. First, it shapes the distribution of q at initialization and second, it balances the assignment for the different experts. This balance is crucial for the MoE formulation as it helps prevent starving some experts while over-populating others.This loss can also be used during the training process when a ground truth segmentation is available and a segment-per-expert is desired. Note that, as can be expected, in some of our experiments this yielded lower reconstruction performance (additional constraint to satisfy) and therefore is not used in the reconstruction pipeline, however, having each expert correspond to a semantic entity may have benefits for downstream tasks.Note that manager pretraining is independent of the signal and therefore acts as an initialization.MoE loss. After pretraining the manager, we reconstruct the signals by training both manager and experts (including encoders) for a set amount t all of the training iterations using the MoE reconstruction loss: The reconstruction loss is then averaged over all input samples.L Recon-MoE (x) = 1 N experts Nexperts i=1 q i • Φ (i) e Φ E e (x) -y gt (x) 2 (4) (a) (b) (c) (d) (e) (f) (g) (h)During the final t e number of training iterations, we train only the experts in order to relax some of the instability caused by concurrently changing the manager assignments. In this stage, the parameters θ m , θ m E and θ e E are frozen and only the θ e (i) get updated. 
mixture of experts for implicit neural representations: 
 We present a novel approach for INRs that overcomes limitations by naively applying the mixture of experts method of Jacobs et al. [16, 30] in the reconstruction setting. Specifically, direct application fails to produce satisfactory results because the manager network is unable to fully take advantage of the supervisory signal provided to the experts. Our approach adapts the MoE architecture to share contextual information between manager and experts, and introduces a mechanism for pre-training the manager that avoids bad local minima and balances expert assignment.Preliminaries and Vanilla MoE INRs. The original formulation of Jacobs et al. [16, 30] encourages different sub-networks, called experts, to specialize on a subset of the data. This is done using a manager network that outputs a vector of probabilities for the selection of each expert q i . Let the parameters be θ m for the manager, and θ e (i) for the parameters for the i-th expert.As a straightforward extension and baseline, we implemented a naive extension (vanilla) of the Mixture of Experts to INRs as depicted in Figure 1b . In this vanilla formulation the manager's expert selection probability q = Φ m (x; θ m ) and the final reconstructed signal output Φ(x) = Φ (j) (x; θ (j) ), where j = argmax{q 1 , . . . , q N }, are modeled as MLPs. However, for implicit neural representations, this formulation does not capitalize on the clear benefits of subdividing the input space since it is susceptible to converging to local minima. Therefore we propose the neural experts formulation.Neural Experts. Our proposed architecture, illustrated in Figure 1c , is composed of two modules: the Manager and Experts modules. The experts act as the signal reconstructors while the manager acts as a routing function, indicating which expert should be used for each input sample x. For brevity, in the sequel, we will omit parameters from the MLP function Φ(x; θ) and denote it simply as Φ(x).1. Experts: The experts module is subdivided into two components, both modeled using multilayered perceptrons (MLP). The encoder, parameterized by θ e E , processes input samples x to produce an intermediate representation Φ e E (x), here the subscript refers to the module (experts), while the superscript denotes the encoder component within it (Encoder). This representation is then fed into the next component, which includes N experts different experts. Each expert has its own architecture and is parametrized by its own set of parameters θ e (i) , to output a reconstructed signal per expert Φ e (i) (Φ e E (x)).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
image reconstruction: 
 We conduct a comprehensive evaluation of image reconstruction on the full Kodak dataset [12] (24 images) in Table 1 . The reconstruction task aims to encode the input signal (image in this case) into the neural network. At test time, the coordinates are fed into the network and the reconstruced image is compared to the ground truth.The results show that our approach provides a significant boost in performance compared to other baselines. In this experiment we compare the proposed Neural Experts approach to prominent MLP architectures with Sof tP lus [2, 3] , Sof tP lus + F F [41] , Sine [39] and F IN ER [24] activations. For a fair comparison, we also implement a 'Wider' variant of these architectures that matches the INR's width with the combined number of elements over all experts. This results in an increased number of parameters for the 'Wider' variations. The results, presented in Figure 2 show that the proposed Neural Experts approach provides a significant boost in performance despite having fewer parameters than the 'Wider' baselines.In Table 2 Comparison to baselines. To demonstrate the versatility of our Neural Experts, we follow SIREN [39] and show its application to audio signal reconstruction. We evaluate the performance on raw audio waveforms of varying length clips of music (bach), single person speech (counting), and two person speech (two speakers). Table 3 shows the mean-squared error of the converged models. The results show that the proposed MoE architecture is able to outperform the MLP-based representations with over 40% fewer parameters. Our Neural Experts approach was able to converge quicker to a representation which can be played with barely noticeable distortion. All architectures were trained for 30k iterations, however noticeable performance gaps are already evident at 10k iterations (including manager pretraining).In Figure 3 we show qualitative results of the reconstructed audio signal of the two speakers signal compared to the ground truth. The results show that while reconstruct the signal quite well, the proposed approach yields lower errors. For more audio reconstruction visualizations please see the supplemental material.Using speaker identity as auxilary supervision. An interesting observation can be made from Figure 4 where we compare the reconstruction of the two speakers example with and without speaker identity segmentation information. Here, segmentation refers to labeling the time span for each one of the different speakers. The results show that the manager is able to learn this segmentation well, essentially allocating an expert per speaker (and an expert to background noise and gaps). Surprisingly, despite the additional segmentation constraint, this yields slightly better MSE score of 0.15 (compared to 0.16 without segmentation), however we witnessed that the segmentation supervision introduces slower convergence time (achieving comparable results only after ∼ 15k iterations. This result highlights possible future applications involving editing capabilities of INRs. We evaluate the surface reconstruction performance of our method by fitting signed distance functions (SDFs) on four shapes from the Stanford 3D scanning repository 2 . We focus on the quality of the reconstructed surface of the shape (i.e., the zero level set of the SDF). Therefore, we follow the setup from BACON [22] and optimise points primarily around the surface of the shape.In particular, we sample points from the surface and add Laplacian noise with two different standard deviations to give fine samples (samples close to the surface of the shape) and coarse samples (samples far from the surface of the shape). At these locations we optimise for the error between the SDF values and their ground truth SDF value. We provide further implementation details in the supplemental material. We compare our method to a SIREN network of similar number parameters at two different sizes, called Small and Large. The small and large variants have 8 layers with 256 and 512 hidden units respectively. Our method uses 8 experts and random pretraining, with the Large size having 512 units in the encoder, 64 units in the experts and 128 units in the manager, and the Small size having 256 units in the encoder, 32 units in the experts and 64 units in the manager. As we only care about where the surface is (or equivalently, the segmentation into inside and outside) we report IoU between the inside of the predicted shape and the inside of the ground truth shape. This is calculated on a 128 3 grid as per BACON. However, as the volume of the shapes are quite large, IoU is not a good measure of the quality of their boundary [10] . As a result, following the segmentation literature we use Trimap IoU [20, 9, 10] Results on the Thai Statue shape. Our method noticeably captures more detail in the toes, nostrils, and eye. The error colormap shows that our method produces a mesh with far less large errors (lighter indicates higher distance to the ground truth surface), and the expert segmentation shows our method provides a subdivision of the space.Conditioning. We analyze the effect of different conditioning options including no conditioning (Naive MoE), max or mean pooling over the expert encoder output and adding the output to the manager's encoder output, and concatenating the encoder outputs. The results, reported in Table 5 show that without conditioning, the method does not perform as well. Most importantly, concatenating the outputs of the encoders yields the best performance. Manager pretraining. We experiment with several different manager pretraining options including none, grid, random, kmeans segmentation, and SAM segmentation [19] pretraining, presented in Figure 6 . The pretraining aims to give a specific and predetermined expert assignment by the manager at initialization, so grid means that each pixel is assigned to the same expert based on a grid pattern, random uses a fixed sample from a uniform distribution for each expert, and the segmentation variants use a segmentation algorithm to determine the assignment. As an additional baseline we also report a constant manager that does not train at all and defacto acts as a constant routing component. Figure 6 also shows the manager assignment after the full training procedure. It can be seen that the manager is able to move away from its initialization and learn to cluster regions based on the input signal.The results, presented in Table 6 , show that pretraining the manager is crucial for the convergence of the proposed Neural Experts and that random pretraining performs best. It seems that a balanced assignment plays an important role, as highlighted by the grid and random results. Surprisingly, semantic segmentation does not provide a significant benefit for the reconstruction task, however, may still be useful for other tasks since it provides an INR per semantic entity (the expert). Note that except for SAM and Kmeans, pretraining the manager is independent of the reconstruction signal and can be done once and used as an initialization.We evaluate the performance of our method given the same architecture for the encoders and manager (excluding the manager's output layer) but with different numbers of experts. The PSNR results for the astronaut image are reported in that as more experts are introduced, the training becomes slightly noisier due to multiple pixel reassignments at every iteration. The results show a trade-off between the number of parameters and performance with more than double the training time for seven experts compared to two experts while also demonstrating an increase in PSNR of 20.55. Therefore, for our final model we opted to use four experts as a compromise between the desired performance and reasonable training time. It is important to highlight that while the number of experts has a significant impact at training time (since all experts are evaluated for training), the impact on inference time is negligible since only the selected expert is evaluated.Experts vs Layers. We evaluate the performance of our method with different expert count and expert architectures (layers count and width) while keeping the number of parameters approximately constant (∼ 366k). The results, reported in Table 8 , show a trend that less layers and more experts yield better performance for image reconstruction. Note that for our experiments we chose to use 4 experts with two layers as it provided robust performance across different modalities but for the specific case of image reconstruction 6 experts with a single layer yield the best performance.Parameter allocation. In Table 9 , we report results of the parameter allocation experiment, exploring the balance between the expert encoder, manager, and experts. With an approximately constant parameter count, we vary layer and element numbers to meet specific ratios. Results show robust performance across allocations, except when the manager is larger, as expected due to reduced signal capacity. See supplementary for architecture details. 
conclusion: 
 In this paper, we introduced a novel approach to implicit neural representations (INRs) through the incorporation of a mixture of experts (MoE) architecture. Our method addresses the limitations of traditional single-network INRs by enabling the learning of local piece-wise continuous functions. This approach facilitates domain subdivision, local fitting, and opens the potential for localized editing, which is a significant advancement over existing global constraint methods.Our results demonstrate that the Neural Experts framework significantly enhances accuracy, and memory efficiency across various reconstruction tasks, including 3D surfaces, image, and audio reconstruction. The incorporation of conditioning and pretraining methods for the manager network further improves are key elements, ensuring that the network reaches the desired solutions more effectively. Through extensive evaluations, we have shown that our MoE-INR approach outperforms traditional non-MoE methods in both performance and computational efficiency.Future work will focus on further exploring the potential of local editing capabilities. For example, our approach can be used alongside diffusion models to provide local signal generations. Additionally, our method can be extended to more complex and larger-scale models utilizing parallelization tools, e.g., sharding. We believe that the principles and techniques introduced in this paper can pave the way for more advanced and efficient INRs, facilitating broader adoption and application in diverse fields such as computer vision, graphics, and signal processing.Limitations and negative impact. Our method builds upon and extends existing INR approaches, and thus, it may inherit some limitations of the underlying networks. For example, it is known that SoftPlus MLPs have difficulty fitting low-frequency signals and therefore our SoftPlus MoE INR has the same limitation. Moreover, while increasing the number of experts in the current formulation enables parallelization and reduces computation time during inference, it also leads to slower training times as each additional expert requires more computations during training. This limitation is well known for MoE architectures and have been addressed in the literature [38, 21] for non-INR architectures and we believe this is an interesting avenue for future works.Our proposed Neural Experts approach enhances signal reconstruction in speed and accuracy, but it also brings potential societal impacts. Efficiently encoding data into neural network formats raises concerns about digital impersonation, unauthorized data replication, and harmful content creation. Additionally, learning 3D shapes as signed distance functions (SDFs) facilitates rendering objects, which could be exploited for DeepFakes and deceptive content, posing ethical and privacy risks. While these potential misuses are speculative and require advancements beyond our current method, it is important to remain vigilant and develop safeguards to ensure responsible use of this technology.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code is not available with the submission for anonymity reasons, however we will release the code in a github repository after acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The supplemental material contains all specific implementation details and the main paper contains an ablation study detailing how key parameters were selected.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Unlike most learning-based methods, our approach encodes signals using networks. Therefore, it is trained and tested on the same signal rendering error bars and significance tests difficult to justify. However, we report the mean and standard deviation on multiple different signals which gives some indication to the robustness of our approach.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: In the supplemental material we specify the GPUs used for performing the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: At the end of the paper there is a section discussing negative and positive impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does not poes such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, every baseline method and dataset used were cited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Below we provide the architecture and training details used in the different experiments throughout the paper as well as additional experimental results and visualizations.A.1 Image reconstruction Implementation details. In Section 4.1 we reported the performance of the proposed approach compared to other prominent architectures. For our approach we used 4 experts, 2 hidden layers for encoder and 2 for the experts. The manager has a similar architecture with 2 layers for the manager encoder and 2 for the final manager block. Each layer has 128 elements. All models were trained using an Adam optimizer, a learning rate of 10 -5 with exponential decay. All models are trained for 30K iterations where for our approach we use t all = 80% and t e = 20%, i.e. we train all parameters for the first 24K iteration and just the experts for the remaining 6K iterations. These models were trained on an NVIDIA A5000 GPU. All input images to the INR were cropped to be in a 1:1 ratio, scaled down by a factor of four and their coordinates were scaled to be in the unit cube to accelerate training (following [22] ).For a fair comparison we run a vanilla MLP with Sine or Softplus activations with a total of 4 hidden layers and 128 elements in each layer. However, this yields significantly fewer parameters so we implement a 'Wider' version that will have a similar 'width' as all experts combined width. Therefore the 'Wider' version has 512 elements for the last two hidden layers. This yields improved performance compared to the vanilla (naive) MoE implementation. However, when compared to our Neural Experts, even with its higher parameter count 'Wider' underperforms.Additional results. We report the performance of various activation functions and approaches on image reconstruction in Table 10 . We extend the results in the main paper and include results with InstantNGP as an example of newer hybrid INRs that use parameters for the encoding. The results show that our Neural Experts SIREN and Neural Experts FINER achieve superior performance compared to InstantNGP (89.35 and 90.84 PSNR vs 84.56 PSNR) with an order of magnitude fewer parameters (366k vs 7.7M). In the InstantNGP experiment, we use their default architecture for image experiments (16 encoding levels, 2 parameters per level, maximum cache size of 2 19 , and a 2 hidden layer decoder with 64 neurons). Note that most of InstantNGP's parameters are spatial parameters (hash encoding), specifically 99.86% of all parameters. This comparison between our Neural Experts SIREN/FINER and InstantNGP shows that the spatial parameters (parameters dedicated for specific spatial regions, so the experts and the hash encoding) are effective, but if used with patterns and heuristics, many of the spatial parameters become redundant and underutilized quickly. This motivates the learning of the spatial regions to achieve a parameter efficient approach (our Neural Experts method).Extending InstantNGP to include the MoE architecture is non-trivial as the hash encoding parameters, which make up over 99% of the total parameters, are used in the beginning of the network (to provide a specialized encoding to then go through a tiny MLP decoder). The obvious Neural Experts extension (which requires a shared input encoding), is to apply MoE to that tiny decoder, which we call Naive Neural Experts InstantNGP. We observe that the baseline InstantNGP performs better than this variant. However, the shared input hash encoding parameters are still over 99% of the total parameters. In the parameter allocation experiment Table 11 , the allocation ratio between the expert encoder, the experts and the manager plays an important role in the MoE's performance, with the best performing ratio being 14%, 55% and 31% respectively. So this naive result is to be expected. Constructing a more fair distribution of parameters would require a fundamental change to the InstantNGP backbone.Either we greatly increase the parameters of the expert and manager (which would make the total parameters another order of magnitude larger) or we need a way to split the hash encoding parameters into the experts. Both of these extensions are outside the scope of the current work and we leave this for future work.In Table 11 we extend Table 9 to provide exact architecture details and include the sizes of each component: expert encoding, experts, manager encoding and manager. These are given in the form of width x layers except for experts which is given as number of experts x ( width x layers ).Here, we also provide the results of our method on additional images from the Kodak dataset [12] . The results show that our method achieves improved performance consistently. Here, we provide qualitative results of our method compared to prominent baselines in Figure 12 .The results show that our proposed approach provides reduced errors. We also provide the resulting audio reconstruction .wav files in our supplementary material .zip file.Note that the architectural and training details for the audio signal reconstruction experiment are the same as in the image reconstruction experiments. Implementation Details. We first scale the input shape points to fit within the [-1, 1] 3 cube as this is the recommended domain for SIREN [39] , and take the domain to be the [-1.2, 1.2] 3 cube to ensure spacing around the scaled points. As this is larger than the [-0.5, 0.5] 3 cube used in BACON [22] , we double the standard deviation they use for the coarse and fine samples (4 • 10 -2 and 4 • 10 -6 respectively). We use an L1 loss and weight all points equally rather than an L2 loss and weight closer points more as done in BACON, as we find the former works better (for both the baseline and our method). We use 10k surface samples, 10k fine samples and 10k coarse samples per batch. We optimize our network for 30000 iterations using Adam, starting from a learning rate of 5 • 10 -3 and decreasing the learning rate by 0.9999 at each iteration. We run our surface reconstruction experiments on a single RTX 3090 (24GB VRAM).In BACON, the ground truth SDF for a point is approximated by finding the three closest points in the input and averaging their normal to determine the sign (whether the initial point is inside or outside the shape), while taking the distance to the closest point as the unsigned distance. We find that this causes spurious inside regions along medial axes where the mean of the normals cancel out, and instead use the closest point.The random pattern in the manager pretraining is done by dividing the domain into a 64 3 grid and assigning random experts for each grid cell. The IoU is computed on a 128 3 grid of the domain as per BACON, taking the grid points that predicted as inside as the set of interest. Trimap IoU with distance d is computed by only using grid points whose distance to the surface is less than or equal to d (as determined by the ground truth SDF at those grid points).Visualization is done by evaluating on a 512 3 grid and running marching cubes [25] .Further Results and Visualizations. We report IoU and Trimap IoU with different distance d in Table 12 and Table 13 . We also provide visualizations for each shape in Figure 13 , Figure 14 , Figure 15 and Figure 16 .Output Mesh Error Colormap Expert SegmentationFigure 13 : Results on the Armadillo shape. Our method performs better around the mouth and forearm region where there are high levels of detail. SIREN Large fits an oversmooth surface plus small inside surfaces in order to capture indentations, leading to poor Trimap IoU performance. Justification: We included a limitation section at the end of the paper, highlighting the drawbacks of the proposed approach. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: The paper conforms with the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://graphics.stanford.edu/data/3Dscanrep/ 


Decision: **Summary:**
Implicit neural representations (INRs) excel in tasks like image, shape, audio, and video reconstruction but usually rely on a single network, imposing global constraints. Contrary to the traditional, this paper introduced a mixture of experts (MoE), enabling local, piece-wise learning that improves domain fitting. Integrating MoE into INR frameworks boosts speed, accuracy, and memory efficiency, with new methods for gating network conditioning and pretraining further enhancing performance. The research direction is both intriguing and novel, making it well-suited for presentation at the conference.

**Strengths:** may include:
- "Neat architectural design" (aVob)
- Strong ablation study (aVob, VtKc, ij4R)
- "Consistently" outperformance (aVob)
- "Well-written and easy to follow" (VtKc, ij4R, ekS9)
- "a novel MoE-based architecture," "is novel" (ij4R, ekS9)

**Weaknesses:**
Initially, reviewers raised concerns about having "more parameters" than the baselines (aVob), the need for further discussion on comparative methods (e.g., spatial functa, KiloNeRF, InstantNGP, Gaussian Activation Function, WIRE, FINER) by the Reviewers aVob, VtKc, ij4R, and ekS9, "unstable convergence" (aVob), and "misalignment between the experimental results and the motivations" (VtKc). 

**Justifications for Acceptance:**
However, after the rebuttal and discussions with the reviewers, the authors provided experimental evidence and explanations for their decisions, which led to acceptance from all the reviewers. Especially, AC appreciates the authors' efforts and quick response to the request for additional evidence demonstrating the broad impact of their work by providing InstantNGP results, which sufficiently resolved the major issues raised. Based on this, AC is pleased to recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces a MoE architecture for INRs, enhancing scalability, local fitting, and parallelization. Traditional INRs use a single network, imposing global constraints, while this method learns several local expert functions, subdividing the domain. Key contributions include a novel manager architecture and initialization method for better convergence and domain subdivision without ground truth. It show improved performance and efficiency over traditional methods, across image, audio, and 3D surface reconstruction,
strength_weakness: strength_weakness: - It requires more parameters compared to the vanilla model. While it is fair to compare it with the wider version of the vanilla model, it is unclear if the proposed model still performs well when it has the same number of parameters as the vanilla model (not wider version). Tab.3 alleviates this concern to some extent.

- In addition to comparing with the vanilla model, it would be good to include a discussion on various INR methods that apply locality bias (e.g., spatial functa).

- As shown in the convergence graphs in the appendix, it shows more unstable convergence compared to the baseline.
questions: questions: - Please see the above weaknesses.
- In Tab.1, what is the PSNR of Neural Experts SIREN with the same number of parameters as SIREN?
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The paper presents a novel INR framework that leverages the Mixture of Experts (MoE) technique. The proposed strategy consists of an expert and a manager branch. Each branch has an encoder that processes the input coordinate and extracts an embedding. By processing the two encoder embeddings, the manager predicts the probability of which of the N experts should be used for extracting the signal. They show how the proposed INR framework achieves better reconstructions than SIREN on several modalities, such as audio, image, or 3D surfaces. They also propose a manager pre-training strategy, which is necessary to exploit all the experts effectively.
strength_weakness: strength_weakness: W1) The major weakness of the paper is the misalignment between the experimental results and the motivations of this research:

 a- In the introduction (L26-28), the authors correctly point out that, in traditional INRs, each coordinate needs to be processed by the whole network. Even though they claim that this problem can be solved by MoE INRs, in the proposed architecture, the input coordinate needs to be processed by the full manager and the encoder of the expert branch. The only saved computation is the one from the final expert, a much smaller network than the others. Thus, the saved computation looks minimal to standard SIREN (considering the same total weights). Moreover, no experiments regarding computational efficiency and the advantages of parallelized inference are necessary to motivate this claim. Maybe, instead of talking about the absolute efficiency of the proposed approach, it is better to show the better trade-offs in terms of efficiency and reconstruction quality than SIREN.

b- In the introduction (L28-30), the authors claim that standard INRs extract features vastly different for close spatial coordinates (i.e., locality problem). I am unaware of studies that formally investigate this INR property. Thus, I suggest adding a reference work or validating it with experiments. Moreover, the authors claim MoE INRs can learn local piece-wise functions (L259 in the conclusion section). Thus, they do not suffer from the problem above. Yet, the experiments show something different. For instance, by looking at Figure 3c, different experts predict audio signals for close temporal coordinates. I can notice the same behavior in the last column of Figure 6, in which many distinct experts predict pixels in the upper part of the image.

W2 ) The idea of using MoE resembles the idea of KiloNeRF [1]. In that case, the routing strategy is not learned and depends only on the input 3D coordinate, and each expert focuses on a pre-determined spatial region. I think the authors should add this reference, explaining the pros and cons of the two kinds of approaches.

W3) In recent years, INR frameworks have been proposed to be faster and more efficient than MLP-only techniques such as SIREN. For instance, hybrid INRs such as InstantNGP [2] (hash grid + MLP) can be used as the base framework to speed up computation when learning INRs of images and surfaces or NeRFs. The paper should also include more recent competitors than SIREN.


[1] Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. ICCV 2021.
[2] Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 2022.
questions: questions: Q1) In Table 1, does the vanilla MoE baseline employ Softplus activations, while the final strategy uses Sine activations? In this case, the comparison is unfair and does not validate the superiority of their approach to the vanilla one.

Q2) Can the authors also include the Chamfer Distance metric in Table 3?



I like the paper's core idea. However, the author's response to my concerns will greatly influence my final rating.
limitations: limitations: The paper includes a discussion on limitations.
paper_summary: paper_summary: This paper proposes a new architecture for implicit neural representations (INRs) based on the mixture-of-experts (MoE) architecture. This new architecture is differs from traditional MoE architectures in that now all the experts have a shared encoder and expert-specific decoders, while the manager also now has an encoder-decoder architecture, with the manager decoder taking as input both the manager encoder's representation as well as the experts' shared encoder representation. The authors also provide a pre-training method for the manager. The method is evaluated on image reconstruction, audio reconstruction, and 3D shape reconstruction by fitting signed distance functions (SDFs).
strength_weakness: strength_weakness: One of the major weaknesses of this paper is the experimental evaluation. The method does not compare against other methods that propose a new INR architecture (e.g. Gaussian activation function [1], WIRE [2], FINER [3]) or a standard MLP with positional encoding. The experimental evaluation is not also very robust, as only small datasets are used (Kodak, only 3 audio recordings, only 4 shapes) and more complicated tasks investigated by similar works are not considered (for example, WIRE [2] and FINER [3] both include evaluation on neural radiance fields). 

The proposed MoE architecture also did not show improvements when used with softplus activation. 

This paper also did not include other metrics normally used to evaluate the tasks, such as SSIM and LPIPS for 2D image fitting (e.g. FINER [3]). 

Minor point: inclulding the number of parameters in Table 1 may be helpful since comparisons between the number of parameters is discussed in the text.

References
1. Ramasinghe, Sameera, and Simon Lucey. "Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
2. Saragadam, Vishwanath, et al. "Wire: Wavelet implicit neural representations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
3. Liu, Zhen, et al. "FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: 1. Can this work be combined with traditional ReLU MLPs with positional encoding or MLPs with activation functions other than sine?
2. How does this work compare to other novel INR architectures (e.g. WIRE, FINER, etc)?
limitations: limitations: Limitations and negative impact are addressed by the authors.
paper_summary: paper_summary: This paper proposes a mixture of experts (MoE) approach for INRs, which allows the learning of local piece-wise continuous functions by subdividing the domain and fitting locally. The incorporation of a MoE architecture enhances speed, accuracy, and memory efficiency. They also propose a novel manager architecture and initialization that enable domain subdivision without ground truth.
strength_weakness: strength_weakness: 1. Missing some closely-relative works. I encourage the authors to have a detailed discussion of previous MoE INRs [1,2] and decomposition/partition-based INRs [3,4].
2. Lacking some key comparison experiments with decomposition/partition-based INRs. The authors only compare their method with the baseline SoftPlus and SIREN (and their wider version). However, some related works [4] have also shown the INR based on pre-determined masks can also outperform the wider version of SIREN.  I encourage the authors to experimentally compare your method with [4] to illustrate the necessity of learnable partition regions.
3. A detailed ablation study on the hyper-parameters of the MoE INRs is missing, such as the layer of encoder, manager, and experts. Given a fixed number of parameters, how to allocate the parameters to the three modules remains unknown.

[1] Zhao, Jianchen, et al. "MoEC: Mixture of Experts Implicit Neural Compression." arXiv preprint arXiv:2312.01361 (2023).
[2] Wang, Peihao, et al. "Neural implicit dictionary learning via mixture-of-expert training." International Conference on Machine Learning. PMLR, 2022.
[3] Rebain, Daniel, et al. "Derf: Decomposed radiance fields." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[4] Liu, Ke, et al. "Partition speeds up learning implicit neural representations based on exponential-increase hypothesis." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
questions: questions: 1. Could you please discuss how to choose the number of experts giving a fixed number of parameters and iteration time? Is it better to have more experts with each one having fewer parameters or fewer experts with each one having more parameters?
2. I wonder whether it is possible to apply your method to NeRF since there are no supervised signals for the 3D ground truth.
3. When comparing the Vanilla MoE INR and your Neural Experts, have you kept their total parameters similar (smaller experts for your Neural Experts due to the extra parameters needed by the encoders)?
limitations: limitations: The limitations and potential negative societal impact of their work have been discussed in the paper.

Comments: title: New results following discussion from Reviewer VtKc (W3)/Area chair 2LXj, and Reviewer ekS9
Official Comment by Reviewer ekS9
comment: I thank the authors for the responses. I appreciated the experiments with InstantNGP. Most of my concerns are addressed. Thus, I will raise my score to Weak Accept (6). 

I expect the authors to insert the new results with InstantNGP and update the introduction to align more with the considerations in W1a and W1b.
**Results with Random Pretraining the Vanilla SIREN MOE and discussion of MoEC**
We did not apply the random pretraining method for the Vanilla SIREN MOE as it is a contribution of our method that we proposed. We give results when adding the random pretraining to the Vanilla SIREN MOE in the updated Table 1 (put as a new general comment above). We see a significant increase in performance when adding random pretraining to the Vanilla SIREN MoE (62.98 vs 74.28 PSNR) while still having lower performance than our full model (89.35 PSNR). This shows that both random pretraining and the shared input encoder are important parts of the method. 

While the MoEC architecture is similar to our Vanilla SIREN MoE baseline, it does not follow the pure MoE framework from [16,18,28] that our method uses. Instead they use the "sparsely-gated mixture of experts layer" from [35] ([28] in their paper) where conditional computation is done within the network to produce a single output (note the shared decoder in their method, compared to the expert decoders in ours). Thus they cannot use the pure MoE loss, so similar to [35] they use a balancing loss to ensure load balancing and sparsity. Furthermore they only show the efficacy of their method on a single domain (one biomedical dataset) and one base architecture, while we show the efficacy of our method on multiple domains and multiple base architectures. However their motivation is similar to ours: show that using spatial parameters with learnable partitions can improve performance over using them with fixed partitions when constraining the number of parameters.
**Results with InstantNGP**: We give results with InstantNGP in the updated Table 1 (put as a new general comment above). The results show that our Neural Experts SIREN and Neural Experts FINER achieve superior performance compared to InstantNGP (89.35 and 90.84 PSNR vs 84.56 PSNR) with an order of magnitude fewer parameters (366k vs 7.7M). In the InstantNGP experiment, we use their default architecture for image experiments (16 encoding levels, 2 parameters per level, maximum cache size of 2^19, and a 2 hidden layer decoder with 64 neurons). Note that most of InstantNGP's parameters are spatial parameters (hash encoding), specifically 99.86% of all parameters. This comparison between our Neural Experts SIREN/FINER and InstantNGP shows that the spatial parameters (parameters dedicated for specific spatial regions, so the experts and the hash encoding) are effective, but if used with patterns and heuristics (like InstantNGP or the other methods discussed in lines 70-76) many of the spatial parameters become redundant and underutilized quickly. This motivates the learning of the spatial regions to achieve a parameter efficient approach (our Neural Experts method).

Extending InstantNGP to include the MoE architecture is non-trivial as the hash encoding parameters, which make up over 99% of the total parameters, are used in the beginning of the network (to provide a specialized encoding to then go through a tiny MLP decoder). The obvious Neural Experts extension (which requires a shared input encoding), is to apply MoE to that tiny decoder, which we call Naive Neural Experts InstantNGP. Results are reported in the updated table 1, and we observe that the baseline InstantNGP performs better than this variant. However, the shared input hash encoding parameters are still over 99% of the total parameters. As shown in our response to Reviewer ekS9, the parameter allocation ratio between the input encoder, the experts and the manager plays an important role in the MoE's performance, with the best performing ratio being 14%, 55% and 31% respectively. So this naive result is to be expected. Constructing a more fair distribution of parameters would require a fundamental change to the InstantNGP backbone. Either we greatly increase the parameters of the expert and manager (which would make the total parameters another order of magnitude larger) or we need a way to split the hash encoding parameters into the experts. Both of these extensions are outside the scope of the current work. 

Overall, these results show that learning regions for spatial parameters, as in our Neural Experts, achieve better performance with significantly fewer parameters.
**New Table 1** (page 5 of submission, updated for the second time) now updated to have comparison with InstantNGP as the base architecture and with random pretraining the Vanilla SIREN MOE.

| **Method**| **Parameters** | **PSNR Mean**| **PSNR Std**| **SSIM Mean**| **SSIM Std**| **LPIPS Mean** | **LPIPS Std** |
|-|-|-|-|-|-|-|-|
| Softplus| 99.8k| 19.51| 2.95| 0.7158 | 0.1106| 4.08e-1| 8.33e-2 |
| Softplus Wider| 642.8k | 20.91| 3.12| 0.7798 | 0.0899| 3.38e-1| 8.44e-2 |
| Ours Neural Experts SoftPlus| 366.0k | 20.62| 3.12| 0.7628 | 0.0946| 3.53e-1| 8.47e-2 |
| Softplus + FF | 99.8k| 28.97| 3.30| 0.9433 | 0.0193| 7.59e-2| 1.70e-2 |
| Softplus + FF Wider | 642.8k | 29.48| 3.91| 0.9436 | 0.0245| 7.74e-2| 2.30e-2 |
| Ours Neural Experts SoftPlus + FF | 366.0k | 31.66| 3.16| 0.9652 | 0.0180| 4.13e-2| 1.57e-2 |
| SIREN | 99.8k| 57.23| 2.46| 0.9991 | 0.0005| 5.78e-4| 5.09e-4 |
| SIREN Wider | 642.8k | 77.50| 5.32| 0.9996 | 0.0005| 3.08e-4| 3.04e-4 |
| Vanilla SIREN MOE | 349.6k | 62.98| 4.16| 0.9993 | 0.0005| 4.53e-4| 3.88e-4 |
| Vanilla SIREN MoE + Random Pretraining **(New)** | 349.6k | 74.28 | 7.36| 0.9996 | 0.0004 | 3.05e-4 | 2.88e-4 |
| Ours Neural Experts SIREN small | 98.7k| 63.42| 7.09| 0.9992 | 0.0007| 9.96e-4| 2.40e-3 |
| Ours Neural Experts SIREN | 366.0k | 89.35| 7.10| **0.9997** | 0.0004| 2.49e-4| 2.93e-4 |
| FINER | 99.8k| 58.08| 3.04| 0.9991 | 0.0005| 7.47e-4| 1.31e-3 |
| FINER Wider | 642.8k | 80.32| 5.40| 0.9996 | 0.0004| 2.49e-4| 2.46e-4 |
| Ours Neural Experts FINER | 366.0k | **90.84**| 8.14| **0.9997** | 0.0004| **2.46e-4**| 2.48e-4 |
| InstantNGP **(New)** | 7.7M | 84.56 | 5.62| 0.9996 | 0.0003 | 4.28e-4 | 5.37e-4 |
| Naive Neural Experts InstantNGP **(New)** | 7.7M | 75.14 | 2.57| 0.9996 | 0.0004 | 4.07e-4 | 4.35e-4 |
I appreciate the authors' effort in providing such a detailed rebuttal. 

One of my major concerns is the comparison of your paper with "MoEC: Mixture of Experts Implicit Neural Compression". It seems that their model is the vanilla MOE in your paper, right? Furthermore, I find that Random Pretraining for manager is very important to performance. When you conduct the vanilla MOE experiment, have you also applied the Random Pretraining for manager strategy? If not, I wonder whether the Random Pretraining for manager strategy can improve the performance of vanilla MOE.
Regarding W3, could you provide the results using InstantNGP as the reviewer VtKc requested? Considering the potential impact of this work, we look forward to seeing this comparison.
Thank you for the authors' response. After reading all the reviews and their responses, I found several potential weaknesses I had not noticed. But I think the authors addressed most of the major things pointed out. Their new experimental results addressed my concerns and the contribution about 'benefit of optimizing the regions and how to make this work in practice' is convincing to me.

I agree that recent INR methods that apply locality bias mentioned in the reviews have different main goals, but I'm still unsure if this method can have a meaningful impact on INR-based applications without such comparisons.

Therefore, I'd like to maintain my rating of 'weak accept'.