Neural Experts: Mixture of Experts for Implicit Neural Representations
Authors: Yizhak Ben-Shabat
Chamin P Hewa Koneputugodage
Sameera Ramasinghe
Stephen Gould
Venue: NeurIPS 2024 poster (2024)
Keywords: Implicit Neural Representation
Surface Reconstruction
Mixture of Experts
Primary area: deep_learning_architectures
TL;DR: We propose a mixture of experts architecture for reconstructing various signals (3D surfaces, images, audio) using a neural representation.

Abstract: Implicit neural representations (INRs) have proven effective in various tasks including image, shape, audio, and video reconstruction. These INRs typically learn the implicit field from sampled input points. This is often done using a single network for the entire domain, imposing many global constraints on a single function. 
In this paper, we propose a mixture of experts (MoE) implicit neural representation approach that enables learning local piece-wise continuous functions that simultaneously learns to subdivide the domain and fit it locally. 
We show that incorporating a mixture of experts architecture into existing INR formulations provides a boost in speed, accuracy, and memory requirements. Additionally, we introduce novel conditioning and pretraining methods for the gating network that improves convergence to the desired solution. 
We evaluate the effectiveness of our approach on multiple reconstruction tasks, including surface reconstruction, image reconstruction, and audio signal reconstruction and show improved performance compared to non-MoE methods. Code is available at our project page https://sitzikbs.github.io/neural-experts-projectpage/ .

Neural Experts: Mixture of Experts for Implicit Neural Representations
Yizhak Ben-Shabat"
Chamin Hewa Koneputugodage" The Australian National University chamin. hewaCanu edu. au
Roblox, The Australian National University sitzikbs@gmail. com
Sameera Ramasinghe
Stephen Gould
Amazon, Australia
The Australian National University s stephen goulddanu. edu. au
sameera. rameera.ramasinngnnn.. aameerra.raansssnhhheeeeellaaai edu, au
Abstract
Implicit neural representations (INRs) have proven effective various tasks in- cluding image, shape, audio, and video reconstruction. These INRs typically learn the implicit field from sampled input points. This often done using a single function. In this paper, we propose mixture of experts (MoE) implicit neural representation approach that enables learning local piece-wise continuous functions that simultaneously learns to subdivide the domain and fit it locally. We show that incorporating mixture of experts architecture into existing INR formulations provides a boost in speed, accuracy, and memory requirements. Additionally, we introduce novel conditioning and pretraining methods for the gating network that improves convergence to the desired solution. We evaluate the effectiveness of our approach on multiple reconstruction tasks, including surface reconstruction, image reconstruction, and audio signal reconstruction and show improved per- formance compared non-MoE methods. Code is available our project page https://sitzikbbs g. github... io wooo/nurallee -exprrteeeeetseeeeoo weeerts-projectte projectpage/.
single network for the entire domain, imposing many global constraints
1 Introduction
Implicit neural representations (INRs) are a type of neural network that can encode a wide range of signal types, including images, videos, 3D models, and audio [31, 27, 39, 22]. Instead of storing discretely sampled signals (e.g., on a uniform grid), INRs approximate signals using continuous function defined by neural network. Given an input coordinate, the network optimized to estimate the signal value at that coordinate. These neural representation networks have gained popularity because they can effectively fit even complex high-dimensional signals for various tasks, including 3D reconstruction [31, 27, 2, 3, 14, 5], novel-view synthesis [28, 4J. neural rendering [43], and pixel alignment [34].
Current neural representation models, while powerful, are often limited by their architecture, which typically involves multi-layer perceptrons (MLPs). This design has inherent drawbacks. The first is a parallelization and scale limitation: since INRs represented by MLP process each coordinate by the whole network, all parameters have contribute to the output of every point in the domain, making parameter optimization difficult. The second is a locality limitation: a desirable property of INRs is for features change rapidly (to allow modelling of sharp boundaries) which does arise naturally in optimized MLPs due to spectral bias [41]. Consequently, pure-MLP networks inhibit the ability scale, fit the signal, and perform local operations on signals represented by the network.
Equua contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Recent advances in large language models (LLMs) have shown that facilitating scaling is crucial for performance [17, 45, 42]. LLMs, this is often done using mixture of experts (MoE) architecture [11, 16, 30, 38, 21J. MoEs provide inherent parallelizability. enabling efficient computation across different computation hardware. Additionally, MoEs inherently subdivide the input data, allowing the network to focus on distinct regions or features for more effective and specialized learning. While MoEs have proven their merit in various tasks (c.g., normal estimation [6], multiclass classification [8], and LLMs), they have far been overlooked for INRs.
In this paper we introduce Neural Experts, novel mixture of experts architecture for implicit neural representations (MoE INRs). This architecture can be applied to any existing MLP-based INR with enhanced levels of control over the network. Specifically, MoE INRs offer the flexibility to either explicitly control locality or allow it to be learned implicitly while also enabling new levels of parallelization. One intuitive explanation for the effectiveness of MoEs for INRs that while traditional INRs approximate using continuous functions, MoEs can naturally represent using piecewise continuous functions, facilitated by the gating function of the manager network which has global perspective of the signal throughout training. This effectively partitions the input space into regions of expertise for individual experts.
We demonstrate the effectiveness of our proposed Neural Experts approach on a variety of applications including the representation of images, audio, and 3D shapes. Our work takes an important step towards making neural representations more flexible and effective for a wide range of tasks.
Specifically, the key contributions of this work are:
Introducing Mixture of Experts for Implicit Neural Representations, which simultaneously subdivides the domain, reconstructs signals, and enables parallelization.
Deriving a novel manager architecture and initialization that enable domain subdivision without ground truth.
Demonstrating the effectiveness of our approach diverse set of applications in image, audio, and 3D surface reconstruction, showing improved reconstruction performance with fewer parameters compared to non-MoE methods.
2
Related Work
Implicit Neural Representations. Implicit Neural Representations (INRs) are continuous function approximators often based multilayer perceptrons (MLPs). Their continuous nature benefits many reconstruction tasks but is particularly advantageous when dealing with irregularly sampled signals, such point clouds [31, 2, 14]. However, standard MLPs have been shown to perform poorly when fitting signals with high frequencies [41,39] A common method to address this is to apply frequency encoding in the MLP, where frequencies are explicitly provided to the network. This can either be done by Fourier feature encoding [28, 41] or by using activation functions such as sinusoidal [39], Gaussian [32], wavelet [35] and sinc [37] functions. However these approaches are sensitive the frequency bandwidth at initialization, and lack control of the bandwidth [5, 22]. Thus improvements include supplying large bandwidth and reducing it over optimization using regularization [5] or explicit frequency decomposition [48, 22].
Another approach is to specialise subsets of the parameters for certain regions of the domain. This can be done by specialising parameters for each cell in a grid of the domain [7], which can be improved by making it hierarchical [40], adaptive [26] operate
Laplacian pyramids [36]. Another method is to have a set of weights that get chosen based on the input's 's spatial position with a predetermined pattern [15, 23], or by hashing [29]. However the former is inefficient parameter-wist for general use, where the complexity of the signal could be localized to specific regions, and the latter does not specialise the parameters important regions naturally.
Our method, on the other hand, allows the locally operating parameters (expert subnetworks), to change their local region during training. This is done by their joint optimization with the manager network that subdivides the domain.
Mixture of Experts The mixture of experts framework [16, 18, 30] has shown to be very effective for different tasks. Its strengths stems from its architectural design that subdivides a network into multiple experts and manager network (also referred to as gating routing network), and its loss

---

that encourages the manager to give higher weighting to better performing experts. This allows the network subdivide the input space based on the task's loss and encourages different experts to specialize in reconstructing specific signals over that space.
Many works have focused the MoE framework including formulating the experts as Gaussian processes [44], adding experts sequentially [1]. introducing hierarchy [47], and ensembles- like formulation [13]. Recent advancements have shown that it is crucial for scaling up large language models [17, 45, 42]. In particular, since the capacity of a network is limited by the number of parameters it has, Shazeer et al. [38] propose MoE layers as a general purpose neural component to drastically scale parameters without a proportional increase in computation. This allows for improved parallelization and scaling [21], but cannot use the original MoE loss and requires a balancing loss to ensure load balancing and sparsity. Various works propose to use this layer within INRs [49, 46], however their use of the layers do not allow for sharp boundaries like the original MoE formulation.
Given the advantages of MoEs, we propose to apply similar principals to INRs. Unlike a straightfor- ward extension, we will show that training an INR-based manager requires careful initialization and conditioning for improved performance.
It is important to note that the MoE framework can only work for INRs within task that gives loss per instance (which the MoE framework then changes to a weighted sum of losses over experts). Thus while the MoE framework can be used for many INR tasks (such as signal representation and reconstruction), it cannot be used for NeRF. Rebain et al. [33] show a NeRF-specific solution that significantly departs from the MoB-frameworr.
3 Mixture of Experts for Implicit Neural Representations
We present a novel approach for INRs that overcomes limitations by naively applying the mixture of experts method of Jacobs et al. [16, 30] in the reconstruction setting. Specifically, direct application fails to produce satisfactory results because the manager network is unable to fully take advantage of the supervisory signal provided to the experts. Our approach adapts the MoE architecture share contextual information between manager and experts, and introduces mechanism for pre-training the manager that avoids bad local minima and balances expert assignment.
Preliminaries and Vanilla MoE INRs. different sub-networks, called experts, to specialize on subset of the data. This done using a manager network that outputs vector of probabilities for the selection of each expert q:. Let the parameters be 0n for the manager, and (.()) for the parameters for the i-th expert.
The original formulation of Jacobs et al. [16, 30] encourages
As straightforward extension and baseline, we implemented a naive extension (vanilla) of the Mixture of Experts to INRs as depicted in Figure 1b. this vanilla formulation the manager's expert selection probability q = 2m (z; 6m.) and the final reconstructed signal output ((x) x (a;; 801), where argmax (g). .. 9N ). are modeled as MIPs. However, for implicit neural representations, this formulation does not capitalize on the clear benefits of subdividing the input space since susceptible to converging to local minima. Therefore we propose the neural experts formulation.
Neural Experts. Our proposed architecture, illustrated in Figure 1c, composed of two modules: the Manager and Experts modules. The experts act as the signal reconstructors while the manager acts as a routing function, indicating which expert should be used for each input sample x. For brevity, in the sequel, we will omit parameters from the MLP function ((2; 6) and denote it simply as ((x).
The experts module subdivided into two components, both modeled using multi- layered perceptrons (MLP). The encoder, parameterized by 0.=. processes input samples to produce an intermediate representation & (1r), here the subscript refers to the module (experts), while the superscript denotes the encoder component within it (Encoder). This representation is then fed into the next component, which includes Nexpents different experts. Each expert has its own architecture and is parametrized by its own set of parameters 0,(), to output a reconstructed signal per expert $ (d. (x)).
Experts:
2. Manager: The manager module also composed of two components, both also modeled using MLPs. The manager encoder, parametrized by 0g. receives the input samples and outputs an intermediate representation & (xc). A key component of our approach to condition the manager using the signal. To do that, we concatenate the expert encoder
3

---

Coordinates x
Monager
Encoder
Cooedranne
Cooedirates
MLP 07
MLP
MLP tn
MLP
9:11
eoveffilt
+
Signal ()x;
Signal
Signal
Traditional INR
Vanilla MoE INR
(c) Our Neural Experts
Figure/ I: Illustration comparing between INR architectures for (a) traditional INR, (b) Vanilla MoE INR and (c) the proposed Neural Experts. Two key elements our approach include the conditioning and pretraining of the manager that improve signal reconstruction with fewer parameters.
output with the manager encoder output and feed that into the next MLP. This results in the manager's 's final output q {q: = 1..... Nexpents, which provides the selection criteria of each expert. More formally, let & (x) @ 4 be the concatenation of outputs from the manager encoder and expert encoder, respectively. Then
q(x) 4m ( (x) @ 4 (x))
(I)
is the selection probability vector for all experts.
Finally, let j argmax (egmaa(91...... (9I.... gN denote the index of the expert with largest selection probability, Then the reconstructed signal is given by
((x)
( (x))
It is important to note that the expert outputs are chosen based on the manager's prediction before expert computation executed. This approach facilitates parallelization of expert computations (at inference) by routing samples exclusively to the most suitable expert.
Manager pretraining. For INRs, the initialization is crucial for performance [39, 2, SJ. Since the manager network is essentially another INR we propose to pretrain it to output random uniform expert assignment. For that, we generate the random uniform assignment Useg as ground truth and train the manager using cross entropy loss for each input coordinate,
Lacg (x) (g( 0 0m. )
(scg(=))
The segmentation loss is then averaged over all input samples.
This loss provides two main benefits. First, it shapes the distribution of q at initialization and second, it balances the assignment for the different experts. This balance is crucial for the MoE formulation as it helps prevent starving some experts while over-populating others.
This loss can also be used during the training process when a ground truth segmentation is available and segment-per-expert is desired. Note that, as can be expected, in some of our experiments this yielded lower reconstruction performance (additional constraint satisfy) and therefore is not used the reconstruction pipeline, however, having each expert correspond to a semantic entity may have benefits for downstream tasks.
Note that manager pretraining is independent of the signal and therefore acts as an initialization.
MoE loss. After pretraining the manager, we reconstruct the signals by training both manager and experts (including encoders) for set amount tall of the training iterations using the MoE reconstruction loss:
Nespst
L (2x)
gi
($(1)) (0.(x:) - )r((x))
(4)
N Nexpents

---

70
20
Epada
(a)
(b)
(c)
(d)
(h)
Figure 2: Image reconstruction. Qualitative (left) and quantitative (right) results. Showing image reconstruction (top), gradients (middle), and laplacian (bottom) for (a) GT. (b) SoftPlus, (c) SoftPlus Wider, (d) Our SoftPlus MoE, (e) SIREN, (f) SIREN Wider, (g) Naive MoE, and (h) Our SIREN MoE. The quantitative results (right) report PSNR as training progresses and show that our Neural Experts architecture with Sine activations outperforms all baselines.
The reconstruction loss is then averaged over all input samples.
During the final f. number of training iterations, we train only the experts in order to relax some of the instability caused by concurrently changing the manager assignments. this stage, the parameters Om, 0 and 0g frozen and only the 0.(0) get updated.
4
Neural Experts Experimental Results
4.1 Image reconstruction
We conduct a comprehensive evaluation image reconstruction on the full Kodak dataset [12] (24 images) in Table 1. The reconstruction task aims to encode the input signal (image in this case) into the neural network. At test time, the coordinates are fed into the network and the reconstruced image is compared to the ground truth.
The results show that our approach provides a significant boost in performance compared other baselines. In this experiment we compare the proposed Neural Experts approach to prominent MLP architectures with Sof' [2, 3), Soft Plus + FF [41], Sine [39] and F N ER [24] activations. For fair comparison, we also implement a 'Wider' variant of these architectures that matches the INR's width with the combined number of elements over all experts. This results in an increased number of parameters for the 'Wider' variations. The results, presented in Figure 2 show that the proposed Neural Experts approach provides significant boost in performance despite having fewer parameters than the 'Wider' baselines.
In Table 2 we present additional experiments focused on the Sine activation function. We report results across several architecture variants, including smaller version of our Neural Experts model, baseline Vanilla MoE INR, and version of the baseline model enhanced with random pretraining. The results reveal several key trends: first, our method shows substantial improvement over the naive Vanilla MoE model; second, random pretraining significantly benefits the Vanilla MoE. model's performance; and finally, our smaller version of the Neural Experts model remains competitive. Addi- tional qualitative and quantitative results (including comparisons to hybrid methods like InstantNGP [29]), as well as specific architecture and training details, are available in the supplemental material.
4.2 Audio signal reconstruction
Comparison to baselines. To demonstrate the versatility of our Neural Experts, we follow SIREN [39] and show its application to audio signal reconstruction. We evaluate the performance on raw audio waveforms of varying length clips of music (bach), single person speech (counting), and two person speech (Iwo speakers). Table 3 shows the mean-squared error of the converged models. The results show that the proposed MoE architecture is able to outperform the ML.P-ased representations with over 40% fewer parameters. Our Neural Experts approach was able to converge quicker to representation which can be played with barely noticeable distortion.
5

---

Activation
Method
Params.
PSNR
SSIM
LPIPS
Mean
Std
Mean
Std
Mean
Std
Softplus Softplus
Base
99.8k
19.51
2.95
0.7158
0.1
4.08e-1
8.33e-2
Wider
642.8k
20.91
3.12
0.7798
0.0899
3.38c-1
8.44c-2
SoftPlus
Ours
366.0k
20.62
3.12
0.7628
0.0946
3,53c-1
8.47c-2
Softplus FF
Base
99.8k
28.97
3.30
0,9433
0.0193
7.59e-2
1.70e-2
Softplus FF
Wider
642.8k
29.48
3.91
0.9436
0.0245
7.74c-2
2.30c-2
SoftPlus + FF
Ours
366.0k
31.66
3.16
0,9652
0.0180
4.13c-2
1.57c-2
Sine
Base
99.8k
57.23
2.46
0.9991
0.0005
5.78e-4
5.09e-4
Sine
Wider
642.8K 8k
77.50
5.32
0,9996
0.0005
3.08e-4
3.04e-4
Sine
Ours
366.0k
89.35
7.10
0.9997
0.0004
2.49c-4
2.93e-4
FINER
Base
99.8k
58.08
3.04
0.9991
0.0005
7.47e-4
1.31e-
FINER
Wider
642.8k
80.32
5.40
0.9996
0.0004
2.49e-4
2.46e-4
FINER
Ours
366.0k
90.84
8.14
0,9997
0.0004
2.46e-4
2.48e-4
Table I: Image reconstruction with different activations. Reporting performance of various
activations the Kodak dataset [12]. Our approach outperforms the baselines (Base, Wider).
Method
Params.
PSNR
SSIM
LPIPS
Mean
Std
Mcan
Std
Mean
Std
99.8k
57.23
2.46
0.9991
0.0005
5.78e-4
5.09e-4
Ours small
98.7
63.42
7.09
0.9992
0.0007
9.96e-4
2.40e-3
Wider
642.8k
77.50
5.32
0.9996
0.0005
3.08e-4
3.04e-4
Vanilla MoE
349.6k
62.98
4.16
0.9993
0.0005
4.53e-4
3.88c-4
Vanilla MoE + Random Pretraining
349.6k
74.28
7.36
0.9996
0.0004
3.05e-4
2.88c-4
Ours
366.0k
89.35
7.10
0.9997
0.0004
2.49c-4
2.93c-4
Table 2: Image reconstruction architecture ablations with Sine activation. Comparing our
method to baselines on the Kodak dataset [12] with Sine activations. Our method shows substantial improvement over the naive Vanilla MoE model, random pretraining significantly enhances the Vanilla MoE model's performance, and our smaller Neural Experts model remains competitive.
All architectures were trained for 30k iterations, however noticeable performance gaps are already evident at 10k iterations (including manager pretraining).
In Figure 3 we show qualitative results of the reconstructed audio signal of the two speakers signal compared to the ground truth. The results show that while reconstruct the signal quite well, the proposed approach yields lower errors. For more audio reconstruction visualizations please see the supplemental material.
Using speaker identity auxilary supervision. An interesting observation can be made from Figure 4 where we compare the reconstruction of the two speakers example with and without speaker identity segmentation information. Here, segmentation refers labeling the time span for each one of the different speakers. The results show that the manager is able to learn this segmentation well, essentially allocating an expert per speaker (and an expert to background noise and gaps). Surprisingly, despite the additional segmentation constraint, this yields slightly better MSE score of 0.15 (compared to 0.16 without segmentation), however we witnessed that the segmentation supervision introduces slower convergence time (achieving comparable results only after 15k iterations. This result highlights possible future applications involving editing capabilities of INRs.
Architecture
Bach
Counting
Two Speakers
parameters
SIREN
0.71
36.6
2,06
100K
SIREN Wider
0.49
15.9
0.51
642K
Our SIREN MoE
0.12
1.72
0.16
365K
Table 3: Audio reconstruction. Reporting mean squared error (MiE), by 10 -4 for brevity. The results show that the proposed Neural Experts method is able significantly outperform non- MoE architectures while utilizing fewer parameters.

---

(a) SIREN
(b) SIREN Wider
(c) Our SIREN MoE
Figure 3: Audio reconstruction visualization. Two speakers audio reconstruction presented. Within each waveform block, the rows represent the ground truth, reconstruction, and error visu- alization from top to bottom. For our Neural Experts we color code the different experts on the reconstructed waveform.
(a) With segmentation supervision
(h) No segmentation supervision
Figure Speaker identity supervision experiment. Our Neural experts audio signal reconstruction with and without speaker identity segmentation supervision on the two speaker waveform. Colors represent expert number. The results show that the manager network is able allocate an expert to each speaker while not compromising the reconstruction quality.
4.3 Surface reconstruction results
We evaluate the surface reconstruction performance our method by fitting signed distance functions (SDFs) four shapes from the Stanford 3D scanning repository". 2 We focus the quality of the reconstructed surface the shape (i.e., the zero level set of the SDF). Therefore, we follow the setup from BACON [22] and optimise points primarily around the surface of the shape.
In particular, we sample points from the surface and add Laplacian noise with two different standard deviations to give fine samples (samples close to the surface of the shape) and coarse samples (samples far from the surface of the shape). At these locations we optimise for the error between the SDF values and their ground truth SDF value. We provide further implementation details in the supplemental material.
We compare our method to a SIREN network similar number parameters at two different sizes, called Small and Large. The small and large variants have 8 layers with 256 and 512 hidden units respectively. Our method uses 8 experts and random pretraining, with the Large size having 512 units in the encoder, 64 units in the experts and 128 units in the manager, and the Small size having 256 units in the encoder, 32 units in the experts and 64 units in the manager. As we only care about where the surface (or equivalently, the segmentation into inside and outside) we report IoU between the inside of the predicted shape and the inside of the ground truth shape. This is calculated on 1283 grid as per BACON. However, as the volume the shapes are quite large, IoU is not good measure of the quality of their boundary [10]. As result, following the segmentation literature we use Trimap IoU [20, 9, 10] with different boundary distances d, where Trimap IoU with distance d calculates IoU only on the grid points that are within distance d of the surface.
We show results with Trimap IoU (d 0.001) and Chamfer distance in Table and a qualitative comparison in Figure 5. Our method significantly outperforms the SIREN baseline at each model size, showing that our mixture experts more capable at capturing fine surface detail. Interestingly, our Small model performs better than the Large SIREN bascline on most shapes. We show more results (including different values of d) in the supplemental material.
inteps://graphio: stanford. wwwp///at//Dsaaareeeeeee
7

---

Method
Params.
Armadillo
Dragon
Lucy
Thai Statue
Mean
T-IoU
de
T-JoU
de
T-loU
de
T-loU
de
T-IoU
dr
SIREN I Large
1.3M
0.6981
1.4983
0.5517
2.2367
0.7958
1.2030
0.6191
5.8465
0.6662
5.4029
Ours Large
1.3M
0.9020
1.4975
0.7500
2.1340
0.8881
1.1322
0.7318
5,4084
0.8180
5,0905
SIREN Small
396k
0.5968
2.9165
0.5345
2.3944
0.6787
1.2583
0.5395
6.7273
0.5874
6.1553
Ours Small
323k
0.8200
1.5086
0.6800
2.2102
0.8272
1.1047
0.5787
5.9721
0.7265
5.1845
Table 4: 3D Shape reconstruction. Reporting the Trimap IoU for d
0.001
(I-IoUf) and Chamfer
distance xle5 (do-) for different shapes. The results show a significant boost in performance
compared to a larger MLP-based model with the same activation.
SIREN Large
Our Neural Experts Large
Output Mesh
Error Colormap
Final Expert Segmentation
Figure 5: 3D surface reconstruction. Results on the Thai Statue shape. Our method noticeably captures more detail in the toes, nostrils, and eye. The error colormap shows that our method produces a mesh with far less large errors (lighter indicates higher distance to the ground truth surface), and the expert segmentation shows our method provides subdivision of the space.
4.4 Ablation study
Conditioning, We analyze the effect of different conditioning options including conditioning (Naive MoE), max or mean pooling over the expert encoder output and adding the output to the manager's 's encoder output, and concatenating the encoder outputs. The results, reported in Table show that without conditioning, the method does not perform as well. Most importantly, concatenating the outputs of the encoders yields the best performance.
Pretraining
PSNR
Conditioning
PSNR
Fixed subdivision
53.39
No conditioning
64.52
Fixed random
53.58
max
77.3
None
56.31
mcan concatenate
77.62
SAM
58.88
81 .17
Kmeans
63.62
Table 5: Conditioning ablation. Without con- ditioning yields the worst performance however concatenation the experts encoder and manager encoder outperforms variations.
Grid
67.82
Random
81.17
Table 6: Pretraining analysis. Pretraining the manager output random expert assignment map is crucial for our method's performance.

---

Subdivision
None
SAM
Kmeans
Grid
Random
Figure 6: Manager pretraining. Visualizing the experts selected by the manager after the pretraining
stage (top row) and after the full network training (bottom row) for different pretraining ablations.
# Experts
Params.
Time
PSNR
2 266K
18.3
65.57
3
316K
22.3
71.70
Number of layers
4 366K
26.6
81.17
Experts
5
416K
29.7
77.08
79.39-E8.74
77.78+8.18 62.23 65.32-6.51
466K
33.3
82.68
89.35+7.100 81.19+8.04 74.36=7.31 92.32+7.80
7
S16K
36.5
82.34
99.1515.32 92.84.1.95 87.8011..98 75.26-9..02
Table 7: Number of experts ablation.
96.13.7.955 90.72.110.6 83.05 66.5718.72
There a trade-off between increasing the number of experts (leading to higher pa- rameters count and training time), and the improvement in PSNR performance. Time iteration time (ms).
Table 8: Experts vs. Layers ablation. The results show trend that fewer layers and more experts yield better performance.
Manager pretraining. We experiment with several different manager pretraining options including none, grid, random, kmeans segmentation, and SAM segmentation [19] pretraining, presented in Figure The pretraining aims to give specific and predetermined expert assignment by the manager at initialization, so grid means that each pixel is assigned to the same expert based random uses fixed sample from uniform distribution for each expert, and the segmentation variants use segmentation algorithm to determine the assignment. As an additional baseline we also report constant manager that does not train at and defacto acts as a constant routing component. Figure also shows the manager assignment after the full training procedure. It can be seen that the manager able to move away from its initialization and learn to cluster regions based on the input signal. The results, presented in Table 6, show that pretraining the manager is crucial for the convergence of the proposed Neural Experts and that random pretraining performs best. seems that a balanced assignment plays important role, highlighted by the grid and random results. Surprisingly, semantic segmentation does not provide a significant benefit for the reconstruction task, however, may still be useful for other tasks since it provides INR per semantic entity (the expert). Note that except for SAM and Kmeans, pretraining the manager is independent of the reconstruction signal and can be done
grid pattern,
and used as initialization.
Number Experts. evaluate the performance of our method given the same architecture for the encoders and manager (excluding the manager's output layer) but with different numbers of experts. The PSNR results for the astronaut image are reported in Table 7. important to note
Allocation Type
Expert Encoder
Experts
Manager
Parameters
PSNR
Larger Manager
14%
38%
48%
366. lk
$3.05=110.1
Larger Experts (Ours)
14%
55%
31%
366.0k
89.3547.10
Larger Encoder
54%
29%
17%
368.3k
88.12-16.92
Balanced
32%
34%
34%
368.0k
87.86.18.98
Table 9: Parameter allocation ablation. Neural Experts perform comparably (regardless of alloca- tion), except the case where the majority of the parameters are allocated to the manager.

---

that as more experts are introduced, the training becomes slightly noisier due to multiple pixel reassignments at every iteration. The results show trade-off between the number of parameters and performance with more than double the training time for seven experts compared to two experts while also demonstrating an increase in PSNR of 20.55. Therefore, for our final model we opted to use four experts as a compromise between the desired performance and reasonable training time. It is important to highlight that while the number of experts has a significant impact at training time (since all experts are evaluated for training), the impact on inference time is negligible since only the selected expert is evaluated.
Experts vs Layers. We evaluate the performance of our method with different expert count and expert architectures (layers count and width) while keeping the number of parameters approximately constant (~ 366k). The results, reported in Table 8, show a trend that less layers and more experts yield better performance for image reconstruction. Note that for our experiments we chose to use 4 experts with two layers as it provided robust performance across different modalities but for the specific case of image reconstruction experts with single layer yield the best performance.
Parameter allocation. In Table 9, we report results of the parameter allocation experiment, exploring the balance between the expert encoder, manager, and experts. With an approximately constant parameter count, we vary layer and element numbers to meet specific ratios. Results show robust performance across allocations, except when the manager is larger, as expected due to reduced signal capacity. See supplementary for architecture details.
5 Conclusion
In this paper, we introduced a novel approach to implicit neural representations (INRs) through the incorporation of a mixture of experts (MoE) architecture. Our method addresses the limitations of traditional single-network INRs by enabling the Icarning of local piece-wise continuous functions. This approach facilitates domain subdivision, local fitting, and opens the potential for localized editing, which is a sigmificant advancement over existing global constraint methods.
Our results demonstrate that the Neural Experts framework significantly enhances accuracy, and memory efficiency across various reconstruction tasks, including 3D surfaces, image, and audio reconstruction. The incorporation of conditioning and pretraining methods for the manager network further improves are key elements, ensuring that the network reaches the desired solutions more effectively. Through extensive evaluations, we have shown that our MoE-INR approach outperforms traditional non-MoE methods both performance and computational efficiency.
Future work will focus on further exploring the potential of local editing capabilities. For example, our approach can be used alongside diffusion models to provide local signal generations. Additionally, our method can be extended to more complex and orrger-scale models utilizing parallelization tools, e.g., sharding. We believe that the principles and techniques introduced in this paper can pave the way for more advanced and efficient INRs, facilitating broader adoption and application in diverse fields such as computer vision, graphics, and signal processing.
Limitations and negative impact. Our method builds upon and extends existing INR approaches, and thus, it may inherit some limitations of the underlying networks. For example, it is known that SoftPlus MLPs have difficulty fitting low-frequency signals and therefore our SoftPlus MoE INR has the same limitation. Moreover, while increasing the number of experts in the current formulation enables parallelization and reduces computation time during inference, it also leads to slower training times each additional expert requires more computations during training. This limitation is well known for MoE architectures and have been addressed in the literature [38, 21] for non-INR architectures and we believe this an interesting avenue for future works.
Our proposed Neural Experts approach enhances signal reconstruction in speed and accuracy, but it also brings potential societal impacts. Efficiently encoding data into neural network formats raises concerns about digital impersonation, unauthorized data replication, and harmful content creation. Additionally, learning 3D shapes as signed distance functions (SDFs) facilitates rendering objects, which could be exploited for DeepFakes and deceptive content, posing ethical and privacy risks. While these potential misuses are speculative and require advancements beyond our current method, it is important to remain vigilant and develop safeguards to ensure responsible use of this technology.
10

---

References
[1] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate; Lifelong learning with network of experts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3366-3375, 2017. 3
[2] Matan Atzmon and Yaron Lipman. SAL: Sign Agnostic Learning of shapes from raw data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2565-2574, 2020. 1,2, 4,5
[3] Matan Atzmon and Yaron Lipman. SALD: Sign Agnostic Learning with Derivatives. In International Conference on Learning Representations, 2021. 1,5
[4] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021. 1
[5] Yizhak Ben- Seaba,, Chamin Hewa Koneputugodage, and Stephen Gould. Digs: Divergence guided shape implicit neural representation for unoriented point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19323-19332, 2022. 1, 2,
[6] Yizhak Ben- Shabat, Michael Lindenbaum, and Anath Fischer. Nesti-net: Normal estimation for unstruc- tured 3d point clouds using convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 101 10112-10120, 2019.
[7] Rohan Chabra, Jan E Lenssen, Eddy Ilg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, Proceedings, Part XXIX 16, pages 608-625. Springer. 2020."
[8] Ke Chen, Lei Xu, and Huisheng Chi. Improved learning algorithms for mixture of experts in multiclass classification. Neural networks, 1229):229-12522, 1999. 2
[9] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin P. Murphy, and Alan Loddon Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Transactions Pottern Analysis and Machine Intelligence, 2016."
[10] Bowen Cheng, Ross Girshick, Piotr Dollár. Alexander C. Berg, and Alexander Kirillov. Boundary IoU: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[11] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, al. Glam: Afficient scaling of language models with mixture- of-experts. International Conference on Machine Learning, pages 5547-5569, PMIR, 2022.
[12] Eastman Kodak Company. Kodak Lossless True Color Image Suite. http:
kkodak/
5, 6, 14,
[13] Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1409 1418, 2016.
[14] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit Geometric Regularization for learning shapes. In International Conference Machine Learning, volume 119, pages 3789-3799.
PMLR, 2020. 1,2
[15] Zekun Hao, Arun Mallya, Serge Belongie, and Ming- Yu Liu. Implicit neural representations with levels- of-experts. Advances Neural Information Processing Systems, 35:2564-2576, 2022. 2
[16] Robert A Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79 87, 1991. 2,
[17] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 04088, 2024. 2, 3
[18] Michael I Jordan and Robert A Jacobs. Hicrarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994. 2
[19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023, 9
11

---

[20] Pushmeet Kohli, L' L'ubor Ladicky, and Philip H. S. Torr. Robust higher order potentials for enforcing label consistency. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008
[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. [GS] Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations. 2021. 2, 3, 10
[22] David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition, pages 16252-16262, 2022. 1, 2, 14, 19
[23] Ke Liu, Feng Liu, Haishuai Wang, Ning Ma, Jiajun Bu, and Bo Han. Partition speeds up learning implicit neural representations based on exponential-incrent hypothesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5474-5483, 2023.
[24] Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning in implicit neural representation by pariable-periodit activation functions. In Proceedings of the IEEE/CVF Computer Vision and Pattern Recognition Conference, 2024.
[25] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. Proceedings of the 14th annual conference on Computer graphics and interactive techniques, 1987. 19
[26] Julien N.P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstcin. Acorn: Adaptive coordinate networks for neural representation. In ACM Truns. Graph. (SIGGRAPHI), 2021. 2
[27] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowazin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4460 4470, 2019.
[28] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020. 1,2
[29] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1-15, 2022. 2,5
[30] Steven Nowlan and Geoffrey E Hinton. Evaluation of adaptive mixtures of competing experts. Advances
in Neural Information Processing Systems, 3, 1990. 2, 3
[31] Jeong Joon Park, Peter Florence. Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019. 1,2
[32] Samecra Ramasinghc and Simon Lucey. Beyond periodicity: Towards unifying framework for activations in coordinate-mlps. In European Conference on Computer Vision, pages 142-158. Springer, 2022. 2
[33] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In JEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 14148-14156, 2021. 3
[34] Shunsuke Saito, Zeng Huang, Ryota Natsume. Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304-2314, 2214,,14,222 2019. 1
[35] Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In P Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 18507-18516, 2023. 2
[36] Vishwanath Saragadam, Jasper Tan, Guha Balakrishnan, Richard G Baraniuk, and Ashok Veeraraghavan. Miner: Multiscale implicit neural representation. European Conference on Computer Vision, pages 318-333. Springer, 2022. 2
[37] Hemanth Saratchandran, Sameera Ramasinghe, Violetta Shevchenko, Alexander Long, and Simon Lucey. A sampling theory perspective on activations for implicit neural representations. arXiv preprint arXiv:2402.0m 05427, 2024. 2
12

---

[38] Noam Shazeer, *Azalia Mirhoseini. *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gade -atedd mixture-of-experts layer. In International Conference on Learning Representations. 2017. 2,3,
[39] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances Neural Information Processing Systems, 33.7462-7473, 2020. 1, 4,
[40] Towaki Takikawa, Jocy Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real time rendering with implicit 3D shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[41] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances Neural Information Processing Systems, 33:7537-7547, 2020. 1, 2,
[42] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint 24Xv::2033.88295 2024. 2,
[43] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pages 701-727. Wiley Online Library, 2020. 1
[44] Lucas Theis and Matthias Bethge. Generative image modeling using spatial Istms. Advances in Neurul Information Processing Systems, 28, 2015. 3
[45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anna Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini wxXiv:2302.13971 2023. 3,
[46] Peihao Wang. Zhiwen Fan, Tianlong Chen, and Zhangyang Wang. Neural implicit dictionary via mixture- of-expert training. In International Conference Machine Learning, 2022. 3
[47] Bangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-Fei. Hicrarchical mixture of classification experts uncovers interactions between brain regions. Advances in Neural Information Processing Systems, 22, 2009. 3
[48] Wang Yifan, Lukas Rahmann, and Olga Sorkine-homung. Geometry-connisstent neural shape representation with implicit displacement fields. International Conference on Learning Representations. 2022. 2
[49] Jianchen Zhao, Cheng-Ching Tseng, Ming Lu, Ruichuan An, Xiaobao Wei, He Sun, and Shanghang Zhang. Moee: Mixture experts implicit neural compression. 2023. 3
13

---

A Appendix supplemental material
Below we provide the architecture and training details used in the different experiments throughout the paper as well as additional experimental results and visualizations.
A.1 Image reconstruction
Implementation details. In Section 4.1 we reported the performance of the proposed approach compared to other prominent architectures. For our approach we used 4 experts, 2 hidden layers for encoder and 2 for the experts. The manager has similar architecture with 2 layers for the manager encoder and 2 for the final manager block. Each layer has 128 elements. All models were trained using an Adam optimizer, a learning rate of 10 with exponential decay. All models are trained for 30K iterations where for our approach we use tan = 80% and te 20%, i.e. we train all parameters for the first 24K iteration and just the experts for the remaining 6K iterations. These models were trained on an NVIDIA A5000 GPU. All input images to the INR were cropped to be in 1:1 ratio, scaled down by a factor of four and their coordinates were scaled to be in the unit cube to accelerate training (following [22]).
For fair comparison we run a vanilla MLP with Sine or Softplus activations with total of 4 hidden layers and 128 elements in each layer. However, this yields significantly fewer parameters so we implement 'Wider' version that will have a similar 'width' as all experts combined width. Therefore the 'Wider' version has 512 elements for the last two hidden layers. This yields improved performance compared to the vanilla (naive) MoE implementation. However, when compared to our Neural Experts, even with its higher parameter count 'Wider' underperforms.
Additional results. We report the performance of various activation functions and approaches on image reconstruction in Table 10. We extend the results in the main paper and include results with InstantNGP as example of newer hybrid INRs that use parameters for the encoding. The results show that our Neural Experts SIREN and Neural Experts FINER achieve superior performance compared to InstantNGP (89.35 and 90.84 PSNR vs 84.56 PSNR) with an order of magnitude fewer parameters (366k vs 7.7M). In the InstantNGP experiment, we use their default architecture for image experiments (16 encoding levels, 2 parameters per level, maximum cache size of 219, and a 2 hidden layer decoder with 64 neurons). Note that most of InstantNGP's parameters are spatial parameters (hash encoding). specifically 99.88 99.86% of all parameters. This comparison between our Neural Experts SIREN/FINER and InstantNGP shows that the spatial parameters (parameters dedicated for specific spatial regions, so the experts and the hash encoding) are effective, but if used with patterns and heuristics, many of the spatial parameters become redundant and underutilized quickly. This motivates the learning of the spatial regions to achieve a parameter efficient approach (our Neural Experts method).
Extending InstantNGP to include the MoE architecture non-trivial as the hash encoding parameters, which make up over 99% of the total parameters, are used in the beginning of the network (to provide a specialized encoding then go through a tiny MLP decoder). The obvious Neural Experts extension (which requires a shared input encoding), is to apply MoE to that tiny decoder, which we call Naive Neural Experts InstantNGP. We observe that the baseline InstantNGP performs better than this variant. However, the shared input hash encoding parameters are still over 99% of the total parameters. In the parameter allocation experiment Table 1, the allocation ratio between the expert encoder, the experts and the manager plays an important role in the MoE's performance, with the best performing ratio being 14%, 55% and 31% % respectively. So this naive result to be expected. Constructing a more fair distribution of parameters would require a fundamental change to the InstantNGP backbone. Either we greatly increase the parameters of the expert and manager (which would make the total parameters another order of magnitude larger) we need way to split the hash encoding parameters into the experts. Both of these extensions are outside the scope of the current work and we leave this for future work.
In Table 11 we extend Table 9 to provide exact architecture details and include the sizes of each component: expert encoding, experts, manager encoding and manager. These are given in the form of width x layers except for experts which given as number of experts ( width x layers) ).
Here, we also provide the results of our method on additional images from the Kodak dataset [12]. The results show that our method achieves improved performance consistently.
14

---

Activation
Method
Params.
PSNR
SSIM
LPIPS
Mean
Std
Mean
Std
Mcan
Std
Softplus Softplus
Base
99.8k
19.51
2.95
0.7158
0.1106
4.08e-1
8.33e-2
Wider
642.8k
20.91
3.12
0.7798
0.0899
3.38e-1
8.44c-2
SoftPlus
Ours
366.0k
20.62
3.12
0.7628
0.0946
3.53e-1
8.47c-2
Softluus FF
Base
99.8k
28.97
3,30
0.9433
0.0193
7.59e-2
1.70e-2
Softplus FF
Wider
642.8k
29.48
3.91
0.9436
0.0245
7.74e-2
2.30c-2
SoftPlus FF
Ours
366.0k
31.66
3.16
0,9652
0.0180
4.13e-2
1.57c-2
Sine
Base
99.8k
57.23
2.46
0,9991
0.0005
5.78e-4
5.09e-4
Sine
Wider
642.8k
77.50
5.32
0.9996
0.0005
3.08c-4
3.0cc44
Sine
V. MoE
349.6k
62.98
4.16
0.9993
0.0005
4.53e-4
3.88c-4
Sine
V. MoE + RP
349.6k
74.28
7.36
0,9996
0.0004
3.05e-4
2.88c-4
Sine
Ours small
98.7k
63.42
7.09
0,9992
0.0007
9.96e-4
2.40e-3
Sine
Ours
366.0k
89.35
7.10
0.9997
0.0004
2.49e-4
2.93e-4
FINER
Base
99.8k
58.08
3.04
0.9991
0.0005
7.47c-4
1.31c-
FINER
Wider
642.8k
80.32
5,40
0,9996
0.0004
2.49c-4
2.46c-4
FINER
Ours
366.0k
90.84
8.14
0.9997
0.0004
2.46c-4
2.48e-4
InstantNGP
Base
7.7M
84.56
5.62
0.9996
0.0003
4.28c-4
5,37c-4
InstantNGP
Naive NE
7.7M
75.14
2.57
0.9996
0.0004
4.07e-4
4.35c-4
Table 10: Image reconstruction. Comparing all approaches the Kodak dataset [12]. Our approach
reconstructs the image more faithfully and outperforms the baselines. FF: Fourier Features, RP: Random Pretraining, V. MoE: Vanilla MoE.
Architecture
Percentages Enc.
Allocation Type
Exp. Enc., Exp., Man. Enc., Man. Exp.
Exp.
Man.
Parameters
PSNR
Larger Manager
128x2, 4x(102x)), 128x2, 196x2
14%
38%
48%
366.1k lk
83.05:110.11
Larger Experts (Ours)
128x2, 4x(12xx), 128x2, 128x2
14%
55%
31%
366.0k
89.35:7.100
Larger Encoder
256x2, 4x(68x2), 96x2, 68x2
54%
29%
17%
368.3k
88. 18.992
Balanced
196x2, 4x(85x2), 128x2, 128x2
32%
34%
34%
368.0k
87.86
Table 11: Parameter allocation ablation. We update Table 9 to include the sizes for each component.
20
13502
(a)
(b)
(c)
(d)
(c)
(f)
(g)
(h)
Figure Image reconstruction qualitative (left) and quantitative (right) results. Showing image reconstruction (top), gradients (middle), and laplacian (bottom) for (a) GT. (b) SoftPlus, (c) SoftPlus Wider, (d) Our SoftPlus MoE, (e) SIREN, (f) SIREN Wider, (g) Naive MoE, and (h) Our SIREN MoE. The quantitative results (right) report PSNR as training progresses and show that our MoF architecture outperforms all baselines.
15

---

10
5000 10000
3000
(a)
(b)
(c)
(d)
(c)
f
(h)
Figure 8: Image reconstruction qualitative (left) and quantitative (right) results. Showing image reconstruction (top), gradients (middle), and laplacian (bottom) for (a) GT, (b) SoftPlus, (c) SoftPlus Wider, (d) Our SoftPlus MoE, (e) SIREN, (f) SIREN Wider, (g) Naive MoE, and (h) Our SIREN MoE. The quantitative results (right) report PSNR as training progresses and show that our MoE
architecture outperforms baselines.
100
10
2000
25000
(a)
(b)
(c)
(e)
(1)
g
Epeda
Figure 9: Image reconstruction qualitative (left) and quantitative (right) results. Showing image reconstruction (top), gradients (middle), and laplacian (bottom) for (a) GT, (b) SoftPlus, (c) SoftPlus Wider, (d) Our SoftPlus MoE, (e) SIREN, (f) SIREN Wider, (g) Naive MoE, and (h) Our SIREN MoE. The quantitative results (right) report PSNR as training progresses and show that our MoE architecture outperforms all baselines.
2140 10000)
13000
20000
25000
30000
(b)
(c)
(1)
(h)
Figure 10: Image reconstruction qualitative (left) and quantitative (right) results. Showing image reconstruction (top), gradients (middle), and laplacian (bottom) for (a) GT. (b) SoftPlus, (c) SoftPlus Wider, (d) Our SoftPlus MoE, (e) SIREN, (f) SIREN Wider, (g) Naive MoE, and (h) Our SIREN MoE. The quantitative results (right) report PSNR as training progresses and show that MoE architecture outperforms all baselines.
16

---

180
PSOK
15580
2000
2500
(a)
(b)
(c)
(f)
(g)
(h)
1pech
Figure 11: Image reconstruction qualitative (left) and quantitative (right) results. Showing image reconstruction (top), gradients (middle), and laplacian (bottom) for (a) GT, (b) SoftPlus, (c) SoftPlus Wider, (d) Our SoftPlus MoE, (e) SIREN, (f) SIREN Wider, (g) Naive MoE, and (h) Our SIREN MoE. The quantitative results (right) report PSNR as training progresses and show that our MoE architecture outperforms all baselines.
17

---

A.2 Audio signal reconstruction experiments
Here, we provide qualitative results our method compared prominent baselines in Figure 12. The results show that our proposed approach provides reduced errors. We also provide the resulting audio reconstruction wav files in our supplementary material .zip file.
Note that the architectural and training details for the audio signal reconstruction experiment are the same in the image reconstruction experiments.
-44
60
-55
(a) SIREN
(b) SIREN Wider
(c) Our SIREN MoE
Figure 12: Audio reconstruction visualization. Bach, counting, and two spreakers audio presented in the top, middle and bottom row respectively. Within each waveform block, the rows represent the ground truth, reconstruction, and error visualization from top to bottom. For our Neural Experts we color code the different experts on the reconstructed waveform.
18

---

Mean
Method
# params
IoU
IoU(0.1)
loU(0.01)
IoU(0.001)
SIREN Large
1.5M
0.9865
0,9886
0.9576
0.6662
MoE Large
1.3M
0.9912
0.9936
0.9770
0.8180
SIREN Small
396k
0.9786
0,9802
0.9235
0.5874
MoE Small
323k
0.9834
0.9853
0.9476
0.7265
Table 12: IoU and Trimap IoUs of 3D reconstruction. IoU(d) indicates Trimap IoU with boundary
region of radius (IoU is computed within the region of distance d of the ground truth boundary).
Armadillo
Dragne
Lacy
Thai Statue
Mean
Method
IoU
loU(0.001)
IoU
loU10.001)
IoU
loU10.001)
IoU
loU(0.001)
loU000001)
SIREN Large
1.5M
0.9965
0.6981
0.9861
0:5517
0.9930
0.7958
0.9705
0.6191
0.9865
0.6662
McE Large
1.M
0,9982
0.9020
0.9915
0,7500
0,9970
0.8881
0.9782
0.7318
0,9912
0.8180
SIREN Small
396k
0.9920
0.5968
0.5345
0.9855
0.5395
MoE Small
323k
0.9977
0,8200
0.9899
0.6800
0,9950
0.8272
0.9509
0.5787
0,9834
0.7265
Table 13: IoU and Trimap IoU of 3D reconstruction. IoU(d) indicates Trimap IoU with boundary
region of radius (IoU is computed within the region of distance d of the ground truth boundary).
A.3 Surface Reconstruction
Implementation Details. We first scale the input shape points to fit within the [-1, 1|3 cube this is
the recommended domain for SIREN [39], and take the domain to be the [-1.2, 1.2] cube to ensure spacing around the scaled points. As this is larger than the [-0.5, 0.5) cube used BACON [22], we double the standard deviation they use for the coarse and fine samples (4 10 and 4 10-6 respectively). We use an L1 loss and weight all points equally rather than an L2 loss and weight closer points more as done in BACON, as we find the former works better (for both the baseline and our method). We use 10k surface samples, 10k fine samples and 10k coarse samples per batch. We optimize our network for 30000 iterations using Adam, starting from a learning rate of 5 10-3 and decreasing the learning rate by 0.9999 at each iteration. We run our surface reconstruction experiments on single RTX 3090 (24GB VRAM).
In BACON, the ground truth SDF for a point approximated by finding the three closest points sign (whether the initial point is inside or outside the shape), while taking the distance the closest point as the unsigned distance. We find that this causes spurious inside regions along medial axes where the mean of the normals cancel out, and instead use the closest point.
in the input and averaging their normal to determine the
The random pattern in the manager pretraining is done by dividing the domain into 64 grid and assigning random experts for each grid cell. The loU is computed per BACON, taking the grid points that predicted as inside as the set interest. Trimap loU with distance is computed by only using grid points whose distance to the surface is less than equal to d (as determined by the ground truth SDF at those grid points).
128 grid of the domain as
Visualization is done by evaluating a 512 grid and running marching cubes [25].
Further Results and Visualizations. We report IoU and Trimap IoU with different distance in Table 12 and Table 13. We also provide visualizations for each shape in Figure 13, Figure 14, Figure 15 and Figure 16.
19

---

SIREN Large
Our Neural Experts Large
SIREN Large
Our Neural Experts Large
Output Mesh
Error Colormap
Expert Segmentation
0.000
0.002
0.004
0.006
0.008
0.010
Figure 13: Results on the Armadillo shape. Our method performs better around the mouth and forearm region where there are high levels of detail. SIREN Large fits an oversmooth surface plus small inside surfaces in order to capture indentations, leading to poor Trimap IoU performance.
20

---

SIREN Large
Our Neural Experts Large
SIREN Large
Our Neural Experts Large
Output Mesh
Error Colormap
Expert Segmentation
0.002
0.008
Figure 14: Results on the Dragon shape. Our method captures more detail around the jaws such as indentations.
21

---

SIREN Large
Our Neural Experts Large
SIREN Large
Our Neural Experts Large
Output Mesh
Error Colormap
Expert Segmentation
0.001
0.002
0.004
0.005
Figure 15: Surface reconstruction results on the Lucy shape. Our method performs overall better but in particular at the torch region where there are high levels of detail.
22

---

SIREN Large
Our Neural Experts Large
SIREN Large
Our Neural Experts Large
Output Mesh
Error Colormap
Expert Segmentation
0.001
0.002
0.004
Figure 16: Results on the Thai Statue shape. Our method noticeably captures more detail in the toes, nostrils, and eye.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We proposed a new method for a mixture of experts -based implicit neural representation and provide ample evaluations to demonstrate its effectiveness in various tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: included a limitation section at the end of the paper, highlighting the drawbacks of the proposed approach.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
24

---

Answer: [NA]
Justification: The paper does not propose a new theory and therefore does not include its assumptions and results. We propose new method and specify the full pipeline, well as the mathematical formulation of it.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The full details for reproducing the results presented in the paper are available. The main paper contains the principals and main architecture of the method while specific implementation details are available in the supplemental. The code will be made publicly available upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
25

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is not available with the submission for anonymity reasons, however we will release the code in a github repository after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ct cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The supplemental material contains specific implementation details and the main paper contains an ablation study detailing how key parameters were selected.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Unlike most learning-basee methods, our approach encodes signals using networks. Therefore, it is trained and tested the same signal rendering error bars and significance tests difficult to justify. However, we report the mean and standard deviation on multiple different signals which gives some indication to the robustness of our approach.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the supplemental material specify the GPUs used for performing the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eurrrp.eeeee cc cc/ppblic////cccccccccccccccccct htt.:hhhhiiiieieeeeeeee...
Answer: [Yes]
Justification:
The paper conforms with the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
27

---

At the end of the paper there is section discussing negative and positive impacts.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not poes such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, every baseline method and dataset used were cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
28

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "GIREWRcLxn", "reviewer": "Reviewer_aVob", "summary": "This paper introduces a MoE architecture for INRs, enhancing scalability, local fitting, and parallelization. Traditional INRs use a single network, imposing global constraints, while this method learns several local expert functions, subdividing the domain. Key contributions include a novel manager architecture and initialization method for better convergence and domain subdivision without ground truth. It show improved performance and efficiency over traditional methods, across image, audio, and 3D surface reconstruction,", "questions": "- Please see the above weaknesses.\n- In Tab.1, what is the PSNR of Neural Experts SIREN with the same number of parameters as SIREN?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It demonstrates a neat architectural design along with a robust ablation study.\n- It consistently shows performance improvement across various tasks, and the performance with respect to the number of parameters is also superior.\n- It is interesting that random initialization, which includes no inductive bias in the manager pretraining process, outperforms initializations like SAM.", "weaknesses": "- It requires more parameters compared to the vanilla model. While it is fair to compare it with the wider version of the vanilla model, it is unclear if the proposed model still performs well when it has the same number of parameters as the vanilla model (not wider version). Tab.3 alleviates this concern to some extent.\n\n- In addition to comparing with the vanilla model, it would be good to include a discussion on various INR methods that apply locality bias (e.g., spatial functa).\n\n- As shown in the convergence graphs in the appendix, it shows more unstable convergence compared to the baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: We have added results in the updated Table 1 in General Comments (see \"Ours Neural Experts SIREN small **(New)**\"). This version has a width of 68 in each layer (encoding, experts and manager) instead of 128, leading to 98,686 parameters. Note that it still outperforms the SIREN baseline, which has 99k parameters. \n\nW2: See comment in General Comments.\n\nW3: Yes, the convergence is unstable as we optimizing both the parameters and the regions. However our results show that even with unstable convergence we have a consistent performance increase from our baselines, and thus show the promise of this line of work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0NlZwfPKj5", "reviewer": "Reviewer_VtKc", "summary": "The paper presents a novel INR framework that leverages the Mixture of Experts (MoE) technique. The proposed strategy consists of an expert and a manager branch. Each branch has an encoder that processes the input coordinate and extracts an embedding. By processing the two encoder embeddings, the manager predicts the probability of which of the N experts should be used for extracting the signal. They show how the proposed INR framework achieves better reconstructions than SIREN on several modalities, such as audio, image, or 3D surfaces. They also propose a manager pre-training strategy, which is necessary to exploit all the experts effectively.", "questions": "Q1) In Table 1, does the vanilla MoE baseline employ Softplus activations, while the final strategy uses Sine activations? In this case, the comparison is unfair and does not validate the superiority of their approach to the vanilla one.\n\nQ2) Can the authors also include the Chamfer Distance metric in Table 3?\n\n\n\nI like the paper's core idea. However, the author's response to my concerns will greatly influence my final rating.", "limitations": "The paper includes a discussion on limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "-\tThe original idea might be conducive to new research in this direction.\n-\tThe paper is well-written and easy to follow. \n-\tSupervising experts with semantic losses and obtaining networks specialized in specific semantic areas of the input signal might unlock several applications for INRs and ease their interpretability.\n-\tThe proposed framework achieves good reconstruction performance.\n-\tThe ablations in Table 4 and Table 5 are very insightful.", "weaknesses": "W1) The major weakness of the paper is the misalignment between the experimental results and the motivations of this research:\n\n a- In the introduction (L26-28), the authors correctly point out that, in traditional INRs, each coordinate needs to be processed by the whole network. Even though they claim that this problem can be solved by MoE INRs, in the proposed architecture, the input coordinate needs to be processed by the full manager and the encoder of the expert branch. The only saved computation is the one from the final expert, a much smaller network than the others. Thus, the saved computation looks minimal to standard SIREN (considering the same total weights). Moreover, no experiments regarding computational efficiency and the advantages of parallelized inference are necessary to motivate this claim. Maybe, instead of talking about the absolute efficiency of the proposed approach, it is better to show the better trade-offs in terms of efficiency and reconstruction quality than SIREN.\n\nb- In the introduction (L28-30), the authors claim that standard INRs extract features vastly different for close spatial coordinates (i.e., locality problem). I am unaware of studies that formally investigate this INR property. Thus, I suggest adding a reference work or validating it with experiments. Moreover, the authors claim MoE INRs can learn local piece-wise functions (L259 in the conclusion section). Thus, they do not suffer from the problem above. Yet, the experiments show something different. For instance, by looking at Figure 3c, different experts predict audio signals for close temporal coordinates. I can notice the same behavior in the last column of Figure 6, in which many distinct experts predict pixels in the upper part of the image.\n\nW2 ) The idea of using MoE resembles the idea of KiloNeRF [1]. In that case, the routing strategy is not learned and depends only on the input 3D coordinate, and each expert focuses on a pre-determined spatial region. I think the authors should add this reference, explaining the pros and cons of the two kinds of approaches.\n\nW3) In recent years, INR frameworks have been proposed to be faster and more efficient than MLP-only techniques such as SIREN. For instance, hybrid INRs such as InstantNGP [2] (hash grid + MLP) can be used as the base framework to speed up computation when learning INRs of images and surfaces or NeRFs. The paper should also include more recent competitors than SIREN.\n\n\n[1] Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. ICCV 2021.\n[2] Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1)a: The limitation we are mainly interested in is learning capacity rather than computation time. For traditional INRs, as each coordinate needs to be processed by the whole network, all parameters have to contribute to the output of every point in the domain, making parameter optimization difficult. For our proposed network, we have subnetworks that are specialised for particular regions of the domain, and thus it is easier to optimise towards better solutions (as our results show). We will make this more clear in our revised version. As the reviewer mentions, saved computational time would be minimal (and highly dependent on hardware architecture), hence we have not explored this.\n\nW1)b: We apologize for the poor wording regarding locality limitation. What we meant is that a desirable property of INRs is for features to change rapidly (to allow modelling of sharp boundaries). This can't be achieved in networks without explicit spectral bias (which we mention in lines 62-66). We will update the paper to clarify this. Our method gives an alternative way for handling sharp changes, i.e., by changing the expert, which is a more natural way to handle discontinuous signals. This is consistent with your observations in Figures 3c and 6. \n\nW2): We will add this reference. Apart from the obvious difference (ours is an MoE so can learn different regions, while they have fixed regions in a grid structure), KiloNeRF's priority is post optimization rendering speed, hence their simple structure. Their architecture leads to insufficient performance when optimizing directly (see their Figure 3), so they rely on distillation to train. Our method on the other hand is an alternative for the original network that optimizes directly with better signal reconstruction performance, which is not the case for KiloNeRF. Furthermore we show improvement over grid based methods in Table 5.\n\nW3): See general comment. We leave combining our MoE approach with hash based methods as future work. Also note that our updated table 1 has comparision with FINER from CVPR2024 (see general comment).\n\nQ1): No, vanilla MoE was with Sine activations, we will change the name to Vanilla SIREN MoE to make it more clear\n\nQ2): We have added the Chamfer distance now (see table in general comment).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I4bIAuzhXO", "reviewer": "Reviewer_ij4R", "summary": "This paper proposes a new architecture for implicit neural representations (INRs) based on the mixture-of-experts (MoE) architecture. This new architecture is differs from traditional MoE architectures in that now all the experts have a shared encoder and expert-specific decoders, while the manager also now has an encoder-decoder architecture, with the manager decoder taking as input both the manager encoder's representation as well as the experts' shared encoder representation. The authors also provide a pre-training method for the manager. The method is evaluated on image reconstruction, audio reconstruction, and 3D shape reconstruction by fitting signed distance functions (SDFs).", "questions": "1. Can this work be combined with traditional ReLU MLPs with positional encoding or MLPs with activation functions other than sine?\n2. How does this work compare to other novel INR architectures (e.g. WIRE, FINER, etc)?", "limitations": "Limitations and negative impact are addressed by the authors.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "This paper proposes a novel MoE-based architecture for INRs together with a novel pre-training strategy for the MoE manager. \n\nThe empirical results are good and there is an ablation study on the major components. \n\nThe paper is well-written and easy to understand. \n\nMany details relating to reproducibility are provided in the supplemental material.", "weaknesses": "One of the major weaknesses of this paper is the experimental evaluation. The method does not compare against other methods that propose a new INR architecture (e.g. Gaussian activation function [1], WIRE [2], FINER [3]) or a standard MLP with positional encoding. The experimental evaluation is not also very robust, as only small datasets are used (Kodak, only 3 audio recordings, only 4 shapes) and more complicated tasks investigated by similar works are not considered (for example, WIRE [2] and FINER [3] both include evaluation on neural radiance fields). \n\nThe proposed MoE architecture also did not show improvements when used with softplus activation. \n\nThis paper also did not include other metrics normally used to evaluate the tasks, such as SSIM and LPIPS for 2D image fitting (e.g. FINER [3]). \n\nMinor point: inclulding the number of parameters in Table 1 may be helpful since comparisons between the number of parameters is discussed in the text.\n\nReferences\n1. Ramasinghe, Sameera, and Simon Lucey. \"Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n2. Saragadam, Vishwanath, et al. \"Wire: Wavelet implicit neural representations.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n3. Liu, Zhen, et al. \"FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**One of the major weaknesses of this paper is the experimental evaluation ...**\n- **vs other architectures**: All the listed architectures are about changing the activation function. Our proposal is orthogonal to that as it can work with any activation function, and we have shown our method with two activation functions. We have added two more, Softplus+FF and FINER, in the updated Table 1 (see General Comments). We also explain why we cannot test our method on NeRFs in the General Comments section above.\n- **Dataset Size**: Our experiment dataset sizes are similar to the papers you listed. In each of those, relevant experiments (Image,SDF,Audio) are done on similar or smaller sizes (only FINER's SDF experiments has more: 5 shapes to our 4 shapes). In fact many experiments and comparisons are done on as small as 1 instance. (FINER: 16 2D images and 5 SDF shapes, Gaussian activations: comparisons on 1-3 images per property, WIRE: 1 image, 1 3D shape, comparisons on 1 image per property). We compare on 24 images, 4 SDF shapes, and 3 audio samples. \n\n**The proposed MoE architecture also did not show improvements when used with softplus activation.**\nSoftplus activations alone is not expressive enough. We give new results with Softplus+FF (Fourier Features) in the updated Table 1 (see general comment).\n\n**This paper also did not include other metrics normally used to evaluate the tasks, such as SSIM and LPIPS for 2D image fitting (e.g. FINER [3]).**\n- We have added these metric in the updated Table 1 (see General Comments)\n\n**Q1. Can this work be combined with traditional ReLU MLPs with positional encoding or MLPs with activation functions other than sine?**\nYes, we give new results with Softplus+FF (Fourier Features) in the updated Table 1 (see general comment).\n\n**Q2. How does this work compare to other novel INR architectures (e.g. WIRE, FINER, etc)?**\nAs mentioned before, those are orthogonal contributions as those methods change the activation function, and we can use any activation function in our framework. See our new results with FINER in the updated Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ozQOq5dApc", "reviewer": "Reviewer_ekS9", "summary": "This paper proposes a mixture of experts (MoE) approach for INRs, which allows the learning of local piece-wise continuous functions by subdividing the domain and fitting locally. The incorporation of a MoE architecture enhances speed, accuracy, and memory efficiency. They also propose a novel manager architecture and initialization that enable domain subdivision without ground truth.", "questions": "1. Could you please discuss how to choose the number of experts giving a fixed number of parameters and iteration time? Is it better to have more experts with each one having fewer parameters or fewer experts with each one having more parameters?\n2. I wonder whether it is possible to apply your method to NeRF since there are no supervised signals for the 3D ground truth.\n3. When comparing the Vanilla MoE INR and your Neural Experts, have you kept their total parameters similar (smaller experts for your Neural Experts due to the extra parameters needed by the encoders)?", "limitations": "The limitations and potential negative societal impact of their work have been discussed in the paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The proposed MoE INR has a good performance compared to baselines.\n3. The idea of delivering MoE as a learnable partition region for INR fitting with randomized initialization is novel to me. From the ablation study, the randomized initialization improves the performance a lot.", "weaknesses": "1. Missing some closely-relative works. I encourage the authors to have a detailed discussion of previous MoE INRs [1,2] and decomposition/partition-based INRs [3,4].\n2. Lacking some key comparison experiments with decomposition/partition-based INRs. The authors only compare their method with the baseline SoftPlus and SIREN (and their wider version). However, some related works [4] have also shown the INR based on pre-determined masks can also outperform the wider version of SIREN.  I encourage the authors to experimentally compare your method with [4] to illustrate the necessity of learnable partition regions.\n3. A detailed ablation study on the hyper-parameters of the MoE INRs is missing, such as the layer of encoder, manager, and experts. Given a fixed number of parameters, how to allocate the parameters to the three modules remains unknown.\n\n[1] Zhao, Jianchen, et al. \"MoEC: Mixture of Experts Implicit Neural Compression.\" arXiv preprint arXiv:2312.01361 (2023).\n[2] Wang, Peihao, et al. \"Neural implicit dictionary learning via mixture-of-expert training.\" International Conference on Machine Learning. PMLR, 2022.\n[3] Rebain, Daniel, et al. \"Derf: Decomposed radiance fields.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[4] Liu, Ke, et al. \"Partition speeds up learning implicit neural representations based on exponential-increase hypothesis.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: Thank you for bringing these important works to our attention, we will address them in our paper.\n\nW2: We compare with [4,new] on their dataset (300 test images from LSUN bedrooms). We use the same learning rate and same measure (PSNR after 300 steps for SIREN). Note that these have not converged at 300 iterations (see also Fig. 5 in [4,new]), hence for the experiments in our paper we report after 30k steps (also done for the last row in the table).\n\n| **Method**                        | **Mean** | **Std** | **Parameters** | **Steps** |\n|-----------------------------------|----------|---------|----------------|-----------|\n| SIREN [4,new]                         | 21.211   | -       | 790.0k         | 300       |\n| SIREN-PoG [4,new]                     | 23.864   | -       | 793.6k         | 300       |\n| SIREN-PoS [4,new]                     | 24.485   | -       | 793.6k         | 300       |\n| Ours Neural Experts SIREN         | 30.50    | 2.73    | 366.0k         | 300       |\n| Ours Neural Experts SIREN         | 54.98    | 2.56    | 366.0k         | 30000     |\n\nW3:\nWhile keeping the number of overall parameters fairly constant, we experiment with how to allocate parameters between the components. The first column is \n`width x layers for the input encoding, num experts x (width x layers) for the experts themselves, width x layers for the manager encoding, width x layers for the manager` and the second column is \n`% parameters for input encoding, % parameters for the experts, % parameters for the manager`.\nWe can see that as long as the majority of the parameters is not for the manager (one third of the total params or less) our neural experts SIREN model performs similarly.\n\n\n| **Layers**                        | **Pecentages** |**Parameters**  | **PSNR**        |\n|-----------------------------------|----------------|----------------|-----------------|\n| 128x2, 4x(128x2), 128x2, 128x2    | 14%, 55%, 31%  | 366.0k         | 89.35±7.10      |\n| 196x2, 4x(85x2), 128x2, 128x2     | 32%, 34%, 34%  | 368.0k         | 87.86±8.98      |\n| 128x2, 4x(102x2), 128x2, 196x2    | 14%, 38%, 48%  | 366.1k         | 83.05±10.11     |\n| 256x2, 4x(68x2), 96x2, 68x2       | 54%, 29%, 17%  | 368.3k         | 88.12±6.92      |\n\nQ1: We now experiment with changing the number of experts vs number of layers within the experts. Like the previous table, we keep the parameters as similar as possible (~366k) by changing the layer width of experts. The results show that less layers and somewhat high number of layers is best, with 6 experts and 1 layer performing the best out of all the configurations we tried. Our chosen configuration of 4 experts and 2 layers is somewhat consistent with this rule of thumb.\n\n| Number of experts \\ Number of layers |     1       |      2       |    4        |     8      |\n|--------------------------------------|-------------|--------------|-------------|------------|\n| 2                                    | 79.39±8.74  | 77.78±8.18   | 72.23±6.92  | 65.32±6.51 |\n| 4                                    | 92.32±7.80  | 89.35±7.10   | 81.19±8.04  | 74.36±7.31 |\n| 6                                    | 99.15±5.32  | 92.84±7.95   | 87.80±8.98  | 75.26±9.02 |\n| 8                                    | 96.13±7.95  | 90.72±10.61  | 83.05±10.31 | 66.57±8.72 |\n\n\n\nQ2: It is not possible to use our method for NeRFs, for the reason the reviewer points out, see our discussion in the General Comments section above.\n\nQ3: The parameters are similar 349584 for vanilla SIREN MOE vs 365968 for our neural experts SIREN (so 95.5\\% of the params of our model, or our model is a 1.05x increase in parameters).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Comments 
We thank the reviewers for their insightful comments. We include requested additional experimental results here and response to common questions. Specific questions are addressed to each reviewer below.

**New Table 1** (page 5 of submission) now updated to have comparison with Softplus + FF (Fourier features [38]) based networks and FINER based networks [1, new], a version of our method with similar parameter count to SIREN, and SSIM and LPIPS metrics.

| **Method**| **Parameters** | **PSNR Mean**| **PSNR Std**| **SSIM Mean**| **SSIM Std**| **LPIPS Mean** | **LPIPS Std** |
|---------------------------------------------|----------------|----------------|---------------|----------------|---------------|----------------|---------------|
| Softplus| 99.8k| 19.51| 2.95| 0.7158 | 0.1106| 4.08e-1| 8.33e-2 |
| Softplus Wider| 642.8k | 20.91| 3.12| 0.7798 | 0.0899| 3.38e-1| 8.44e-2 |
| Ours Neural Experts SoftPlus| 366.0k | 20.62| 3.12| 0.7628 | 0.0946| 3.53e-1| 8.47e-2 |
| Softplus + FF **(New)** | 99.8k| 28.97| 3.30| 0.9433 | 0.0193| 7.59e-2| 1.70e-2 |
| Softplus + FF Wider **(New)** | 642.8k | 29.48| 3.91| 0.9436 | 0.0245| 7.74e-2| 2.30e-2 |
| Ours Neural Experts SoftPlus + FF **(New)** | 366.0k | 31.66| 3.16| 0.9652 | 0.0180| 4.13e-2| 1.57e-2 |
| SIREN | 99.8k| 57.23| 2.46| 0.9991 | 0.0005| 5.78e-4| 5.09e-4 |
| SIREN Wider | 642.8k | 77.50| 5.32| 0.9996 | 0.0005| 3.08e-4| 3.04e-4 |
| Vanilla SIREN MOE | 349.6k | 62.98| 4.16| 0.9993 | 0.0005| 4.53e-4| 3.88e-4 |
| Ours Neural Experts SIREN small **(New)** | 98.7k| 63.42| 7.09| 0.9992 | 0.0007| 9.96e-4| 2.40e-3 |
| Ours Neural Experts SIREN | 366.0k | 89.35| 7.10| **0.9997** | 0.0004| 2.49e-4| 2.93e-4 |
| FINER **(New)** | 99.8k| 58.08| 3.04| 0.9991 | 0.0005| 7.47e-4| 1.31e-3 |
| FINER Wider **(New)** | 642.8k | 80.32| 5.40| 0.9996 | 0.0004| 2.49e-4| 2.46e-4 |
| Ours Neural Experts FINER **(New)** | 366.0k | **90.84**| 8.14| **0.9997** | 0.0004| **2.46e-4**| 2.48e-4 |
Note that for the new results, our method outperforms the baselines.

1 (new). Liu, Zhen, et al. "FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

**Table 3** We also add Chamfer results to Table 3 (page 7 of the submission).

| **Method**                    | **# params** | **Armadillo**   | **Dragon**      | **Lucy**        | **Thai Statue** | **Mean**        |
|-------------------------------|--------------|-----------------|-----------------|-----------------|-----------------|-----------------|
| SIREN Large                   | 1.5M         | 1.4983e-05      | 2.2367e-05      | 1.2030e-04      | 5.8465e-05      | 5.4029e-05      |
| Our Neural Experts Large      | **1.3M**     | **1.4975e-05**  | **2.1340e-05**  | **1.1322e-04**  | **5.4084e-05**  | **5.0905e-05**  |
| SIREN Small                   | 396k         | 2.9165e-05      | 2.3944e-05      | 1.2583e-04      | 6.7273e-05      | 6.1553e-05      |
| Our Neural Experts Small      | **323k**     | **1.5086e-05**  | **2.2102e-05**  | **1.1047e-04**  | **5.9721e-05**  | **5.1845e-05**  |

Consistent with Trimap IoU (reported in the submission), our method outperforms the baseline on Chamfer distance.

**Discussion about newer INR methods that apply locality bias (such as hybrid methods)**: We discuss such methods in lines 70-76. These methods greatly improve signal reconstruction performance and often also faster as reviewer VtKc mentions. However, as they choose regions in a deterministic or heuristical way [7,15,24,27,33,37], in order to improve task importance many regions are required, leading to these methods being very parameter inefficient (see InstantNGP [27] Fig 2 and 4). Rather than looking into ways to make more efficient patterns or heruistics for assigning spatial parameters, we consider fixing the number of experts (region specific parameter sets) and allowing their region of influence to be optimized, so that for a smaller number of parameters we can get better performance. Thus, while our method is of the same category as those methods (specializing parameters spatially) our method has a different goal: to show the benefit of optimizing the regions and how to make this work in practice. In fact we show that what is required to make this work can be counterintuitive (needs encoder and random pretraining) so we believe this work is of great interest to the community.

**Using our approach with NeRFs**
Extending our MoE method to NeRFs is an intersting future direction as it cannot natively fit within an MoE framework. In an MoE framework, we change a loss per instance $\mathcal{L}_i=\ell(f(x_i),y_i)$ to a weighted sum of losses over experts $\mathcal{L}_i=\sum_j q_j \ell(f_j(x_i),y_i)$ where $f$ is our model, $f_j$ are the expert models, $q_j$ is the expert weights, and $x_i,y_i$ are the current instance's input and ground truth. Thus as reviewer ekS9 points out, we need a 3D ground truth signal. However NeRF only has supervision per ray (color for each pixel), and the rendering model accumulates points along a ray. Thus there is no way to specify a loss per expert, as the points along a ray may be assigned to different experts. While we could then specialise experts for each view direction, we cannot specialise experts for each 3D point with this formulation. This may be addressed by modifying the formulation, e.g., through a variational bound, but is beyond the scope of the current work.

Meta-review: **Summary:**
Implicit neural representations (INRs) excel in tasks like image, shape, audio, and video reconstruction but usually rely on a single network, imposing global constraints. Contrary to the traditional, this paper introduced a mixture of experts (MoE), enabling local, piece-wise learning that improves domain fitting. Integrating MoE into INR frameworks boosts speed, accuracy, and memory efficiency, with new methods for gating network conditioning and pretraining further enhancing performance. The research direction is both intriguing and novel, making it well-suited for presentation at the conference.

**Strengths:** may include:
- "Neat architectural design" (aVob)
- Strong ablation study (aVob, VtKc, ij4R)
- "Consistently" outperformance (aVob)
- "Well-written and easy to follow" (VtKc, ij4R, ekS9)
- "a novel MoE-based architecture," "is novel" (ij4R, ekS9)

**Weaknesses:**
Initially, reviewers raised concerns about having "more parameters" than the baselines (aVob), the need for further discussion on comparative methods (e.g., spatial functa, KiloNeRF, InstantNGP, Gaussian Activation Function, WIRE, FINER) by the Reviewers aVob, VtKc, ij4R, and ekS9, "unstable convergence" (aVob), and "misalignment between the experimental results and the motivations" (VtKc). 

**Justifications for Acceptance:**
However, after the rebuttal and discussions with the reviewers, the authors provided experimental evidence and explanations for their decisions, which led to acceptance from all the reviewers. Especially, AC appreciates the authors' efforts and quick response to the request for additional evidence demonstrating the broad impact of their work by providing InstantNGP results, which sufficiently resolved the major issues raised. Based on this, AC is pleased to recommend acceptance.
Decision: Accept (poster)