Implicit Neural Representations with Levels-of-Experts
Authors: Zekun Hao
Arun Mallya
Serge Belongie
Ming-Yu Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Implicit neural representations
neural fields
coordinate-based networks
hybrid representations
positional encoding

Abstract: Coordinate-based networks, usually in the forms of MLPs, have been successfully applied to the task of predicting high-frequency but low-dimensional signals using coordinate inputs. To scale them to model large-scale signals, previous works resort to hybrid representations, combining a coordinate-based network with a grid-based representation, such as sparse voxels. However, such approaches lack a compact global latent representation in its grid, making it difficult to model a distribution of signals, which is important for generalization tasks. To address the limitation, we propose the Levels-of-Experts (LoE) framework, which is a novel coordinate-based representation consisting of an MLP with periodic, position-dependent weights arranged hierarchically. For each linear layer of the MLP, multiple candidate values of its weight matrix are tiled and replicated across the input space, with different layers replicating at different frequencies. Based on the input, only one of the weight matrices is chosen for each layer. This greatly increases the model capacity without incurring extra computation or compromising generalization capability. We show that the new representation is an efficient and competitive drop-in replacement for a wide range of tasks, including signal fitting, novel view synthesis, and generative modeling.

Implicit Neural Representations with Levels-of-Experts
Zekun Hao*!
Arun Mallya "NVIDIA amallya@nvidia.comm com
Serge Belongie' Cormell University a @jb344@cormell.cm edu
Ming-Yu Liu"
hz4720cornell.edi
m
Abstract
Coordinate-basee networks, usually in the forms of MLPs, have been successfully applied to the task of predicting high-frequency but low-dimensional signals using coordinate inputs. To scale them model large-scale signals, previous works resort to hybrid representations, combining coordinate-based network with grid-based representation, such as sparse voxels. However, such approaches lack compact global latent representation in its grid, making it difficult to model a distribution of signals, which is important for generalization tasks. To address the limitation, propose the Levels-of offelssof-Experts (LoE) framework, which is novel coordinate- based representation consisting of an MLP with periodic, position- dependent weights arranged hierarchically. For each linear layer of the MIP, multiple candidate values of its weight matrix are tiled and replicated across the input space, with different layers replicating at different frequencies. Based the input, only one of the weight matrices is chosen for each layer. This greatly increases the model capacity without incurring extra computation compromising generalization capability. We show that the new representation is an efficient and competitive drop-in replacement for wide range of tasks, including signal fitting, novel view synthesis, and generative modeling.
Introduction
There has been a growing interest in representing diw-dimensional but high-frequency signals, such images, videos, and 3D scenes, with fully-connected neural networks. common paradigm is to use coordinate-base predicts the data value at the specified location (30)
multilayer perceptron (MLP) that takes coordinate positions as input and
< 49] Compared explicit representations
such as point clouds and voxel grids, this kind of implicit neural representation (INR) is memory
efficient and can model a distribution of signals for conditional synthesis tasks 2 26
34 37
thanks to its ability to learn a compact and meaningful latent space.
However, scaling up INR to better represent higher-resolution signals distribution of signals, like a distribution of images, is challenging because the mapping can highly nonlinear. To increase the model capacity deal with the complexity, the dimensions of activations or deeper by stacking more layers. Unfortunately, both options will dramatically increase the computation needed at each data point. Recently, Rebain el al. [40] have shown that this results in an undesirable trade-off because the representation power gain diminishes quickly with increased width or depth. We further explore and analyze this issue in Section 4.2
either make the MLP wider by increasing
Many recent works bypass this scaling problem by using a hybrid representation [4] 33
24 25
38, 40 41 47 48). A discrete data structure, such sparse voxels, decomposes the space into grids. Within each grid, a lightweight MLP conditioned on the grid embedding produces local detail scale finer than the grid resolution. However, such approach has two major limitations: (1) The reliance on smooth interpolation grid embeddings
or
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Waight Map
FC1
FC2
FC3
FC4
(i) Gray code
(ii) Random order
Dynamic Waight MLP
FC1
PC1
FC2
FC2
FC3
FC4
PC4
(iii) Different #weights per tile
(iv) Smooth interpolation
(a) MLP with dependentt weights
(b) Examples of different weight tiling pattems
Figure 1: The Levels-of of-Experts (LoE) framework. (a) A position- dependent MLP with ID input, y = f(x, (x)) (activation functions omitted for brevity). this example, each fully-connected (FC) layer has two candidate weight matrices (marked in blue and orange, shade denotes layer depth), arranged in periodical and hierarchical manner. According to the input location, one of the weight copies is selected for each layer. (b) A variety of hierarchical tiling patterns can be used with LoE. They are not confined to (i) specific alignment, and can have varying (ii) orders of granularity, or (ii) length of the repetend, and (iv) can be generalized matrices. Thus, each layer at a different level, or scale, has number of experts with their own weight matrices, specializing different regions of the input space.
smooth interpolation across the weight
28 48 to encourage continuity across the grid boundaries can negatively impact computation efficienc;; and, (2) As the underlying signal described by multiple distributed features stored in grid, which are associated with fixed locations, it lacks global, compact latent representation unless it employs another expensive model to generate the grid embedding itself [5) (88 42). Still, this only possible in limited cases, e.g., regular grid, without any sparsity, pruning, or hashing.
output-domain tiling and blending 19
Our approach extends the idea of hybrid representation by storing the weights of an MLP on multi-resolution tiled grid. Conceptually, for each of the linear layers, we assign multiple independent copies of its weights, arranged in a tiling pattern and repeated to fill the space as visualized in Figure [] This partitions space so that each copy of weights only needs to handle inputs within certain periodical intervals, essentially making the weights of the MLP position-dependen We refer to this proposed framework as Levels-of-Exee of-exesttsss (LoE). Each layer at a different depth, level, has number of experts with their weight matrices, specializing in different partitions of the input space, depending on the tiling pattern used.
Similar to Fourier features (30 49), we use different grid resolutions for each layer or depth, so that the weight of different layers repeats at different frequencies. This arrangement has several desired properties: (i) While the weight of each layer is repeating, the learnable combined parameterization of all the layers helps avoid repetition over the input range of interest, and (ii) large number of uniquely parameterized intervals of the combined model can be obtained and tailored to the underlying problem. In fact, in Section 4.1 we show that our model can scene even without any input position encoding the position-dependent weight can itself serve as form of positional encoding! We show that when compared to dividing the space to use different MLPs 440 41 our layer-level tiling approach expensive smooth interpolation or blending, while at the same time improving the representation power. Finally, compared to non-repeating grid embeddings (24 25 47), approach encourages the learning of generalizeable mapping, as shown Sections 4.1 and 4.2 and also improves parameter efficiency, particularly benefiting generative modeling tasks, as shown in Section 4.4
reduce output discontinuities without relying computationally
Our model has computational cost comparable to a regular MLP of identical architecture, at same time being more expressive. Although the parameter count, and thus representation power, is greatly amplified by the use of position-dependent weights, only a single copy of weights is active at each input. In practice, this can be implemented efficiently with an off-the-shed shelf fused gather-GEMM,

---

seatter operator [ with little speed loss. Our method is drop-in replacement in many applications without the need for any further modifications. To summarize, we make the following contributions:
1. We introduce novel hybrid implicit neural representation that is parameterized by a hierarchy of
and periodic weights (Section 3
2. We extensively study the effect of various design decisions including the periodicity and hierarchy of weights, weight interpolation methods, and the use of input encodings (Section
3. We demonstrate the efficiency and representation power of our architecture on challenging tasks including high-resolution image fitting, video fitting, novel-view synthesis, and image generation (Section 4
2 Related Work
Implicit neural representations (INRs). INRs represent a signal with a pointwise (coordinate- neural network that takes coordinate as the input and predicts the data value at the location specified by the input coordinate. With INRs, one can query continuous locations independently and efficiently, which is desired property for many learning, graphics, and vision tasks (50). INRs have been used in representing images 2
46
49, shapes [T3 29 37
and scenes 3 30 35] 36, 45).
Earlier INRs were based on MLPs with ReLU activations 37 46 and often failed to represent high-
frequency detail in the underlying signal. Recent works have greatly addressed the issue by leveraging
better input encoding designs (30 [4], activation functions 33 44 network architectures [9] 23
Our approach, a position dependent MLP architecture, is orthogonal to these approaches and can potentially be used in conjunction with them to achieve better results.
Hybrid representations. Several works propose combining INRs with explicit discrete structure representations to improve both the computation and memory efficiency for modeling large and complex signals. Such a hybrid approach often partitions the input space into smaller regions based on the adopted discrete structure representation, which results in local parameterization, or decomposes the input space to low-rank subspaces [3] 6). Various discrete structure representations for local parameterization have been explored, including regular grids 19 33 38 41), sparse voxels [15] 24), voronoi cells [40], octrees 25 47), convex parts [8], and learned shape elements II. We can even use another neural network to predict the parameterization, which enables generalization [5] 3X 42
With the hybrid approach, one first obtains a local feature with the discrete structure and the coordinate, which is then inputted to the pointwise MLP to get the data value.
Note that the discrete nature of the hybrid representation calls for special and often costly designs to ensure smooth transitioning across the subdivision boundaries. One popular approach is to smoothly interpolate the grid feature 24
38 47 before handing it off to the MLP. Such an approach only requires one MLP evaluation per sample, but the cost of interpolation can be high for high- dimensional features on a high- nigh-dimensional grid. Several approaches 4 19 28 48) allow the MLP to predict signals beyond the grid boundarics that multiple predictions of the same coordinate from nearby grids can be evaluated and smoothly blended in the output domain, but they are more expensive to compute. There also hybrid representations that completely abandoned smooth interpolation and achieved considerable speedup [41]. However, it requires distillation from a larger, pretrained network mitigate discontinuity artifacts. Our approach is also a hybrid approach. It enjoys computation and memory efficiency but does not suffer from the boundary interpolation issue.
Neural networks with input-dependent weights. Our method can be regarded as special type of hybrid representation that use a multi-level tiled grid to parameterize the weights of the pointwise MLP. Depending where the input coordinate lies on each level of the grid, different combination of weights is used for the network. Reiser et al. [1] shares the same high-level idea of having coordinate-depenneet network weights, but they learn completely disjoint networks for different grid locations, while we use a hierarchical and periodical structure. We will show the importance of our hierarchical and periodical structure in obtaining smooth and expressive representation.
In a broader context, neural networks with data-dependent weights have been used for modeling 3D animation 7 17 as a form of a collection of experts and for solving differential equations [32 to represent solutions. Our work is different as we use layer-wise hierarchical parameterization and designed for hybrid neural implicit representation.
3

---

3 Method
A typical coordinate-basee multi-layer perceptron (MIP) can be described stack of layers, f:p- (g o o g
& o g² (P),,
where p is the input coordinate which the MLP is being evaluated, y is input mapping. such as the sine-cosine positional encoding [30], Φ is a non-linear activation function, and g' x - W'x + ba is the 7th linear layer, which performs affine transformation the input x, parameterized by a weight matrix W and a bias vector b². During training, W and b' are optimized via gradient descent to fit the MLP to the data.
In our Levels-of-Eat -xppertsss (LoE) approach, instead of regarding each W as a single learnable matrix, we additionally model it function v (-) of the input coordinate p. The resulting dynamic- weightt linear layer has the form, h' : (x. p) (p)x + b', where x are the inputs to the layer, and p are the location which the MLP is being evaluated. By replacing the traditional linear layers g in the MLP with weightt layers h', we obtain an MLP with input-dependent weights,
f:p- (p) do h (p)
Φ h (p) (p))
(2)
As the resulting position-dependee weight matrix has a much higher dimension compared to its input and output vectors and will be evaluated at a large number of query points, it is important for the weight generation functions v/² (p) to be fast, inexpensive, and yet expressive. This rules out the popular -reett-preiition networks used in hypernetwork-basse
approaches [14), in which one has to predict a high-dimensional weight per position. Instead, we use a simple, lightweight function, specifically coordinate interpolation-baser method. Multiple candidate values for the weight matrix are stored in a regular grid (tile) and interpolated in a cyclic manner based on the input coordinates.
Consider the case where we have a grid containing N matrices { (W..... W WN-1) where i is the layer depth, and N is nonnegative integer. We are only interested in the case that N > as N = 1 reduces to the original pointwise MLP formulation. Given a 1D coordinate p = (p), the input-dependent weight for layer W is computed as
W 2 (p) - (p) Σ B
(3)
where a' and 3 are hyperparameters that adjust the scale and translation of the grid for each layer and B, is the blending function that computes the blending coefficient for the j-th candidate. The blending coefficient can take many different forms. For linear and nearest interpolations, they are defined as follows:
B Blincar (g) max(0, ((2 + 1 j) mod - 1))
(4)
{1 [g| mod N -
Bysarrst J.N
(q)
(5)
otherwise.
Note that here mod denotes positive remainder operation: a mod b = a bl We also note that the above equations can easily be extended to multi-dimensional coordinate spaces.
For linear interpolation, regardless of the tile resolution, only 2 of the blending coefficients are non-zero for each coordinate in our ID example. On the other hand, the nearest interpolation scheme only has a single non-zero coefficient for each coordinate. This sparsity allows a fast and efficient implementation of performing the dynamic-weight linear layer computation for batched inputs: for each candidate weight matrix, Wi where we only gather input vectors that have BJNN > 0 at a time, perform matrix multiplication and scaling, and finally scatter the results to the output matrix.
Empirically, we find it helpful to have different layers of our MLP with different spatial frequencies on the grid. This can be easily achieved by using different set of a and 8 per layer. Using different frequencies at different layers gives an inductive bias to the MLP to capture different repetition patterns. It also serves via weight sharing at different locations. We show this is particularly useful in reducing artifacts for novel view synthesis tasks (see Section 4.3).
form of regularization that encourages the learning of smooth mapping
non-exhaustive list of potential grid arrangements-a grid arrangement corresponds to set of [(a', (B))) are presented in Fig. We note it is even possible to use randomized tiling pattern by

---

transforming the grid with a random affine transformation, while still seeing significant performance gain compared to a regular MLP, as evident by the Random Affine experiment in Table I Unless otherwise mentioned, we arrange the grids in a progressively growing fashion throughout the paper. We start with the first grid (corresponding to the first MLP layer) covering the full input space without repetition, and progressively subdivide the grids using additional layers. This is shown in Fig, Fig.Ta)) a This arrangement partitions the input space into uniform-size grids, with each one having a unique combination of weight matrices. A comparison of different grid arrangements is included in the supplementary material.
4 Experiments
In this section, we validate LoE on 4 challenging tasks. In the first two experiments, we fit our model to high-resolution image and video data, evaluate its performance, and study the effect of various design components. Then in Section. 4.3] we evaluate our model on the indirectly supervised, novel-view synthesis task and study its inductive bias. Finally, in Section. 4.4, we demonstrate its generalization capability by training a generative adversarial network (GAN), All the code will be made publicly available.
4.1 Fitting to a figgh-reooution Image
We study the effect of our hierarchical weight tiling on model capacity and computational efficiency by fitting networks to high-resolution image of size 8192> x 8192 [25] pixels. An image is considered as a set of pixels (p., (p)))) represented by their 2-D coordinates P: = (xx,, ys) and RGB colors (p::) e R The model - f()) takes the coordinate as input and predicts the color at the given coordinate. The goal is to fit the model to the data by minimizing the loss: L2 = Σ, - 1?
Table D compares our model with several baseline methods and ablations. Our main method signifi- cantly outperforms baseline methods that do not use position-dependent weights. Despite sharing the same network architecture and computational cost, an MLP using the sine-cosine positional encoding (PE) as the input mapping (30) performed 12dB worse than our model. We also compare with hybrid model that learns an input coordinate embedding (CE) 2 for the MLP. Their fitting quality is significantly lower than ours at the same parameter count while incurring a higher computational cost. This suggests that learning position-dependent weight is more effective than learning grid of embeddings.
We believe that the effectiveness of our method partially comes from the use of a hierarchical and periodic tiling pattern, which encourages the learning of periodic, spatially-shared features. In Table [ the interleaved model uses periodic weight tiling scheme but lacks the multi-scale arrangement (Fig. dd)). Effectively, multiple independent MLPs are learned, each handling a pixel-skipped subset of the image. This only improves the performance slightly compared to the PE MLP baseline. On the other extreme, in the chunked experiment, we partition the input space into uniformly sized chunks and use independent MLPs to handle each chunk. Although the fitting quality is improved, there are large variations in errors across different chunks, as shown in Fig. c). In fact, the chunk boundaries are visible in the fitted image, indicating continuity issues.
Our method achieves the best performance by having the tiled weights repeat wide range of intervals. This allows a more efficient data representation by exploiting periodicity at a wide range of frequencies and allows a more adaptive distribution of model capacity and fitting error that is less dependent on the geometry of the data. We also compare our piecewise constant weight parameter- ization against the smooth, piecewise linear variant, implemented by the bilinear interpolation of weights, shown in Fig. 2(b). Despite having 4 times the computational cost, the fitting quality of the smooth variant is only slightly better (+0.6ldB) than the faster, piecewise constant version while also sharing similarly homogeneous error distribution. This indicates that by using the piecewise constant parameterization, the full performance of the tiled weight models can be enjoyed only a fraction of the cost. Surprisingly, the tiled weight model is able to achieve reasonable performance even without the use of any input position encoding. the Constant Inpul (no PE) experiment shown in Table I1 we feed constant vector to the first layer instead of the coordinate encoding. In fact, the position-dependent tiled weight itself is already form of positional encoding. is able to identify a large number of unique intervals in the input space (up to II ni, where n, is the number of candidate
5

---

(a) Uurs
(b) Ours Billnear
(c) Chunked
(d) Interteaved
(e) Fixed Weight
Ground Truth
Simplifiet Til Patteams
Ervor Maps (ESNNI)
Magnified view
(44.2)
(44.9)
(43.1)
(33.8)
(32.3)
Figure 2: Comparison errors while fitting to a 64MP image. Our method with discontinuous weights (a) has low and uniformly distributed error comparable to (b), the more expensive version that bilinearly interpolates the weights. (c) and (d) use an ensemble of networks without hierarchical weight tiling, resulting in high error variation, discontinuities, and low fitting quality.
Table 1: Comparison of parameter count, computational cost in number of
(MACs) per sample, and fitting quality on the 64MP color image Pluto shown above [25]. All the models use the same number of layers and hidden channels. The model size of our method is chosen to be comparable to ACORN [25]. For the coordinate embedding (CE) baseline, we evaluate multiple grid resolutions and report the best result.
Parameters
MACs
PSNR (dB)
SSIM
PE MLP 30
0.59M
0.57M
32.34 39.65
0.869
PE CE 2
9.37M
0.65M
0.967
Shown in Fig.
Periodic Tiling
Multi- scale
X
Interleaved
9.37M
0.57M
33.80
0.876
(c)
x
Chunked
9.37M
0.57M
43.13
0,980
Random Affine
9.37M
0,57M
42.08
0.973
Constant Input (no PE)
8.92M
0.56M
39.48
0.955
(b)
Ours Bilinear
9.37M
2.28M
44.85
0.985
Ours
9.37M
0.57M
44.24
0.983
weights of layer i). Related ideas of using periodical weights a coordinate- based network are also found in MFN [9] and BACON (23).
4.2 Fitting to a Video
We fit our model to a video with 300 frames and resolution of 512 x 512. In this case, each pixel in the video is associated with 3D coordinate p; (#;, 36, 44). The quantitative results
shown in Table 2] and visual comparisons in Fig. 3 Compared to fixed-weight models such as PE MLP SIREN [44, the capacity of our model grows favorably with increased input dimensions, without incurring extra computation. For higher-dimension dimensional weight tiles. In this case, we use combination of 2 and 4 tiles, which amplifies the number of parameters by 8x and 64x compared to regular linear layer with the same number of channels. This cannot be done in fully implicit model without greatly increasing the computation. For example, in the SIREN-L experiment, we attempt to increase the model capacity of SIREN by quadrupling the hidden channel count. The resulting model needs 15x more computation, yet quality is still lacking. This confirms the diminishing return phenomenon associated with coordinate MI.Ps 400. Compared embedding- hybrid representations such as coordinate embedding (CE), which in this case include dense 643 64 grid that store the position-dependent features and
problems, we can simply use higher

---

trilinearly interpolated, the conclusion in Sec 4.1 still holds that our approach performs significantly better under the same parameter count.
(a) GT
(t)) Olns
(c) SIREN
(d) PE
Table 2: Comparison of model size, computation and fitting quality a short video. All of the models have 4 hidden layers and 256 hidden chan- nels, except for SIREN-L, which has 1024 hidden channels.
30,9
29:13
27.33
Params.
MACs
PSNR(dB)
PE 330
279K
0.28M
27.33
PE + CE 22
17.1M
0.29M
35.83
SIREN 44
265K
0.26M
29.13
Figure 3: Visual comparison of video fitting re- sults. Numbers indicates PSNR (dB). All the mod- els have the same computational cost.
SIREN-L
4.21M
4.20M
37.71
Ours
16.9M
0.28M
39.98
4.3
Novel View Synthesis
So far LoE model has demonstrated improved quality in fitting to high-resolution signals via direct supervision. Here, we examine if the model has the desirable inductive bias to work in an under- constrained setting with indirect supervision. We evaluate task, where the network models color e and opacity a at each 3D location and under different view directions. This kind of volumetric 3D representation is also known neural radiance fields (NeRF) 30 Given an image, each pixel in the image can be associated with a ray r(t) = o + td. The color of the pixel can be obtained by sampling points along the ray, querying the neural network at these points to obtain color and opacity C,, f(r(t)), d), and perform volumetric rendering via numerical integration 27
method on novel view synthesis
C(r) Σ T. exp(-o;5;)) where T, exp (-Σ
1-1 00 and 6; 14+1 tq-
The training is done by minimizing the photometric loss between the rendered colors and ground truth pixel values: Le = >> [[C(rr))))))))))))))(( Calla.
We compare method with baselines on the Tanks and Temples dataset 21 24 which contains 133 training images at resolution 1920 1080. All the models compared have 4 hidden layers and 256 hidden channels. We present the quantitative results in Table 3lnd and include visual comparison in weights significantly
Fig. 4 As shown in the zoom-in view, the use of hicrarchical, sets the otherwise identical PE MLP baseline apart by reproducing much better detail.
has been observed that when the input space is partitioned into grids, and independent MLPs are learned for each grid location, there will exhibit significant free space artifacts in the result [41] Our model does not have such a problem, despite similarly having position-dependent weights. To gain better understanding, weight grids. Despite having a lot more parameters, the ablated model, which indeed suffering from free space artifacts, performed far worst than the baseline PE MLP (Fig. 5). This shows that the use of hicrarchically tiled weights is important for providing a good inductive bias for the task.
experimented with model that lacked hierarchical arrangement in its
4.4 Image Generation with GANs
In this section, we demonstrate the generalization capability of the LoE model on the challenging image generation task. The coordinate-basee models are used as the generators for the generative adversarial networks (GAN) [12. More specifically, the model f(p, z) takes both a coordinate p and noise vector z as input, and map them to color value. To generate an image, fixed z is used and the network is queried at every pixel location p E P. We denote the process of generating full image as G(z) [p(p. x)Ipe P). Images of different appearances can be generated by sampling the noise vector from fixed distribution z Pz. An additional discriminator network / is used to
provide the training signal. We use hinge loss (2 the GAN objective.
7

---

GI
Cuus
PE MIP
GI
Ouu's
PE MLP
IN CEL
IN CEL
IN CEL
The family one nanure' masterpiecee
The family one nature aaterriecnss
Loui
Loui
Low
Figure 4: Visual comparison of novel view synthesis results from model as well as the baseline model. Below each image there are local crops that better show the detail. Our model produces extremely sharp detail compared to the bascline while having the same computational cost. Both models share the same model architecture, with the only difference being the use of hierarchical weight tiling.
Table 3: Comparison novel view synthesis quality the Family scene. All the models have the same computational cost of 315KMACs per sample.
#Params.
PSNR(dB)
SSIM
PE MLP 30
317K
30.50
0.900
Ours
No multiscale
No Hier.
17.8M
27.73
0.861
Figure 5: Free-space artifacts in the ablated model that uses tiled weights but lacks the multiscale arrangement. The weight grids in all the layers aligned and repeated at the same interval.
Ours
17.8M
31.46
0.936
Ours
Baseline
Figure 6: Comparison of image generation results on FFHQ dataset. Images in the top row are generated by our model. Images in the bottom row are generated by baseline model with coordinate embedding. Both models have comparable parameter counts and computational costs.
To reduce computation and encourage easy reproduction, we use simplified setting with lightweight components. For the generator, we use an 8-layer network with residual connections. The noise vector is directly fed into the first layer. For the discriminator, we multi-resolution patch discriminator [18] with spectral normalization [3I] The model trained on the Flickr Faces-HQ (FFHQ) dataset [20] at resolution of 256 x 256. For the baseline method, we use coordinate embedding 256 x 256 resolution and 256 channels in order to obtain a parameter count comparable to the LoE model. Please refer to the supplementary material for full implementation detail and larger-scale experiments.
We report the model size, computational cost and image quality measured in Fréchet inception distance (FID) [66 in Table and provide a gallery of sample images in Fig. Our method not only achieves better FID but also produces images free of fixed noise pattern artifacts (shown in Fig.

---

Table Comparison of image genera- tion quality FFHQ dataset.
#Params
MACs
FID
PE + CE 2
19.9M
1.39M
23.5
Ours
19.6M
1.26M
18.3
Figure 7: Examples of the fixed noise pattern artifacts the images generated with the baseline PE + CE method. Our method does suffer from this issue.
5 Discussion
In this work, we demonstrated new type of hybrid implicit representation, called Levels-of-Experts (LoE), which is parameterized by hierarchical and periodic, position-dependent weights that are arranged on levels of repeating grids. The new representation offers great versatility, improving the performance compared to fully implicit models while at the same time having a comparably low computational footprint and the same of use. Compared to previous grid-based hybrid representations, the LoE model demonstrates good parameter efficiency and generalization capability.
wide range of tasks. Our method provides greatly increased model capacity
Limitations. The nearest- pnarest-interpolation variant of our model has undefined derivatives and discontinu- ities when crossing the grid boundaries. Even though the smooth interpolation variants can be used in these scenarios, compared to SIREN, it does not have smooth, high-order derivatives, limiting its use applications such as solving differential equations. Similar to previous works, our method required prior knowledge of the underlying signal in order to choose suitable grid scales.
Broader Impact. Our model is orthogonal to various other approaches of improving or extending the increasingly popular implicit neural representations (INRs), such as better activation functions [39], better input encoding [49] the use of hypernetworks 14 45 or combination with other pre- or post-processing CNNs 3 Our method enable higher fidelity data (image, video, volume, etc.) synthesis, representation, and compression at reduced computational costs compared to prior works. Like prior works, our method can be misused negative ends, including for deepfakes.

---

References
CUTLASS: CUDA templates for linear algebra subroutines. htttps///gttubb.co////..L.. 2019.
[2] Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image generators with anddttinaall-inddeeedder pixel synthesis. CVPR, 2021.
[3] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-NeRF: multiscale representation for anti-aliann anti-aliasing neural radiance fields. ICCV, 2021.
[4] Rohan Chabra, Jan E Lenssen, Eddy 1lg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In ECCV, 2020.
[5] Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry-aware 3d generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-16133, 2022.
[6] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
[7] Javier Dehesa, Andrew Vidler, Julian Padget, and Christof I Lutteroth. Grid-functioned neural networks, In ICML, 2021.
[8] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. CvxNet: Learnable convex decomposition. In CVPR, 2020.
[9] Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and Zico Kolter. Multiplicative filter networks. In ICLR, 2020.
[10] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3D shape. In CVPR, 2020.
[11] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In ICCV, 2019.
[12] Ian Goodfellow. Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In NewrIPS, 2014.
[13] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In JCML, 2020.
[14] David Ha, Andrew Dai, and Quoe V Le. Hyper/Netorkss. In ICLR, 2016.
[15] Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Ya Yu Liu. GANeraft: Unsupervised 3D Neural Rendering of Minecraft Worlds. In ICCV, 2021.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. NeurlPS, 2017.
[17] Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character control. ACM TOG, 2017.
[18] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. sanage-to-imaee translation with conditional adversarial networks. In CVPR, 2017.
[19] Chiyu Jiang, Avncesh Sud, Amecsh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3D scenes. In CVPR, 2020.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
[21] Arno Knapitsch, Jaesik Park, Qian Yi Zhou, and Vladien Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4): 13, 2017.
[22] Jac Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv preprint arXiv: 100:: 028000028444488 2017.
[23] David B. Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein, Bacon: landteimited coordinate networks for multiscale scene representation. In CVPR, 2022.
10

---

[24] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NeurlPS, 2020.
[25] Julien N.P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. ACORN: Adaptive coordinate networks neural representation. ACM TOG, 2021.
[26] Ricardo
Noha Radwan, Mchdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild; Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
[27] Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 122:999-108, 1995.
[28] Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In ICCV, 2021.
[29] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupaney networks: Learning 3D reconstruction in function space. In CVPR, 2019.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[31] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
[32] Ben Moscley, Andrew Markham, and Tarje Nissen- Meyer. Finite basis informed neural networks (FBPINNs): scalable domain decomposition approach for solving differential equations. arXiv preprint arXiv:2107.0787), 2021.
[33] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies primitives with a multiresolution hash encoding. ACM TOG, 2022.
[34] Michael Niemeyer and Andreas Geiger. GIRAFFE: Representing seenes as compositional generative neural feature fields. In CVPR, 2021.
[35] Michael Niemeyer, Lars Mescheder, Michael Oechsle. and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In CVPR, 2020.
[36] Michael Occhsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In ICCV, 2019.
[37] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. CVPR, 2019.
[38] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In ECCV, 2020.
[39] Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for activations in coordinate-M1L MLPP. arXiv preprint arXiw:2111.15135, 2021.
[40] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. DeRF: Decomposed radiance fields. In CVPR, 2021.
[41] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In ICCV, 2021.
[42] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. ICCV, 2019.
[43] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative radiance fields for 3D-aware image synthesis. In NewrIPS, 2020.
[44] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NewrIPS, 2020.
[45] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-structure aarrr neural scene representations. In NeurlPS, 2019.

---

[46] Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines, 2007.
[47] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. CVPR, 2021.
[48] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar, Block-nerf: Scalable large seene neural view synthesis, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248-848888 82558, 2022.
[49] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keu Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurlPS, 2020.
[50] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shigin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields visual computing and beyond. Eurographics, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Included in Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Included in S5ction
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We include all
the details needed to reproduce the results the supplemental material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Fully specified in the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [Yes] The error bars are included in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Included the supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In the supplemental material. All the data we used are publicly available.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We include code snippets in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] All the data are permissively licensed.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes]] The particularly relevant, publicly available FFHQ datasct is originally obtained from permissively licensed sources.
12

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We did not crowdsource or use human subjects.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "T7sr0LgM17j", "reviewer": "Reviewer_9VaW", "summary": "The submission proposes a new Neural Architecture for coordinate-based networks.\nMotivated by previous work that decomposes the domain and trains distinct networks within them, a so-called\"Level of Experts\" architecture is proposed.\nInstead of distinct networks for each subarea, the authors propose to assign each weight matrix of a layer to multiple parts of the signal domain with periodic recurrence via a query function. The number of available weight matrices and the frequency of their occurrence can vary for each layer of the architecture, i.e. the result is a hierarchy of position-dependent and periodic weights. \nThis is accomplished via first shifting and scaling the position, and then passing it through a modulo-based operation that maps the position to a grid of weight matrices.\nTo avoid discontinuities, the query function does not return a single weight matrix, but instead a linear or bilinear interpolation of the 'nearest' (with respect to the query function) weight matrices.\nNotably, the proposed architecture does not require an additional positional encoding at the input.\n\nExperiments:\n1.  The expressivity of the architecture for fitting a high-resolution (8192x8192) image as well as a video (512x512; 300 frames) is demonstrated. Comparisons are made with related architectures such as MLPs with positional encoding and MLPs with sinusoidal activations (SIREN).\nDifferent settings for the query function are evaluated, each of which corresponds to different \"patterns\" in the signal space.\n2.  A Novel View Synthesis task (\"Tanks and Temples dataset\") using NERF. Comparisons include MLPs+positional encoding and a nonhierarchical setting (i.e. a decomposition of the domain into chunks for independent networks). Focus is drawn to the spatial artifacts of the simple domain decomposition, whereas the proposed method resolves these issues with the interpolation of the weights.\n3. Image Generation using GANs, with the proposed architecture as the generator.\n\nIn all settings of 1.) and 2.) the \"Level of Experts\" architecture provides higher PSNR and SSIM while having comparable computational cost to a simple MLP with positional encoding.\nSimilar results are obtained for the generative model in 3.), here showing a higher FID with comparable computational costs.\nFor the interpolation between the weight matrices, the linear interpolation shows similar performance to the bilinear case, while having significantly lower computational costs. ", "questions": "The method of [1] is mentioned to have a similar high-level idea, and its limitations are indirectly shown with the chunked settings.  I think the argument for the advantage of the \"Levels-of-Experts\" could be made even stronger, by highlighting that the architecture of [1] requires the \"Chunked\" network to be distilled from a pre-trained network. \n\n[1] Reiser, Christian, et al. \"Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "limitations": "The authors sufficiently address the limitations in terms of the required prior information about the signal for selecting the grid scales.\nIn addition, the limited applicability to settings that require (higher-order) derivatives (e.g. Physics Informed Neural Networks) is mentioned, which is a result of the underlying interpolation method between the weight matrices.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Significance:\n- (Medium) An general architecture for coordinate-based Neural Networks is proposed, outperforming comparable state-of-the-art in a wide range of tasks while preserving low computational costs.\n- (Medium) The proposed framework is mostly orthogonal to existing methods for improving implicit representations and appears to be relatively simple to implement, offering itself to practical use and future work.\n\n### Originality\n- To the best of my knowledge, the proposed use of periodically reoccurring weight matrices in each layer of the \"Levels-of-Experts\" is novel.\n- The related work seems to sufficiently cover the existing literature and competitive methods. It should however be noted that I have limited familiarity with this specific subfield and might not be aware of newer work.\n\n### Quality\n- The paper seems to the best of my knowledge to be technically sound, however, it is mostly of a practical nature providing a novel architecture/framework.\n- The experiments cover a wide range of tasks, and are generally sound, showcasing both quantitative and qualitative results and advantages compared to existing approaches.\n- No code is included, although later publication is indicated. This disagrees with the given answer in Checklist 3. a).  \n\n### Clarity\n- The paper is well written, clearly structured, and in general enjoyable to read.\n- The method is clearly described. It should be clear how to reimplement the proposed levels of experts given the paper.\n\nIn summary, the paper is well written and the proposed method is sound.\nThe contribution is of mainly practical nature, but its clear advantages are shown in extensive experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BSDEme1jeOR", "reviewer": "Reviewer_SVJM", "summary": "The paper addresses the compactness and efficiency issue in grid based signal representations and propose a Levels-of-Experts (LoE) framework that hierarchically arrange the weights of the MLP from a set of candidate weight matrix for high fidelity data (image, video, volume) representation.", "questions": "I do not fully understand why the proposed representation is able to reduce computational costs compared to prior works (L296). If I understand it correctly, the linear interpolation between weights is extremely consumed (k times number of layer times nodes of each layer), and the computational costs can be significantly larger for high dimensional data (dim>1).", "limitations": "Limitations have been discussed in detail.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "What's good:\n1) The paper organization and presentation are clear.\n2) The idea of position-dependent weight is novel and interesting.\n3) The paper provides qualitative and quantitated evaluations on image and video regression, radiance field reconstruction and image generation. The experiments are sufficient to demonstrate the effectiveness.\n\nTo be improved:\n1) The abstract claims that the key advance of the proposed method is its compact global latent representation compared to most recent grid based approaches, however, I couldn't find any related discussion or experiment on why global representation is better. Maybe a continue or high order derivative could be one of the advantages?\n2) More comparison results and details discussions with recent approaches would be appreciated, such as BACON[23], MFN [9] and instant-NGP [33].\n3) Seems the reference to TensoRF is incorrect, please check.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AOHXPnV5-Ya", "reviewer": "Reviewer_Girn", "summary": "The paper proposes an algorithm for performing implicit neural representation in a hybrid fashion, trading off between the discrete and continuous counterpart through a hierarchical tiling structure. The authors propose a fast tiling mechanism determining which MLP system to use at what hierarchical level (layer). The authors showcase that this methodology of using multiple experts at every layer with a fast expert-accessing mechanism leads to performance benefits on modelling high resolution images and videos, and ultimately outperforms the baseline counterparts on the tasks of novel view synthesis as well as GAN-based image generation tasks.", "questions": "I have a few clarification questions that might help me understand the work better.\n\n* Why is there such a big difference in the number of parameters between PE MLP system and the others?\n* Further, why are the number of parameters same in PE + CE with the proposed models? The proposed model introduces N different weight matrices at every layer, so I would assume that the number of parameters would grow by $\\times N$\n* In the novel view synthesis, how is only *indirect supervision* provided? Is there not ground-truth supervision based on the true color and density at the different points on the ray? Some clarification on this would be helpful.", "limitations": "The work can be substantially improved by averaging the performance of the proposed algorithm over multiple different data-points as opposed to just a single example. Further, the authors should do some form of variance analysis to make sure that the results are statistically significant. Finally, for some reason a lot of the Figures have captions that talk about top and bottom row, while the figure only has one row (eg. Figure 4)\n\nBeyond this, there are no potential negative societal impacts that I can think of which stem from this work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The authors propose an intuitive extension to coordinate-based MLP systems by allowing for coordinate-dependent weight matrices.\n* The choice of expert chosen at any point in the grid and hierarchy is obtained through an extremely fast algorithm.\n* The results show improvements of the proposed approach over the baselines, often with lower memory cost.\n\n**Weaknesses**\n\n* The proposed work introduces certain additional hyper-parameters, namely $(\\alpha^i, \\beta^i)$, ablations regarding which are lacking in the draft.\n* The authors claim that their approach leads to dynamic expert selection; however the proposed algorithm only lays out a static tiling based on which expert selection is done. I would recommend the readers to refer to *Is a Modular Architecture Enough?* (Mittal et. al 2022) or *Recurrent Independent Mechanisms* (Goyal et. al 2019) as some of the works that look into *dynamic* expert selection, as opposed to a static grid tiling.\n* Are the results of the experiments in Section 4.1 - 4.3 averaged over different high resolution images or different video samples or is it only based on a single example?\n* The work is lacking in quantifying the variance of the results, which can be obtained through multiple seeded runs as well as multiple data-points.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8RhVUwWvxfj", "reviewer": "Reviewer_yDin", "summary": "A Mixture of Experts framework is proposed that uses a dynamic weight MLP with position-dependent weights for solving various coordinate based tasks such as signal fitting, novel view synthesis and generative modeling. Unlike other hybrid representations that divide the space to use different MLPs, the proposed method uses layer-level tiling in the network architecture which produces smoother boundary interpolations. The results adequately establish the efficacy of the approach on 4 challenging tasks compared to strong baselines. ", "questions": "1. How did you come up with the optimal hierarchical tiling pattern.\n2. What are the ranges of alpha and beta. How do these hyperparameters affect the continuity at the boundaries.", "limitations": "The limitations have been adequately addressed in the paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is very well-written, with relevant references and easy-to-follow narration.\n2. The method is simple and general enough such that can be plugged into many different coordinate based tasks, as shown in the experiments.\n3. The evaluations show significant gains in accuracy as compared to the prior art.\n\nWeaknesses:\n1. The paper claims that the LoE formulation could be used to model large-scale signals. None of the experiments reflect this claim.  \n2. The network design of hierarchical and tiled weights seem handcrafted. It is not clear if there is a way to come up with the correct pattern without trial-and-error.\n3. It will be interesting to see comparison to more recent encoding strategies such as Instant-NGP's multi-resolution hash encoding etc. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept