Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: A Mixture of Experts framework is proposed that uses a dynamic weight MLP with position-dependent weights for solving various coordinate based tasks such as signal fitting, novel view synthesis and generative modeling. Unlike other hybrid representations that divide the space to use different MLPs, the proposed method uses layer-level tiling in the network architecture which produces smoother boundary interpolations. The results adequately establish the efficacy of the approach on 4 challenging tasks compared to strong baselines. 
strengths_and_weaknesses: Strengths:
1. The paper is very well-written, with relevant references and easy-to-follow narration.
2. The method is simple and general enough such that can be plugged into many different coordinate based tasks, as shown in the experiments.
3. The evaluations show significant gains in accuracy as compared to the prior art.

Weaknesses:
1. The paper claims that the LoE formulation could be used to model large-scale signals. None of the experiments reflect this claim.  
2. The network design of hierarchical and tiled weights seem handcrafted. It is not clear if there is a way to come up with the correct pattern without trial-and-error.
3. It will be interesting to see comparison to more recent encoding strategies such as Instant-NGP's multi-resolution hash encoding etc. 
questions: 1. How did you come up with the optimal hierarchical tiling pattern.
2. What are the ranges of alpha and beta. How do these hyperparameters affect the continuity at the boundaries.
limitations: The limitations have been adequately addressed in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper addresses the compactness and efficiency issue in grid based signal representations and propose a Levels-of-Experts (LoE) framework that hierarchically arrange the weights of the MLP from a set of candidate weight matrix for high fidelity data (image, video, volume) representation.
strengths_and_weaknesses: What's good:
1) The paper organization and presentation are clear.
2) The idea of position-dependent weight is novel and interesting.
3) The paper provides qualitative and quantitated evaluations on image and video regression, radiance field reconstruction and image generation. The experiments are sufficient to demonstrate the effectiveness.

To be improved:
1) The abstract claims that the key advance of the proposed method is its compact global latent representation compared to most recent grid based approaches, however, I couldn't find any related discussion or experiment on why global representation is better. Maybe a continue or high order derivative could be one of the advantages?
2) More comparison results and details discussions with recent approaches would be appreciated, such as BACON[23], MFN [9] and instant-NGP [33].
3) Seems the reference to TensoRF is incorrect, please check.
questions: I do not fully understand why the proposed representation is able to reduce computational costs compared to prior works (L296). If I understand it correctly, the linear interpolation between weights is extremely consumed (k times number of layer times nodes of each layer), and the computational costs can be significantly larger for high dimensional data (dim>1).
limitations: Limitations have been discussed in detail.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The submission proposes a new Neural Architecture for coordinate-based networks.
Motivated by previous work that decomposes the domain and trains distinct networks within them, a so-called"Level of Experts" architecture is proposed.
Instead of distinct networks for each subarea, the authors propose to assign each weight matrix of a layer to multiple parts of the signal domain with periodic recurrence via a query function. The number of available weight matrices and the frequency of their occurrence can vary for each layer of the architecture, i.e. the result is a hierarchy of position-dependent and periodic weights. 
This is accomplished via first shifting and scaling the position, and then passing it through a modulo-based operation that maps the position to a grid of weight matrices.
To avoid discontinuities, the query function does not return a single weight matrix, but instead a linear or bilinear interpolation of the 'nearest' (with respect to the query function) weight matrices.
Notably, the proposed architecture does not require an additional positional encoding at the input.

Experiments:
1.  The expressivity of the architecture for fitting a high-resolution (8192x8192) image as well as a video (512x512; 300 frames) is demonstrated. Comparisons are made with related architectures such as MLPs with positional encoding and MLPs with sinusoidal activations (SIREN).
Different settings for the query function are evaluated, each of which corresponds to different "patterns" in the signal space.
2.  A Novel View Synthesis task ("Tanks and Temples dataset") using NERF. Comparisons include MLPs+positional encoding and a nonhierarchical setting (i.e. a decomposition of the domain into chunks for independent networks). Focus is drawn to the spatial artifacts of the simple domain decomposition, whereas the proposed method resolves these issues with the interpolation of the weights.
3. Image Generation using GANs, with the proposed architecture as the generator.

In all settings of 1.) and 2.) the "Level of Experts" architecture provides higher PSNR and SSIM while having comparable computational cost to a simple MLP with positional encoding.
Similar results are obtained for the generative model in 3.), here showing a higher FID with comparable computational costs.
For the interpolation between the weight matrices, the linear interpolation shows similar performance to the bilinear case, while having significantly lower computational costs. 
strengths_and_weaknesses: ### Significance:
- (Medium) An general architecture for coordinate-based Neural Networks is proposed, outperforming comparable state-of-the-art in a wide range of tasks while preserving low computational costs.
- (Medium) The proposed framework is mostly orthogonal to existing methods for improving implicit representations and appears to be relatively simple to implement, offering itself to practical use and future work.

### Originality
- To the best of my knowledge, the proposed use of periodically reoccurring weight matrices in each layer of the "Levels-of-Experts" is novel.
- The related work seems to sufficiently cover the existing literature and competitive methods. It should however be noted that I have limited familiarity with this specific subfield and might not be aware of newer work.

### Quality
- The paper seems to the best of my knowledge to be technically sound, however, it is mostly of a practical nature providing a novel architecture/framework.
- The experiments cover a wide range of tasks, and are generally sound, showcasing both quantitative and qualitative results and advantages compared to existing approaches.
- No code is included, although later publication is indicated. This disagrees with the given answer in Checklist 3. a).  

### Clarity
- The paper is well written, clearly structured, and in general enjoyable to read.
- The method is clearly described. It should be clear how to reimplement the proposed levels of experts given the paper.

In summary, the paper is well written and the proposed method is sound.
The contribution is of mainly practical nature, but its clear advantages are shown in extensive experiments.
questions: The method of [1] is mentioned to have a similar high-level idea, and its limitations are indirectly shown with the chunked settings.  I think the argument for the advantage of the "Levels-of-Experts" could be made even stronger, by highlighting that the architecture of [1] requires the "Chunked" network to be distilled from a pre-trained network. 

[1] Reiser, Christian, et al. "Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
limitations: The authors sufficiently address the limitations in terms of the required prior information about the signal for selecting the grid scales.
In addition, the limited applicability to settings that require (higher-order) derivatives (e.g. Physics Informed Neural Networks) is mentioned, which is a result of the underlying interpolation method between the weight matrices.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes an algorithm for performing implicit neural representation in a hybrid fashion, trading off between the discrete and continuous counterpart through a hierarchical tiling structure. The authors propose a fast tiling mechanism determining which MLP system to use at what hierarchical level (layer). The authors showcase that this methodology of using multiple experts at every layer with a fast expert-accessing mechanism leads to performance benefits on modelling high resolution images and videos, and ultimately outperforms the baseline counterparts on the tasks of novel view synthesis as well as GAN-based image generation tasks.
strengths_and_weaknesses: **Strengths**

* The authors propose an intuitive extension to coordinate-based MLP systems by allowing for coordinate-dependent weight matrices.
* The choice of expert chosen at any point in the grid and hierarchy is obtained through an extremely fast algorithm.
* The results show improvements of the proposed approach over the baselines, often with lower memory cost.

**Weaknesses**

* The proposed work introduces certain additional hyper-parameters, namely $(\alpha^i, \beta^i)$, ablations regarding which are lacking in the draft.
* The authors claim that their approach leads to dynamic expert selection; however the proposed algorithm only lays out a static tiling based on which expert selection is done. I would recommend the readers to refer to *Is a Modular Architecture Enough?* (Mittal et. al 2022) or *Recurrent Independent Mechanisms* (Goyal et. al 2019) as some of the works that look into *dynamic* expert selection, as opposed to a static grid tiling.
* Are the results of the experiments in Section 4.1 - 4.3 averaged over different high resolution images or different video samples or is it only based on a single example?
* The work is lacking in quantifying the variance of the results, which can be obtained through multiple seeded runs as well as multiple data-points.
questions: I have a few clarification questions that might help me understand the work better.

* Why is there such a big difference in the number of parameters between PE MLP system and the others?
* Further, why are the number of parameters same in PE + CE with the proposed models? The proposed model introduces N different weight matrices at every layer, so I would assume that the number of parameters would grow by $\times N$
* In the novel view synthesis, how is only *indirect supervision* provided? Is there not ground-truth supervision based on the true color and density at the different points on the ray? Some clarification on this would be helpful.
limitations: The work can be substantially improved by averaging the performance of the proposed algorithm over multiple different data-points as opposed to just a single example. Further, the authors should do some form of variance analysis to make sure that the results are statistically significant. Finally, for some reason a lot of the Figures have captions that talk about top and bottom row, while the figure only has one row (eg. Figure 4)

Beyond this, there are no potential negative societal impacts that I can think of which stem from this work.
review_title: null

Meta-review: metareview: This paper presents a framework for position-dependent MLPs where the weights in each layer depend on the input coordinate periodically, with hierarchically tiled periodic weight patterns across successive layers. The paper shows that such models outperform prior work on a variety of tasks involving data representation. The reviewers all agree that the proposed approach is novel and highly effective, and the paper is clear and compelling. I accordingly recommend acceptance.
confidence: Certain
Decision: Accept