Title: Neural Experts: Mixture of Experts for Implicit Neural Representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Implicit Neural Representation

Surface Reconstruction

Mixture of Experts
Abstract: Implicit neural representations (INRs) have proven effective in various tasks including image, shape, audio, and video reconstruction. These INRs typically learn the implicit field from sampled input points. This is often done using a single network for the entire domain, imposing many global constraints on a single function. 
In this paper, we propose a mixture of experts (MoE) implicit neural representation approach that enables learning local piece-wise continuous functions that simultaneously learns to subdivide the domain and fit it locally. 
We show that incorporating a mixture of experts architecture into existing INR formulations provides a boost in speed, accuracy, and memory requirements. Additionally, we introduce novel conditioning and pretraining methods for the gating network that improves convergence to the desired solution. 
We evaluate the effectiveness of our approach on multiple reconstruction tasks, including surface reconstruction, image reconstruction, and audio signal reconstruction and show improved performance compared to non-MoE methods. Code is available at our project page https://sitzikbs.github.io/neural-experts-projectpage/ .


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a mixture of experts (MoE) approach for INRs, which allows the learning of local piece-wise continuous functions by subdividing the domain and fitting locally. The incorporation of a MoE architecture enhances speed, accuracy, and memory efficiency. They also propose a novel manager architecture and initialization that enable domain subdivision without ground truth.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n2. The proposed MoE INR has a good performance compared to baselines.\n3. The idea of delivering MoE as a learnable partition region for INR fitting with randomized initialization is novel to me. From the ablation study, the randomized initialization improves the performance a lot.\n\nWEAKNESSES:\n1. Missing some closely-relative works. I encourage the authors to have a detailed discussion of previous MoE INRs [1,2] and decomposition/partition-based INRs [3,4].\n2. Lacking some key comparison experiments with decomposition/partition-based INRs. The authors only compare their method with the baseline SoftPlus and SIREN (and their wider version). However, some related works [4] have also shown the INR based on pre-determined masks can also outperform the wider version of SIREN.  I encourage the authors to experimentally compare your method with [4] to illustrate the necessity of learnable partition regions.\n3. A detailed ablation study on the hyper-parameters of the MoE INRs is missing, such as the layer of encoder, manager, and experts. Given a fixed number of parameters, how to allocate the parameters to the three modules remains unknown.\n\n[1] Zhao, Jianchen, et al. \"MoEC: Mixture of Experts Implicit Neural Compression.\" arXiv preprint arXiv:2312.01361 (2023).\n[2] Wang, Peihao, et al. \"Neural implicit dictionary learning via mixture-of-expert training.\" International Conference on Machine Learning. PMLR, 2022.\n[3] Rebain, Daniel, et al. \"Derf: Decomposed radiance fields.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[4] Liu, Ke, et al. \"Partition speeds up learning implicit neural representations based on exponential-increase hypothesis.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\nQUESTIONS:\n1. Could you please discuss how to choose the number of experts giving a fixed number of parameters and iteration time? Is it better to have more experts with each one having fewer parameters or fewer experts with each one having more parameters?\n2. I wonder whether it is possible to apply your method to NeRF since there are no supervised signals for the 3D ground truth.\n3. When comparing the Vanilla MoE INR and your Neural Experts, have you kept their total parameters similar (smaller experts for your Neural Experts due to the extra parameters needed by the encoders)?\n\nLIMITATIONS:\nThe limitations and potential negative societal impact of their work have been discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new architecture for implicit neural representations (INRs) based on the mixture-of-experts (MoE) architecture. This new architecture is differs from traditional MoE architectures in that now all the experts have a shared encoder and expert-specific decoders, while the manager also now has an encoder-decoder architecture, with the manager decoder taking as input both the manager encoder's representation as well as the experts' shared encoder representation. The authors also provide a pre-training method for the manager. The method is evaluated on image reconstruction, audio reconstruction, and 3D shape reconstruction by fitting signed distance functions (SDFs).\n\nSTRENGTHS:\nThis paper proposes a novel MoE-based architecture for INRs together with a novel pre-training strategy for the MoE manager. \n\nThe empirical results are good and there is an ablation study on the major components. \n\nThe paper is well-written and easy to understand. \n\nMany details relating to reproducibility are provided in the supplemental material.\n\nWEAKNESSES:\nOne of the major weaknesses of this paper is the experimental evaluation. The method does not compare against other methods that propose a new INR architecture (e.g. Gaussian activation function [1], WIRE [2], FINER [3]) or a standard MLP with positional encoding. The experimental evaluation is not also very robust, as only small datasets are used (Kodak, only 3 audio recordings, only 4 shapes) and more complicated tasks investigated by similar works are not considered (for example, WIRE [2] and FINER [3] both include evaluation on neural radiance fields). \n\nThe proposed MoE architecture also did not show improvements when used with softplus activation. \n\nThis paper also did not include other metrics normally used to evaluate the tasks, such as SSIM and LPIPS for 2D image fitting (e.g. FINER [3]). \n\nMinor point: inclulding the number of parameters in Table 1 may be helpful since comparisons between the number of parameters is discussed in the text.\n\nReferences\n1. Ramasinghe, Sameera, and Simon Lucey. \"Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n2. Saragadam, Vishwanath, et al. \"Wire: Wavelet implicit neural representations.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n3. Liu, Zhen, et al. \"FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\n1. Can this work be combined with traditional ReLU MLPs with positional encoding or MLPs with activation functions other than sine?\n2. How does this work compare to other novel INR architectures (e.g. WIRE, FINER, etc)?\n\nLIMITATIONS:\nLimitations and negative impact are addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel INR framework that leverages the Mixture of Experts (MoE) technique. The proposed strategy consists of an expert and a manager branch. Each branch has an encoder that processes the input coordinate and extracts an embedding. By processing the two encoder embeddings, the manager predicts the probability of which of the N experts should be used for extracting the signal. They show how the proposed INR framework achieves better reconstructions than SIREN on several modalities, such as audio, image, or 3D surfaces. They also propose a manager pre-training strategy, which is necessary to exploit all the experts effectively.\n\nSTRENGTHS:\n-\tThe original idea might be conducive to new research in this direction.\n-\tThe paper is well-written and easy to follow. \n-\tSupervising experts with semantic losses and obtaining networks specialized in specific semantic areas of the input signal might unlock several applications for INRs and ease their interpretability.\n-\tThe proposed framework achieves good reconstruction performance.\n-\tThe ablations in Table 4 and Table 5 are very insightful.\n\nWEAKNESSES:\nW1) The major weakness of the paper is the misalignment between the experimental results and the motivations of this research:\n\n a- In the introduction (L26-28), the authors correctly point out that, in traditional INRs, each coordinate needs to be processed by the whole network. Even though they claim that this problem can be solved by MoE INRs, in the proposed architecture, the input coordinate needs to be processed by the full manager and the encoder of the expert branch. The only saved computation is the one from the final expert, a much smaller network than the others. Thus, the saved computation looks minimal to standard SIREN (considering the same total weights). Moreover, no experiments regarding computational efficiency and the advantages of parallelized inference are necessary to motivate this claim. Maybe, instead of talking about the absolute efficiency of the proposed approach, it is better to show the better trade-offs in terms of efficiency and reconstruction quality than SIREN.\n\nb- In the introduction (L28-30), the authors claim that standard INRs extract features vastly different for close spatial coordinates (i.e., locality problem). I am unaware of studies that formally investigate this INR property. Thus, I suggest adding a reference work or validating it with experiments. Moreover, the authors claim MoE INRs can learn local piece-wise functions (L259 in the conclusion section). Thus, they do not suffer from the problem above. Yet, the experiments show something different. For instance, by looking at Figure 3c, different experts predict audio signals for close temporal coordinates. I can notice the same behavior in the last column of Figure 6, in which many distinct experts predict pixels in the upper part of the image.\n\nW2 ) The idea of using MoE resembles the idea of KiloNeRF [1]. In that case, the routing strategy is not learned and depends only on the input 3D coordinate, and each expert focuses on a pre-determined spatial region. I think the authors should add this reference, explaining the pros and cons of the two kinds of approaches.\n\nW3) In recent years, INR frameworks have been proposed to be faster and more efficient than MLP-only techniques such as SIREN. For instance, hybrid INRs such as InstantNGP [2] (hash grid + MLP) can be used as the base framework to speed up computation when learning INRs of images and surfaces or NeRFs. The paper should also include more recent competitors than SIREN.\n\n\n[1] Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. ICCV 2021.\n[2] Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 2022.\n\nQUESTIONS:\nQ1) In Table 1, does the vanilla MoE baseline employ Softplus activations, while the final strategy uses Sine activations? In this case, the comparison is unfair and does not validate the superiority of their approach to the vanilla one.\n\nQ2) Can the authors also include the Chamfer Distance metric in Table 3?\n\n\n\nI like the paper's core idea. However, the author's response to my concerns will greatly influence my final rating.\n\nLIMITATIONS:\nThe paper includes a discussion on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper introduces a MoE architecture for INRs, enhancing scalability, local fitting, and parallelization. Traditional INRs use a single network, imposing global constraints, while this method learns several local expert functions, subdividing the domain. Key contributions include a novel manager architecture and initialization method for better convergence and domain subdivision without ground truth. It show improved performance and efficiency over traditional methods, across image, audio, and 3D surface reconstruction,

STRENGTHS:
- It demonstrates a neat architectural design along with a robust ablation study.
- It consistently shows performance improvement across various tasks, and the performance with respect to the number of parameters is also superior.
- It is interesting that random initialization, which includes no inductive bias in the manager pretraining process, outperforms initializations like SAM.

WEAKNESSES:
- It requires more parameters compared to the vanilla model. While it is fair to compare it with the wider version of the vanilla model, it is unclear if the proposed model still performs well when it has the same number of parameters as the vanilla model (not wider version). Tab.3 alleviates this concern to some extent.

- In addition to comparing with the vanilla model, it would be good to include a discussion on various INR methods that apply locality bias (e.g., spatial functa).

- As shown in the convergence graphs in the appendix, it shows more unstable convergence compared to the baseline.

QUESTIONS:
- Please see the above weaknesses.
- In Tab.1, what is the PSNR of Neural Experts SIREN with the same number of parameters as SIREN?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15