Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  This paper studies the problem of learning the gauge transformation in an end-to-end manner within learning the neural scene representation setting.  In this context, gauge transformations in  continuous and discrete cases are learned. The developed learning paradigm, which is generic,  maps a 3D point to a continuous coordinate or a discrete index in the target gauge, respectively.
This work also introduces  the concept of Information Regularization (InfoReg) from the principle of information conservation during gauge transformation. This regularization is performed by maximizing the mutual information between gauge parameters.  The paper also claims that the proposed top-k gauge is useful in achieving a tradeoff between learning collapse and computation cost.

strength_and_weaknesses: ++ The paper is generally well written and easy to follow. The presented method is intuitive and meaningful. Graphical illustrations of Fig. 1, 2 , and 4 are particularly helpful.

++ Sections 3.2 and 3.3 are well motivated and offer new theoretical insights.

The empirical discovery and analysis on top-k gauge, reported in Section 3.3 and Fig. 7 are particularly interesting.  

++ The supplementary material with additional pseudo code and discussion regarding the method’s limitations is helpful.


-- Comparison with SOTA method on Table 3, although highlights the benefit in terms of model size, is limited to only two scenes. It is difficult to conclude the benefit of the proposed method over SOTA without more exhaustive experiments.

--  The claim of  “trade-off between collapse and cost” on the basis of Figure 7 (in Section 4.2.2) is somewhat misleading.  The reported PSNR for the “original” method may be poor but cannot be said to collapse. As such, the reported gap in PSNR (which is not very significant) may not generalize well across scenes.

--  Qualitative results reported in Fig 8 and 9 (in the supplementary materials) are difficult to understand.

--  The limitation on “output layer dimension ranging from 2^14 to 2^24” is indeed of a serious concern.

clarity,_quality,_novelty_and_reproducibility: Please refer the strengths above.
summary_of_the_review: 
I generally like the addressed problem and the approach made in the paper. However, it is currently difficult to know the benefit of the proposed method. This is mainly due to the limited experimental results. I suggest authors report more exhaustive experiments highlighting the contribution empirically.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author proposes a general framework for the neural gauge field, including continuous and discrete cases. To solve the collapse problem, the author proposes a regularization term - InfoReg. In practice, the regularization term regulates the target distribution to a uniform distribution. The author also proposes a new top-k gauge mechanism that achieves a trade-off between model collapse and computational cost. Experiments show the method outperforms other regularization methods and gauge transformation methods. The method also outperforms SOTA in rendering tasks, but with higher computation costs. 
strength_and_weaknesses: Strength:
+ The proposed method proves to be effective and has the potential to be adopted by future "gauge transformation" frameworks. 

+ The paper is well-written and easy to follow for me. 

Weakness:
+ The performance of the rendering tasks at inference time sacrifices computational cost. 
clarity,_quality,_novelty_and_reproducibility: Clarity:
+ Top-K strategy helps with distribution collapse. The paper claims the Top-K strategy helps with distribution collapse. The experiment shows the strategy improves rendering quality but doesn't provide evidence that it helps with distribution collapse. Also, it will be better if the paper can use a quantitative metric to evaluate the degree of "collapse". 

+ Training Time. I'm wondering what the training time of this method compares with other methods. As the EMD calculation takes a long time, will the regularization terms make the training time longer than other methods? 

+ Equation 1: i_x ∈ R^1 => i_x ∈ N^1

+ Uniform Distribution Range: For the continuous case, the range of q(y) is R^2 or [0, 1]^2?

+ Other manifolds: Tha paper assumes the target distribution to be a uniform distribution. Does the method also apply to other distributions? It will be interesting if these results can be provided, to show the generalizability of the method. 

Quality
+ Fair.

Novelty 
+ Fair.

Reproducibility
+ Fair


summary_of_the_review: This paper proposes a general method for gauging transformation, which is intellectually interesting and could be potentially applied to other frameworks and applications. My major concern is the higher training and inference time for high-quality results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a way to learn gauge transformer for neural fields applications (especially modeling neural radiance field for novel view synthesis). The main contribution of the work is to propose a unified framework to reason about different NeRF paradigms in the language of gauge transformation and proposes a regularization to enable learning such transformation without suffering from collapsing in local minimum. Empirical results suggests that the proposed regularization indeed improve performance.
strength_and_weaknesses: Strength:
1. The paper provides a novel perspective that unifies the coordinate transform in instant-NGP, tensorRF, and triplanes under the framework of gauge transform. 
2. The proposed regularization shows improvement over existing regularization methods provided by prior works.
3. The paper shows that with the novel Top-4 Gauge, the method is able to perform comparably with SOTA.

Weakness:
1. Evaluation. It's not very clear to me that the using downstream novel view synthesis is the best way to evaluate Neural Gauge Fields. I believed that it's essential to show that the proposed method doesn't lag behind SOTA performance, but it's also important to show case what this new formulation should allow. I will encourage the authors to include other applications, potentially in the field of learning texture, materials, BRDF etc.
2. Writing. In the introduction and method section, it's hard for me to follow the motivation why is it beneficial to think of instant-NGP, tensorRT in the framework of gauge transform. This is not clear to me what's the theoretical benefit (cleaner? easier to reason about different mappings?) or the empricial advantages. I would also encourage the authors to include a background section about gauge transform as it can benefit the readers without enough backrgound.
3. The authors compared to instant-NGP / tensorRT, both of these two methods are very memory efficient and fast to optimize. I wonder if Neural Gauge Field subsumes these formulation, does the same benefit carries through? It would be nice to see time profiling and memory profiling if possible.
clarity,_quality,_novelty_and_reproducibility: Please see the strength & weakness section.
summary_of_the_review: My main concern about the paper is about the evaluation. The paper mainly evaluate the performance of Neural Gauge Field using novel view synthesis, without reporting optimization time, memory, or other applications (which I believed the true use of Neural Gauge Field can lay). The secondary concern is regarding it's not clear why we need the unified language in terms of gauge transform to describe different formulation such as instang-ngp, triplanes, and/or tensorRF. 


-------
Update: I decided to raise my score since the authors address a number of concerns I raised.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the feature index or coordinate mapping function in radiance field reconstruction as a gauge transformation problem. The core idea of the manuscript is to optimize the transformation by maximizing the mutual information and encouraging uniform coordinate distribution with an EM distance regulation term. Extensive experiments on syntheses and real object datasets demonstrate that the proposed method achieves better performance in terms of rendering quality and compactness compared to the recent SOTA fast radiance field reconstruction approaches (instant-NGP and TensoRF ) and space wrapping baed radiance field reconstruction methods (Non-Rigid Neural Radiance Fields and NeuTex).
strength_and_weaknesses: What's good:
1) The idea of transforming coordinate learning into a distribution regulation problem is interesting and elegant.
2) The usage of KL divergence and EM distance loss for coordinate space mapping seems novel and interesting.
3) The paper provides a detailed mathematical derivation of the relationship between mutual information maximization and the proposed KL divergence loss, but I could not read over the derivation due to this emergency review.
4) The proposed method shows better quantitative rendering scores than the previous approach.

To be improved:
1) The overall pipeline is unclear to me, a pipeline figure could really help to better understand the proposed method.
2) The paper lacks a key highlight that motivates me to use it in practice, it's good to perform better than the other two space-wrapping-based approaches, but the scores still have a big gap with SOTA and suffer from high computation cost. Would be better if the authors can figure out some unique properties in the proposed method. How does it perform for radiance field editing and rendering speed? Can it combine with the traditional rasterization rendering framework for fast rendering since the features are distributed in 2D space?
3) Table 3 is tricky to me, why not use the default configuration in the TensoRF and Instant-NGP? Seems the total parameters in Instant-NGP are significantly smaller than TensoRF under your configuration but the model size in instant-NGP is larger than TensoRF: 2^14 * 8 * 2 (even assuming 2^14 entries for each level) vs. 64 * 64 * 27 * 3, please correct me if any misunderstanding to the counting. And could be better to list the training time to have a better side-by-side comparison.
4) The reference to the convolutional occupancy field is missing, which is early work that also uses space wrapping.
5) Lacking the reference, discussion and comparison with "Variable Bitrate Neural Fields", which also learn a coordinate wrapping function.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clear mostly.

Novelty: Yes as commented above

Reproducibility: not sure since the implementation section is not clear to me.
summary_of_the_review: The proposed regulation terms are novel and interesting to me, and happy to see the authors can clarify the questions listed in the "to be improved" section in the rebuttal period.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework for neural gauge fields in the context of radiance fields and experimentally shows that the proposed regularisation is superior to alternative ones. This work also introduced the idea of a top-k gauge that gradually alleviates the need for regularisation.
strength_and_weaknesses: Strengths
-----------

Overall I believe the idea of exploring to learn a gauge transformation in an end-to-end manner is an important research direction, and this work makes contributions in this regard.

Results show better performance than state-of-the-art methods.


Weaknesses
--------------
Overall, this paper was hard to evaluate because I am not an expert in the neural radiance field and because I believe the presentation requires improvement. The elaboration on existing methodologies and new ideas is not in a way that presents the reader with an abstraction of the theory behind recent advances to then clearly and concisely show their contributions.





clarity,_quality,_novelty_and_reproducibility: Although I believe this work could present new ideas, I feel the presentation requires significant improvement to evaluate novelty as not an expert in the topic.
summary_of_the_review: Despite they could be a significant theoretical contribution in trying to present a new framework for gauge fields, I could not really judge the impact and novelty in the way this paper is written.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper provides a unifying perspective on a recent trend in learning Neural Fields (TensorRF, Instant-NGP) where a pre-defined mapping is first used to map 3D points to a different space over which a neural field is learned. This work proposes a mechanism (and best practices e.g. regularization schemes) to instead learn this mapping for both, continuous and discrete cases, and demonstrates consistent improvements by doing so. Given the empirical results and the interesting theoretical perspective in an increasingly relevant area, the AC feels this paper would benefit the community. All the reviewers (except one with low confidence) recommend acceptance, and the AC concurs.
justification_for_why_not_higher_score: While I am certain about the acceptance decision, I think the paper could also be a spotlight. The primary reason for leaning towards a poster is that none of the reviewers are strongly championing the work, and that the empirical gains, while consistent, are not overwhelming and do rely on some tunable hyperparameters.
justification_for_why_not_lower_score: The unifying perspective provided for parametrizing 3D neural fields via intermediate mappings should be useful and interesting to the community. Moreover, the consistent benefits of learning such a mapping as well as the use of Infogain as a regularizer in discrete and continuous settings show that these could be adapted across scenarios.
Decision: Accept: poster