General Neural Gauge Fields
Authors: Fangneng Zhan
Lingjie Liu
Adam Kortylewski
Christian Theobalt
Venue: ICLR 2023 poster (2023)
Keywords: NeRF
Neural Scene Representation

Abstract: The recent advance of neural fields, such as neural radiance fields, has significantly pushed the boundary of scene representation learning. Aiming to boost the computation efﬁciency and rendering quality of 3D scenes, a popular line of research maps the 3D coordinate system to another measuring system, e.g., 2D manifolds and hash tables, for modeling neural fields. The conversion of coordinate systems can be typically dubbed as \emph{gauge transformation}, which is usually a pre-defined mapping function, e.g., orthogonal projection or spatial hash function. This begs a question: can we directly learn a desired gauge transformation along with the neural field in an end-to-end manner? In this work, we extend this problem to a general paradigm with a taxonomy of discrete and continuous cases, and develop an end-to-end learning framework to jointly optimize the gauge transformation and neural fields. To counter the problem that the learning of gauge transformations can collapse easily, we derive a general regularization mechanism from the principle of information conservation during the gauge transformation. To circumvent the high computation cost in gauge learning with regularization, we directly derive an information-invariant gauge transformation which allows to preserve scene information inherently and yield superior performance.

Published as a conference paper at ICLR 2023
GENERAL NEURAL GAUGE FIELDS
Fangneng Zhan, Lingjie Liu, Adam Kortylewski, Christian Theobalt Max Planck Institute for Informatics, 66123, Germany {fzhan, 11iu, akortyle, jeeooaat]@@pii
ABSTRACT
The recent advance of neural fields, such as neural radiance fields, has signifi- cantly pushed the boundary of scene representation learning. Aiming boost the computation efficiency and rendering quality of 3D scenes, a popular line of research maps the 3D coordinate system to another measuring system, e-g- 2D manifolds and hash tables, for modeling neural fields. The conversion of coordi- nate systems can be typically dubbed as gauge transformation, which is usually a pre-defined mapping function, e.g., orthogonal projection spatial hash function. This begs a question: can we directly learn desired gauge transformation along with the neural field in end-to-end manner? In this work, extend this prob- lem to a general paradigm with a taxonomy of discrete & continuous cases, and develop end-to-end learning framework to jointly optimize the gauge trans- formation and neural fields. To counter the problem that the learning of gauge transformations can collapse easily, we derive a general regularization mechanism from the principle of information conservation during the gauge transformation. To circumvent the high computation cost gauge learning with regularization, we directly derive an intoommttio-invarrian gauge transformation which allows to preserve scene information inherently and yield superior performance.
INTRODUCTION
Representing 3D scenes with high efficiency and quality has been stand-tan target in computer vision and computer graphics research. Recently, the implicit representation of neural radiance fields (Mildenhall et al. 2021) has shown that a 3D scene can be modeled with neural networks, which achieves compelling visual quality and low memory footprint. However, suffers from long training times. The explicit voxel-based methods (Yu et al. 2021a Sun et al., (022) emerged with faster convergence but higher memory requirements, due to the use of the 3D voxel grids as scene representation. To strike good balance between computation efficiency and rendering quality, EG3D (Chan et al. 2022) proposed to project the 3D coordinate system to a tri-plane system. Along with this line of research, TensoRF (Chen et al. (222)) factorizes 3D space into compact low-rank tensors; Instant-NGP (Müller et al., (022) models the 3D space with multi-resolution hash grids to enable remarkably fast convergence speed.
These recent works (e.g., EG3D and Instant-NGP), share the same prevailing paradigm by converting the 3D coordinate of neural fields another coordinate system. Particularly, a coordinate system of the scene (e.g., 3D coordinate and hash table) can be regarded as a kind of gauge, and the conver- sion between the coordinate systems can be referred Notably, existing gauge transformations in neural fields are usually pre-defined functions (e.g.. or- thogonal mappings and spatial hash functions (Teschner al., 2003)), which are sub-optimal for modeling the neural fields as shown in Fig. [ In this end, learnable gauge transformation is more favored as can be optimized towards the final objective. This raises an essential question: how to learn the gauge transformations along with the neural ficlds. Some previous works explore special case of this problem, e.g., Neu (Xiang et al. 202I) and NeP (Ma et al., 2022) aim to transform 3D points into continuous 2D manifolds. However, general and unified learning paradigm for various gauge transformations has not been established or explored currently.
gauge transformation (Moriyasu) 1983).
In this work, we introduce general Neural Gauge Fields which unify various gauge transformations in neural fields with a special focus on how to leam the gauge transformations along with neural fields. Basically, gauge is defined by gauge parameters and gauge basis, e.g., codebook indices

---

Published as a conference paper at ICLR 2023
3D Scene Points
Orthogonal Mapping
Orthogonal Mapping2
Learned Mapping
Multi-view Images
Conceptual illustration of gauge transformation from 3D point coordinates to 2D plane. Instead of naively employ a pre-defined orthogonal mapping which incurs overlap on the 2D) plane, the proposed neural gauge fields aim to learn the mapping along with neural fields driven by multi-view synthesis loss.
Figure 1:
and codebook vectors, which can be continuous or discrete. Thus, the gauge transformations for neural fields can be duly classified into continuous cases (e.g., triplane space) and discrete cases (e.g., a hash codebook space). We then develop general learning paradigms for continuous cases and discrete cases, which map 3D point continuous coordinate a discrete index in the target gauge, respectively.
As typical cases, study continuous mapping from 3D space the 2D plane and discrete mapping from 3D space to 256 discrete vectors. As shown in Fig. X and 3 we observed that naively opti- mizing the gauge transformations with the neural fields severely suffers from gauge collapse, which means the gauge transformations will collapse to small region in a continuous case or collapse to a small number of indices in a discrete case Baevski al. (2019); Kaiser al. (2018). To regular- ize the learning of gauge transformation, a cycle consistency loss has been explored in Xiang et al. ; Ma et al. (2022) to avoid many-to-one mapping; structural regularization is also adopted in Tretschk al. (2021) to preserve local structure by only predicting a coordinate offset. However, the cycle consistency regularization tends to be heuristic without grounded derivation while the struc- tural regularization is constrained to regularize continuous cases. In this work, introduce more intuitive Information Regularization (InfoReg) from the principle of information conservation dur- ing gauge transformation. By maximizing the mutual information between gauge parameters, successfully derive general regularization forms for gauge transformations. Notably, a geometric uniform distribution and discrete uniform distribution are assumed as the prior distribution of con- tinuous and discrete gauge transformations. respectively, where an Earth Mover's distance and KL divergence are duly applied to measure the distribution discrepancy for regularization.
Learning the gauge transformation with regularization usually incurs high computation cost which infeasible for some practical applications such as fast scene representation. In line with relative information conservation, we directly derive an Information- Invariant (InfoInv) gauge transforma- tion which allows to preserve scene information inherently gauge transformations and obviate the need for regularizations. Particularly, the derived Infolnv coincides with the form of position encod- ing adopted NeRF [Mildenhall et al. 2021), which provides certain rationale for the effectiveness of position encoding in neural fields.
The contributions of this work are threefold. First, we develop a general framework of neural gauge fields which unifies various gauge transformations in neural fields, and gives general learning forms for continuous and discrete gauge transformations. Second, we strictly derive regularization mech- anism for gauge transformation learning from the perspective of information conservation during gauge transformation that outperform earlier heuristic approaches. Third, we present an information nested function to preserve scene information inherently during gauge transformations.
2 RELATED WORK
Recent work has demonstrated the potential of neural radiance fields (Mildenhall et al. 202 and its extensions for multifarious vision and graphics applications, including fast view synthesis (Liu al.
2020 Yu et al. 202 Hedman et al. 2021 Lindell et al. 2222 Neff et al. 2021: Yu et al. 202
Reiser et al. 2021 Sun al. (2022), generative models (Schwarz et al. 2020 Niemeyer & Geiger
2021 Gu et al. 2022; Chan et al. | [202] Or-El et al. 2022),, surface reconstruction (Wang et al. 2021
Oechsle et al. 2021 Yariv al. 2021 etc. Under this context, various gauge transformations have
2

---

Published as a conference paper at ICLR 2023
3D Scone Image
Lorning Comiinsus
Tearning Coniineus Cycle Learning Cuntinucus Straleg Learning Contimoos
Infoleeg
Figure 2: Continuous gauge transformation from 3D scene to 2D square. Without regularization, the learned gauge transformation will collapse to small region in the 2D square. Including cycle regularization (Xiang et al. (021) or structural regularization (StruReg) Tretschk et al. (2021) don't t
fully solve the problem, and our InfoReg achieves clearly better regularization performance.
Leaming Discrete
Learning Discrete Cycle
Leamingg Discrete InfoReg
Learning Top-4
Learning Top-4 InfoRog
Discrete gauge transformation from 3D scene to 256 discrete vectors. Without regulariza- tion, the learned gauge transformation will collapse to a small number of vector indices (horizontal axis). The learning with cycle regularization (Xiang et al., 2021 is still significantly collapse-prone, while our InfoReg enables to alleviates the collapse substantially.
Figure 3:
been intensely explored neural fields for different purposes, e.g., reducing computation cost (Chan 2022), fast convergence (Chen et al. 2022 Müller et al. 2022), editable UV map (Xiang al., Ma et al. 2022).
et al.
2021
A pre-defined mapping function is usually adopted gauge transformation. A simple case is thogonal mapping, which projects 3D space into a continuous low-dimension space. For instance, EG3D (Chan et al., 2022, Peng al., 2020) orthogonally projects 3D space into single plane or triplane space for scene modelling. TensoRF (Chen et al. 2022) further extends this line of research for fast and efficient optimization by mapping the 3D scene space into set of oow-dieession ten- sors. With pre-defined spatial hash function, the recent Instant-NGP (Müller et al. (022)) maps 3D grid points into hash table indices and achieves a speedup of several orders of magnitude. Besides, PREF (Huang et al., (022) explores to transform 3D scene points a set of Fourier basis via Fourier transformation, which helps to reduce the aliasing caused by interpolation.
Some works also explore learning the gauge transformation for downstream tasks in neural fields. For example, NeuTex (Xiang et al., (02T)) and NeP (Ma et al., (022) propose to learn the mapping from 3D points to 2D texture spaces; GRAM (Deng et al.) 2022) learns to map the 3D space into set of implicit 2D surfaces; VQ-AD Takikawa et al. (2022) enables compressive scene representa- tion by mapping coordinates to discrete codebook, which
discrete case of gauge transformation. However, all above works suffer from gauge collapse without regularization or designed initializa- tion. Besides, popular line of research (Tretschk al. [222]] Tewari et al. [022] Park al.) [0200] Pumarola et al. 2020 Liu et al. 2022 Peng et al.) (021) learns the deformation from the actual 3D space into another (canonical) 3D space, where structural regularization is usually applied by predicting a deformation offset instead of the absolute coordinate.
3
METHODS
This section will start with the introduction of neural gauge fields framework, followed by the deriva- tion of learning regularization based information conservation. The nested gauge transformation which aims to preserve scene information inherently is then introduced and derived. Some discus- sion and insights for the application of gauge transformations drawn in the end.
3

---

Published as a conference paper at ICLR 2023
3.1 NEURAL GAUGE FIELDS FRAMEWORK
In normal usage, gauge defines measuring system, e.g., pressure gauge and temperature gauge. Under the context of neural fields, a measuring system (i.e., gauge) is a specification of parameters to index neural field, e.g.. 3D Cartesian coordinate system, triplane in EG3D) (Chan et al., 2022), hash table in Instant-NGP (Müller et al. (022). The transformation between different measuring systems dubbed as Gauge Transformation.
Based on this intuition, we introduce the general framework of Neural Gauge Fields which consists of a gauge transformation and a neural field. A neural gauge field aims transform the original space to another gauge system to index neural field. This additional transform could introduce certain bonus to the neural field, e.g., low memory cost, high rendering quality, or explicit texture, depending on the purpose of the model. The gauge transformation could be pre-defined function e.g., an orthogonal mapping in EG3D and spatial hash function in Instant-NGP. or directly learned by the neural network along with neural fields. Intuitively, compared with pre-defined function, learnable gauge transformation is more favorable as it can be optimized towards the final use case of the neural field and possibly yields better performance. For learning neural gauge fields, we disambiguate between two cases: continuous (e.g.. 2D plane and sphere surface) and discrete (e.g., hash table space) mappings.
Continuous Gauge Transformation. As the target gauge system is continuous, the gauge trans- formation can be handled by directly regressing the coordinate (i.e., gauge parameters) in the target gauge. For a point æ E RR3 in the 3D space and target gauge, a gauge network .M (a MLP structure or transformation matrix) can be employed predict the target coordinate x' Mxx)aa x' E RN,, where N is the coordinate dimension of the target gauge. Then, the predicted coordinate x' can be fed into an MLP or used to look up feature vectors model neural fields.
Discrete Gauge Transformation. Without loss of generality, we assume there is codebook with uN]. and the discrete gauge transformation aims transform a 3D
N discrete vectors [01, u., ..... point to a discrete codebook index (i.e., gauge parameter). In line with the setting of Instant-NGP, we divide the 3D space into a grid of size M M M. For each grid point x, we apply a gauge network M to predict its discrete distribution Pz E R over the N discrete vectors, followed by Top-k (k=1 by default) operation to select the Top-k probable indices i,, ... ig as below:
Ps M(x), $1, ...
i ,
T(p-
P.R
(1)
The produced indices usually used to lookup corresponding vectors from the codebook | and the features of other 3D points can be acquired by interpolating the nearby grid point vectors. As the Argmax in Top-k operation is not differentiable, reparameterization trick (Bengio et al.l|2211))
is applied to approximate gradient by replacing Argmax with Softmax in back-propagation. The pseudo code of the forward & backward propagation of discrete cases is given in Algorithm
We now have established the general learning paradigms for continuous and discrete gauge transformations. In our setting, the gauge trans- formations are optimized along with the neural field supervised by the per-pixel color loss be- tween the rendering and the ground-trutt multi- view images. However, found the learning of gauge transformations is easily stuck in lo- cal minimal. Specifically, the gauge transfor- mation tends to collapse to a small region in continuous cases as in Fig. [ or collapse small number of indices in discrete cases as in Fig. 3. highlighting the need for an additional regularization to enable an effective learning of continuous and discrete gauge transformations, which we will introduce in the following.
Algorithm 1 Pseudo code of forward & backward
propagation in learning discrete gauge transfor- mation.
Input: A 3D point æ. predicted distribution Pa
[pi, P2. pNN, codebook V.
Forward propagation:
1. index = Argmax(Pa))
2. index hard = One Hot(index)
feature lookup: v, Matmul(index.hart V)
Backward propagation:
1. index soft = Softmax(P2.
2. feature lookup: Matmulfindex.sott V)
Output: feature e...
AActually, the predicted indices can also he fed into an MI. for embedding, but this manner tends to be meaningless in practical applications and is seldomly adopted.
4

---

Published as a conference paper at ICLR 2023
3.2 REGULARIZATION FOR LEARNING GAUGE TRANSFORMATIONS
Some previous works (iiang et al. 2021 Ma et al. 2022) leverage a cycle consistency loss to regu- larize the mapping, preventing many-to-one mapping (gauge collapse) of 3D points. Alternatively, structural regularization is also employed in some works (Tretschk et al. 2021 Tewari et al. 2022) to preserve local structure by predicting a residual offset instead of the absolute coordinate.
However, the cycle loss only works for certain continuous cases and shows poor performance for the regularization of discrete cases, while the structural regularization cannot be applied to discrete cases at all. In this work, we strictly derive a general regularization mechanism for gauge transformation from the perspective of information conservation which superior to the heuristic regularizers used in prior work. Specifically, although the gauge of the scene has been changed, the intrinsic informa- tion of the scene is expected be preserved during gauge transformation, which can be achieved by maximizing the mutual information between the original 3D points and target gauge parameters.
With the gauge network M for gauge transformation, the sets of original 3D points and target gauge parameters can be termed as X and Y = M(X), respectively. Then, the mutual information between X and Y can be formulated as:
p(r,y) drdy // p((l=)((z) log I(X,)) /f p(z, y) log p(z)p(y)
p(ylæ) plv)
drdy.
where p(æ) and p(y) are the distributions of original 3D point x and target gauge parameter y, p(ylæ) is the conditional distribution of y produced by the gauge transformation of x. To avoid that the distribution of Y collapses into local region or small number of indices, we assume that obeys a prior uniform distribution q(y). and the KL divergence between p(y) and ((y) can be p(y) dy. Then, the final regularization loss can translate written as K (p(()))la))) = p(y) log d(vr into the following term (see detailed derivation in the Appendix):
min { + - p(g
+ [[[K([(pte])]]]](]]]]]]] (3)
where 7 and denote the weight of regularization and prior distribution discrepancy.
Regarding the first term K (T.(p(y)=) p()) (p(())(=)) in Eq. 3 we can replace the unbounded KI.
divergence with the upper-bounder IS divergence as: JS(p(y) ((())))()))()((
Lemma Jensen-Shannon Mutual Information Estimator (Nowozin et al. 2016):
JS(P,Q) max T
(Re-pte)--
- (awgg()) [sp([ (x)))
(4)
where sp(z) = log(1 + e)). For
we thus have:
max
E(r wwt-:pthppttnllsppp11101 ,))] [e (ooppt))poolsp(11 [()[],)))]]
(5)
where T denotes a network which minimizes the distance of positive pairs and maximizes the dis- tance of negative pairs. Please refer section 4.1 for detailed implementation of T. Actually, CycleReg is special implementation of InfoReg by discarding the prior distribution discrepancy.
Now the first term of Eq. (3) has been well solved, We then turn to the second term K|K [(p(v)=)]<<()) (i.e., the KL divergence between p(ylz) and (yy). This term can be solved according to the types of prior uniform distributions q(v) assumed in continuous and discrete cases.
Gauge Transformation. To prevent that the gauge transformation collapses to local region, we expect the prior distribution q(y) is a uniform distribution over the target gauge space. Without loss of generality. we assume the target gauge space is a 2D plane as shown in Fig. E (a). Then, we uniformly sampled points on the 2D plane as discrete approximation of the prior uniform distribution, which is denoted by 7(v) = [ui, U2, wn.. u, E IR2). With a set of 3D points transformed into 2D points, the conditional distribution p(y|z) can also be approximated by sampling/ l points from the predicted 2D points according to radiance contribution (see A.3), which denoted by p(y|2) [t]. U2, ..... 2n,, 0s E RR?. Considering 7(y) and p(y (z) are distributions over geometric space, KL divergence is inapposite to measure their discrepancy the distribution geometry is not concerned in KL divergence. We thus resort to a geometry- aware metric: Earth
Continuous
Mover's Distance (EMD) (Rubner al. 2000
5

---

Published as a conference paper at ICLR 2023
One-ilot
Initial State
Middle State
Final State
Use Hot
P((x)
KL Divergence
EbI Distance
One-Hlet
BLy
(n) EM Distance for Continuous Cases
kn1
NeT
(h) KT. for Discrete Cases
Figure 4: (a) Earth Mover's distance to regularize the discrepancy between p(y(æ) and 7(y) in continuous case (2D plane), where p(ylar) and 7(y) predicted 2D points within batch and uni- formly sampled points in the 2D plane, respectively. (b) KL divergence to regularize the distribution discrepaney in discrete cases (N basis vectors), where p(ylær) and gly) the average of predicted one-hot distributions and a discrete uniform distribution, respectively.
Specifically, cost matrix C' and transport matrix M with size of (h, h) be defined where each entry Cij in C gives the distance between points i; and uj. each entry Mi in represents the amount of probability moved between points V; and uj. Then, the EMD between 7(y) and p(ylz)) E Ej=1 Miy The distribution of predicted points M
can formulated as: LEMD = min
will be regularized to fit the uniform distribution by minimizing LEMD as shown in Fig. (a).
Discrete Gauge Transformation. For discrete gauge transformation,
expect the prior distribu- tion (v) to be a uniform distribution over the discrete vectors (i.e., the gauge basis), so that the target gauge parameters will not collapse
small number of vector indices. Thus, the prior distribution L) for N discrete vectors. As the one-hot distribution y; is predicted from 3D points as described in Algorithm 1. the conditional distribution p(y) ar) can be approximated by the average of all predicted one-hot distributions over a training batch, which is denoted by p(y(z) > P:/N where B is the number of samples within training batch. Then KL divergence between ((v) and p(yla) (r) can be duly computed as shown in Fig. (b).
((v) can be denoted by (v)) = +v= N :.... .VV
3.3 INFORMATION NOOOMATTION-IVARIANTT GAUGE TRANSFORMATION
Learnnng gauge transformation and applying regularization preserve information usually incur additional computation cost, which is not desired in many applications. We thus introduce an Information Invariant (InfoInv) gauge transformation which enables preserve scene information inherently and avoids the cumbersome process of learning regularization.
Specifically, for two positions m, n, we apply a gauge transformation g to yield their new gauge parameters fm) and f(n). As aiming to preserve the scene information, the relative position relationship between points should preserved accordingly, i.e.:
< gm))) g(n) >
Cos(g(m).g(0 [[((m)]] [l((n)]
h(mm - n).
(6)
where Cos and <> denotes cosine similarity and inner product, simplicity, we assume g(m) and g(n) are two-dimension vectors. Thus, < g(m), g()) > can be written as R[g(m)g" (n)) where R symbolises the real part of a complex value. With the exponential
function of m n. For
form of complex number, we have:
g(m) = (0(m)
g(n) (l9()))le
Then Eq. (6) can be translated as Cos(g(m), g(n))
[l s(a) -((()
Rig (m))) (w)
1m(m n²). One of the closed-form solution of ((m) can derived as (m)) m0, which further
cos
In implementa- tion, [l9(m)] can be approximated by grid absorbed by network, 0 is pre-defined learned parameter. The 2-dimension case of g(m) can be easily generalized to higher dimension by stacking multiple 2-dimension cases, whose form coincides with the heuristic position encoding adopted in
yields g(m) = (m)( (m00), Specifically, ei(m0) can be written as
sin(m))

---

Published as a conference paper at ICLR 2023
Continuous Gauge Transformation
Discrete Gauge Transformation
Regularizations
S-NeRF
S-NSVF
DTU
T&T
S-NeRF
S-NSVF
DTU
T&T
Baseline (w/o Rogularization)
0.021
0.017
0.014
0.019
0.102
0.094
0.063
0.078
+StruReg (Tretschk et 2021)
0.372
0.360
0.325
0.354
N/A
N/A
N/A
+CycleReg Xiang al. 2021 +InfoReg
0.547
0.562
0.568
0.522
0.105
0.086
0.076
0.088
0.793
0.752
0.772
0.806
0.992
1.000
0.996
1.000
Table 1: The PSNR performance with different regularizations for the learning of continuous gauge transformation (3D space + 2D plane) and discrete gauge transformation (3D space + 256 vectors).
S-NeRF
S-NVSF
DTU
T&T
Gauge Transformations
PSNRT
SSIM
PSNRT
SSIMI
PSNRT
SSIMF
PSNR1
SSIM1
Orthogonal Mapping
18.20
0.816
19.32
0.833
23.31
0.845
19.97
0.829
Learning Continuous Gauge
30.74
0.919
32.51
0.949
27.06
0.873
25.95
0.867
Spatial Hash Function
27.06
0.902
29.81
0.932
0.876
26.22
0.869
Learning Discrete Gauge
26.56
0.893
29.13
0.925
26.21
0.863
24.79
0.855
Learning Top-4 Gauge
29.39
0.917
31.39
0.942
28.77
0.887
27.14
0.892
Table Performance of different methods (pre-defined or learned) for gauge transformations. and * indicate continuous gauge transformation (3D space - 2D plane) and discrete gauge trans- formation (3D space + 256 vectors). Orthogonal mapping and spatial hash function are typical pre-defined functions for continuous and discrete gauge transformations, respectively.
cos(mo)
NeRF Miidenhall et al. 2021) The yielded parameters (lg(m)) and to index neural fields MLP networks feature grids.
can be leveraged sin(mo)
3.4 DISCUSSION
As general framework, learning gauge transformations with InfoReg Infolnv gauge transforma- tion can be applied in various computer vision or computer graphic tasks.
As concrete examples, learning gauge transformation with InfoReg can be applied to 3D space >> 2D plane mapping to learn explicit and editable textures (Xiang et al.) (2021), or 3D space - code- book mapping to yield compressed neural fields (Takikawa al.) 2022). However, with relatively high computation cost, learning gauge transformation with InfoReg cannot be applied to fast scene representation or reconstruction naively. We leave this efficient extension as our future work.
As kind of gauge transformation, Infolnv be applied to index neural fields directly with multi- scale setting (depending on how many different are selected). Notably, InfoInv allows to pre- serve scene information inherently without the cumbersome of regularization, which means it can be seamlessly combined with other gauge transformations (e.g.. triplane projection and spatial hash function) to achieve fast scene representation and promote rendering performance.
EXPERIMENTS
4.1
IMPLEMENTATION
The gauge network. M for learning gauge transformations can be a MLP network or a transformation matrix, depending on the specific downstream application. For the mapping from 3D space to plane to get (view-dependent) textures, the neural field is modeled by MLP-based network which takes predicted 2D coordinates, i.c., output the gauge network, and a certain view to predict color and density. For the mapping from 3D space to discrete codebooks, the neural field is modeled by looking up features from the codebook, followed by a small MLP of two layers to predict color and density. By default, the codebook has two layers and each layer contains 256 vectors with 128 dimensions. In line with Instant-NGP (Müller et al. 2022). the 3D space also divided into two-level 3D grids with size 16 16 x 16 and 32 32 32 for discrete gauge transformation.

---

Published as a conference paper at ICLR 2023
Zeared Mapping
Reoked
Onhegoeal Mapping
Recdond Teamed Readena Mappleg
Kankerci
Mapping
(b) Mapping
Figure 5: Qualitative results of continuous gauge transformation from 3D space to 2D plane on (a) 360 view scenes and (b) limited view scenes.
Space
27.
Infolleg
Cycle Loss
InfoReg Cycle Lows
26.5
InfoReg
2.
StruReg
Original
Original
611
26.
Original
25-
25.
IntoRgg Strukeg
25.
253
Pegrankoon Vange
Dicnppncy Werght
Decrepercy Voriiee
Figure 6: Ablation study of the regularization weights and prior distribution discrepancy weight.
4.2 EVALUATION METRICS
The introduced neural gauge fields focus on learning gauge transformation, the experiments are therefore performed to evaluate the performance of gauge transformations. By default, the experi- ments are conducted on continuous gauge transformation from 3D space to 2D plane and discrete gauge transformation from 3D space to 256 codebook vectors. Specifically, the effectiveness of regularization be evaluated by measuring the space occupancy in the target gauge space, which refers to the percentage of mapped area in continuous cases or the utilization ratio of codebook in discrete cases. Besides, gauge transformation can be applied to index radiance field, the gauge transformation can be evaluated by the novel view synthesis performance of a radiance field.
4.3 EVALUATION RESULTS
Regularization Performance. We conduct the comparison of different regularization methods in- cluding structural regularization (StruReg), Cycle regularization (CycleReg). and our InfoReg. As tabulated in Table [ the proposed InfoReg outperforms other regularization clearly in both contin- and discrete cases. Particularly, StruReg and CycleReg can bring certain improvements for continuous cases, while StruReg can not be applied to discrete cases and CycleReg only brings marginal gain for discrete cases.
Pre-defined vs Learned Mapping. As shown in Table 2 we compare the performance of the pre-defined functions and the learned mapping for gauge transformation. For continuous cases, the learned mapping with InfoReg outperforms the pre-defined orthogonal mapping clearly. For discrete cases, the standard learned discrete mapping (top-1 gauge) is slightly inferior to the pre- defined spatial hash function, while the top-4 variant of the learned discrete mapping surpasses the pre-defined function substantially.
Qualitative Results. To demonstrate the superiority of learned gauge transformation over pre- defined function (i.c., orthogonal mapping), we visualize the mapping results from 3D space to 2D plane in Fig. Fig. (a) illustrates the mapping the challenging 360 view scenes. The learned mapping can still well capture the scene texture, while the orthogonal mapping presents poor mapping performance and even fails in certain case. Fig. (b) compares the mapping limited view scenes. The orthogonal mapping is still prone to lose much texture information thanks to the inherent inflexibility.
8

---

Published as a conference paper at ICLR 2023
Rende fno Performance (256 Vectors)
Codebookk Uniizaton
Infolest West on Contirucss Case
InfoNest on screte Case
OrightalRRog
256
Original
204
(Intoltv)
onfotew)
24
14
2000
4043
633
500
1000
2339 6930 3009
4000
Figure 7: The left two figures: rendering performance and codebook utilization with different values in top-k gauges. The right two figures: the performance gain with the inclusion of Infolnv.
Model Training Details
Synthetic-NeRF
Synthetic-NST
Models
Steps,
Time(s)4
PSNRT
SSIM
PSNR
SSIMT
TensoRF
30k
708
32.32
0.958
35.32
0.976
TenfoRE+Infolny
30k
721
32.96
0.962
35,81
0.979
Instant-NGP
30k
458
32,43
0.960
35,43
0.977
IGstant-NGP+Infolnv
30k
496
32.91
0.962
36.01
0.981
Table 3:
The rendering performance gain by combining our information-invarian gauge trans- formation to SOTA methods on Synthetic-NeRF and SyntheticNSSVE. Notably, only very slight computation and memory cost introduced with the including of neural gauge fields.
Top-k Ablation. As defined in Eq. (T) different k can be selected for discrete gauge transformation. Figs. 7 (1) and (2) ablate the rendering performance and codebook utilization with different values. The rendering performance and codebook utilization both are improved consistently with the increase k, which means the gauge collapse is gradually alleviated. The performance gain brought by InfoReg is also narrowed with the increase of k. We conjecture that larger k leads to higher information capacity which alleviates the information loss in gauge transformation.
InfoReg Parameter Study. Fig. examines the effect of the regularization weight 7 and prior discrepancy weight in Eq. on the Lego scene. With the increase of regularization weight, the regularization performance of InfoReg is improved consistently, although it is saturated with a large weight. The regularization performance presents a positive correlation with a small prior discrepancy weight and negative correlation with large prior discrepancy weight.
InfoInv Performance. To demonstrate the power of intematiionnnvaaiian gauge transformation, we apply Infolnv to both grid-based and MLP-based neural fields for novel view synthesis. As shown in Figs. 7 (3) and (4), both convergence speed and final performance of triplane projection and spatial hash function are improved with the including of Infolnv. Besides, we also combine our Infolnv with SOTA methods including TensoRF and Instant-NGP. As shown in Table 3 the inclusion of Infolnv boost SOTA methods consistently with preferred model size. Besides, we combine InfoInv with widely adopted transformations including triplane projection and spatial hash function. Specifically, [[g(m)]] approximated by the indexed feature original TensoRF or Instant-NGP, cos (m0) and sin(m() are used to index auxiliary grid. As shown in Table 3 Infolnv benefits these SOTA methods consistently with ignorable (or very slight) increase of training time and model size.
5 CONCLUSION
This work introduces a general framework of neural gauge fields for unifying various gauge transfor- mations neural fields. With a target to optimize the gauge transformation along with neural fields, we establish the learning paradigm for continuous and discrete gauge transformations and derive regularization mechanism to solve the issue of gauge collapse from the perspective of information preservation. As optimizing gauge transformation with regularization usually incurs high computa- tion cost, a nested gauge transformation is introduced to perform instant transformation and preserve scene information with negligible computation cost. The extensive experiments and thorough analy- sis show that the derived gauge transformation methods outperform their vanilla counterparts, which demonstrates the potential of gauge transformations in neural fields.

---

Published as a conference paper at ICLR 2023
6 ETHICS STATEMENT
Our neural gauge fields focus on technically advancing neural scene representations and are not biased towards specific race, gender, or region. As general framework, neural gauge fields can be applied to various fields, e.g., virtual reality, artistic creation, and game design. On the other hand, the achievements in the image synthesis quality lurk the risk of being misused, e.g.. illegal impersonation of actors, or face synthesis for malicious purposes. Therefore, it is also of importance to explore safeguarding technologies that can reduce the potential for misuse. One potential way to use our approach well as other image synthesis approaches as a discriminator to identify the synthesized images automatically. This has been actively explored by the community, c.g. Mirsky & Lee (021). Alternatively, one should explore watermarking technologies identify synthesized images. Besides, access control should also be carefully considered when deploying the relevant techniques alleviate the risk of misuse. Finally, believe creating appropriate regulations and laws is essential to solving the issue of technology misuse while promoting its positive effects on technology development.
REPRODUCIBILITY STATEMENT
We attach the source code of neural gauge fields in the supplementary material. Additionally, provide detailed implementations in the manuscript that enable a full reproduction of the experimen- tal results. For example, we specify the choices of rendering resolution, model architecture, and the codebook setting for discrete gauge transformation. All datasets used in our experiments are pub- licly accessible. Finally, all pre-trained models will be made publicly available upon the publication of this work. The implementations will enable researchers to reproduce the results as well as extend this work to future research.
REFERENCES
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 308.3488..32. 2013.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5799-5809 2021.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awan 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123-16133 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. arXiv preprint arXiv:2203.095/1 2022.
Yu Deng, Jiaolong Yang. Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10673-10684 2022.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d aware generator for high-resolution image synthesis. In International Conference on Learning Repre- sentations, 2022.
Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 5875-5884, 2021.
Binbin Huang, Xinhao Yan, Anpei Chen, Shenghua Gao, and Jingyi Yu. Pref: Phasorial embedding fields for compact neural representations. arXiv preprint arXiv:2205. 13524, 2022.
10

---

Published as a conference paper at ICLR 2023
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Con- ference Machine Learning, pp. 2390-23999 PMLR, 2018.
David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15556-1456,, 2021.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-15663, 2020.
Lingji Liu, Mare Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Trans. Graph.(ACM SIGGRAPH Asia), 2021.
Li Ma, Xiaoyu Li, Jing Liao, Xuan Wang, Qi Zhang, Jue Wang, and Pedro Sander. Neural parame- terization for dynamic human head editing. arXiv preprint arXiv:2207.0022 000000, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 106, 2021.
Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: A survey, ACM Computing Surveys (CSUR), 20,
Keihachiro Moriyasu. An elementary primer for gauge theory. World Scientific, 1983.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. arXiv preprint arXiv:2201.05955 2022.
Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards real-time render- ing of compact neural radiance fields using depth oracle networks. In Computer Graphics Forum, 2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11453-11 11644, 2021.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: using variational divergence minimization. Advances neural information processing systems, 29, 2016.
Training generative neural samplers
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for nuet reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5599, 2021.
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: Resoluteoo DD-Consistent Image and Geometry Generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13503-13513, June 2022.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven Seitz, and Ricardo Martin-Brualla. Deformable neural radiance fields. arXiv preprint arXiv:2011.12948. 2020.
Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14314-14323, 2021.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision, pp. 523-540. Springer, 2020.
11

---

Published as a conference paper at ICLR 2023
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes, 2020.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger, Kilonerf: Speeding neural radiance fields with thousands of tiny mlps. In International Conference on Computer Vision (ICCV), 2021.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as metric for image retrieval. International journal of computer vision, 40(2):99-12) 12199911111-1111 2000.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154- 20166, 2020.
Cheng Sun, Min Sun, and Hwann-Tzee Tzong Chen. Direct voxel grid optimization: Super-fast con- vergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5459-5469, 2022.
Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Ja- cobson, and Sanja Fidler. Variable bitrate neural fields. ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-9, 2022.
Matthias Teschner, Bruno Heidelberger, Matthias Müller, Danat Pomerantes, and Markus H Gross. Optimized spatial hashing for collision detection of deformable objects. In Vmv, volume 3, pp- 47-54, 2003.
Ayush Tewari, Xingang Pan, Ohad Fried, Maneesh Agrawala, Christian Theobalt, et al. Disen- tangled3d: Learning 3d generative model with disentangled geometry and appearance from monocular images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1516-1525, 2022.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEF/CVF International Con- ference on Computer Vision, pp. 12959-12970, 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Robert Wald. General relativity. 2010.
Peng Wang. Lingjic Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 14:27171-2771 2021.
Fanbo Xiang. Zexiang Xu, Milos Hasan, Yannick Hold-Geoffin Kalyan Sunkavalli, and Hao Su. Neutex: Neural texture mapping for volumetric neural rendering. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
11 7119-7128, 2021.
Lior Yariy, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Thirty-Fifhh Conference on Neural Information Processing Systems, 2021.
Alex Yu, Sara Fridovich-Keil Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05131, 2021a.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5752-5761, 2011b.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 GAUGE TRANSFORMATION BACKGROUND
In normal usage, gauge means a measuring instrument or measuring system, c.g., pressure gauge and temperature gauge. The transformation between different gauges (e.g.. Fahrenheit and Celsius temperature) can be termed as Gauge Transformation. In physics, a gauge is a particular choice specification of vector and scalar potentials which will generate physical field. A transforma- tion from one physical field configuration to another is called a gauge transformation. The gauge transformations for classical general relativity Wald (20T0) arbitrary coordinate transformations.
Under the context of neural fields, the measuring systems (i.e., gauge) of the 3D scene can be ex- tended to more general definition, e.g., 3D Cartesian coordinate system, 2D coordinate system, hash table space, and frequency space. The transformation between different scene measuring system is dubbed as Gauge Transformation, and the neural fields with certain gauge transformation termed as Neural Gauge Fields as shown in Fig. g
(x.y))
Triplane (x.2 [x,]]
Contimaous Cases
((N.y.))
Spherical (8.9)
Feature or MLP
(x,
Original
Tuget Gouge
Codebook [index]
Discrute Cases
Fourier Space [-.,, cos (2 P)....
Neural Field
Gauge Transformation
Scene Representation
The framework neural gauge fields which consists a neural field and a gauge transformation. The gauge transformation maps original 3D coordinate to another measuring system (e.g., 2D plane and codebook) for scene representation via feature grid or a MLP network.
Figure 8:
A.2 DERIVATION OF INFOREG
According to Eq.
2, the final optimization loss can be formulated as:
{ - // p(ul=))(z)) log plylar) p(y)
dady - plv) log ply) {(v)
Greg min p(u(x)
dy
(8)
where y and denote the weight of regularization and prior distribution discrepancy. Then, we have:
/f p(ylx))((x)) log
p(ula dady + /f p(xlw))(w) log p(v)p(ulz) p(y)
Lreg = min p(y(x)
dxdy
((v)p(y(x))
// p(ul))p(()))
(7 + +) log r(ll:)
min
p(y) + e log a(v)
dxdy
p(ul#)
+7) ((X,Y) + [[KT(o(ol=))l((0)) p(ulz)
min
(9)
For (XX,)) we have:
((X,))= // p()))))(z)) log p(ul=)((z)
(Tbp(le))(()])))))ee))) p(y)p(z)
(10)
Thus, get:
- ( + BKKTTTT {(TTTTT(((())))))(])]
Greg min p(y(z)
(11)
which can be further solved described in the main paper.
13

---

Published as a conference paper at ICLR 2023
A.3 RADIANCE CONTRIBUTION
As 3D points near the scene surface are of more significance in mutual information, we leverage the radiance contribution weights per shading point to weight the mutual information term as in Xiang al. (202T). Specifically, with volume density a and radiance c of 3D scene, the RGB color 7 can be computed by aggregating the radiance values of shading points on the ray as below:
i-1
1= 1-ETO Tx-- oo =
Σ Oj 1=1
(12)
where denotes the index of shading point on the ray, d, and c; denote the density and ra- diance at shading point i, 8, denotes the distance between two consecutive points, Ti is the transmittance. Thus, the radiance contribution weight of a shading point can formulated as Wi = T1(1 - cxp((o,,)). The w; is applied to weight the positive pairs in Eq. (5).
A.4 DIFFERNNTAALE TOP-K
With predicted discrete distribution Ps- [P1, Pa. PN.,, the top-k operation aims to select the k most probable index in a differentiable way so that the gradient can be prokppppppaated. For prowadd-roppggation, we use the efficient top-k operation implemented in PyTorch to select the top- values and indices. For pack-propagation, we apply the reparameterization trick (Bengio et al.) 2013) to approximate the gradient by replacing the top-k operation with Softmax. The detailed pseudo code of the differentiable top-k operation is given in Algorithm
Algorithm 2 Pseudo code of differentiable top-k operation.
Input: k values top-k, 3D point z, predicted distribution (pL, [p,,P2, Pa. PN). codebook Forward propagation:
1. topk_init=(0, 0, ... R
index, value Top-k(P=)
topk_hard topk niitsccaterrinnex value)
value_sum = value.sum()
5. topk hard topk hard + value sum
feature lookup:
Matmul(topk hard,
Backward propagation:
1. index soft = Softmax(P.)
feature lookup: Matmul(index soft. V/)
Output: feature Uh.
A.5
IMPLEMENTATION
We implemented our neural gauge fields and re-implemented all compared methods without cus- tomized CUDA kernels. All models are optimized for 150k steps with a batch size of 1024 pixel rays.
Efficient Gauge Transformation. The proposed neural gauge fields be applied benefit isting models without increasing their computation cost and training speed significantly. Here, we provide several strategies to achieve it: First, instead of using MLPs learn the gauge transfor- mation, we can directly optimize tensors for transformation, e.g., 2D offset grids to indicate the coordinate transformation TensoRF, 1D vectors for grid points to indicate codebook index dis- tribution in Instant-NGP. Thus, the convergence speed of gauge transformation can be aligned with grid-based models like TensoRF and Instant-NGP. Second, gauge transformation learning can be performed in the middle training stage instead of the full training process, then the learned trans- formation is frozen for ensuing model training. Thus, the computation cost introduced by learning gauge transformation (and regularization) can be ignorable compared with the total training time. Third, we empirically find that the prior distribution regularization term in InfoReg has been enough avoid gauge collapse in most cases, which is much more efficient than the complete InfoReg.
14

---

Published as a conference paper at ICLR 2023
A.6 LIMITATIONS
This work focuses the fundamental problem of learning gauge transformation along with neural fields and is orthogonal the SOTA methods for representation, e-g-. TensoRF and Instant- NGP. Although we demonstrate that the learned gauge transformation can outperform its pre-defined counterpart (c.g., orthogonal mapping and spatial hash function). there are still some challenges to apply this method to the SOTA models.
For instance, the original Instant-NGP employs a 16-layer codebook with size ranging from 2 to 2 When replacing the spatial hash function in Instant-NGP with the learne counterpart, the gauge network size will be extremely huge with an output layer dimension ranging from 2 to 2 One possible solution is firstly grouping the codebook into several sub-blocks with smaller size, followed by smaller gauge network to learn the mapping to each block.
A.7 DISCUSSION & FUTURE WORK
As general framework, the proposed neural gauge fields can be extended in several directions. Firstly, this work focuses on neural fields for novel view synthesis, which can be easily applied to downstream tasks, e.g., surface reconstruction with SDE dynamic scenes modelling. Secondly, our InfoReg solves the gauge collapse in discrete gauge transformation, and can be potentially applied to the field of discrete representation learning (e.g., VQ-VAE (Van Den Oord al.) 2017)) which also suffers from codebook collapse. Thirdly, this work mainly explores the pure pre-defined or learned
mapping, and how to combine pre-defined mapping and learned mapping to yield better performance also of great potential. Please refer to Appendix A.6 for more discussion of limitations.
A.8 MORE QUALITATIVE RESULTS
Fig. 9 compares the performance of InfoReg with Euclidean distance and cosine distance, the Eu- clidean distance is prone preserve better texture structure compared with Cosine distance although they both cover the 2D space and achieve similar rendering performance.
Fig. 10 shows additional samples of leaning continuous gauge transformation with different regu- larizations.
Except for novel view synthesis. we also apply our neural gauge fields the downstream task of texture editing as shown in Fig. [T] Specifically, we learn the mapping from 3D space (points on object surface) sphere surface which visualized by cubemap for texture editing.
15

---

Published as a conference paper at ICLR 2023
3D Scene Image
Euclidean Distance
Cosine Distance
The results (3D space to 2D plane) of different distance metrics (Fuclidean distance or Cosine distance) used in network T in the proposed InfoReg. Euclidean distance leads to better preservation local texture structure compared with Cosine distance.
Figure 9:
3D Seene Image
Without Regularization
With Cycle Regularization
With InfoReg
Figure 10:
The results (3D space to 2D plane) of different learning regularizations. Our InfoReg
leads to the best mapping results by utilizing the full plane.
3D Scene Image
Leamndd Texture
Edited Texture-A
Edited Texture-B
Texture-A View
Texture-A View2
Texture-A View3
Texture-A View4
Texture-B View
Texture-B View2
Texture-B View3
Texture-B View4
Figure 11: The application of neural gauge fields for texture editing. A gauge transformation learned from 3D space to sphere surface (visualized by cubemap). By editing the texture of cubemap, the model is able to render multi-view images of the corresponding texture.
16


Reviews:
[{"review_id": "tQA50qIxpr", "reviewer": "Reviewer_HA8S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " This paper studies the problem of learning the gauge transformation in an end-to-end manner within learning the neural scene representation setting.  In this context, gauge transformations in  continuous and discrete cases are learned. The developed learning paradigm, which is generic,  maps a 3D point to a continuous coordinate or a discrete index in the target gauge, respectively.\nThis work also introduces  the concept of Information Regularization (InfoReg) from the principle of information conservation during gauge transformation. This regularization is performed by maximizing the mutual information between gauge parameters.  The paper also claims that the proposed top-k gauge is useful in achieving a tradeoff between learning collapse and computation cost.\n", "summary_of_the_review": "\nI generally like the addressed problem and the approach made in the paper. However, it is currently difficult to know the benefit of the proposed method. This is mainly due to the limited experimental results. I suggest authors report more exhaustive experiments highlighting the contribution empirically. \n", "strength_and_weaknesses": "++ The paper is generally well written and easy to follow. The presented method is intuitive and meaningful. Graphical illustrations of Fig. 1, 2 , and 4 are particularly helpful.\n\n++ Sections 3.2 and 3.3 are well motivated and offer new theoretical insights.\n\nThe empirical discovery and analysis on top-k gauge, reported in Section 3.3 and Fig. 7 are particularly interesting.  \n\n++ The supplementary material with additional pseudo code and discussion regarding the method’s limitations is helpful.\n\n\n-- Comparison with SOTA method on Table 3, although highlights the benefit in terms of model size, is limited to only two scenes. It is difficult to conclude the benefit of the proposed method over SOTA without more exhaustive experiments.\n\n--  The claim of  “trade-off between collapse and cost” on the basis of Figure 7 (in Section 4.2.2) is somewhat misleading.  The reported PSNR for the “original” method may be poor but cannot be said to collapse. As such, the reported gap in PSNR (which is not very significant) may not generalize well across scenes.\n\n--  Qualitative results reported in Fig 8 and 9 (in the supplementary materials) are difficult to understand.\n\n--  The limitation on “output layer dimension ranging from 2^14 to 2^24” is indeed of a serious concern.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nlJtUkR4ir", "reviewer": "Reviewer_mQvE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework for neural gauge fields in the context of radiance fields and experimentally shows that the proposed regularisation is superior to alternative ones. This work also introduced the idea of a top-k gauge that gradually alleviates the need for regularisation.", "summary_of_the_review": "Despite they could be a significant theoretical contribution in trying to present a new framework for gauge fields, I could not really judge the impact and novelty in the way this paper is written. ", "strength_and_weaknesses": "Strengths\n-----------\n\nOverall I believe the idea of exploring to learn a gauge transformation in an end-to-end manner is an important research direction, and this work makes contributions in this regard.\n\nResults show better performance than state-of-the-art methods.\n\n\nWeaknesses\n--------------\nOverall, this paper was hard to evaluate because I am not an expert in the neural radiance field and because I believe the presentation requires improvement. The elaboration on existing methodologies and new ideas is not in a way that presents the reader with an abstraction of the theory behind recent advances to then clearly and concisely show their contributions.\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Sz1IapXgs8", "reviewer": "Reviewer_2BXg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author proposes a general framework for the neural gauge field, including continuous and discrete cases. To solve the collapse problem, the author proposes a regularization term - InfoReg. In practice, the regularization term regulates the target distribution to a uniform distribution. The author also proposes a new top-k gauge mechanism that achieves a trade-off between model collapse and computational cost. Experiments show the method outperforms other regularization methods and gauge transformation methods. The method also outperforms SOTA in rendering tasks, but with higher computation costs. ", "summary_of_the_review": "This paper proposes a general method for gauging transformation, which is intellectually interesting and could be potentially applied to other frameworks and applications. My major concern is the higher training and inference time for high-quality results. ", "strength_and_weaknesses": "Strength:\n+ The proposed method proves to be effective and has the potential to be adopted by future \"gauge transformation\" frameworks. \n\n+ The paper is well-written and easy to follow for me. \n\nWeakness:\n+ The performance of the rendering tasks at inference time sacrifices computational cost. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G8iKGkbf2r", "reviewer": "Reviewer_Bfkm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the feature index or coordinate mapping function in radiance field reconstruction as a gauge transformation problem. The core idea of the manuscript is to optimize the transformation by maximizing the mutual information and encouraging uniform coordinate distribution with an EM distance regulation term. Extensive experiments on syntheses and real object datasets demonstrate that the proposed method achieves better performance in terms of rendering quality and compactness compared to the recent SOTA fast radiance field reconstruction approaches (instant-NGP and TensoRF ) and space wrapping baed radiance field reconstruction methods (Non-Rigid Neural Radiance Fields and NeuTex).", "summary_of_the_review": "The proposed regulation terms are novel and interesting to me, and happy to see the authors can clarify the questions listed in the \"to be improved\" section in the rebuttal period.", "strength_and_weaknesses": "What's good:\n1) The idea of transforming coordinate learning into a distribution regulation problem is interesting and elegant.\n2) The usage of KL divergence and EM distance loss for coordinate space mapping seems novel and interesting.\n3) The paper provides a detailed mathematical derivation of the relationship between mutual information maximization and the proposed KL divergence loss, but I could not read over the derivation due to this emergency review.\n4) The proposed method shows better quantitative rendering scores than the previous approach.\n\nTo be improved:\n1) The overall pipeline is unclear to me, a pipeline figure could really help to better understand the proposed method.\n2) The paper lacks a key highlight that motivates me to use it in practice, it's good to perform better than the other two space-wrapping-based approaches, but the scores still have a big gap with SOTA and suffer from high computation cost. Would be better if the authors can figure out some unique properties in the proposed method. How does it perform for radiance field editing and rendering speed? Can it combine with the traditional rasterization rendering framework for fast rendering since the features are distributed in 2D space?\n3) Table 3 is tricky to me, why not use the default configuration in the TensoRF and Instant-NGP? Seems the total parameters in Instant-NGP are significantly smaller than TensoRF under your configuration but the model size in instant-NGP is larger than TensoRF: 2^14 * 8 * 2 (even assuming 2^14 entries for each level) vs. 64 * 64 * 27 * 3, please correct me if any misunderstanding to the counting. And could be better to list the training time to have a better side-by-side comparison.\n4) The reference to the convolutional occupancy field is missing, which is early work that also uses space wrapping.\n5) Lacking the reference, discussion and comparison with \"Variable Bitrate Neural Fields\", which also learn a coordinate wrapping function.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G-ybuQXDz0j", "reviewer": "Reviewer_hK9t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a way to learn gauge transformer for neural fields applications (especially modeling neural radiance field for novel view synthesis). The main contribution of the work is to propose a unified framework to reason about different NeRF paradigms in the language of gauge transformation and proposes a regularization to enable learning such transformation without suffering from collapsing in local minimum. Empirical results suggests that the proposed regularization indeed improve performance.", "summary_of_the_review": "My main concern about the paper is about the evaluation. The paper mainly evaluate the performance of Neural Gauge Field using novel view synthesis, without reporting optimization time, memory, or other applications (which I believed the true use of Neural Gauge Field can lay). The secondary concern is regarding it's not clear why we need the unified language in terms of gauge transform to describe different formulation such as instang-ngp, triplanes, and/or tensorRF. \n\n\n-------\nUpdate: I decided to raise my score since the authors address a number of concerns I raised.", "strength_and_weaknesses": "Strength:\n1. The paper provides a novel perspective that unifies the coordinate transform in instant-NGP, tensorRF, and triplanes under the framework of gauge transform. \n2. The proposed regularization shows improvement over existing regularization methods provided by prior works.\n3. The paper shows that with the novel Top-4 Gauge, the method is able to perform comparably with SOTA.\n\nWeakness:\n1. Evaluation. It's not very clear to me that the using downstream novel view synthesis is the best way to evaluate Neural Gauge Fields. I believed that it's essential to show that the proposed method doesn't lag behind SOTA performance, but it's also important to show case what this new formulation should allow. I will encourage the authors to include other applications, potentially in the field of learning texture, materials, BRDF etc.\n2. Writing. In the introduction and method section, it's hard for me to follow the motivation why is it beneficial to think of instant-NGP, tensorRT in the framework of gauge transform. This is not clear to me what's the theoretical benefit (cleaner? easier to reason about different mappings?) or the empricial advantages. I would also encourage the authors to include a background section about gauge transform as it can benefit the readers without enough backrgound.\n3. The authors compared to instant-NGP / tensorRT, both of these two methods are very memory efficient and fast to optimize. I wonder if Neural Gauge Field subsumes these formulation, does the same benefit carries through? It would be nice to see time profiling and memory profiling if possible.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster