Generalized Fourier Features for Coordinate-Based Learning of Functions on Manifolds
Authors: Carlos Esteves
Tianjian Lu
Mohammed Suhail
Yi-fan Chen‎
Ameesh Makadia
Venue: ICLR 2022 Submitted (2022)
Keywords: positional encoding
fourier features
coordinate-based mlp

Abstract: Recently, positional encoding of input coordinates has been found crucial to enable learning of high-frequency functions with multilayer perceptrons taking low-dimensional coordinate values.  In this setting, sinusoids are typically used as a basis for the encoding, which is commonly referred to as "Fourier Features".  However, using sinusoids as a basis assumes that the input coordinates lie on Euclidean space.  In this work, we generalize positional encoding with Fourier features to non-Euclidean manifolds.  We find appropriate bases for positional encoding on manifolds through generalizations of Fourier series.  By ensuring the encodings lie on a hypersphere and that the appropriate shifts on the manifold preserve inner-products between encodings, our model approximates convolutions on the manifold, according to the neural tangent kernel (NTK) assumptions. We demonstrate our method on various tasks on different manifolds: 1) learning panoramas on the sphere, 2) learning probability distributions on the rotation manifold, 3) learning neural radiance fields on the product of cube and sphere, and 4) learning light fields represented as the product of spheres.

Under review as a conference paper at ICLR 2022
GENERALIZED FOURIER COORDINATE-BASED LEARNING OF FUNCTIONS ON MANIFOLDS
FEATURES
FOR
Anonymous authors Paper under doublc-blind review
ABSTRACT
Recently, positional encoding of input coordinates has been found crucial to en- able learning of high-frequency functions with multilayer perceptrons taking low- dimensional coordinate values. In this setting, sinusoids are typically used as ba- sis for the encoding, which is commonly referred to as "Fourier Features". How- ever, using sinusoids basis assumes that the input coordinates lie on Euclidean space, In this work, we generalize positional encoding with Fourier features to non -on-Euclidean manifolds. We find appropriate bases for positional encoding manifolds through generalizations of Fourier series. By ensuring the encodings lie a hypersphere and that the appropriate shifts on the manifold preserve inner- products between encodings, our model approximates convolutions on the mani- fold, according the neural tangent kernel (NTK) assumptions. We demonstrate our method on various tasks on different manifolds: 1) learning panoramas on the sphere, learning probability distributions on the rotation manifold, 3) learning neural radiance fields on the product of cube and sphere, and 4) learning light fields represented as the product of spheres.
INTRODUCTION
Recent breakthroughs on learning representations 3D shapes (Mescheder et al. 2019 Park et al.,
2019 Sitzmann et al. 2019) scenes (Mildenhall et al. 2020) employ the so-called "coordinate- based networks, which take low-dimensional coordinates as inputs and approximate a continuous function. These sometimes called "implicit models". when the approximated function implicitly represents the desired output; typical example is using a signed distance function to represent a 3D shape (Park al. 2019).
Perhaps the most important recent advancement in this line of research NeRF (Mildenhall et al,, 2020).
One of the reasons for NeRF's impressive performance the positional encoding input coordinates using sinusoidals of various frequencies, a technique that has been widely adopted (Liu
2020 Schwarz et al. 2020) Yariv et 2020) and studied (Tancik et al., 2020 Zheng al. The sinusoidals typically used for positional encoding are elements of orthonormal bases for functions on Euclidean spaces. Our key observation is that, to generalize this idea to con-Euclidean manifolds, we should use orthonormal basis functions on the manifold.
al.
2021
Figure illustrates the evaluation of the Euclidean basis functions on spherical coordinates, which break orthogonality, results in uneven frequency distribution and singularities near the poles. The spherical harmonics are orthonormal basis functions for the sphere and do not exhibit these undesired properties.
NeRF (Mildenhall et al., 2020) and subsequent works avoid the problem just described by parametrizing the view direction as unit vector in H instead of two angles. We gue that this overparametrization unnecessarily increases the dimensionality of the problem, since the number of basis functions for fixed bandwidth b grows exponentially with the number of dimensions d. Another relevant example is IPDF (Murphy et al.] 2021). which takes inputs on the 3-dimensional manifold SO(3) in the form of flattened rotation ma- trix (a vector in R²), and applies positional encoding by sparsely sampling the space of (b²) basis elements. We show that our approach improve results for both NeRF and IPDF.

---

Under review as a conference paper at ICLR 2022
Our basic idea is find an orthonormal ba- sis for the space of functions on the manifold of interest, and introduce principled methods for choosing a subset of the (possibly infinite) basis elements which the input coordinates are evaluated. We describe mathematical tech- niques to obtain bases for large classes man- ifolds and show experiments on a variety of them.
Our main contributions are:
We introduce principled way to apply po- sitional encoding for coordinate-based learning of freeeeryyy functions on manifolds.
Figure I: On Euclidean spaces, sinusoids form an orthonormal basis (left) and are used for po- sitional encoding in coordinate-basee MLPs. The same functions, when applied to other manifold, distorted and not orthogonal. We propose to use orthonormal basis functions for the underlying manifold for positional encoding. For the sphere, we use the spherical harmonics basis (right).
We prove that our approach is shift-invariant under the light of the neural tangent kernel (NTK) theory (Jacot et al. 2018: Tancik et al.) 2020) and for the approprite "shift" on the manifold, which implies a manifold convolu- tional behavior.
Our experiments show the advantages of the proposed methodology different applications and manifolds: 1) learning panoramas the sphere, 2) learning probability distributions on the rotation manifold, 3) learning neural radiance fields on the product of cube and sphere, and 4) learning light fields represented the product of spheres.
RELATED WORK
The idea of positional encoding using a Fourier basis appeared as early as Rahimi & Recht (2008) They introduced the so-called random Fourier features accelerate training of kernel methods. The idea is to approximate shift-invariant kernel with random directions sampled from its Fourier transform, on which the inputs are evaluated. More recently, positional encoding has become popu- lar for sequence modeling in natural language processing and often appears in attention layers and transformers as a way to encode the order the input tokens. Gehhing et al. (2017) use a learned embedding while Vaswani et al. (2017) use the Fourier basis for encoding input token positions. Alternative approaches were introduced by Xu et al. (2019): Wang et al. (202)).
Rahaman al. (2019) demonstrated that neural networks tend to learn low frequencies more easily than high frequencies. a property called "spectral bias". It also shows, for a simple task of ap- proximating 1D function, that projecting the input coordinate into basis of sinusoids facilitates learning high frequencies. NeRF (Mildenhall et al., (020)) reproduced this finding in the challenging task of photorealistic novel view synthesis from a collection of images. They showed that positional encoding is crucial to achieving photorealism, and significantly outperformed the previous state of the art, which included other coordinate-based MLPs (Sitzmann al. 2019). Concurrently, Zhong et al. 2020) demonstrated the usefulness of Fourier encoding for reconstruction of 3D protein com- plexes.
Tancik et al. (2020) conducted an in-depth study of positional encoding for coordinate- MLPs. They show that, under the Neural Tangent Kernel (NTK) theory (Jacot al. 2018), the positional encoding using sinusoidal basis results in stationary kernel, which in turn can be interpreted as a convolutional (shift-invariant) reconstruction filter, desirable for signals on Euclidean spaces. The theory also explains the "spectral bias" (Rahaman et al.] 2019) of these models via the rapidly decaying NTK eigenvalues for MLPs. In practice, Tancik et al. (2020) propose randomly sampling the frequencies of the Fourier basis elements for the encoding, and show that it outperforms previ- ous coordinate- -aasa MLPs with no positional encoding (Mescheder et al. 2019), or axis-aligned positional encoding (Mildenhall et al. 2020).
2

---

Under review as a conference paper at ICLR 2022
While groundbreaking results and insightful theoretical developments have been demonstrated, most of the attention so far has been focused on coordinate-bais based MLPs for functions on Euclidean spaces. In this paper, we focus on the non-Euc -uclidudd case.
One alternative to Fourier features was introduced by SSizmmann et al. (2220) who use the sine function the MLP nonlinearity. Major differences are that the sinusoidal is applied every layer, and the frequencies are defined by the network MLP weights. Zheng et al. (2021) studied the trade-off between memorization and generalization in positional encoding, showing the former is related to the rank of the embedding matrix and the latter is related to the distance preservation of the embedding. They also propose another alternative Fourier features that consists of sampling a Gaussian at fixed offsets from the input.
3 BACKGROUND
NeRF (Mildenhall et al., 2020) recently demonstrated the importance of positional encoding for coordinate- learning of high frequency functions. For a coordinate I = [21, 22, x3] E IR the following map applied before the multilayer perceptron (MLP),
24 + {sin(2 TT;)) cos(2" Tx,), (m(22TT.)) cos(2" ... sin(2" moy), cos(22 (1)
Recall that {sin(m x)} U {cos(n })} with m, n. E 3 form an orthonormal basis for functions defined on compact subset of g So Eq. ① corresponds to evaluating basis functions of different frequencies at input coordinates; we will refer to this method as "Euclidean encoding". In the particular case of Eq. ( only axis-aligned. powers-of-two frequencies are used, so the basis for bandwidth 2 is only sparsely sampled. [Tancik et al. (2020) found it better to randomly sample the frequency space, which results in basis elements that are not axis aligned, but still a sparse sample of the complete basis.
In NeRF MMileenhall et al., 2020), a subset of the input encodes the view direction. A direction can be associated with point the surface of the unit sphere, non-Euclidean manifold. In order to apply the Euclidean encoding to such input, (Mildenhall et al. 2020 represent the direction as a 3D unit vector instead, which increases the input dimensionality from 2D to 3D, and causes the positional encoding to sample from a set of O(b²) basis functions, instead of the minimal ( for bandwidth b. In IPDF (Murphy et al. 2021), the input coordinates are flattened 3 x 3 rotation matrices, which correspond to points on the 3D rotation manifold SO(3). The Euclidean encoding then corresponds sampling basis functions R This results in sampling from space of O(b))
basis functions instead of ((b3).
Our key observation is that positional encoding of input coordinates should reflect the geometry of the underlying manifold. We propose to use the appropriate orthonormal basis functions on the manifold to implement this idea. While for many applications the Euclidean manifold is the correct one, for others it is not, as exemplified above. In the following sections we will show how the appropriate basis can be found for different types of manifolds, how to actually choose a subset of the basis to use in the encoding, and demonstrate experimentally the benefits of using them.
4 METHOD
4.1 OVERVIEW
We consider neural network that takes as inputs points on an n-dimensional manifold M. We assume there exists an orthonormal basis B (b:: M + 13 for the space of scalar functions on the manifold (MM) Our goal is to apply the map y: M + R to positionally encode x E M before feeding it to the network, where l is the encoding dimension and
(x) (b)(x) B
B' B).
The first question that arises is how to find the basis for given manifold. There is no general solution, but specific solutions exist for some large classes of manifolds. For compact groups (not necessarily abelian), the Peter-Wey, theorem establishes generalization of the Fourier series that
3

---

Under review as a conference paper at ICLR 2022
results in a countable set of orthonormal basis functions. It gives rise to the Wigner-D matrix ele- ments that are used as a basis for SO(3) in Section 4.3 For locally compact abelian groups, the Pontryagin duality can be used, which is generalization of the Fourier transform. For Riemannian manifolds, an orthonormal basis can he obtained as eigenfunctions of the Laplace-Beltran operator.
For the sphere 5², these are the spherical harmonics used in Section 4.2
4.2
The neural tangent kernel (NTK) theory (Jacot et al. 2018: Arora et al. 2019 Lee et (211)) shows that, under certain conditions, an MLP trained for regression f: R RR converges the solution of a kernel regression [=) R Σ, Wiyjk([r æi), where k is the NTK, w; the kernel regression weights, and the sum is over the whole dataset. Moreover, when the inputs z have constant norm, the kernel depends only on inner products: k(z,8) y) = k(x' y)) In other words, it is rotation- invariant. Tancii al. (2020) leverages this theory to explain the success of Fuclidean encoding it transforms the votation-invarian kernel in a shift-invariant one, which results in approximating convolution operation over the whole training set for inputs in a flat space.
In this paper, the inputs do not lie on Euclidean space, the shift-invariance in the sense of trans- lations in Euclidean space is not appropriate. For the manifolds considered. we will seek positional encodings that have constant norm (lie on 5*) and such that the natural shift on the manifold also preserves the inner products between positionally encoded inputs. Then, the NTK should approx- imate convolution the manifold. We'll show in the following sections that these properties constrain the selection of basis elements.
4.2 THE SPHERE $
The spherical harmonics are eigenfunctions of the Laplace- Beltrami operator on the sphere $2 and thus form an orthonormal basis for the space of square-integrable functions L²(()). The spherical harmonic of degree é and order m is given by
Laplace-Beltrami
Y (0, Φ) = a' Pi (cos 0) ime
(3)
where 8, á are the colatitude and longitude angles, Pf the associate Legendre polynomial of degree and order m, with -f m < l. The degree l indicates the angular frequency, and a is normalization constant.
Under the NTK umbrella, an MIP taking points on the sphere encoded as 3D unit vectors will converge to kernel regression with the kernel depending only on inner products. This convenient because the natural shift on the sphere is a rotation, which preserves inner products. We want to select a subset of the spherical harmonics for positional encoding that maintains these properties: 1) preserves inner-products, and 2) has constant norm.
Proposition 1. For given l, let Y [Y Y spherical harmonics of degree f. Let Ra represent the point x E S2 rolated by R E SO(3). Then, for any x. y E 52 and R E (O(3).
.Y²) be vector concatenating all the 2
Proof. See Appendix A.I
Since [ ( + 1)/(4T), the vector (x)) has the same norm for any x, and the NTK depends only on inner products.
These results
easily extended to concatenation of harmonics of multiple degrees, which sug- gests a strategy for selecting the basis elements for positional encoding: choose any set of degrees L N and use all harmonics for the chosen degrees: B' (Y & E L, -6, S m f.).
For simplicity, we typically choose a degree mmax such that L = | 1 Emaz). Note that the maximum degree f is the only hyperparameter that needs to be selected for both strategies. In contrast, the Gaussian encoding from Tancik et al. (2020) requires both the encoding size and scale hyperparameters.
4

---

Under review as a conference paper at ICLR 2022
4.3 THE ROTATION MANIFOLD SO(3)
The rotation manifold (3) is non-abelian and compact. The latter property guarantees a Fourier series for the manifold according the Pete theorem, given by the Wigner-D matrices. The Wigner-D matrices are unitary irreducible representations of SO(3) and its matrix elements form an orthonormal basis for L2(OO(()) (3)). The elements for the (26 + 1) (2f + 1) matrix D of degree l are
D (a, 3, = e ima (3)e",
(4)
where a, 3, 7 are ZYZ Euler angles representing the rotation, - S m, n S and di (B) real and proportional to the Jacobi polynomial Pk Pl (cos (3), with k, a, being functions of e, m, n.
Since D' is unitary for all L, for any of degrees L N. an encoding concatenating all matrix elements as follows has constant norm,
B' = (D m. E -4 S m. S.).
(5)
The natural shift (3) is also rotation (the group acting itself). Therefore, similarly to the discussion in Section 4.2 it is desirable that the inner products between encodings are invariant with respect rotations of the input.
Proposition 2. For a given f, let vec(D*) be the vectorization of the Wigner-D matrix Dt, Then, for any R. Rí, Rg SO(3),
(vec(D'((R))) vec(D (D (Ra)))
Proof. See Appendix A.2
This also holds for a concatenation of multiple vec(D4), the map in Eq. 5 besides constant norm, also has rotation-invariant inner products, as desired. practice, instead an arbitrary set of degrees, we choose an max and use all degrees 1 < l fmax-
4.4 PRODUCT OF SPHERES Sa x S2
The coordinates for a point in 5 x S2 can be obtained by the concatenation of coordinates on each sphere: (01, Φ1, f2 Ø2). A basis for the product manifold can be obtained by product of the basis functions for the sphere. Thus, we have the following orthonormal basis functions for L LS2 x S²),
Y mi.mng (01. Φ1, 02 Φ2) = y: (01, Φ1 Y (02, Φ2),
(6)
with Ye as defined Eq. We will use the following form interchangeably to reduce notation, yh, mi my (x)) yr (x1) yre (122), where 1T, T2 E S².
For a given set of pairs of degrees C N x N. we consider the following basis functions, consisting of selecting all possible (mi, m2) given (L,, Ej).
B'
(Yfri ma.m,
(Gr, E L, -4 S m4 S l,, and - S m, S6}
(7)
Proposition 3. For a given pair (f1, l2), let yh = (Y&l, } be a vector containing all the (261 + 1)(20g 1) products of spherical harmonics of degrees l and lz. For any x1, D2 S?, the norm of f is constant: [[ (21, (
((2h, + 12 1)/(4x).
Proof. See Appendix A.3
Since the encodings have constant norm, we are interested in whether the inner product between encodings is preserved under input shift, which would lead to the previously discussed desirable properties that follow from the NTK. For the case of S x S?, an appropriate shift consists in applying the same rotation both spheres. For example, in experiment in Section 5.4, the inputs on 52 x 5 represent a pair of points on a sphere, so makes sense that the appropriate shift should rotate both points.
5

---

Under review as a conference paper at ICLR 2022
Figure 2: Regressed spherical panoramas. From left to right: no positional encoding, axis-aligned, Gaussian, spherical harmonics (ours) and the ground truth.
Proposition Consider the vector of orthonormal functions V as defined Proposition The inner product between encodings is invariant to rotations of the inputs:
(2112(211,10) 12), y (y), v2)) Ey (Ræ1, Rx2), Y (Ryn, Ryz))
for any pair (z1, F2) E 52 52 and (y1, y2) S 5?, and rotation (O(3).
Proof. See Appendix A.4
Therefore, we can construct basis for positional encoding using Eq. 77. In practice, we set maximum degree frax and use all pairs under it: L {(e, Eÿ)
E, lmax). }.
5 EXPERIMENTS
5.1 SPHERICAL PANORAMAS
Following Tancik et al. (2020); Sitzmann et al. (2020), we evaluate the learning of high- frequeeeqess on simple task of training MLP regress image from its input coordinates. Since we are interested muc-Euelidean manifolds, we evaluate the methods on 10 spherical panoramas from SUN360 (Xiao et al. (2012),, randomly sampling 1/4 of the pixels the training set, with probabilities proportional to pixel areas in the input spherical grid.
We follow the image resolution, architecture design, and training schedule of Tancik et al. 2020) and replace their positional encoding with ours, where we use all (famx + (f + 2)/2 spherical harmonics up f 16. Table [ shows the test peak signal-to-noise ratios (PSNRs). Figure depicts the results. Note the large distortions on the poles with Gaussian encoding; these are the areas with large distortion when assuming an Euclidean manifold.
Table 1: Test PSNRs (dB) for spherical panorama regression. We show averages and standard deviations (in parenthesis, referring to last digit) over three runs. for no encoding, axis-aligned Mildenhall et al. 2020), Gaussian (Tancik et al. 2020). and our spherical harmonics encoding.
No encoding
Axis-aligned
Gaussian
Ours
Test PSNR (dB)
17.91(3)
20.95(1)
26.48(7)
27.08 (3)
5.2 PROBABILITY DISTRIBUTIONS ON SO(3)
Recently, Murphy et al, (202T) introduced IPDF, a model that learns to predict probability distri- butions SO(3) to represent the object pose from image. The main part of the model is

---

Under review as a conference paper at ICLR 2022
Table Estimating the pose distribution of symmetric objects SYMSOL DBN refers to "Deep Bingham Networks" DDeng et al. (2020), IPDF shows the original results from Murphy et al. 2021))
and IPDF* is our modified version. Our model significantly outperforms the baseline in both metrics.
log-iikel (1)
Spread [deg] ())
avg.
cyl
tet
cube
ico
avg.
cyl
tct
cubc
ico
DBN
-1.48
0.16
-0.95
0.27
-4.44
-2.45
22.4
10.1
15.2
16.7
40.7
29.5
IPDF
4.10
4.45
4.26
5.7
4.81
1.28
3.96
1.4
1.4
4.0
8.4
IPDF#
5.22
6,45
5.58
6.33
6.53
1.21
4.36
1.16
1.15
5.48
4.64
9.38
Ours-5
5.70
6.47
5.77
6.93
6.95
2.40
2.03
1.78
1.12
2.51
1.92
2.81
Figure 3: Predicted orientation distributions for the cube and icosahedron. Top row shows "IPDF*" and the bottom shows ours. The baseline has several areas high probability away from the equiv- alent ground truths (marked with solid lines), while our method produces clean distribution. We omit the ground truth annotations for the icosahedron to reduce clutter. Refer to Murpuy et al. (2021) for details about the visualization.
coordinate-baat based MLP taking rotation matrices inputs. Euclidean positional encoding is applied and shown to improve upon no encoding.
In this experiment, show that using Wigner-D matrix elements the basis for the encoding significantly outperforms the Euclidean encoding. We encode the input rotation using all basis elements up degree é 5 by evaluating Eq. (4 using the "Spherical Functions" open source package to perform the computations.
We use the same architecture Murphy et al. (2021), and train the model on the SYMSOL I dataset, which was introduced by for pose estimation of symmetric solids. We slightly modify the training scheme IPDF by adding two randomly sampled points near each ground truth and also maximizing the logllikeli of them both. This stabilizes the training and greatly improves the log-likelihood metric. We denote this modified version "IPD and also apply it to our approach.
Table 2 shows the results. The log-likelihood is the average over all equivalent ground truth poses, and the "spread" metric can seen as the expected angle error to the closest ground truth (Murphy) al. 2021). Figure compares predictions from our method and the baseline and shows that our approach produces more accurate outputs.
5.3 NEURAL RADIANCE FIELDS
NeRF MMildenhall et al. 2020) has recently shown unprecedented quality neural rendering and inspired numerous advancements on the field. Part of the success of NeRF is due to the positional encoding of input coordinates. Specifically, the inputs are point in Euclidean space IR3 and
https:///ithhi httea://ggthbb..oo//m///////eeee

---

Under review as a conference paper at ICLR 2022
Table NeRF with spherical encoding on the Blender dataset. "NeRF*" inserts the direction the 6th layer instead of the 8th. Our method using spherical harmonics for encoding shows small but statistically significant improvements over the baselines. We report PSNRs in dB, with the standard deviations of the last digits shown within parenthesis.
avg-
chair
drums
ficus
hotdog
lego
materials
mic
ship
31.73
33.52
30.19
34.43
29.27
NeR
31.93(1)
34. (S)
25.32(3)
30.37
37.24(3)
(33)2((2)
31.35(6)
34.62(7)
29.17(5)
Ours
32. 32.11(1)
34.48(6)
25.41(4)
(0.69(2)
37.42(9)
33.29(3)
31.40(4)
34.84(4)
29.37(7)
Ground Truth
No Encoding
Axis- Exis-Aligned
Ours
Rendering spherical light fields Legol.5k. Our method using products of spherical har- monics for positional encoding captures high frequency details and produces considerably sharper images as compared baselines.
Figure 4:
a direction () E S2, and the network approximates density function at æ. (xx) and the color c(x, 0, o) of the light ray passing through at direction (6, Φ). The Euclidean positional encoding applied independently to x and to the 3D unit vector corresponding to (0, Φ).
In this experiment, replace NeRF's Euclidean positional encoding direction d with encoding based on the spherical harmonics. We used all the harmonics up to degree fmax = 4, Evaluating them on device with the jax. scipy package (Bradbury al. 2018).
In NeRF, the directional information intentionally included only at the last layer, which can be interpreted strong Lambertian inductive bias. Therefore the effect of the directional information and its positional encoding in the overall metrics is expected to be small. We noticed that inserting the directional information earlier the 6th layer instead of 8th slightly increases performance. Our experiments (Table 3) show small, but consistent and statistically significant improvement when using the spherical harmonics encoding. We repeat each experiment five times and report average PSNRs and standard deviations.
5.4 SPHERICAL LIGHT FIELDS
classic approach for novel view synthesis, which consists of interpolating the radiance at unseen rays based large database of given rays, without estimating the scene geometry. Sitzmann et al. (2021) recently combined learning of light fields and coordinate-based MLPs.
Light field rendering (evvyy Hanrahan 1996)
By assuming the radiance along a ray is constant, rays can identified with 4D vector coor- dinates. There are numerous parametrizations in the literature. In this experiment, consider the
8

---

Under review as a conference paper at ICLR 2022
Table 4: Rendering spherical light fields. Axis-aligned refers to the Euclidean positional encoding from Mildenhall et al. 2020) and our method uses product of spherical harmonics.
PSNR (7)
SSIM (t)
LPIS (L)
encoding
Axis Aais-aligned
Ours
encoding
Axis aligned
encoding
Axis aligned
24.94
27.42
29.14
0.82
0.87
0.91
0.16
0.13
0.10
one introduced by Caaaahort et al. (1998), which represents a ray hitting the scene with the two in- tersections between ray and sphere bounding the scene. Therefore, each light ray has coordinates (01, Φ1, 02 (22) E S x S2.
We introduce coordinate-based MLP learn this spherical light field, which enables rendering novel views by querying the ray coordinates for a desired viewpoint. The underlying manifold is a product of spheres, 5 S and we use the orthonormal basis for this manifold for positional encoding, as described in Section 4.4
We use all pairs of harmonics up to mmx = 4.
Rendering the light field directly with an MLP can orders of magnitude faster than approaches that use ray tracing (for example, NeRF needs to evaluate the MLP more than 100 times to render a single pixel). However, since light fields not model the underlying scene geometry, there no inherent notion of multi-view consistency in the functions learned. This hampers a light field rendering models ability to generalize to novel views from few samples. Consequently, akin to classical light field methods, we render large number of views for training. We sample 1.5k HEALPix (Górski et al. 20055 Zonca et al., 2019) points on a hemisphere and render the lego scene from Mildenhall al. (020) with cameras at these points, directed towards the origin.
We train light field models on this dataset using different positional encodings. Table 4 shows the results. We observe improvement of nearly 1.7 dB when using the proposed encoding when compared to axis-aligned encoding. Figure shows that our method improves considerably the rendering quality, retaining more high-frequency details compared baselines.
6 LIMITATIONS
Our method is sensitive to the choice of basis elements. Selecting maximum frequency that is too high leads to too many elements in the encoding and overfitting. while low frequencies tend to underfit. Similar limitations were observed by Tancik al. (2020), where the "scale" parameter, which is related to the frequency of the basis elements, needs to be carefully tuned.
Evaluating the special functions that compose the basis used for positional encoding is generally computationally expensive. This is amortized when the inputs lie on fixed grid (as in the spherical panoramas) can be pre-computed from the dataset (like the NeRF (Mildenhall et al. (2020) and light field experiments). This, combined with the fast on-device jax. scipy implementation of the spherical harmonics results in almost no overhead for these experiments. For IPDF (Murphy al.] 2021), however, the training grid is sampled randomly, and there is no on-device implementation, which results half the training speed. During inference the grid is fixed but very large, requiring a 32 Gb device for fast evaluation.
CONCLUSION
This paper introduced method to apply positional encoding for learning freeufrreuency functions on non-Euclidean manifolds. It generalizes the widely used "Fourier Features" by employing or- thonormal basis on the manifold for the encoding, as opposed to the sinusoidals that are basis func- tions for Euclidean spaces. The neural tangent kernel (NTK) theory guides our design; we ensure the encodings lie on a hypersphere and that their inner-products is preserved under the appropriate shift on the manifold. We demonstrated that our approach has advantages with respect to the standard Euclidean encodings in multiple tasks and manifolds.
http:://hht http://healpis wtw.alll...xoo.ooroooooo soureeeorge.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
We have presented generalized Fourier features for non-Euclidean manifolds. We have intentionally targeted broad set of applications, and focus has been presenting the core mathematical underpinnings of the technique with the objective of making it accessible to a wide audience.
REPRODUCIBILITY STATEMENT
We provide the experimental details necessary for reproducing the results the appropriate subsec- tions in Section and Appendix B. We note that the implementations are fairly straightforward as most cases we are simply making small modifications to the positional encoding in existing open source experiment code (e.g. swapping the positional encoding input in NeRF or IPDF). We commit to open sourcing the code for our experiments upon publication.
REFERENCES
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grainee analysis of optimization and generalization for overparameterized two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 322-332. URL https:: https::/prooeeding htt.:eddiiinnnm.....ss.eeee mwlr.ppess/v9//aroraaaa
PMLR, 09-15 Jun 2019.
html
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //githull com wwtpo/////oooo/a/oooooooee
Emilio Camahort, Apostolos Lerios, and Donald Fussell. Uniformly sampled light fields. In Euro- graphics Workshop on Rendering Techniques, pp. 117-130. Springer, 1998.
Haowen Deng, Mai Bui, Nassir Navab, Leonidas Guibas, Slobodan Ilic, and Tolga Birdal. Deep Bingham Networks: Dealing with Uncertainty and Ambiguity in Pose Estimation. arXiv preprint arXiv:2012.11002. 2020.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1243-1252. PMLR, 06-11 Aug 2017. URL
https: httpss///ccceeinegesemee .mlr.
press/v70/ge wwtp///geeringgggg. htt.mngg......lllll...m.mm.m.
K. M. Górski, F. Hivon, A. J. Banday, B. D. Wandelt, F. K. Hansen, M. Reinecke, and M. Bartel- mann. HEALPix: A Framework for High-Resolution Discretization and Fast Analysis of Data Distributed on the Sphere. The Astrophysical Journal, 622:759-771, April 2005. doi:
10.1086/427976.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
wiw.eeeeeiiigg......ee. meurips. cc.cc// Paper. pdf
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL www.thttt//ttm httttpsssppppcpppiiitgtooo/o///t neeeeeneuiiie httpp//pee//11///ieeeeeeee Paper pdf
0 0
Mare Levoy and Pat Hanrahan. Light field rendering. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques, pp. 31-41, 1996.
10

---

Under review as a conference paper at ICLR 2022
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 15651-15663. Curran Asso- ciates, Inc., 2020. URL
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 405-421, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58452-4
Kieran A Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: Non-parametric representation of probability distributions the rotation mani- fold. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Confer- Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7882-7893. PMLR, 18-24 Jul 2021. URL murphy21a. h
mll...mrr... press/v139/ htr.oeerdlgggs......rre///////
Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf:
Learning continuous signed distance functions for shape representation. In JEEE Con- ference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 165-174, 2019. doi: 10.1 199.119R..000..0000
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR, 09-15 Jun 2019. TRL htttps://prooeeeiingg mmr. press
htmm
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2008. URL httppp::://pooeeeteeeetee
wwtgissnneuiipp.. cc/ Paap:e55-5pppppppp.......
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin
(eds.), Advances in Neural Information Processing Systems, volume 33, pp- 20154-20166. Cur- ran Associates, Inc., 2020. htRtttts::////ttttt URL/attps:: www.sre.e.ppp..ee.. wtt.:ppp.cppppppepeeepee wttpp/pp..ccppppp/////.../.
fi 1144:922223211112222222111111113 Pappeeeedddd--p.ppp...p..
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-awan neural scene representations. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1119-1130, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 7462-7473. Curran Associates, Inc., 2020. URL https:://prcceediiiiiit/ neuriiss cc cc/ppaper/222//11eeee -atssscoo---a-p..... haperrpdf.
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Du- rand. Light field networks: Neural scene representations with single-evaluation rendering. In arXiv, 2021.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. NeurlPS, 2020.

---

Under review as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL
www.eeeediiiinneeniieeeeeee nww.inurrp.... htppscc//aaprr/217//1111 #ffe2/BBrccee1ID000ccccccee&.. Paper. pdf
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simon- sen. Encoding word order in complex embeddings. In International Conference on Learning Representations, 2020. URL
h hattp:///ppeneeeiee....eee/eeeee
Jianxiong Xiao, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Recognizing scene viewpoint using panoramic place representation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2695-2702. IEEE, 2012.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. In Advances in Neural Information Processing Systems, pp. 15889-15899, 2019.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appear- ance. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 2492-2502. Curran Asso- ciates, Inc., 2020. URL bttp hwwp:eeedddiin....e.e.eeee ctlpps.cc/ei cc//paper/2220/iile (2772etc3b0000aa3333666666666622
Jianqiao Zheng, Sameera Ramasinghe, and Simon Lucey, Rethinking positional encoding. CoRR, 2021. httptttt:///....rrgpp///e/////et
6256vv1
Ellen Zhong, Tristan Bepler, Joseph H. Davis, and Bonnie Berger. Reconstructing continuous dis- tributions of 3d protein structure from cryo-em images. In International Conference on Learning Representations, 2020. URL
Andrea Zonca, Leo Singer, Daniel Lenz, Martin Reinecke, Cyrille Rosset, Eric Hivon, and Krzysztof Gorski. healpy: equal area pixelization and spherical harmonies transforms for data the sphere in python. Joumna of Open Source Software, 4(35); 1298, March 2019. doi: 10.21 1011000ss..01200 01298
URL
A PROOFS
A.1 PROOF OF P
Proof. A formula for the rotation of spherical harmonics is given by
(8)
for all x E S and SO(3), where D is Wigner- matrix (with elements as defined in Eq.
Since Tt is unitary, preserves inner products. Then
(Rx). Y' (2g)) = =D"( (x(()))))))))))))))))))))))))))) =
12

---

Under review as a conference paper at ICLR 2022
Remark Since (x) = (- (z), can slightly modify the encodings to be real-valued and use only m 2 components,
(('))) E (=)
-ESmSE
= -x((=)
E (v) + Σ (x))((()))
tsm<o
Osmse
=Yo(=)Y)
Σ
Y (r)
-m (1)
Σ (x)
OcmS!
0<mSE
-o(())o
Σ
(
(
0<mst
= -r(())
Σ 0<mSE
where R and 3 are the real and imaginary parts, respectively.
This suggests the following alternative encoding, which also has constant norm and rotation- invariant inner products.
B'= 1Y: V, (23(X()) V23((2 (4E))
(9)
A.2 PROOF PROPOSITION 2
Proof. From the representation property of the Wigner-D matrices, we have
(10)
In the following, the identity vec(B)" vec(A) t)(A"B), where x" is the conjugate- transpose of x.
(V))(D)(RR))), vec(DD ve(DD((R)))ee<((((RR))
=((D(()R))'(((RR))
-I((D((R)'D''()
-vve(DD (R1)) vee (R))))
13

---

Under review as a conference paper at ICLR 2022
Remark Similarly to the case with the spherical harmonics, in practice real-valued en- coding using only m S basis elements, by leveraging that D mm = m- m
(vec(D'((I))) veel (R2)))
Σ Σ D (R)) m.n (.
Σ D D,, (m, - Dr D (Ri)DD_m,(PP2)+
osmse
Σ (o, (R2) +
ocnge
E
De m. Σ
1 (Ri) mm (R2) + Dt D'mm. ( ( Dm (R2)+
OSME! Oenst
(Ri))
(R) + De (Ri)D -m. -m (R22)
(B,) D (R2)
Σ 2R
D (B1)
+ Σ 28
Di (R1) D (B2) (B,)
0CMSE
0Ense
Σ
Σ
2R
D D (Ri) (Pa))
2R (
oemstoense
Σ SSSE
(
R
Di +
Σ
Σ 2
0emsE -tnnse
Since R(ab) (b) + (b) we obtain an encoding with constant norm and rotation- invariant inner products as follows,
B' (R(DD 1 =nng44u
(3(D))) LEL, SnJ
(V2AR(D(,,)). 14 -4 nn 4JU (V23(D(h,)) 14 m -45 44)
(II)
A.3 PROOF PROPOSITION
Proof. The elements of y (11, .22) coincide with the matrix elements of y ( with y as defined in Proposition
Let [ F be the Frobenius norm X. Then,
22 (z1, x22)
+1 1r( (
V 4x
1 V(20, + +1)(262 +)) 4m
A.4 PROOF OF PROPOSITION 4
Proof. We write f2 as the vectorization of the outer product, with Y as defined in Proposition Then use the identity (vec(A), vec( (B)) tr( B* A),
!
yh (2) (
14

---

Under review as a conference paper at ICLR 2022
the spherical harmonics rotation formula, that the Wigner-D is unitary, and the trace eyclic property,
(Y 2 (Rx1, R Y ( Ry2)) = (Roo (Rzy) vec(Y
ver (Ry1) (Ryz)"))
(Ry)))
(((""""(( (Rx2)")
(vec(Y (21) vec (ec (1), (2
B EXPERIMENTAL DETAILS
B. SPHERICAL PANORAMAS
Spherical PSNR Since pixels sampled equiangular grid the sphere have different areas, we use a simple quadrature rule to evaluate the PSNR. For co-latitude and longitude Φ the mean squared error (MSE) can be defined
2x
MSE
Σ sin():)
Σ 95
[I(0, 0.) - K(0,, 0,)?
(12)
NNNoN 9,
π
Σ sin(O.) Σ o,, - ((0,, 0,
(13)
2N N
where 7 represents an image, K is the noisy approximation of and N₂ and N₂ the number of sampling points along co-latitude and longitude, respectively. The PSNR (in dB) can thus be defined
as
MAX]
PSNR = 10 log
(14)
MSE
20 log10 (MAX,) - 10 log1o (MSE),
(15)
where the maximum possible pixel value of the image I is MAX, 255 (each pixel value is represented by eight bits).
Additional visualizations Figure show additional testing output images with both baseline and the proposed encoding methods.
B.2 SPHERICAL LIGHT FIELDS
For the light field rendering task, we train three models using encoding, axis-aligned encoding and the proposed product of spherical harmonies encoding. The model is trained to predict the color, given a light field representation of ray in space. The training signal to train the model is computed as the mean-square error between rendered views and ground truth images. We use an MLP with 12 layers, 1024 channels, ReLU activation for intermediate layers and a sigmoid on the output. The models are trained for 250k iterations with a batch size 8192 using an Adam optimizer with default settings. We use learning rate schedule with warm-up for 2.5k iterations followed by an exponential decay. The intial and final learning rates are set to 2 x 10 and 2 x 10 respectively. We set the maximum degree to be 4 for the axis-aligned well
encoding.
15

---

Under review as a conference paper at ICLR 2022
Figure 5: Testing output images(from left to right) with no, Euclidean, Gaussian, and spherical- harmonics encoding and the ground truth (the last column).
JaleWaa LUUvCH SWGTW
Figure 6: Samples from the "Spherical Text" dataset.
C EXTRA EXPERIMENTS
In this section, we show results on second spherical image dataset for sensitivity analysis to the most important hyperparameters. This dataset is based on the "Text" images of Tancik et al. (2020) which consists characters of different colors and sizes. is ideal for investigate positional encod- ing methods due to the abundance of sharp edges (high frequency content).
We use their 32 original images and perform an stereographic projection to the sphere at random orientations create spherical images. We refer to this dataset "Spherical Text".
16

---

Under review as a conference paper at ICLR 2022
We also introduce stronger baselines that take 3D points on the sphere, and thus respect the manifold structure, in contrast to the baselines in Section 5.1 The following baselines are considered this section.
Similar to the baseline in Section 5.1 Input coordinates are Cartesian, treating the spherical image as a planar image. For a desired encoding size 2M, and maximum frequency L, each coordinate is encoded independently as + fsim (2 2/ 2)) (20 2'24) }. where M and 1,2. Follows Mildenhall et al. (2020)
Axis-aligned 2D
j = kL/ (M 1) for
Gaussian 2D
Similar to the "Axis-aligned 2D", but the tuple coordinates is encoded jointly
as - (2m b, 2)} U (2tr by =)}, where {b, are vectors randomly sampled from a centered normal distribution of variance o² and < M.. Follows Tancik et al. (2020).
In the case, instead treating the spherical image as planar image, we treat it sphere embedded in I& and take the 3D coordinates of points on the sphere as inputs. The encoding is the same as the case, with E 1, 2,3, and encoding size 3M. This type of encoding used in the view-direction component in NeRF Mildenhall et al. (2020), which is spherical function.
Axis-aligned 3D
Gaussian 3D Similarly to "Axieealigned 3D", we apply the Gaussian encoding to 3D points on the sphere. The difference with respect to the 2D case is that by is 3 x instead of 2 x 1.
C.I BASELINE SENSITIVITY TO FREQUENCY AND SCALE
Table 5: Sensitivity analysis the maximum frequency L and Gaussian scale a on the "Spherical
Text" dataset. Encoding size is fixed to 2M 512. The bascline models are highly sensitive to the hyperparameter values. We include our spherical harmonics encoding as reference; for Lmax = 22 the encoding size is slightly lower than 2M, for bma 23 it is slightly higher.
/Seae/Scale
PNSR
Axis-aligned
= 5
25.82
Axis-aligned 3D
= 4
26.54
Axis-aligned 3D
L -=3 3
25.81
Axis-aligned 2D
L = 7
25.80
Axis- Axis-aligned 2D
L 6
26.43
Axis-aligned 2D
5
26.31
Gaussian 3D
10
25.99
Gaussian 3D
27.38
Gaussian 3D
= 6
28.17 (200)
Gaussian 3D
4
28.32(25)
Gaussian 3D
2
27.41
Gaussian 2D
12
26.84
Gaussian 2D
10
26.83
Gaussian 2D
0 =
26.9
Gaussian 2D
6
26.82
Ours
max = 22
28.45(5)
Ours
L m 23
28.60(9)
One downside of all baselines is that there two hyperparameters be tuned: the encoding size 2M and the scale a or maximum frequency L. First, we evaluate the sensitivity of to a and L by keeping the encoding size constant and equal to 2M 512. Table 5 shows that the baseline
17

---

Under review as a conference paper at ICLR 2022
methods are highly sensitive to these hyperparameter values, which partially explains method's advantages.
C.2 SENSITIVITY TO MAXIMUM FREQUENCY
The 2D baselines have the major downside of ignoring the manifold topology. The 3D baselines remove this limitation, but "Axis-aligned 3D" suffers from another limitation. By increasing the dimensions from two to three while keeping the encoding size constant, each dimension must be sampled more coarsely. This problem is exacerbated in IPDF (Murphy et al. 202T). which uses 9D parametrization
3D manifold.
The "Gaussian 3D" baseline mitigates this issue by sampling directions with components all dimensions. As shown in T it produces the best results among the baselines, and approaches the performance of our spherical harmonies encoding.
In this section, we thoroughly evaluate the performance of our method against "Gaussian 3D" for different encoding sizes, which determine the maximum frequency of our method. For our spherical harmonics encoding, this is the only hyperparameter to be set, while for the Gaussian we also have pick the scale Figure shows that outperform the baseline for all evaluated encoding sizes.
29
28
27
PSNR [dB]
26
25
Ours
==
0= 8
24
0= 2
23
200
300
700
800
encoding size
Figure 7: Comparison against "Gaussian 3D" for different sizes of encoding and Gaussian scales, on reconstructing the "Spherical Text" dataset. Our spherical harmonics encoding outperforms the Gaussian Fourier features in all scenarios. The Gaussian encoding best scale depends the encod- ing size, which highlights the difficulty to tune these hyperparameters.
C.3 IMPORTANCE SHIFT-INVARIANEE
Another major difference between our method and "Gaussian 3D", is that our encodings have con- stant norm and rotation-invarian inner products, thus satisfying the NTK conditions for the outputs to approximate convolution the manifold (shift-invariance).. This is appropriate for the tasks we consider, where the relative importance between training inputs (coordinates) should not depend on their absolute position. The Gaussian encoding does not satisfy these conditions and results in translation- invarrint encoding, which is not appropriate for points on the sphere.
In this section, show the effects of intentionally breaking these two properties in model. We sample a vector, fixed during training, from a standard normal distribution with the same dimensions the encoding. We then replace the positional encoding by its pointwise multiplication with this
18

---

Under review as a conference paper at ICLR 2022
vector of random factors. The resulting encoding does not have constant norm cotation-invartin inner products.
TableBishows the performance the "Spherical Text" dataset, with and without the random factors. We notice a consistent drop in the average PSNR when faar 12.
Table 6: Average accuracy on reconstructing the "Spherical Text" dataset, when random factors are applied to the positional encoding vector, breaking the constant and rotation- rotation-invariant inner products properties. The performance drops consistently for max degrees above 12.
Max degree
PSNR
Ours
12
27.40(14)
Ours random factors
12
27.43(17)
Ours
16
27.82(11)
Ours x random factors
27.43(13)
Ours
22
28.45(5)
Ours x random factors
28.13(15)
Ours
28
28.79(8)
Ours x random factors
28
28.34(8)
19


Reviews:
[{"review_id": "fJqD15IwMAF", "reviewer": "Reviewer_BYCg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the paper is well written, and the experiments seem to be convincing.  However, I have some major concerns. \n\n- First, the author motivated their work (i.e. this paper)  by citing that the original NerF paper used 3-dimensional direction to describe the incoming light direction, hence suffer from over-parameterization. Similarly,  it used the same argument to against the iPDF paper.    This is not true.  \n\nMore importantly,  it is not clear why achieving \"shift-invariance\" , \"minimal parameterization\" or even the orthonomality property matters for the purpose of positional encoding.   The purpose for the common sinusoidal positional encoding is to guide the learning process to move away from the \"spectral bias\" and to make the learning of high frequency information easier.   Even if the inputs are not parameterized minimally or not shift-invariant, as long as the  basis functions representation contains high frequency fluctuations the above purpose can be met.  \n\n Of course, one may argue that having a proper choice of orthonormal bases would result in some convolution-like behaviour but I don't understand why this is relevant at this stage of positional encoding.   Conversely, one can still use non-orthonormal basis for encoding (e.g. for Fourier features, multiple all  \\sin functions by a factor of 2 and leave the \\cos functions the same as before, or add identity mapping to basis functions), yet it will not necessarily lead to a worse performance.    As far as the positional encoding is concerned, sometimes having non-minimal redundant and even duplicating information ( lifting) can  be beneficial.   After all, what it matters is whether this can lessen the Spectral Bias of the deep neural network.   The new \"theory\" developed in this paper is nice to have, but isn't necessarily practically relevant. \n\nSecondly, The paper only includes three  special (and simple) instances non-Euclidean manifolds.   The three example manifolds (sphere, rotation, and product of sphere) are similar and special.  Their natural shifts on manifolds are rotation, and can be almost trivially obtained.   Their Fourier series have been studied by previous work. In other words, these results are well known and not novel.  What about other manifolds, where natural shifts are complex, and Fourier series are unknown ?  For a more generic non-Euclidean manifold it is not clear whether or how a `natural shift' can be defined/derived.   This is though a secondary point.  I am not convinced the work is well motivated, for my first point above.   \n", "summary_of_the_paper": "The paper generalizes the commonly used sinusoidal position encoding scheme (such as that in NerF input encoding) to inputs naturally residing in non-Euclidean manifold.\n\nThis is achieved by representing the input coordinates as  projections on  alternative sets of orthornormal bases instead of the trivial Euclidean coordinates.   The authors argue that by doing so, they extend the translational invariance of Fourier features in Euclidean space to non-Euclidian manifolds, resulting in a convolution-like operator that is invariant to the 'natural shift' on these manifolds.  These new position encoding techniques are tested for several tasks and seem to get consistent improvemnts over their Fourier counterparts.\n", "summary_of_the_review": "\nStrength: \n1. This paper presents an interesting idea attempting to extend positional encoding to non-Euclidean manifolds.\n2.   This paper provides experiments for three selected manifolds.\n\nWeakness: \nThe motivation of the proposed work is unclear (see comments above). Moreover, some further arguments are in order. \nExperiment provided only show that the selected base functions have similar properties of Euclidean encoding, but that doesn't explain why the selection of base functions can help extract details. Are they really better than  using sinusoids ? Is it really important to design base functions based on the principle? Take sinusoids as example, if we multiply a constant before sine bases and keep cosine bases still, we break the constant-norm property, but the network can still learn from these inputs and mitigate the spectral bias. The increase of performance is not very obvious compared with Euclidean encoding (0.5-1 PSNR increase). Experiments of how basis elements (such as max frequency) affect the performance is not found.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VjO5lUuK59S", "reviewer": "Reviewer_8frW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- For manifold data, it makes sense to use an encoding that adapts to the underlying manifold structure rather than the extrinsic Euclidean coordinates. The submission makes some contribution in this direction;\n- Performance improvement verified in the examples;\n- The paper discusses multiple application scenarios where the data can be encoded as spheres or their products. \n\nWeakness:\n- The title appears to suggest learning on general manifolds, but the actual discussion is focused on manifolds with a spherical structure. Does that mean there will be some manifolds where the proposed approach is not applicable? In particular, does this approach work for manifolds of arbitrary topology? If not, then the title can be slightly misleading and should be revised.\n- It is true that the Laplace-Beltrami operator can be used to derive an orthonormal basis on Riemannian manifolds. Unfortunately, there is no further discussion for such cases beyond this general statement. Can the author(s) provide some specific examples of how this should be used?\n- The paper confirms that computing the proposed encodings can be expensive. Can the author(s) quantify the increase in computational cost? This should give a more complete picture of the tradeoff between speed and accuracy.", "summary_of_the_paper": "The paper proposes a positional encoding scheme on manifolds for coordinate-based learning, which achieves better performance than Euclidean coordinate encoding.", "summary_of_the_review": "The paper makes a contribution towards a more general positional encoding scheme on manifolds. There needs to be more clarification on the applicability of the method on arbitrary manifolds, as well as the computational cost.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JApdcaWPgM7", "reviewer": "Reviewer_1zs2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\nThe paper provides abundant proof for explaining the key properties of the proposed method. Various empirical study has been provided with illustration to demonstrate the advantage of the method. \n\n## Weakness\n1. The neural network model is not very clearly shown though the main claimed contribution is for the positional encoding. \n2. The link of the proposed model with the NTK is well described. Actually, there is no big evidence provided by the authors about how the theory and model design are connected with the NTK theory.\n3. Since the authors mentioned in Section.6 the importance of selecting an appropriate scale, it would be nice to find complete evidence to support this, i.e., a sensitivity analysis. \nThe underlying method took degrees up to 10 in the paper. I wonder if this choice is enough for most applications, or the value is only restricted by the computational cost? If its the first case, can the authors provide some justifications on it? Otherwise, how bad influence would it have when the required degree is much larger than the model can provide?\n4. In numerical parts, the authors should include more the baselines for comparison.\n5. The paper writing could be further improved. Some examples include \"positional encoding is crucial to achieve (achieving) photorealism\" (Section 2); \"A direction can be associated to (with)...\" (Section 3).", "summary_of_the_paper": "This paper generalizes the positional encoding with Fourier features to non-Euclidean manifolds, which has the rotation invariance for feature extraction. The approximates convolutions on the manifold, according to the neural tangent kernel (NTK) assumptions, are shift-invariant.", "summary_of_the_review": "The theoretical basis of this paper is well supported, but to what degree this method is of practical needs further justification. The whole model of the paper with the proposed positional encoding method is yet to be well stated. The link with the NTK of the proposed method is not evident to readers and may not be the truth as the authors mentioned.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IeY38tVhLcP", "reviewer": "Reviewer_CgLb", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a nice paper!  Very clearly written, nice background.  I learned alot reading it.\n\nThe idea and why you would need to extend Fourier features to non-Euclidean spaces is well motivated and explained.  I appreciate the theoretical approach.\n\nThe examples are compelling and give good intuition for why it helps.\n\nOverall an impressive piece of work, I expect others at ICLR will learn from it as well.\n\n", "summary_of_the_paper": "In this paper the authors extend the Fourier features framework, which has recently had great success in 3D scene rendering, to non-Euclidean spaces, including S2, SO(3) and S2xS2.  The results show improvement over prior methods.\n", "summary_of_the_review": "A good theoretical approach with results to back it up.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8uo-tbx-Kfg", "reviewer": "Reviewer_awek", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. What is the point of NTK in background section? The authors are not using kernel basis\n2. The extensive details about spherical harmonics and Wigner basis can be put into appendix, as it is quite well-known.\n3. Same goes for section 4.4, basis for product manifold is well-known as well, so using spherical harmonics as basis for hypersphere, the basis for product of hyperspheres is straightforward. Proposition 3 is well-known result as well.\n4. Proposition 4 comes from the invariance property of metric on hypersphere, hypersphere can be seen as a quotient space of rotation group and hence with the canonical metric, proposition 4 is straightforward.\n5. The rendering application is section 5.3 is interesting and it is nice to visualize the results by replacing Euclidean encoding with spherical harmonics in NeRF.\n6. The authors did not mention anything regarding how to choose the scale of the basis, although they mentioned it as a limitation. Even Papers like \"Spherical CNN\" by Cohen et al. uses similar basis. It would have been a good contribution if the authors give recipe of choosing maximum bandwidth.", "summary_of_the_paper": "The authors used coordinate based basis for functions on manifold ans shown different application on sphere and rotation manifold.", "summary_of_the_review": "The theoretical contribution is not much as most of the method part can be put into appendix and very well-known. The experiments are proof of concept type, and the paper is essentially using well-known basis for functions of manifold, without giving any recipe for choosing frequency or maximum bandwidth L. The contribution is pretty limited as it's essentially using well-known basis in several applications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Positional encoding of the input coordinates using Fourier basis [as described in (1)] is a common tool in the context of multilayer perceptrons (MLP). The author propose to replace the Fourier basis with one on manifolds M (2), such as the classical spherical harmonics (M=S^2), the Fourier basis on M=SO(3) or on M=S^2 x S^2.

MLPs form an important tool of our times. Unfortunately, as it is elaborated by the reviewers the Fourier basis of the investigated manifolds are widely-studied and the presented results are well-known; the submission lacks novelty.
Decision: Reject