Year: 2021

Abstract:
Recently, positional encoding of input coordinates has been found crucial to enable learning of high-frequency functions with multilayer perceptrons taking lowdimensional coordinate values. In this setting, sinusoids are typically used as a basis for the encoding, which is commonly referred to as “Fourier Features”. However, using sinusoids as a basis assumes that the input coordinates lie on Euclidean space. In this work, we generalize positional encoding with Fourier features to non-Euclidean manifolds. We find appropriate bases for positional encoding on manifolds through generalizations of Fourier series. By ensuring the encodings lie on a hypersphere and that the appropriate shifts on the manifold preserve innerproducts between encodings, our model approximates convolutions on the manifold, according to the neural tangent kernel (NTK) assumptions. We demonstrate our method on various tasks on different manifolds: 1) learning panoramas on the sphere, 2) learning probability distributions on the rotation manifold, 3) learning neural radiance fields on the product of cube and sphere, and 4) learning light fields represented as the product of spheres.

heading: 1 INTRODUCTION
text: Recent breakthroughs on learning representations of 3D shapes (Mescheder et al., 2019; Park et al., 2019; Sitzmann et al., 2019) or scenes (Mildenhall et al., 2020) employ the so-called “coordinatebased” networks, which take low-dimensional coordinates as inputs and approximate a continuous function. These are sometimes called “implicit models”, when the approximated function implicitly represents the desired output; a typical example is using a signed distance function to represent a 3D shape (Park et al., 2019).
Perhaps the most important recent advancement in this line of research is NeRF (Mildenhall et al., 2020). One of the reasons for NeRF’s impressive performance is the positional encoding of input coordinates using sinusoidals of various frequencies, a technique that has been widely adopted (Liu et al., 2020; Schwarz et al., 2020; Yariv et al., 2020) and studied (Tancik et al., 2020; Zheng et al., 2021). The sinusoidals typically used for positional encoding are elements of orthonormal bases for functions on Euclidean spaces. Our key observation is that, to generalize this idea to non-Euclidean manifolds, we should use orthonormal basis functions on the manifold.
Figure 1 (middle) illustrates the evaluation of the Euclidean basis functions on spherical coordinates, which break orthogonality, results in uneven frequency distribution and singularities near the poles. The spherical harmonics are orthonormal basis functions for the sphere and do not exhibit these undesired properties.
NeRF (Mildenhall et al., 2020) and subsequent works avoid the problem just described by parametrizing the view direction as a unit vector in R3 instead of two angles. We argue that this overparametrization unnecessarily increases the dimensionality of the problem, since the number of basis functions for a fixed bandwidth b grows exponentially with the number of dimensions d. Another relevant example is IPDF (Murphy et al., 2021), which takes inputs on the 3-dimensional manifold SO(3) in the form of a flattened rotation matrix (a vector in R9), and applies positional encoding by sparsely sampling the space of O(b9) basis elements. We show that our approach improve results for both NeRF and IPDF.
Our basic idea is to find an orthonormal basis for the space of functions on the manifold of interest, and introduce principled methods for choosing a subset of the (possibly infinite) basis elements on which the input coordinates are evaluated. We describe mathematical techniques to obtain bases for large classes of manifolds and show experiments on a variety of them.
Our main contributions are:
• We introduce a principled way to apply positional encoding for coordinate-based learning of high-frequency functions on manifolds.
• We prove that our approach is shift-invariant under the light of the neural tangent kernel (NTK) theory (Jacot et al., 2018; Tancik et al., 2020) and for the appropriate “shift” on the manifold, which implies a manifold convolutional behavior.
• Our experiments show the advantages of the proposed methodology on different applications and manifolds: 1) learning panoramas on the sphere, 2) learning probability distributions on the rotation manifold, 3) learning neural radiance fields on the product of cube and sphere, and 4) learning light fields represented as the product of spheres.

heading: 2 RELATED WORK
text: The idea of positional encoding using a Fourier basis appeared as early as Rahimi & Recht (2008). They introduced the so-called random Fourier features to accelerate training of kernel methods. The idea is to approximate a shift-invariant kernel with random directions sampled from its Fourier transform, on which the inputs are evaluated. More recently, positional encoding has become popular for sequence modeling in natural language processing and often appears in attention layers and transformers as a way to encode the order of the input tokens. Gehring et al. (2017) use a learned embedding while Vaswani et al. (2017) use the Fourier basis for encoding input token positions. Alternative approaches were introduced by Xu et al. (2019); Wang et al. (2020).
Rahaman et al. (2019) demonstrated that neural networks tend to learn low frequencies more easily than high frequencies, a property called “spectral bias”. It also shows, for a simple task of approximating an 1D function, that projecting the input coordinate into a basis of sinusoids facilitates learning high frequencies. NeRF (Mildenhall et al., 2020) reproduced this finding in the challenging task of photorealistic novel view synthesis from a collection of images. They showed that positional encoding is crucial to achieving photorealism, and significantly outperformed the previous state of the art, which included other coordinate-based MLPs (Sitzmann et al., 2019). Concurrently, Zhong et al. (2020) demonstrated the usefulness of Fourier encoding for reconstruction of 3D protein complexes.
Tancik et al. (2020) conducted an in-depth study of positional encoding for coordinate-based MLPs. They show that, under the Neural Tangent Kernel (NTK) theory (Jacot et al., 2018), the positional encoding using a sinusoidal basis results in a stationary kernel, which in turn can be interpreted as a convolutional (shift-invariant) reconstruction filter, desirable for signals on Euclidean spaces. The theory also explains the “spectral bias” (Rahaman et al., 2019) of these models via the rapidly decaying NTK eigenvalues for MLPs. In practice, Tancik et al. (2020) propose randomly sampling the frequencies of the Fourier basis elements for the encoding, and show that it outperforms previous coordinate-based MLPs with no positional encoding (Mescheder et al., 2019), or axis-aligned positional encoding (Mildenhall et al., 2020).
While groundbreaking results and insightful theoretical developments have been demonstrated, most of the attention so far has been focused on coordinate-based MLPs for functions on Euclidean spaces. In this paper, we focus on the non-Euclidean case.
One alternative to Fourier features was introduced by Sitzmann et al. (2020), who use the sine function as the MLP nonlinearity. Major differences are that the sinusoidal is applied on every layer, and the frequencies are defined by the network MLP weights. Zheng et al. (2021) studied the trade-off between memorization and generalization in positional encoding, showing the former is related to the rank of the embedding matrix and the latter is related to the distance preservation of the embedding. They also propose another alternative to Fourier features that consists of sampling a Gaussian at fixed offsets from the input.

heading: 3 BACKGROUND
text: NeRF (Mildenhall et al., 2020) recently demonstrated the importance of positional encoding for coordinate-based learning of high frequency functions. For a coordinate x = [x1, x2, x3] ∈ R3, the following map was applied before the multilayer perceptron (MLP),
xi 7→ {sin(20πxi), cos(20πxi), sin(21πxi), cos(21πxi), · · · , sin(2Lπxi), cos(2Lπxi)}. (1)
Recall that {sin(m⊤x)} ∪ {cos(n⊤x)} with m, n ∈ Z3 form an orthonormal basis for functions defined on a compact subset of R3. So Eq. (1) corresponds to evaluating basis functions of different frequencies at input coordinates; we will refer to this method as “Euclidean encoding”. In the particular case of Eq. (1), only axis-aligned, powers-of-two frequencies are used, so the basis for bandwidth 2L is only sparsely sampled. Tancik et al. (2020) found it better to randomly sample the frequency space, which results in basis elements that are not axis aligned, but it is still a sparse sample of the complete basis.
In NeRF (Mildenhall et al., 2020), a subset of the input encodes the view direction. A direction can be associated with a point on the surface of the unit sphere, a non-Euclidean manifold. In order to apply the Euclidean encoding to such input, (Mildenhall et al., 2020) represent the direction as a 3D unit vector instead, which increases the input dimensionality from 2D to 3D, and causes the positional encoding to sample from a set of O(b3) basis functions, instead of the minimal O(b2) for a bandwidth b. In IPDF (Murphy et al., 2021), the input coordinates are flattened 3× 3 rotation matrices, which correspond to points on the 3D rotation manifold SO(3). The Euclidean encoding then corresponds to sampling basis functions on R9. This results in sampling from a space of O(b9) basis functions instead of O(b3). Our key observation is that positional encoding of input coordinates should reflect the geometry of the underlying manifold. We propose to use the appropriate orthonormal basis functions on the manifold to implement this idea. While for many applications the Euclidean manifold is the correct one, for others it is not, as exemplified above. In the following sections we will show how the appropriate basis can be found for different types of manifolds, how to actually choose a subset of the basis to use in the encoding, and demonstrate experimentally the benefits of using them.

heading: 4 METHOD

heading: 4.1 OVERVIEW
text: We consider a neural network that takes as inputs points on an n-dimensional manifold M. We assume there exists an orthonormal basis B = {bi : M → R} for the space of scalar functions on the manifold L2(M). Our goal is to apply the map γ : M → Rk to positionally encode x ∈ M before feeding it to the network, where k is the encoding dimension and
γ(x) = {bj(x) | bj ∈ B′ ⊂ B}. (2)
The first question that arises is how to find the basis for a given manifold. There is no general solution, but specific solutions exist for some large classes of manifolds. For compact groups (not necessarily abelian), the Peter-Weyl theorem establishes a generalization of the Fourier series that
results in a countable set of orthonormal basis functions. It gives rise to the Wigner-D matrix elements that are used as a basis for SO(3) in Section 4.3. For locally compact abelian groups, the Pontryagin duality can be used, which is a generalization of the Fourier transform. For Riemannian manifolds, an orthonormal basis can be obtained as eigenfunctions of the Laplace-Beltrami operator. For the sphere S2, these are the spherical harmonics used in Section 4.2.
The neural tangent kernel (NTK) theory (Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019) shows that, under certain conditions, an MLP trained for regression f : Rn → R, converges to the solution of a kernel regression f(x) ≈ ∑ i wiyik(x, xi), where k is the NTK, wi the kernel regression weights, and the sum is over the whole dataset. Moreover, when the inputs x have constant norm, the kernel depends only on inner products: k(x, y) = k′(x⊤y). In other words, it is rotationinvariant. Tancik et al. (2020) leverages this theory to explain the success of Euclidean encoding – it transforms the rotation-invariant kernel in a shift-invariant one, which results in approximating a convolution operation over the whole training set for inputs in a flat space.
In this paper, the inputs do not lie on Euclidean space, so the shift-invariance in the sense of translations in Euclidean space is not appropriate. For the manifolds considered, we will seek positional encodings that have constant norm (lie on Sn) and such that the natural shift on the manifold also preserves the inner products between positionally encoded inputs. Then, the NTK should approximate a convolution on the manifold. We’ll show in the following sections that these properties constrain the selection of basis elements.

heading: 4.2 THE SPHERE S2
text: The spherical harmonics are eigenfunctions of the Laplace-Beltrami operator on the sphere S2 and thus form an orthonormal basis for the space of square-integrable functions L2(S2). The spherical harmonic of degree ℓ and order m is given by
Y ℓm(θ, ϕ) = α ℓ mP ℓ m(cos θ)e imϕ, (3)
where θ, ϕ are the colatitude and longitude angles, P ℓm is the associate Legendre polynomial of degree ℓ and order m, with −ℓ ≤ m ≤ ℓ. The degree ℓ indicates the angular frequency, and αℓm is a normalization constant.
Under the NTK umbrella, an MLP taking points on the sphere encoded as 3D unit vectors will converge to a kernel regression with the kernel depending only on inner products. This is convenient because the natural shift on the sphere is a rotation, which preserves inner products. We want to select a subset of the spherical harmonics for positional encoding that maintains these properties: 1) preserves inner-products, and 2) has constant norm.
Proposition 1. For a given ℓ, let Y ℓ = [Y ℓ−ℓ, Y ℓ−ℓ+1 · · · , Y ℓℓ ] be a vector concatenating all the 2ℓ+1 spherical harmonics of degree ℓ. Let Rx represent the point x ∈ S2 rotated by R ∈ SO(3). Then, for any x, y ∈ S2 and R ∈ SO(3),
⟨Y ℓ(x), Y ℓ(y)⟩ = ⟨Y ℓ(Rx), Y ℓ(Ry)⟩.
Proof. See Appendix A.1. Since ∥∥Y ℓ(x)∥∥ = √(2ℓ+ 1)/(4π), the vector Y ℓ(x) has the same norm for any x, and the NTK depends only on inner products.
These results are easily extended to a concatenation of harmonics of multiple degrees, which suggests a strategy for selecting the basis elements for positional encoding: choose any set of degrees L ⊂ N and use all harmonics for the chosen degrees: B′ = {Y ℓim | ℓi ∈ L, −ℓi ≤ m ≤ ℓi}. For simplicity, we typically choose a degree ℓmax such that L = {ℓ | 1 ≤ ℓ ≤ ℓmax}. Note that the maximum degree ℓ is the only hyperparameter that needs to be selected for both strategies. In contrast, the Gaussian encoding from Tancik et al. (2020) requires both the encoding size and scale as hyperparameters.

heading: 4.3 THE ROTATION MANIFOLD SO(3)
text: The rotation manifold SO(3) is non-abelian and compact. The latter property guarantees a Fourier series for the manifold according to the Peter-Weyl theorem, given by the Wigner-D matrices. The Wigner-D matrices are unitary irreducible representations of SO(3) and its matrix elements form an orthonormal basis for L2(SO(3)). The elements for the (2ℓ+1)× (2ℓ+1) matrix Dℓ of degree ℓ are
Dℓm,n(α, β, γ) = e imαdℓm,n(β)e inγ , (4)
where α, β, γ are ZYZ Euler angles representing the rotation, −ℓ ≤ m, n ≤ ℓ, and dℓm,n(β) is real and proportional to the Jacobi polynomial P (a,b)k (cosβ), with k, a , b being functions of ℓ, m, n.
Since Dℓ is unitary for all ℓ, for any set of degrees L ∈ N, an encoding concatenating all matrix elements as follows has constant norm,
B′ = {Dℓim,n | ℓi ∈ L, −ℓi ≤ m, n ≤ ℓi}. (5)
The natural shift on SO(3) is also a rotation (the group acting on itself). Therefore, similarly to the discussion in Section 4.2, it is desirable that the inner products between encodings are invariant with respect to rotations of the input. Proposition 2. For a given ℓ, let vec(Dℓ) be the vectorization of the Wigner-D matrix Dℓ. Then, for any R, R1, R2 ∈ SO(3),
⟨vec(Dℓ(R1)), vec(Dℓ(R2))⟩ = ⟨vec(Dℓ(RR1)), vec(Dℓ(RR2))⟩.
Proof. See Appendix A.2
This also holds for a concatenation of multiple vec(Dℓi), so the map in Eq. (5), besides constant norm, also has rotation-invariant inner products, as desired. In practice, instead of an arbitrary set of degrees, we choose an ℓmax and use all degrees 1 ≤ ℓ ≤ ℓmax.

heading: 4.4 PRODUCT OF SPHERES S2 × S2
text: The coordinates for a point in S2 × S2 can be obtained by the concatenation of coordinates on each sphere: (θ1, ϕ1, θ2, ϕ2). A basis for the product manifold can be obtained by product of the basis functions for the sphere. Thus, we have the following orthonormal basis functions for L2(S2×S2),
Y ℓ1,ℓ2m1,m2(θ1, ϕ1, θ2, ϕ2) = Y ℓ1 m1(θ1, ϕ1)Y ℓ2 m2(θ2, ϕ2), (6)
with Y ℓm as defined in Eq. (3). We will use the following form interchangeably to reduce notation, Y ℓ1,ℓ2m1,m2(x1, x2) = Y ℓ1 m1(x1)Y ℓ2 m2(x2), where x1, x2 ∈ S2.
For a given set of pairs of degrees L ⊂ N×N, we consider the following basis functions, consisting of selecting all possible (mi, mj) given (ℓi, ℓj),
B′ = {Y ℓi,ℓjmi,mj | (ℓi, ℓj) ∈ L, −ℓi ≤ mi ≤ ℓi, and − ℓj ≤ mj ≤ ℓj}. (7)
Proposition 3. For a given pair (ℓ1, ℓ2), let Y ℓ1,ℓ2 = {Y ℓ1,ℓ2mi,mj} be a vector containing all the (2ℓ1 + 1)(2ℓ2 + 1) products of spherical harmonics of degrees ℓ1 and ℓ2. For any x1, x2 ∈ S2, the norm of Y ℓ1,ℓ2 is a constant: ∥∥Y ℓ1,ℓ2(x1, x2)∥∥ = √(2ℓ1 + 1)(2ℓ2 + 1)/(4π). Proof. See Appendix A.3.
Since the encodings have constant norm, we are interested in whether the inner product between encodings is preserved under input shift, which would lead to the previously discussed desirable properties that follow from the NTK. For the case of S2 × S2, an appropriate shift consists in applying the same rotation to both spheres. For example, in our experiment in Section 5.4, the inputs on S2 × S2 represent a pair of points on a sphere, so it makes sense that the appropriate shift should rotate both points.
Proposition 4. Consider the vector of orthonormal functions Y ℓ1,ℓ2 as defined in Proposition 3. The inner product between encodings is invariant to rotations of the inputs:
⟨Y ℓ1,ℓ2(x1, x2), Y ℓ1,ℓ2(y1, y2)⟩ = ⟨Y ℓ1,ℓ2(Rx1, Rx2), Y ℓ1,ℓ2(Ry1, Ry2)⟩, for any pair (x1, x2) ∈ S2 × S2 and (y1, y2) ∈ S2 × S2, and rotation R ∈ SO(3).
Proof. See Appendix A.4
Therefore, we can construct a basis for positional encoding using Eq. (7). In practice, we set a maximum degree ℓmax and use all pairs under it: L = {(ℓi, ℓj) | 1 ≤ ℓi, ℓj ≤ ℓmax}.

heading: 5 EXPERIMENTS

heading: 5.1 SPHERICAL PANORAMAS
text: Following Tancik et al. (2020); Sitzmann et al. (2020), we evaluate the learning of high-frequencies on a simple task of training an MLP to regress an image from its input coordinates. Since we are interested in non-Euclidean manifolds, we evaluate the methods on 10 spherical panoramas from SUN360 (Xiao et al. (2012)), randomly sampling 1/4 of the pixels as the training set, with probabilities proportional to pixel areas in the input spherical grid.
We follow the image resolution, architecture design, and training schedule of Tancik et al. (2020), and replace their positional encoding with ours, where we use all (ℓmax + 1)(ℓmax + 2)/2 spherical harmonics up to ℓmax = 16. Table 1 shows the test peak signal-to-noise ratios (PSNRs). Figure 2 depicts the results. Note the large distortions on the poles with Gaussian encoding; these are the areas with large distortion when assuming an Euclidean manifold.

heading: 5.2 PROBABILITY DISTRIBUTIONS ON SO(3)
text: Recently, Murphy et al. (2021) introduced IPDF, a model that learns to predict probability distributions on SO(3) to represent the object pose from an image. The main part of the model is a
coordinate-based MLP taking rotation matrices as inputs. Euclidean positional encoding is applied and shown to improve upon no encoding.
In this experiment, we show that using Wigner-D matrix elements as the basis for the encoding significantly outperforms the Euclidean encoding. We encode the input rotation using all basis elements up to degree ℓ = 5 by evaluating Eq. (4), using the “Spherical Functions” 1 open source package to perform the computations.
We use the same architecture of Murphy et al. (2021), and train the model on the SYMSOL I dataset, which was introduced by for pose estimation of symmetric solids. We slightly modify the training scheme of IPDF by adding two randomly sampled points near each ground truth and also maximizing the log-likelihood of them both. This stabilizes the training and greatly improves the log-likelihood metric. We denote this modified version “IPDF*”, and also apply it to our approach.
Table 2 shows the results. The log-likelihood is the average over all equivalent ground truth poses, and the “spread” metric can be seen as the expected angle error to the closest ground truth (Murphy et al., 2021). Figure 3 compares predictions from our method and the baseline and shows that our approach produces more accurate outputs.

heading: 5.3 NEURAL RADIANCE FIELDS
text: NeRF (Mildenhall et al., 2020) has recently shown unprecedented quality on neural rendering and inspired numerous advancements on the field. Part of the success of NeRF is due to the positional encoding of input coordinates. Specifically, the inputs are a point in Euclidean space x ∈ R3 and
1https://github.com/moble/spherical
Table 3: NeRF with spherical encoding on the Blender dataset. “NeRF*” inserts the direction at the 6th layer instead of the 8th. Our method using spherical harmonics for encoding shows small but statistically significant improvements over the baselines. We report PSNRs in dB, with the standard deviations of the last digits shown within parenthesis.
avg. chair drums ficus hotdog lego materials mic ship
NeRF 31.73 34.08 25.01 30.51 36.84 33.52 30.19 34.43 29.27 NeRF* 31.93(1) 34.10(5) 25.32(3) 30.37(4) 37.24(3) 33.29(2) 31.35(6) 34.62(7) 29.17(5) Ours 32.11(1) 34.48(6) 25.41(4) 30.69(2) 37.42(9) 33.29(3) 31.40(4) 34.84(4) 29.37(7)
Ground Truth No Encoding Axis-Aligned Ours
Figure 4: Rendering spherical light fields on Lego1.5k. Our method using products of spherical harmonics for positional encoding captures high frequency details and produces considerably sharper images as compared to baselines.
a direction (θ, ϕ) ∈ S2, and the network approximates a density function at x, σ(x) and the color c(x, θ, ϕ) of the light ray passing through x at direction (θ, ϕ). The Euclidean positional encoding is applied independently to x and to the 3D unit vector d corresponding to (θ, ϕ).
In this experiment, we replace NeRF’s Euclidean positional encoding at direction d with an encoding based on the spherical harmonics. We used all the harmonics up to degree ℓmax = 4, Evaluating them on device with the jax.scipy package (Bradbury et al., 2018).
In NeRF, the directional information is intentionally included only at the last layer, which can be interpreted as a strong Lambertian inductive bias. Therefore the effect of the directional information and its positional encoding in the overall metrics is expected to be small. We noticed that inserting the directional information earlier at the 6th layer instead of 8th slightly increases performance. Our experiments (Table 3) show a small, but consistent and statistically significant improvement when using the spherical harmonics encoding. We repeat each experiment five times and report average PSNRs and standard deviations.

heading: 5.4 SPHERICAL LIGHT FIELDS
text: Light field rendering (Levoy & Hanrahan, 1996) is a classic approach for novel view synthesis, which consists of interpolating the radiance at unseen rays based on a large database of given rays, without estimating the scene geometry. Sitzmann et al. (2021) recently combined learning of light fields and coordinate-based MLPs.
By assuming the radiance along a ray is constant, rays can be identified with a 4D vector of coordinates. There are numerous parametrizations in the literature. In this experiment, we consider the
one introduced by Camahort et al. (1998), which represents a ray hitting the scene with the two intersections between ray and a sphere bounding the scene. Therefore, each light ray has coordinates (θ1, ϕ1, θ2, ϕ2) ∈ S2 × S2. We introduce a coordinate-based MLP to learn this spherical light field, which enables rendering novel views by querying the ray coordinates for a desired viewpoint. The underlying manifold is a product of spheres, S2 × S2, and we use the orthonormal basis for this manifold for positional encoding, as described in Section 4.4. We use all pairs of harmonics up to ℓmax = 4.
Rendering the light field directly with an MLP can be orders of magnitude faster than approaches that use ray tracing (for example, NeRF needs to evaluate the MLP more than 100 times to render a single pixel). However, since light fields do not model the underlying scene geometry, there is no inherent notion of multi-view consistency in the functions learned. This hampers a light field rendering models ability to generalize to novel views from few samples. Consequently, akin to classical light field methods, we render a large number of views for training. We sample 1.5k HEALPix2 (Górski et al., 2005; Zonca et al., 2019) points on a hemisphere and render the lego scene from (Mildenhall et al., 2020) with cameras at these points, directed towards the origin.
We train light field models on this dataset using different positional encodings. Table 4 shows the results. We observe an improvement of nearly 1.7 dB when using the proposed encoding when compared to axis-aligned encoding. Figure 4 shows that our method improves considerably the rendering quality, retaining more high-frequency details as compared to baselines.

heading: 6 LIMITATIONS
text: Our method is sensitive to the choice of basis elements. Selecting a maximum frequency that is too high leads to too many elements in the encoding and overfitting, while low frequencies tend to underfit. Similar limitations were observed by Tancik et al. (2020), where the “scale” parameter, which is related to the frequency of the basis elements, needs to be carefully tuned.
Evaluating the special functions that compose the basis used for positional encoding is generally computationally expensive. This is amortized when the inputs lie on a fixed grid (as in the spherical panoramas) or can be pre-computed from the dataset (like in the NeRF (Mildenhall et al., 2020) and light field experiments). This, combined with the fast on-device jax.scipy implementation of the spherical harmonics results in almost no overhead for these experiments. For IPDF (Murphy et al., 2021), however, the training grid is sampled randomly, and there is no on-device implementation, which results is half the training speed. During inference the grid is fixed but very large, requiring a 32 Gb device for fast evaluation.

heading: 7 CONCLUSION
text: This paper introduced a method to apply positional encoding for learning high-frequency functions on non-Euclidean manifolds. It generalizes the widely used “Fourier Features” by employing orthonormal basis on the manifold for the encoding, as opposed to the sinusoidals that are basis functions for Euclidean spaces. The neural tangent kernel (NTK) theory guides our design; we ensure the encodings lie on a hypersphere and that their inner-products is preserved under the appropriate shift on the manifold. We demonstrated that our approach has advantages with respect to the standard Euclidean encodings in multiple tasks and manifolds.
2http://healpix.sourceforge.net

heading: ETHICS STATEMENT
text: We have presented generalized Fourier features for non-Euclidean manifolds. We have intentionally targeted a broad set of applications, and our focus has been on presenting the core mathematical underpinnings of the technique with the objective of making it accessible to a wide audience.

heading: REPRODUCIBILITY STATEMENT
text: We provide the experimental details necessary for reproducing the results in the appropriate subsections in Section 5 and Appendix B. We note that the implementations are fairly straightforward as in most cases we are simply making small modifications to the positional encoding in existing open source experiment code (e.g. swapping the positional encoding input in NeRF or IPDF). We commit to open sourcing the code for our experiments upon publication.

heading: A PROOFS

heading: A.1 PROOF OF PROPOSITION 1
text: Proof. A formula for the rotation of spherical harmonics is given by
Y ℓ(Rx) = Dℓ(R)∗Y ℓ(x), (8)
for all x ∈ S2 and R ∈ SO(3), where Dℓ is a Wigner-D matrix (with elements as defined in Eq. (4)). Since Dℓ is unitary, it preserves inner products. Then
⟨Y ℓ(Rx), Y ℓ(Ry)⟩ = ⟨Dℓ(R)∗Y ℓ(x), Dℓ(R)∗Y ℓ(y)⟩ = ⟨Y ℓ(x), Y ℓ(y)⟩.
Remark Since Y ℓ−m(x) = (−1)mY ℓm(x), we can slightly modify the encodings to be real-valued and use only m ≥ 0 components,
⟨Y ℓ(x), Y ℓ(y)⟩ = ∑
−ℓ≤m≤ℓ
Y ℓm(x)Y ℓ m(y)
= Y ℓ0 (x)Y ℓ 0 (y) + ∑ −ℓ≤m<0 Y ℓm(x)Y ℓ m(y) + ∑ 0<m≤ℓ Y ℓm(x)Y ℓ m(y)
= Y ℓ0 (x)Y ℓ 0 (y) + ∑ 0<m≤ℓ Y ℓ−m(x)Y ℓ −m(y) + ∑ 0<m≤ℓ Y ℓm(x)Y ℓ m(y)
= Y ℓ0 (x)Y ℓ 0 (y) + ∑ 0<m≤ℓ Y ℓm(x)Y ℓ m(y) + Y ℓ m(x)Y ℓ m(y)
= Y ℓ0 (x)Y ℓ 0 (y) + ∑ 0<m≤ℓ 2ℜ(Y ℓm(x))ℜ(Y ℓm(y)) + 2ℑ(Y ℓm(x))ℑ(Y ℓm(y)),
where ℜ and ℑ are the real and imaginary parts, respectively. This suggests the following alternative encoding, which also has constant norm and rotationinvariant inner products,
B′ = {Y ℓi0 , √ 2ℜ(Y ℓi1 ), √ 2ℑ(Y ℓi1 ), · · · , √ 2ℜ(Y ℓiℓi ), √ 2ℑ(Y ℓiℓi ) | ℓi ∈ L}. (9)

heading: A.2 PROOF OF PROPOSITION 2
text: Proof. From the representation property of the Wigner-D matrices, we have
Dℓ(RR1) = D ℓ(R)Dℓ(R1). (10)
In the following, we use the identity vec(B)∗vec(A) = tr(A∗B), where x∗ is the conjugatetranspose of x.
⟨vec(Dℓ(RR1)), vec(Dℓ(RR2))⟩ = vec(Dℓ(RR1))∗vec(Dℓ(RR2)) = tr(Dℓ(RR2)∗Dℓ(RR1))
= tr(Dℓ(R2)∗Dℓ(R)∗Dℓ(R)Dℓ(R1))
= tr(Dℓ(R2)∗Dℓ(R1))
= ⟨vec(Dℓ(R1)), vec(Dℓ(R2))⟩.
Remark Similarly to the case with the spherical harmonics, in practice we use a real-valued encoding using only m ≤ 0 basis elements, by leveraging that Dℓm,n = (−1)m−nDℓ−m,−n,
⟨vec(Dℓ(R1)), vec(Dℓ(R2))⟩ = ∑
−ℓ≤m≤ℓ ∑ −ℓ≤n≤ℓ Dℓm,n(R1)D ℓ m,n(R2)
= Dℓ0,0(R1)D ℓ 0,0(R2)+∑
0<m≤ℓ
Dℓm,0(R1)D ℓ m,0(R2) +D ℓ −m,0(R1)D ℓ −m,0(R2)+∑
0<n≤ℓ
Dℓ0,n(R1)D ℓ 0,n(R2) +D ℓ 0,−n(R1)D ℓ 0,−n(R2)+∑
0<m≤ℓ ∑ 0<n≤ℓ Dℓm,n(R1)D ℓ m,n(R2) +D ℓ −m,n(R1)D ℓ −m,n(R2)+
Dℓm,−n(R1)D ℓ m,−n(R2) +D ℓ −m,−n(R1)D ℓ −m,−n(R2)
= Dℓ0,0(R1)D ℓ 0,0(R2)+∑
0<m≤ℓ
2ℜ ( Dℓm,0(R1)D ℓ m,0(R2) ) + ∑ 0<n≤ℓ 2ℜ ( Dℓ0,n(R1)D ℓ 0,n(R2) ) +
∑ 0<m≤ℓ ∑ 0<n≤ℓ 2ℜ ( Dℓm,n(R1)D ℓ m,n(R2) ) + 2ℜ ( Dℓm,−n(R1)D ℓ m,−n(R2) ) =
∑ −ℓ≤n≤ℓ ℜ ( Dℓ0,n(R1)D ℓ 0,n(R2) ) +
∑ 0<m≤ℓ ∑ −ℓ≤n≤ℓ 2ℜ ( Dℓm,n(R1)D ℓ m,n(R2) ) .
Since ℜ(ab) = ℜ(a)ℜ(b) + ℑ(a)ℑ(b), we obtain an encoding with constant norm and rotationinvariant inner products as follows,
B′ ={ℜ(Dℓi0,n), | ℓi ∈ L, −ℓi ≤ n ≤ ℓi}∪
{ℑ(Dℓi0,n), | ℓi ∈ L, −ℓi ≤ n ≤ ℓi}∪ { √ 2ℜ(Dℓim,n), | ℓi ∈ L, 0 < m ≤ ℓi, −ℓi ≤ n ≤ ℓi}∪ { √ 2ℑ(Dℓim,n), | ℓi ∈ L, 0 < m ≤ ℓi, −ℓi ≤ n ≤ ℓi}. (11)

heading: A.3 PROOF OF PROPOSITION 3
text: Proof. The elements of Y ℓ1,ℓ2(x1, x2) coincide with the matrix elements of Y ℓ1(x1)Y ℓ2(x2)∗, with Y ℓ as defined in Proposition 1. Let ∥X∥F be the Frobenius norm of X . Then,∥∥Y ℓ1,ℓ2(x1, x2)∥∥ = ∥∥Y ℓ1(x1)Y ℓ2(x2)∗∥∥F
= √ tr((Y ℓ1(x1)Y ℓ2(x2)∗)∗Y ℓ1(x1)Y ℓ2(x2)∗)
= √ tr(Y ℓ2(x2)Y ℓ1(x1)∗Y ℓ1(x1)Y ℓ2(x2)∗)
=
√ 2ℓ1 + 1
4π tr(Y ℓ2(x2)Y ℓ2(x2)∗)
= 1
4π
√ (2ℓ1 + 1)(2ℓ2 + 1).

heading: A.4 PROOF OF PROPOSITION 4
text: Proof. We write Y ℓ1,ℓ2 as the vectorization of the outer product, with Y ℓ as defined in Proposition 1, Y ℓ1,ℓ2(x1, x2) = vec(Y ℓ1(x1)Y ℓ2(x2)∗). Then we use the identity ⟨vec(A), vec(B)⟩ = tr(B∗A),
the spherical harmonics rotation formula, that the Wigner-D is unitary, and the trace cyclic property,
⟨Y ℓ1,ℓ2(Rx1, Rx2), Y ℓ1,ℓ2(Ry1, Ry2)⟩ = ⟨vec(Y ℓ1(Rx1)Y ℓ2(Rx2)∗), vec(Y ℓ1(Ry1)Y ℓ2(Ry2)∗)⟩ = tr((Y ℓ1(Ry1)Y ℓ2(Ry2)∗)∗Y ℓ1(Rx1)Y ℓ2(Rx2)∗)
= tr(Y ℓ2(Ry2)Y ℓ1(Ry1)∗Y ℓ1(Rx1)Y ℓ2(Rx2)∗)
= tr(Dℓ2(R)Y ℓ2(y2)Y ℓ1(y1)∗Dℓ1(R)∗Dℓ1(R)Y ℓ1(x1)Y ℓ2(x2)∗Dℓ2(R)∗)
= tr(Dℓ2(R)Y ℓ2(y2)Y ℓ1(y1)∗Y ℓ1(x1)Y ℓ2(x2)∗Dℓ2(R)∗)
= tr(Dℓ2(R)∗Dℓ2(R)Y ℓ2(y2)Y ℓ1(y1)∗Y ℓ1(x1)Y ℓ2(x2)∗)
= tr(Y ℓ2(y2)Y ℓ1(y1)∗Y ℓ1(x1)Y ℓ2(x2)∗)
= ⟨vec(Y ℓ1(x1)Y ℓ2(x2)∗), vec(Y ℓ1(y1)Y ℓ2(y2)∗)⟩ = ⟨Y ℓ1,ℓ2(x1, x2), Y ℓ1,ℓ2(y1, y2)⟩.

heading: B EXPERIMENTAL DETAILS

heading: B.1 SPHERICAL PANORAMAS
text: Spherical PSNR Since pixels sampled on an equiangular grid on the sphere have different areas, we use a simple quadrature rule to evaluate the PSNR. For co-latitude θ and longitude ϕ, the mean squared error (MSE) can be defined as
MSE = 2π2
4πNθNϕ ∑ θi sin(θi) ∑ ϕj [I(θi, ϕj)−K(θi, ϕj)]2 (12)
= π
2NθNϕ ∑ θi sin(θi) ∑ ϕj [I(θi, ϕj)−K(θi, ϕj)] , (13)
where I represents an image, K is the noisy approximation of I , and Nθ and Nϕ are the number of sampling points along co-latitude and longitude, respectively. The PSNR (in dB) can thus be defined as
PSNR = 10 log10 ( MAX2I MSE ) (14)
= 20 log10 (MAXI)− 10 log10 (MSE) , (15)
where the maximum possible pixel value of the image I is MAXI = 255 (each pixel value is represented by eight bits).
Additional visualizations Figure 5 show additional testing output images with both baseline and the proposed encoding methods.

heading: B.2 SPHERICAL LIGHT FIELDS
text: For the light field rendering task, we train three models using no encoding, axis-aligned encoding and the proposed product of spherical harmonics encoding. The model is trained to predict the color, given a light field representation of ray in space. The training signal to train the model is computed as the mean-square error between rendered views and ground truth images. We use an MLP with 12 layers, 1024 channels, ReLU activation for intermediate layers and a sigmoid on the output. The models are trained for 250k iterations with a batch size of 8192 using an Adam optimizer with default settings. We use a learning rate schedule with warm-up for 2.5k iterations followed by an exponential decay. The intial and final learning rates are set to 2× 10−3 and 2× 10−5 respectively. We set the maximum degree to be 4 for the axis-aligned as well as our encoding.

heading: C EXTRA EXPERIMENTS
text: In this section, we show results on a second spherical image dataset for sensitivity analysis to the most important hyperparameters. This dataset is based on the “Text” images of Tancik et al. (2020), which consists of characters of different colors and sizes. It is ideal for investigate positional encoding methods due to the abundance of sharp edges (high frequency content).
We use their 32 original images and perform an stereographic projection to the sphere at random orientations to create spherical images. We refer to this dataset as “Spherical Text”.
We also introduce stronger baselines that take 3D points on the sphere, and thus respect the manifold structure, in contrast to the baselines in Section 5.1. The following baselines are considered in this section.
Axis-aligned 2D Similar to the baseline in Section 5.1. Input coordinates are Cartesian, treating the spherical image as a planar image. For a desired encoding size 2M , and maximum frequency L, each coordinate zi is encoded independently as zi 7→ {sin(2π 2jzi)} ∪ {cos(2π 2jzi)}, where j = kL/(M − 1) for 0 ≤ k < M and i ∈ 1, 2. Follows Mildenhall et al. (2020).
Gaussian 2D Similar to the “Axis-aligned 2D”, but the tuple of coordinates z is encoded jointly as z 7→ {sin(2π b⊤i z)} ∪ {cos(2π b⊤i z)}, where {bi} are vectors randomly sampled from a centered normal distribution of variance σ2 and 0 ≤ i < M . Follows Tancik et al. (2020).
Axis-aligned 3D In the 3D case, instead of treating the spherical image as a planar image, we treat it as a sphere embedded in R3, and take the 3D coordinates of points on the sphere as inputs. The encoding is the same as the 2D case, with i ∈ 1, 2, 3, and encoding size 3M . This type of encoding is used in the view-direction component in NeRF Mildenhall et al. (2020), which is a spherical function.
Gaussian 3D Similarly to “Axis-aligned 3D”, we apply the Gaussian encoding to 3D points on the sphere. The difference with respect to the 2D case is that bi is 3× 1 instead of 2× 1.

heading: C.1 BASELINE SENSITIVITY TO FREQUENCY AND SCALE
text: One downside of all baselines is that there are two hyperparameters to be tuned: the encoding size 2M and the scale σ or maximum frequency L. First, we evaluate the sensitivity of to σ and L by keeping the encoding size constant and equal to 2M = 512. Table 5 shows that the baseline
methods are highly sensitive to these hyperparameter values, which partially explains our method’s advantages.

heading: C.2 SENSITIVITY TO MAXIMUM FREQUENCY
text: The 2D baselines have the major downside of ignoring the manifold topology. The 3D baselines remove this limitation, but “Axis-aligned 3D” suffers from another limitation. By increasing the dimensions from two to three while keeping the encoding size constant, each dimension must be sampled more coarsely. This problem is exacerbated in IPDF (Murphy et al., 2021), which uses a 9D parametrization of a 3D manifold.
The “Gaussian 3D” baseline mitigates this issue by sampling directions with components on all dimensions. As shown in Table 5, it produces the best results among the baselines, and approaches the performance of our spherical harmonics encoding.
In this section, we thoroughly evaluate the performance of our method against “Gaussian 3D” for different encoding sizes, which determine the maximum frequency of our method. For our spherical harmonics encoding, this is the only hyperparameter to be set, while for the Gaussian we also have to pick the scale σ. Figure 7 shows that we outperform the baseline for all evaluated encoding sizes.
C.3 IMPORTANCE OF SHIFT-INVARIANCE
Another major difference between our method and “Gaussian 3D”, is that our encodings have constant norm and rotation-invariant inner products, thus satisfying the NTK conditions for the outputs to approximate a convolution on the manifold (shift-invariance). This is appropriate for the tasks we consider, where the relative importance between training inputs (coordinates) should not depend on their absolute position. The Gaussian encoding does not satisfy these conditions and results in a translation-invariant encoding, which is not appropriate for points on the sphere.
In this section, we show the effects of intentionally breaking these two properties in our model. We sample a vector, fixed during training, from a standard normal distribution with the same dimensions as the encoding. We then replace the positional encoding by its pointwise multiplication with this
vector of random factors. The resulting encoding does not have constant norm nor rotation-invariant inner products.
Table 6 shows the performance on the “Spherical Text” dataset, with and without the random factors. We notice a consistent drop in the average PSNR when ℓmax > 12.

Review summaries:
The paper generalizes the commonly used sinusoidal position encoding scheme (such as that in NerF input encoding) to inputs naturally residing in non-Euclidean manifold.  This is achieved by representing the input coordinates as  projections on  alternative sets of orthornormal bases instead of the trivial Euclidean coordinates.   The authors argue that by doing so, they extend the translational invariance of Fourier features in Euclidean space to non-Euclidian manifolds, resulting in a convolution-like operator that is invariant to the 'natural shift' on these manifolds.  These new position encoding techniques are tested for several tasks and seem to get consistent improvemnts over their Fourier counterparts.