Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: In this paper the authors extend the Fourier features framework, which has recently had great success in 3D scene rendering, to non-Euclidean spaces, including S2, SO(3) and S2xS2.  The results show improvement over prior methods.

main_review: This is a nice paper!  Very clearly written, nice background.  I learned alot reading it.

The idea and why you would need to extend Fourier features to non-Euclidean spaces is well motivated and explained.  I appreciate the theoretical approach.

The examples are compelling and give good intuition for why it helps.

Overall an impressive piece of work, I expect others at ICLR will learn from it as well.


summary_of_the_review: A good theoretical approach with results to back it up.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors used coordinate based basis for functions on manifold ans shown different application on sphere and rotation manifold.
main_review: 1. What is the point of NTK in background section? The authors are not using kernel basis
2. The extensive details about spherical harmonics and Wigner basis can be put into appendix, as it is quite well-known.
3. Same goes for section 4.4, basis for product manifold is well-known as well, so using spherical harmonics as basis for hypersphere, the basis for product of hyperspheres is straightforward. Proposition 3 is well-known result as well.
4. Proposition 4 comes from the invariance property of metric on hypersphere, hypersphere can be seen as a quotient space of rotation group and hence with the canonical metric, proposition 4 is straightforward.
5. The rendering application is section 5.3 is interesting and it is nice to visualize the results by replacing Euclidean encoding with spherical harmonics in NeRF.
6. The authors did not mention anything regarding how to choose the scale of the basis, although they mentioned it as a limitation. Even Papers like "Spherical CNN" by Cohen et al. uses similar basis. It would have been a good contribution if the authors give recipe of choosing maximum bandwidth.
summary_of_the_review: The theoretical contribution is not much as most of the method part can be put into appendix and very well-known. The experiments are proof of concept type, and the paper is essentially using well-known basis for functions of manifold, without giving any recipe for choosing frequency or maximum bandwidth L. The contribution is pretty limited as it's essentially using well-known basis in several applications.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper generalizes the positional encoding with Fourier features to non-Euclidean manifolds, which has the rotation invariance for feature extraction. The approximates convolutions on the manifold, according to the neural tangent kernel (NTK) assumptions, are shift-invariant.
main_review: ## Strength
The paper provides abundant proof for explaining the key properties of the proposed method. Various empirical study has been provided with illustration to demonstrate the advantage of the method. 

## Weakness
1. The neural network model is not very clearly shown though the main claimed contribution is for the positional encoding. 
2. The link of the proposed model with the NTK is well described. Actually, there is no big evidence provided by the authors about how the theory and model design are connected with the NTK theory.
3. Since the authors mentioned in Section.6 the importance of selecting an appropriate scale, it would be nice to find complete evidence to support this, i.e., a sensitivity analysis. 
The underlying method took degrees up to 10 in the paper. I wonder if this choice is enough for most applications, or the value is only restricted by the computational cost? If its the first case, can the authors provide some justifications on it? Otherwise, how bad influence would it have when the required degree is much larger than the model can provide?
4. In numerical parts, the authors should include more the baselines for comparison.
5. The paper writing could be further improved. Some examples include "positional encoding is crucial to achieve (achieving) photorealism" (Section 2); "A direction can be associated to (with)..." (Section 3).
summary_of_the_review: The theoretical basis of this paper is well supported, but to what degree this method is of practical needs further justification. The whole model of the paper with the proposed positional encoding method is yet to be well stated. The link with the NTK of the proposed method is not evident to readers and may not be the truth as the authors mentioned.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a positional encoding scheme on manifolds for coordinate-based learning, which achieves better performance than Euclidean coordinate encoding.
main_review: Strengths:
- For manifold data, it makes sense to use an encoding that adapts to the underlying manifold structure rather than the extrinsic Euclidean coordinates. The submission makes some contribution in this direction;
- Performance improvement verified in the examples;
- The paper discusses multiple application scenarios where the data can be encoded as spheres or their products. 

Weakness:
- The title appears to suggest learning on general manifolds, but the actual discussion is focused on manifolds with a spherical structure. Does that mean there will be some manifolds where the proposed approach is not applicable? In particular, does this approach work for manifolds of arbitrary topology? If not, then the title can be slightly misleading and should be revised.
- It is true that the Laplace-Beltrami operator can be used to derive an orthonormal basis on Riemannian manifolds. Unfortunately, there is no further discussion for such cases beyond this general statement. Can the author(s) provide some specific examples of how this should be used?
- The paper confirms that computing the proposed encodings can be expensive. Can the author(s) quantify the increase in computational cost? This should give a more complete picture of the tradeoff between speed and accuracy.
summary_of_the_review: The paper makes a contribution towards a more general positional encoding scheme on manifolds. There needs to be more clarification on the applicability of the method on arbitrary manifolds, as well as the computational cost.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper generalizes the commonly used sinusoidal position encoding scheme (such as that in NerF input encoding) to inputs naturally residing in non-Euclidean manifold.

This is achieved by representing the input coordinates as  projections on  alternative sets of orthornormal bases instead of the trivial Euclidean coordinates.   The authors argue that by doing so, they extend the translational invariance of Fourier features in Euclidean space to non-Euclidian manifolds, resulting in a convolution-like operator that is invariant to the 'natural shift' on these manifolds.  These new position encoding techniques are tested for several tasks and seem to get consistent improvemnts over their Fourier counterparts.

main_review: I find the paper is well written, and the experiments seem to be convincing.  However, I have some major concerns. 

- First, the author motivated their work (i.e. this paper)  by citing that the original NerF paper used 3-dimensional direction to describe the incoming light direction, hence suffer from over-parameterization. Similarly,  it used the same argument to against the iPDF paper.    This is not true.  

More importantly,  it is not clear why achieving "shift-invariance" , "minimal parameterization" or even the orthonomality property matters for the purpose of positional encoding.   The purpose for the common sinusoidal positional encoding is to guide the learning process to move away from the "spectral bias" and to make the learning of high frequency information easier.   Even if the inputs are not parameterized minimally or not shift-invariant, as long as the  basis functions representation contains high frequency fluctuations the above purpose can be met.  

 Of course, one may argue that having a proper choice of orthonormal bases would result in some convolution-like behaviour but I don't understand why this is relevant at this stage of positional encoding.   Conversely, one can still use non-orthonormal basis for encoding (e.g. for Fourier features, multiple all  \sin functions by a factor of 2 and leave the \cos functions the same as before, or add identity mapping to basis functions), yet it will not necessarily lead to a worse performance.    As far as the positional encoding is concerned, sometimes having non-minimal redundant and even duplicating information ( lifting) can  be beneficial.   After all, what it matters is whether this can lessen the Spectral Bias of the deep neural network.   The new "theory" developed in this paper is nice to have, but isn't necessarily practically relevant. 

Secondly, The paper only includes three  special (and simple) instances non-Euclidean manifolds.   The three example manifolds (sphere, rotation, and product of sphere) are similar and special.  Their natural shifts on manifolds are rotation, and can be almost trivially obtained.   Their Fourier series have been studied by previous work. In other words, these results are well known and not novel.  What about other manifolds, where natural shifts are complex, and Fourier series are unknown ?  For a more generic non-Euclidean manifold it is not clear whether or how a `natural shift' can be defined/derived.   This is though a secondary point.  I am not convinced the work is well motivated, for my first point above.   

summary_of_the_review: 
Strength: 
1. This paper presents an interesting idea attempting to extend positional encoding to non-Euclidean manifolds.
2.   This paper provides experiments for three selected manifolds.

Weakness: 
The motivation of the proposed work is unclear (see comments above). Moreover, some further arguments are in order. 
Experiment provided only show that the selected base functions have similar properties of Euclidean encoding, but that doesn't explain why the selection of base functions can help extract details. Are they really better than  using sinusoids ? Is it really important to design base functions based on the principle? Take sinusoids as example, if we multiply a constant before sine bases and keep cosine bases still, we break the constant-norm property, but the network can still learn from these inputs and mitigate the spectral bias. The increase of performance is not very obvious compared with Euclidean encoding (0.5-1 PSNR increase). Experiments of how basis elements (such as max frequency) affect the performance is not found.
review_title: null

Meta-review: Positional encoding of the input coordinates using Fourier basis [as described in (1)] is a common tool in the context of multilayer perceptrons (MLP). The author propose to replace the Fourier basis with one on manifolds M (2), such as the classical spherical harmonics (M=S^2), the Fourier basis on M=SO(3) or on M=S^2 x S^2.

MLPs form an important tool of our times. Unfortunately, as it is elaborated by the reviewers the Fourier basis of the investigated manifolds are widely-studied and the presented results are well-known; the submission lacks novelty.
Decision: Reject