SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization
Authors: Amir Hertz
Or Perel
Raja Giryes
Olga Sorkine-hornung
Daniel Cohen-Or
Venue: NeurIPS 2021 Poster (2021)
Keywords: Machine Learning
Deep Learning
Computer Vision
Implicit functions
Positional Encoding

Abstract: Multilayer-perceptrons (MLP)  are known to struggle learning functions of high-frequencies, and in particular, instances of wide frequency bands.
We present a progressive mapping scheme for input signals of MLP networks, enabling them to better fit a wide range of frequencies without sacrificing training stability or requiring any domain specific preprocessing. 
We introduce Spatially Adaptive Progressive Encoding (SAPE) layers, which gradually unmask signal components with increasing frequencies as a function of time and space. The progressive exposure of frequencies is monitored by a feedback loop throughout the neural optimization process, allowing changes to propagate at different rates among local spatial portions of the signal space. We demonstrate the advantage of our method on variety of domains and applications: regression of low dimensional signals and images, representation learning of occupancy networks, and a geometric task of mesh transfer between 3D shapes.

SAPE: Spatially-Aapttive Progressive Encoding for Neural Optimization
Amir Hertz
Or Perel
Tel Aviv University amirhertz@mail. tau. ac i1
Tel Aviv University orr www.ppeee@gma.acomm com
Raja Giryes
Olga Sorkine-Hornung ETH Zurich, Switzerland sorkinešinf. ethz.cl ch
Daniel Cohen-Or Tel Aviv University
Tel Aviv University raja@tauex. tau. ac.
cohenor@gmail.co com
Abstract
Multilayer-perreetroom (MLP) are known to struggle with learning functions of high-frequencies, and in particular cases with wide frequency bands. We present a spatially adaptive progressive encoding (SAPE) scheme for input signals of MLP networks, which enables them to better fit wide range of frequencies without sacrificing training stability or requiring any domain specific preprocessing. SAPE gradually unmasks signal components with increasing frequencies as a function of time and space. The progressive exposure of frequencies is monitored by a feedback loop throughout the neural optimization process, allowing changes to propagate at different rates among local spatial portions of the signal space. We demonstrate the advantage of SAPE on variety of domains and applications, including regression low dimensional signals and images, representation learning of occupancy networks, and geometric task of mesh transfer between 3D shapes.
Introduction
Neural implicit functions have recently emerged as powerful representation paradigm for modeling complex signals. Their continuous nature them apart from typical discrete representations (i.e. pixels, voxels, meshes), allowing to capture high resolution details in various domains such as images 41 43
3d shapes [31 6) and radiance fields 30 32), while retaining reasonably compact representation. In this formulation, a deep neural network is trained with the goal of faithfully mapping input coordinates to a corresponding target domain, effectively learning the representation of signal properties such as magnitude, color, or shape occupancy.
Implementing implicit neural representations with common neural structures, e.g., multilayer perecep- trons with ReLU activations (ReLU MLPs), proves to be challenging in the presence of signals with high frequencies. Consequently, recent works have demonstrated that deep implicit networks benefit from mapping the input coordinates 30 [ or the intermediate features [I| to positional encodings. That is, before feeding them into neural layer, they first transformed to overparameterizee. high dimensional space, typically by multiple periodic functions.
Positional encodings have been shown to enable highly detailed mappings of signals by MLP networks. For example, Fourier Feature Networks 43 suggested to map input coordinates signals to high dimensional space using sinusoidal functions. In their work, they show that the frequency
In this paper, methods that map coordinates higher dimensional space. Not confused with the term "Positional Encoding" coined by 30 43 which refers to a particular mapping scheme this family.
the term "positional encodings" in lower case letters denote the family of encoding
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

FFN =
FFN == 25
SAPE
Figure I: Left: Fourier Features Network (FFN) encoding, tuned to bandwidth of low frequencies in order to fit the smooth snow areas. The same frequency bandwidth yields blurry buildings in the top image. Middle: FFN tuned to a higher bandwidth to fit to the sharp details of the city. The same bandwidth results in the appearance of noisy artifacts in the mountain image. Right: SAPE is able to fit both examples without extra tuning, using the same choice of frequency bandwidth in both cases.
larget Function
emcoding
FEN
progressive encoding
spatially aduptive progressive encrding
Figure 2: ID signal regression. Red: Samples positional coordinates as network input, and signal magnitude labels. Black: Predicted implicit signal at inference time. MLPs with "no encoding" struggle to fit high frequency segments (see appendix for train details). Efforts of static positional encoding models (FFN) to fit high frequency areas of the signal introduce noise the low frequency region. Fitting varying signals with spatial adaptivity allows MLPs to recover all signal frequencies.
of these sinusoidal encodings is the dominant factor in obtaining high quality signal reconstructions. In particular, they present compelling arguments for randomly sampling frequency values from an isotropic Gaussian distribution with a carefully selected scale, providing striking improvement over mapping coordinates directly via standard MLPs.
Despite the success of positional encodings, there are still concerns left unaddressed: (i) Choos- ing the right frequency scale requires manual tuning, oftentimes involving a tedious parameter sweep; (ii) The frequency distribution scale may change between different inputs, and accordingly it becomes harder to tune a "one-fits fone-fits-all" model for signals that are composed of a large range of frequencies (Fig. I (iii) Frequencies are selected for the entire input in global, spatially invariant manner, thus missing an opportunity to better adapt to local high frequencies (Fig. 2).
Our work investigates mitigations to the aforementioned challenges. We study the setting of positional encodings as input to implicit neural networks and present -paapally-Adaative Progressive Encoding (SAPE). SAPE is policy for learning implicit functions, relying on two core ideas: (i) guiding the neural optimization process by gradually unmasking signal components with increasing frequencies over time, and (ii) allowing the mask progression to propagate at different rates among local spatial portions of the signal space. To govern this process, we introduce a feedbuck loop to control the progression of revealed encoding frequencies as bi-variate function of time and space.
Our work enables MLP networks to adaptively fit a varying spectrum of fine details that previous methods struggle to capture in single shot, without involved tuning of parameters or domain specific preprocessing. SAPE excels in learning implicit functions with a large Lipschitz constant, without sacrificing quality of details or optimization stability, in problems that require meticulous configuration to achieve convergence. To highlight the latter, in Section we present the tasks of 2D silhouettes deformation and 3D mesh transfer both require stable optimization from the get-go in order to avoid convergence to sub-optimal local minima.
SAPE is encoding-agnostic: it is not limited to specific positional encoding type. can be easily applied to the learning process of coordinate-basee based neural implicit functions of various domains including images, 2D shapes, 3D occupancy maps and surfaces, showing improvement in all of them.

---

trequence
P
E
feadback
Figure 3: Method overview. Coordinates p are fed into an implicit function network, regressing the signal value ý at position output. An encoder E maps the input coordinates to high dimensional embedding space. Our encoding layer then masks the encoded features. In this example, samples p encoded and progressively exposed during the network training, going from low frequency (top node) to high (bottom node). Node colors indicate the various neuron states: on, off, and partially masked. Finally, the loss between output and target y is fed back to encoder E, which spatially adapts the encoding mask according to the spatial error, indicated by the heatmap on the pufferfish.
Related work
Implicit neural representations. Recent works show the capability of deep neural networks in
representing different functions (e.g.. 2D/3D scenes or objects) as an implicit, memory efficient continuous function
[8]. These networks are used as a signed distance function (SDF) [5] or
occupancy network, either binary 66 28 33 soft volumetric density [26] 30 49].
Several works train such networks to represent collection of 3D shapes via 3D data supervision reconstruct 3D shapes 5
6
16
29
28 31
13 14 18
42 or infer them from 2D images
22
Recent works employ spatial data structures scale the represented shapes size 25 42
Positional encodings (PE), have been suggested as higher dimensional network input for various purposes. Radial basis function (RBF) networks (7] use weighted sums of embedded RBF encodings due to their symmetric property. 47 used random Fourier Features to map time spans. In natural language processing, Transformers 19
46) leverage sinusoidal positional encoding to maintain the order of token sequences. Our work differs by focusing on how to encode the inputs to MLPs in order improve network implicit representations.
24
The closest works to ours are SIREN 41 and Fourier feature networks (FFN to replace the ReLU activations in the network by periodic ones. FFN 4 the network by projecting them to a higher dimensional space using a family of encoding functions. In the appendix provide an illustration of this approach with some examplary used encodings. The analysis of 43 demonstrates that FFN improves the learning of implicit neural representation compared other alternatives. follow-up work in 44 accelerates the training convergence by using a meta-learned initialization of the network weights. The contemporary work of [27] extends SIREN with a modulation network, to allow representing multiple shapes within single network.
SIREN suggests
encodes the inputs to
Concurrently to our work. Park et al. [32] extend NERF to non-rigidly deforming scenes, and BARF 21 extends NERF for cases of imperfect camera poses. Both these works show the advantage of employing coarse-to- to-rse-to-fine annealing linearly over the frequency bandwidth of the positional encoding. The coarse-to- fine approach bears similarity Different from us, these works do not use a feedback loop or spatial encoding, which, as we show in the following, further closes the gap between the regressed function and the ground truth.
progressive frequency approach.
As final remark, SAPE's concept feedback loop resonates with the idea "predictive coding" in computational neuroscience [35] suggesting another angle appreciate the rationale of our method.

---

SAPE
Iteraiinns
Figure 4: Learning progress of SAPE for implicit representation of a 2D image, showing iterations 75, 150, 300 and 3000 (left to right). The beginning of optimization is dominated by low frequencies: spectral bias maintains global optimization stability. As the optimization progresses high frequency mapping takes place, and SAPE fits the finer details of the image. See other methods in the appendix.
3 Preliminaries
The setting of implicit neural representation is commonly formulated as learning mapping function fe. using neural network, from an imput of low dimensional position coordinates p E IR to an output value in the target domain. The network training data are coordinates samples as the input and an expected value of the function, or signal, as the output. Examples of such mappings include pixel locations to intensity values for images, or 3D space coordinates to binary surface occupancy labels.
Achieving such mappings using conventional neural networks with ReLU is hard. It has been shown that ReLU networks exhibit a behaviour of spectral bias [34] 3 36
2 48 4 Networks tend to learn low frequency functions first, as they are characterized by global behaviour, and are therefore more stable to optimize. Spectral bias, however, also prevents networks from properly learning to fit functions that change at high rate, e-g., functions with large Lipschitz constant. When visual domains are concerned, this is mostly evident in delicate details missing from the network output.
To mitigate this deficiency, recent works proposed replace the ReLU activation layers with periodic sine functions [41 or map the input to some higher dimensional space in order to learn mapping with high frequencies 30 43 In the latter approach, the input p is encoded high dimensional embedding by a family of functionals e₁ RR + R, such that:
Epp) (ei (2
where usually n d. Tancik et al.
43 for example, suggest the following encoding:
EFeN (p) = (20,(2 p). sin(2 p),... (2 p). sin(2Tb,,,)))
(2)
where b, are frequency vectors randomly sampled i..d. from Gaussian distribution with standard deviation a. Other examples of positional encodings are included in the appendix.
These embedding schemes facilitate learning of complex functions, at the price of introducing second associated phenomenon: the spectral bias characteristic of the network is reduced. The implication in this case is twofold. Positional encodings can cause neural optimizations processes to become unstable, and consequently converge to a bad local minimum (Fig. D In addition, when fitting functions of varying levels of details/frequencies,, neurons predicting smooth, low frequency areas are still exposed to encoding dimensions high frequency. That, in turn, may complicate the learning process, networks have to learn where to ignore such embedding dimensions.
Indeed, Tancik et al. [43 advocate careful choice the standard deviation of frequency distribution, in their method, showing that using low values yields missing details in the network output, and using values that are very high results in noisy artifacts. Thereupon, for such static encodings the value of requires a parameter sweep per sample.
4 Spatially-Adaptin Progressive Encoding (SAPE)
Our proposed approach, SAPE, is policy for guiding the optimization of implicit neural represen- tations based on input coordinates. SAPE relies on the delicate balance of the two phenomena that govern the learning process: reconciles the effects of the spectral bias and the expressiveness of high frequency encodings in manner that benefits from both. It maintains a stable optimization, without sacrificing the ability to fine signal details. SAPE is composed of two key components: progressive encoding and spatial adaptivity, which allow it to be less sensitive to the encoding frequency bandwidth, i.e., the choice of the standard deviation the case of 43 (Fig. 4. Next, we detail each of the mechanisms that compose SAPE. The full algorithm is included in the appendix.

---

Source Figure 5: Mesh transfer. In this task, we transfer the tessellation of source mesh to target shape (left columns). We estimate an initial transformation using user specified correspondence points (marked in red). Then, we follow with neural optimization tuning process, which brings the source mesh surface to the target shape while keeping the quality of the original tessellation.
Target
No encoding
Sizen
SAPE
Without loss of generality, in our description below we assume the encoding functionals are sorted by their respective Lipschitz constants (LC), i.e. LC((e) I LC(e2) S .... S LC(en). For example, we sort the Fourier features encoding by the frequency value |b:ll- We set the first d dimensions to the identity encodings: (P) = Pt, Vi I d, thereby exposing the network the original input position coordinates p E R as well.
Progressive encoding. A core part of SAPE is progressively fitting the frequencies. The first layer of the network encodes an input position p with set of functionals e,. In order to only part of the functionals at different stages of training. it multiplies the encoding dimensions with a soft mask, which progresses function of the optimization iteration
Epo (p, (eo(p, t), (p, @g( .... (p. a(t))(()).
(3)
where (p, t) = (t) (p) is the progression control for the encoding functional et, regulated by the elements masking vector a+() E [0, 1). During the optimization, we progressively reveal encoding features in manner that allows the encoding neurons to be tuned to different states according to the value of (t), where (t) corresponds to the state on, ax 0 means off and 0 < (t) < denotes partially (Fig.
We define a policy & that controls the progression of the mask vector a(t) by a(t + 1) @ (0)] In this work, we use the following progression rule for the mask of the ith encoding dimension at time step t:
t- T (i d) 0, 1). if d
clamp
Φr [a(()) as(t 11)
(4)
1,
otherwise,
where T/2n represents the number of iterations each encoding dimension takes to progress from 0 to and T' is the maximal number of iterations in the optimization. Essentially, &p performs a linear progression sequentially for each encoding i until it is fully exposed, s.t. (1) 1. When certain encoding dimension mask achieves saturation, we continue to the following one. We allow a technical exception this rule, for proper progression of correlated masks of encoding dimensions sharing the same Lipschitz constant, c.g., pairs of cos, sin in the encoding in Eq. (2) For initialization, the first d positional encoding masks are fully exposed right from the beginning, and in our setting, correspond with the identity encoding functionals.
Notice that previous methods, which introduce all encoding dimensions to the network at once, can now be formulated as a specific case in our framework, where ax (((+1)
Spatial adaptivity. When / > T/2, the progression policy &p achieves global saturation in terms of revealing encoding dimensions, so that the masking vector becomes a(t) = 1. For signals of global- characteristics, c.g., have a low LC', the network is compelled to learn how to cope with undesirable encoding dimensions high frequency to regress a smooth, slowly changing signal. This can lead to sub-optimal outputs, often visible in the form of visual artifacts (Fig. mid-bottom).
Given loss function and convergence threshold E, simple improvement to policy pp may apply early stopping when L < E. However, for signals with varying levels of detail as a function of spatial position p. this improvement not sufficient: early stopping does not oceur due to areas characterized by high frequencies. Relying a global threshold is inherently sub-optimal decision, as when learning an implicit neural function with positional encoding, different spatial segments of the signal in question may converge at different rates (Fig.
5

---

To solve this problem, we set the mask vector to be spatially sensitive: att, p). By design, the new policy PSA progresses the encoding of each spatial location p separately, per encoding dimension i. To simplify our explanation, in the current setting we assume the signal can be discretized to distinct spatial segments on a regular grid, each tracked independently (e.g.. pixels, voxels). The progressive, spatially adaptive policy is controlled by feedback loop, where the independent regression loss of each signal segment is used to control the progression:
(p((a)))) if C(t, 2
SSa (ar(t, p)]: alt +
aa(t. p). otherwise
(5)
For implicit functions, inference time we assume signals can be regressed with "pseudo-continuous"
coordinates p. Therefore, to support coordinates that did not appear during training and have no mask recordings, we extend the estimated parameters of (T, p) continuously over the entire input domain by linear interpolation.
One common example that benefits from spatial adaptivity is natural 2D image containing blurry, out-of-focuu areas in the background together with sharp, detailed foreground objects. To shed more light on the behaviour of our algorithm, Figs. [ and demonstrate examples of spatial heat maps, highlighting the maximal encoding frequency achieved per spatial location, upon convergence.
Sparse grid sampling. We now extend our method to cases where model fo is optimized by coordinates p that do not lie on a regular grid, or when it is simply infeasible to store parameters (t,, p) per sampled coordinate p during training. In this setting, we discretize the input domain and store parameters Qu in a sparse grid G. where for grid of resolution rd C Rd,, denote u the multidimensional grid coordinates. In the forward pass, we ohtain the original encoding mask ax(,, p) per sampled point p by encoding linear interpolation of the masking parameters over the nearest grid nodes (see the inset (a) on the left):
Forward
alt, p) = Σ Wpnn Oxt, u),
(6)
Loss feedback
uEN
where (p) the neighboring grid coordinates in the vicinity of p. and Wpp.u are the interpolation weights for a sample over the multidimensional grid G such that EuENa Wp,u = 1. During
training, the loss L(I, u) at time / for a given grid point u is accumulated over all sampled points affected by (inset (b) above):
C(t, u)
Ep up.u Σ up C(t.
1
Here, L(t, p) is the loss for the original coordinate To obtain the overall training loss, we sum over all the used points for training time t. We note that the linear interpolation used during the forward pass has an added benefit: since only local neighbors participate per update of the original coordinate p. sparse weights allow for efficient accumulations during forward and backward pass updates.
5 Experiments
We evaluate SAPE on a variety of common 2D and 3D regressions tasks. addition, we demon- strate how it may be used to improve the tasks of deforming 2D silhouettes and transferring mesh connectivity between 3D shapes. We demonstrate our results using the Fourier feature encoding 43 (Eq. 2 We emphasize that SAPE is agnostic to the encoding used and applicable to other mapping schemes as well. More examples, as well as the full implementation details appear in the appendix.
5.1 Evaluations
We test SAPE on variety of problems: regression tasks optimized by a direct supervision and geometric tasks optimized by an indirect supervision. All configurations employ 256 unique frequency encodings sampled from a Gaussian distribution. For fair evaluation, the distribution scale and number of neural layers remain the same for all inputs the same problem setting. However, hyperparameter values may vary per problem. For convergence threshold s, we set the values of 1e-3 for regression tasks and le-2 2 for geometric tasks. See the appendixfor full description of implementation details. We compare the settings of 6 MLP configurations:

---

Table 1: Quantitative Evaluation of SAPE against baseline encoded MLPs on various tasks.
Best results displayed in bold, - accounts for deviation among independent experiments. 2D silbooettee (loU)
2D regression (PSNR)
3D occupancy (loU)
Natural images
Text images
Thingi0K
Turbosquid
MPEG7 (loU)
No Encoding
19.72 + 2.77
19.39 2.02
0.899 +: 0.031
0,872 + 0.08
0,81 + 0.177
STREN
27.03
+ 4.28
30.81
1.72
0.964
0.93
0.779 : 0.253
RBFG
25.98 + 3.98
25.91 + 4.03
0.9.1 + 0.022
0.919 = 0.044
0.851 = 0.15
SAPE RBFG
27.06
3.94
28.52
0,966 0.015
0.94 0.03
0.102
FF
25.57
1 4.19
30.47
1 2.11
0.945 1 0.03
0.964 1 0.017
0.873 1 0.142
Progressive
27.01
+: 3.56
30.2
1.83
0.98 4: 0.018
0.943 0.019
0.86 - 0.133
SAPE FF
28.09
= 4.04
31.84
+ 2.15
0.981 + 0.008
0.969 = 0.013
0.928 = 0.095
1) No encoding: Basic ReLU MLP without en- coding. 2) SIREN: An MLP with sine activa- tions based on the implementation of Sitzmann et al. [4]] 3) RBF-grid: An MLP with repeated radial basis function as a first encoding layer. 4) SAPE + RBF-grid. 5) FFN: An MLP network with Fourier features as the first encoding layer, see Eq. 2 6) SAPE + FFN.
Table 2: Evaluation on the mesh transfer task.
Chamfer (L)
Hausdorff (U)
Dirichlet (L)
No Encoding
1.37
1.36
1.3
SIREN
0.74
1.32
4.75
RBFG
0.97
0.87
1.65
SAPE RBFG
0.55
0.71
1.65
HF
1.27
0.94
2.22
SAPE
0.39
0.49
1.82
The bandwidths of encoding functions in 3) and 5) are optimally selected by a grid search over validation set or taken from a public implementation, depending on the task. For the SAPE variants we use double the a of these bandwidths, SAPE is not sensitive a particular value, as long as allows high frequency encodings. The quantitative results are summarized in Tables 1 and 2 Below an overview of each task.
2D image regression. In this task we optimize the networks to map 2D input pixel coordinates, normalized to [-1, 1]2, their RGB values. We conduct the evaluation on the same test sets as Tancik et al. 43 which contain a dataset of natural images and a synthetic dataset of text images. Similar to them, the network is trained on regularly spaced grid containing 25% of the pixels. We the evaluation metric of PSNR over the entire image, compared to the ground truth image. Quantitative results are reported in Table D on the left.
3D occupancy, In this task, we use a similar setting to occupancy networks 28 : the model is trained to classify input 3D coordinate for being inside or outside the training shape. For training, sample 9 million points divided into 3 equal groups: uniformly sampled points in [-1, 1]3, surface points perturbed with random Gaussian noise vectors using 0.1 and o 0.01. We evaluate the quality of the result by estimating the intersection-over--nioo (IoU) with respect to the ground truth shape by sampling additional 1e6 random points and 1e6 more challenging samples near the surface, and report the average IoU score.
We compare the networks on two test sets The first set is composed of 10 selected models from the Thingi dataset [50). The second, more challenging test set is composed of 4 models from TurboSquid] 2 The quantitative results are summarized in Table I middle section. Qualitative results are shown in the appendix.
2D silhouettes. Here, we optimize an MLP to deform a unit circle represented as a polyline to a target 2D point cloud of silhouette shape. Fig. 9 shows number of examples of such target shapes. We start by calibrating the MIP to learn simple mapping from [o, 1] to the 2D unit circle: (p) = (cos(2 sin (2xp)). We then optimize the mapping function by minimizing the symmetric chamfer distance between the network output and the silhouette.
To evaluate the performance of the different methods, we test the networks on 20 shapes from the MPEF7 dataset (20) and measure the intersection over union between the resulting shape and the target shape; the results are reported in Table [ Fig. 9 shows qualitative results of this task and snapshots from the optimization process. Due to the expressiveness of both FFN and SIREN networks, the chamfer loss causes distortions in the early stage of the optimization that cannot be recovered. leading to undesired results. The coarse-to-fine optimization of SAPE allows both avoiding distortions and matching high frequencies of the target shape.
https:: httpp://www.turbbsuuu...i turbosquid.com
7

---

FFN
SAPE 17 20
20
40
40
Figure 6: 2D Image regression comparing different distributions of bandwidths for Fourier Frequen- cies. SAPE is more robust to the choice of a, consistently producing pleasing results in terms of both PSNR and visual quality.
PSNR
PSNR
SAPE
Stamdard deviation of |bbll
Spatial grid resolution
Figure 7: Left: PSNR as function of the Fourier features' standard deviation of the encoding for the example in Fig. 6 Right: PSNR as a function of spatial grid resolution for the example in Fig. g
3D mesh transfer. In this task, we would like to transfer a 3D source mesh to target shape 45 9
38 39,, which is represented by mesh or a point cloud. The MLP receives the vertices of the source mesh and outputs transformed vertices, such that together with the source tessellation, the optimized mesh fits the target shape while respecting the structure of the source mesh in terms of distortion. In addition, we may utilize a set of marked correspondence points between the input shapes that enable the estimation of an initial affine transformation between the source and the target, followed by biharmonic deformation [17] The optimization loss for this task is composed of two terms: A distance loss that measures the symmetric chamfer distance between the optimized mesh and the target shape, and a structural term that measures the discrete conformal energy between the optimized mesh and the source mesh.
We test the different networks on 6 pairs of meshes and evaluate the results by measuring the chamfer and Hausdorff distances between the target shape and the output mesh. The distortion the transferred meshing is measured by Dirichlet energy with respect to the source mesh. Table shows the quantitative results. Qualitative results are shown Fig.
Standard MLPs struggle to fit the target shape and remain close to the source mesh therefore obtaining low Dirichlet energy but also high Chamfer and Hausdorff distances and thus failing the task. SAPE's feedback specifies the regions on the mesh that are distanced from the target shape. That allows the optimization to gradually increase the frequencies used in high curvature areas while avoiding large distortions at the beginning of the optimization, when global deformations take place. For that reason, contrary other methods such FFN and SIREN, SAPE is able avoid solutions of bad local minima and reconstruct the target signal better.
162
642
256
Figure 8: Influence of the spatial grid resolution output quality. Top row: Output of SAPE with grid resolutions of 12, 42, 162, 64 and 256 Bottom left: Interpolated heatmap of maximal frequency unmasked per grid position. Bottom right: Enlarged results of area with range of frequencies.

---

No encoding
Siren
HIN
Iaget shape
encoding
Figure 9: Iterative deformation of 2D silhouettes. Left: comparison of different network configu-
rations when optimizing for the deformation of unit circle to the target shape, represented by polyline. Right: snapshots of the optimization process. Notice that SAPE (bottom) avoids large distortion at the beginning of the optimization, and can therefore fit the delicate details of the target shape as the optimization progresses.
0
Training scene
PEN
FFN
FEN
SAPE 20
Ground truth
SAPE heat map
16
Figure 10: 3D occupancy scene regression. By spatially adapting the volumetric encoding for the occupancy network, SAPE fit small details such as the wheels without introducing noise in smooth or empty regions. To the right, we display cross sections of the 3D heat map SAPE. Notice that smooth surfaces, like the the trees and the roof, utilize lower frequency encodings compared to detailed objects like the car and the slide.
5.2 Ablation
Progressive non-progressive. Similar to the setting described in Section 5.1 train MLP on randomly sampled pixels of images. We map 2D pixel coordinates to RGB values and measure the PSNR over all pixels compared the ground truth image.
Fig. and Fig. 7 (left) show a comparison between the performance of FFN with and without SAPE when training on 25% of the pixels. Our results show that due to progressive-spatial adjustment of the encoding, SAPE is less sensitive to tuning of the standard deviation (a) of distribution of frequencies b; By contrast, FFN suffers from fragile trade-off between underfitting and overfitting, which results in either blurry pixels in high frequency regions or noise in the smooth background areas.
A similar phenomena is shown in the 3D example in Fig- TO In this task, we train an occupancy network using the setting described in [.11 For regulating the level of encoding across the scene our method based on feedback loss during training. Without SAPE, low frequency encodings cannot represent detailed structures like the car object in the scene. Increasing the frequency bandwidth results in noisy surfaces and undesired blob artifacts in empty spaces. By contrast, SAPE achieves better representation of detailed regions, well as of smooth surfaces.
voxel map of resolution 1283 for spatial encoding. This map is updated
Spatial progressive vs. non-spatial progressive. To demonstrate the advantage of using spatially adaptive encoding, compare it to variant of SAPE that globally progresses the frequency level of encodings for all spatial areas equally, and converges when all training samples fit. Fig. shows the ad advantage of using the spatial encoding in SAPE for ID regression task. In addition, Table

---

shows quantitative improvement gained by SAPE applied on FF, compared to global progression (Progressive FF).
Note that the non-spatial variant is equivalent to SAPE with a spatial grid of resolution one. Fig. B and Fig- 7 (right) show the benefit of the spatial encoding component for 2D image regression task (similar to the one above). We also show how increasing the grid resolution affects the quality of the result 2D image regression task. We observe that the quality stays at its highest when the spatial grid resolution approximates the sampling ratio of coordinates during training.
6 Limitations and Future Work
When SAPE is employed with sampling grid, the chosen resolution provides a trade-off between memory and quality. Choosing a grid of low resolution may yield sub-optimal outputs (see examples in the appendix). As SAPE's varying frequencies sampling grid bears some resemblance to multi- resolution analysis and wavelet transforms, future extensions to SAPE may solve this issue by maintaining the masking parameters in a sparse structure such an Octree 42 GMM [10].
When using very high frequencies, SAPE may enhance inherent output noise, as the maximal frequency encoding gets picked for the noisy portions of the signal.
In this work we investigated linear progression policy for unmasking frequency encodings. Followup works may extend our scheme to a more optimized curriculum, for example, using meta-learning.
7 Conclusions
We presented a policy for improving the quality of signals learned by coordinate based optimizations. We rely on two major contributions: progressive introduction of positional encodings, and spatial adaptivity, which allows different rates of progression per signal location. Our method is simple to implement and improves the implicit neural representation result of MLP networks in various tasks.
Given the surge of applications that use positional encodings, and the ease of deploying SAPE, we believe our approach will be useful for many problems in the vision and graphics domains. Specif- ically, the fact that SAPE is mncoiin--agnostic and insensitive to most encoding hyperparameters provides stable training and facilitates the use of positional encodings in novel tasks.
Broader Impact Statement
SAPE is method for boosting the accuracy of learned implicit neural functions. can be leveraged to enable new applications or improve various downstream tasks that involve precise processing of signals, like segmentation, or reconstruction of complex signals, such as streaming video or signed distanced functions. Beyond the domains of vision and graphics, we conjecture SAPE might also benefit applications in speech, audio, and signal processing general. Depending on their use, all mentioned applications may have positive or negative impact. SAPE can be used to facilitate novel applications that require accurate recovery signals, like 3D reconstruction for AR and VR. can also be misused for the purpose of generating fake media, including audio, video and images.
Acknowledgements
We would like to thank Hao (Richard) Zhang. Hadar Averbuch-Elor and Rinon Gal for their insightful comments, and the anonymous reviewers for their helpful remarks. This work was supported in part by the Israel Science Foundation grant no. 2492/20 and by the ERC-StG grant no. 757497 (SPADE).
References
[1] Matan Atzmon and Yaron Lipman. SAL: sign agnostic learning of shapes from raw data. In 2020 IEEF/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2562-2571. IEEF, 2020. doi: 10. 109/CVP442000 20...022202222
URL https: /oi. org/ 10. CVOR220002220 2020.00264
10

---

[2] Oshrat Bar, Amnon Drory, and Raja Giryes. A spectral perspective of neural networks robustness label noise, 2021.
[3] Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Advances in Neural Information Processing Systems, volume 32, 2019.
[4] Yuan Cao, Zhiying Fang. Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. International Joint Conference on Artificial Intelligence, (IJCAI), pages 2205-2211, 8 2021.
[5] Rohan Chabra, Jan E Lenssen, Eddy lgg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, pages 508-62.. Springer, 2020.
[6] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[7] Ch Dash, Ajit Kumar Behera, Satchidananda Dehuri, and Sung-Bae Cho. Radial basis function neural networks: A topical of-te-of-the-art survey. Open Computer Science, 6, 01 2016. doi: 10.1515/comp-20166
2016-0005.
[8] Thomas Davies, Derek Nowrouzezahrai, and Alee Jacobson. On the effectiveness of weight- encoded neural implicit 3d shapes. 2021.
[9] Zhantao Deng, Jan Bednařík, Mathieu Salzmann, and Pascal Fua. Better patch stitching for parametric surface reconstruction. arXiv preprint arXiv:2010.0702/, 2020.
[10] Benjamin Eckart, Kihwan Kim, Alejandro Troccoli, Alonzo Kelly, and Jan Kautz. Accelerated generative models for 3d point cloud data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5497-5505, 2016.
[11] Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy J Mitra, and Michael Wimmer. Points2surf learning implicit surfaces from point clouds. In European Conference on Computer Vision, pages 108- 124. Springer, 2020.
[12] D. Ezuz, Heeren, O. Azencot, M. Rumpf, and M. Ben-Chen. Elastic correspondence between triangle meshes. Computer Graphics Forum, 18(2):121-134, 2019.
[13] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In International Conference on Computer Vision (ICCV), October 2019.
[14] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857-4866, 2020.
[15] Amos Gropp, Lior Yariy, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In International Conference on Machine Learning, volume 1 119, pages 3789-3799, 2020.
[16] Zekun Hao, Hadar Averbuch-Eloo Noah Snavely, and Serge Belongie. Dualsdf: Semantic shape manipulation using two-level representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
[17] Alec Jacobson, Ilya Baran, Jovan Popovic, and Olga Sorkine. Bounded biharmonic weights for real-time deformation. ACM Trans. Graph., 30(4), 2011.
[18] Chiyu "Max" Jiang. Avneesh Sud, Ameesh Makadia. Jingwei Huang, Matthias Niessner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[19] Guolin Ke, Di He, and Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations, 2021. URL, https::t/::// htttp:///peerevieee net/forum?idd09-- F2

---

[20] L.J. Latecki and R. Lakamper. Shape similarity measure based on correspondence of visual parts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(10):1185-1190 2000. doi: 10. 1 179/344478
[21] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. arXiv preprini arXiv:2 2021.
[22] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. arXiv preprint wrXivvI911.00720 2019.
[23] Xuanging Liu, Hsiang-Fu Yu, I. Dhillon, and Cho-Jui Hsich. Learning to encode position for transformer with continuous dynamical model. In JCML, 2020.
[24] Antoine Liutkus, Ondřej Cifka, Shih-Lun Wu, Umut Şimşekli, Yi-Hsuan Yang. and Gaël Richard. Relative positional encoding for transformers with linear complexity, 2021.
[25] Julien N.P. Martel, David B. Lindell, Connor Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural representation. ACM Trans. Graph. (SIGGRAPH), 2021.
[26] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[27] Ishit Mchta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Man- mohan Chandraker. Modulated periodic activations for generalizable local functional represen- tations, 2021.
[28] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[29] Mateusz Michalkiewicz, Jhony Kaesemodel Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations layers in neural networks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4742-4751, 2019. doi:
10.1109//CCV222199000M
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[31] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[32] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Deformable neural radiance fields. arXiv preprins arXiv:2011.12945 2020.
[33] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference Computer Vision (ECCV), 2020.
[34] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310 PMLR, 2019.
[35] Rajesh Rao and Dana Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-fiele effects. Nature neuroscience, 2:79-87, 02 1999. doi: 10. 1038448/4500.
[36] Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In NewrIPS, 2019.
12

---

[37] Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In The IEFE International Conference on Computer Vision (ICCV), October 2019.
[38] Patrick Schmidt, Janis Born, Marcel Campen, and Leif Kobbelt. Distortion-minimizii injective maps between surfaces. ACM Trans. Graph., 38(6), 2019.
[39] Patrick Schmidt, Marcel Campen, Janis Born, and Leif Kobbelt. Inter-surface maps via constant- curvature metrics. ACM Trans. Graph., 39(4), 2020.
[40] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations. In Advances in Neural Information Processing Systems, 2019.
[41] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[42] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[43] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Kan Keil,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
[44] Matthew Tancik, Ben Mildenhall, Terrance Wang. Divi Schmidt, Pratul P. Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate- neural representa- tions. In CVPR, 2021.
[45] Julien Tierny, Joel Daniels, Luis G. Nonato, Valerio Pascucci, and Claudio T. Silva. Inspired quadrangulation. Comput. Aided Des., 43(11):1516-1555 1526, November 2011.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL htttps://proceedinggs www.ediinnnnnnnesneeeeeeeeeeeeee www.neeuriii..m teuurips.cc//pppr/22////11 F Papeeiaaaaaaaaaaaapp..... pdf
[47] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. Advances Neural Information Processing Systems, pages 15889-15899, 2019.
[48] Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In ICONIP, 2019.
[49] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one few images. In CVPR, 2021.
[50] Qingnan Zhou and Alec Jacobson. Thingi10k: A dataset of 10,000 3d-printing models, arXiv preprint 10X1v11600.04797,200 2016.
13


Reviews:
[{"review_id": "w-tC-wzEdDa", "reviewer": "Reviewer_SEHH", "summary": "This article presents a way to perform image regression, by proposing a spatial encoding, which is spatially adaptive (it depends on the position in the image) and progressive (it increases the \"resolution\" we use to encode a point until it does not decrease the loss).\n\nThe complexity of the proposed method is acceptable regarding the results, which are significantly better than concurrent methods.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Detailed summary\n\nThis article presents a way to build a neural network (NN) $f : x \\mapsto f(x)$, where $x$ is the vector of coordinates of a point (in an image), and $f(x)$ is the color of the pixel at $x$. This is done by adding an encoding layer before the NN, which encodes each point $x$ by the value of its Fourier transform at different frequencies. The frequencies are chosen and added to the encoding by \"increasing order\" as long as it decreases the loss, capturing for each $x$ the \"resolution\" to chose. \n\nThis method is inspired by existing works (FFN), which makes it easy to understand.\n\n## Experiments\n\nThe method is tested on various tasks, in 2D and 3D. The main result are given in Table 1, where the proposed method SAPE appears to be the best-performing technique against concurrent ones (including \"no encoding\"). More information about experiments are in the \"limitations\" section.\n\nWhat is the aim of doing regression in a mesh transfer problem?\n\n## Clarity\n\nNo issue\n\n## Significance\n\nImage regression is a recent and studied area of medium importance. The proposed method is a significant improvement of existing methods.\n\n## Rebuttal\n\nI am mostly satisfied with authors' comments. I maintain my rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "## Limitations\n\nInfluence of the variance of the norms of \"frequency vectors\" $\\mathbf{b}$: done in Fig. 6. The choice of a precise variance is far less important in SAPE than in FFN. However, in the appendix, the variance of SAPE is always greater than for FFN and RBF. E.g. Section 5.5, why $\\sigma = 3$ for FFN and RBFG, but $\\sigma = 6$ for SAPE? Section 5.3: SAPE: $\\sigma = 20$; FFN: $\\sigma = 12$, so the rule of \"doubling the bandwidth\" (line 204) is not followed.\n\nInfluence of the grid resolution of SAPE: done in Fig.7. However, no image reconstructed with several grid sizes is shown. Is it possible to take the image of Fig. 10 (or Fig. 6), and reconstruct it with various grid sizes? It would be interesting to add such figure in the appendix.\n\n2D silhouette transfer: Fig. 8 in the article (example where SAPE performs well) comes with Fig. 9 in the appendix (example where SAPE is worse). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "MTWzRSfQRG", "reviewer": "Reviewer_69T8", "summary": "Implicit neural representations are a promising approach to learning compressed representations of data --- for example they can learn to represent an image where the network inputs are the (x,y) coordinates and the outputs are the RGB pixel values. This idea has all sorts of applications in data science and physics. \n\nThis work provides a method for improving the optimization of these implicit neural representations. It addresses the challenge that standard MLPs are biased towards learning low-spatial frequencies, and that a dataset (e.g. an image to compress) can have different frequency distributions in different regions. ", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an excellent, well-written paper. The problem is clearly-defined, supported by nice figures. The approach taken to address the problem is very reasonable. The experiments are well-done and support the conclusions.  Implicit neural representations have a wide range of uses, so this paper could have high impact.  ​\n\nSpecific comments:\n- The first couple paragraphs of the introduction were a little vague. These could be reworded to clarify the topic of the paper. The second sentence of the introduction could use some citations to clarify which models are being referenced.\n-  Line 149: Presumably the loss is computed from the training points. I see why this is reasonable, but one question I had was whether using held-out validation points to perform the early stopping would be better? \n\nEDIT after author rebuttal: I belive the authors appropriately addressed my comments and those of the other reviewers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "CUtgJEgnT1", "reviewer": "Reviewer_6j88", "summary": "The paper proposes a new encoding policy -- termed SAPE -- which enables implicit neural representations to be learned simultaneously for low as well as high frequency portions of an image. SAPE encodes inputs into progressively higher frequencies, guiding its masking of encodings with feedback based on the current prediction mismatch.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Disclaimer upfront: I am not an expert in implicit neural representations (and am frankly not quite sure how I was assigned to review this paper). As such, I will give a layman's high-level perspective without knowledge of likely many relevant papers and leave it up to the area chair to take this factor into account when making a final decision.\n\nStrengths:\n* SAPE demonstrably improves the encoding of images with especially a mix of low (e.g. blurred background) and high frequencies (e.g. detailed foreground object), both in terms of qualitative and quantitative measurements. I especially like that the method is validated on a range of tasks and datasets which speaks to the generality of SAPE.\n* the authors visualize and interpret the inner workings of SAPE quite well, e.g. via progressive refiniment of an image (Fig. 4) and heatmaps of frequency encodings\n* the paper is generally clearly written and relatively straight-forward to follow, even without expert knowledge of the field. The figures are well chosen and adequately demonstrate improvements of the proposed method\n\nCriticisms:\n* marginal quantitative improvements: compared to SIREN, SAPE+FF seems to offer only minor improvements, e.g. 28.09 instead of 27.03 (Table 1). This makes me worried about how hand-picked the images are and what kinds of improvements and shortcomings to expect from SAPE overall. Relatedly, I would have hoped for more discussion (and figures) of the limitations in the main text rather than a minor supplemental section and figure (A7 + Fig. A9). I think you can easily remove/merge one of the many current figures highlighting the method's improvements with one that showcases its limits in ideally more cases than the two supplemental examples (if space becomes an issue, I think you can in the main manuscript compare it against only 1 or 2 other methods rather than all of them).\n* lack of background/motivation: especially for someone unfamiliar with the field, the paper insufficiently motivates why implicit neural representations are exciting. This became clear after reading (elsewhere) about e.g. the independence of the representations to spatial resolution, but to make the paper more accessible to a wider audience, I recommend spending a couple more sentences on making the underlying motivation more clear.\n\nI am happy to increase my score if these concerns are addressed and after learning more about the context of this work from other reviewers.\n\nMinor:\n* some of the datasets used in especially Table 1 (2D image regression) were not clear to me. References in the table, a more detailed description in the text, and examples next to the table headers (if not too small) would be helpful here\n* what are the units in Table 2? What does a difference between 0.5 and 1.0 correspond to, intuitively?\n* How necessary is the partial masking compared to a progressive policy switching directly from off to on?\n* typos: line 66 \"inputs to MLP+s+\"; line 169 \"our _our_\"\n\n\nEDIT after author rebuttal: thank you for your response and addressing my concerns. Assuming the motivation will be expanded and the limitations made more clear, I have increased my score. Due to my unfamiliarity with this domain, I cannot judge what percentile this paper falls into beyond being a \"good paper\", I hope the AC takes this (and other expert reviews) into account when recommending this paper as poster/spotlight/oral.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are lacking, see above.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "C5jMQifaiiN", "reviewer": "Reviewer_nnVB", "summary": "The paper presents a technique for dealing with the problem of choosing bandwidths for positional encodings in implicit functions. The method presented uses higher frequency positional encodings as training progresses and clips the maximum frequency when a quality criterion has been reached for a position (therefore it is spatially adaptive) in order to avoid the appearance of artifacts.\n\nResults on several domains (images, 3d occupancy, and mesh deformations) are presented with good results in all cases.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ideas presented are novel and the exposition is clear in most places (see below). The method addresses a well known and important problem (figs 1 and 2). \n\nHowever the method adds quite a bit of complexity to the model and training, as it introduces a grid of masks $\\alpha(t,u)$ that must be kept track of and interpolated. This adds to the memory requirements when modelling high-dimensional implicit functions (which is acknowledged as a limitation in the conclusions and appendix). One wonders if an implicit function could be used to keep track of $\\alpha$, but the possibility is not discussed. \n\nThe paragraph starting in line 133 is difficult to understand and Equation 4 is difficult to parse. I believe there might be a typo, and 1 should be the top line, and the clamp expression \"otherwise\". Also, I think the rule would make more sense if $\\tau$ was $\\frac{T}{2n}$. Moreover, $\\Phi$ is shown as a function of the previous mask but it actually depends on $t$. What it should do can be guessed from the explanation below, but not easily from the equations. This part of the paper should be double checked and rewritten for clarity.\n\nThe results are compelling, however, most images and occupancy functions look just as good (or better, see Appendix) using Siren.  Siren is only clearly worse in the mesh deformation tasks. \n\nIt is an interesting paper that improves results at the cost of some added complexity to the model.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations a nd societal impact was discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel spatially adaptive progressive encoding layer and demonstrates that it performs well on a variety of tasks. The reviewers found to be interesting and relevant for the NeurIPS community, and I recommend acceptance.
Decision: Accept (Poster)