SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization
Authors: Amir Hertz
Or Perel
Raja Giryes
Olga Sorkine-hornung
Daniel Cohen-Or
Venue: NeurIPS 2021 Poster (2021)
Keywords: Machine Learning
Deep Learning
Computer Vision
Implicit functions
Positional Encoding

Abstract: Multilayer-perceptrons (MLP)  are known to struggle learning functions of high-frequencies, and in particular, instances of wide frequency bands.
We present a progressive mapping scheme for input signals of MLP networks, enabling them to better fit a wide range of frequencies without sacrificing training stability or requiring any domain specific preprocessing. 
We introduce Spatially Adaptive Progressive Encoding (SAPE) layers, which gradually unmask signal components with increasing frequencies as a function of time and space. The progressive exposure of frequencies is monitored by a feedback loop throughout the neural optimization process, allowing changes to propagate at different rates among local spatial portions of the signal space. We demonstrate the advantage of our method on variety of domains and applications: regression of low dimensional signals and images, representation learning of occupancy networks, and a geometric task of mesh transfer between 3D shapes.

SAPE: Spatially-Aapttive Progressive Encoding for Neural Optimization
Amir Hertz
Or Perel
Tel Aviv University amirhertz@mail. tau. ac i1
Tel Aviv University orr www.ppeee@gma.acomm com
Raja Giryes
Olga Sorkine-Hornung ETH Zurich, Switzerland sorkinešinf. ethz.cl ch
Daniel Cohen-Or Tel Aviv University
Tel Aviv University raja@tauex. tau. ac.
cohenor@gmail.co com
Abstract
Multilayer-perreeproom (MLP) are known to struggle with learning functions of high-frequencies, and in particular cases with wide frequency bands. We present a spatially adaptive progressive encoding (SAPE) scheme for input signals of MLP networks, which enables them to better fit wide range of frequencies without sacrificing training stability or requiring any domain specific preprocessing. SAPE gradually unmasks signal components with increasing frequencies as a function of time and space. The progressive exposure of frequencies is monitored by a feedback loop throughout the neural optimization process, allowing changes to propagate at different rates among local spatial portions of the signal space. We demonstrate the advantage of SAPE on variety of domains and applications, including regression low dimensional signals and images, representation learning of occupancy networks, and geometric task of mesh transfer between 3D shapes.
Introduction
Neural implicit functions have recently emerged as powerful representation paradigm for modeling complex signals. Their continuous nature them apart from typical discrete representations (i.e. pixels, voxels, meshes), allowing to capture high resolution details in various domains such as images 41 43
3d shapes [31 6) and radiance fields 30 32), while retaining reasonably compact representation. In this formulation, a deep neural network is trained with the goal of faithfully mapping input coordinates to a corresponding target domain, effectively learning the representation of signal properties such as magnitude, color, or shape occupancy.
Implementing implicit neural representations with common neural structures, e.g., multilayer perecep- trons with ReLU activations (ReLU MLPs), proves to be challenging in the presence of signals with high frequencies. Consequently, recent works have demonstrated that deep implicit networks benefit from mapping the input coordinates 30 [ or the intermediate features [I| to positional encodings. That is, before feeding them into neural layer, they first transformed to overparameterizee. high dimensional space, typically by multiple periodic functions.
Positional encodings have been shown to enable highly detailed mappings of signals by MLP networks. For example, Fourier Feature Networks 43 suggested to map input coordinates signals to high dimensional space using sinusoidal functions. In their work, they show that the frequency
In this paper, methods that map coordinates higher dimensional space. Not confused with the term "Positional Encoding" coined by 30 43 which refers to a particular mapping scheme this family.
the term "positional encodings" in lower case letters denote the family of encoding
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

FFN =
FFN == 25
SAPE
Figure I: Left: Fourier Features Network (FFN) encoding, tuned to bandwidth of low frequencies in order to fit the smooth snow areas. The same frequency bandwidth yields blurry buildings in the top image. Middle: FFN tuned to a higher bandwidth to fit to the sharp details of the city. The same bandwidth results in the appearance of noisy artifacts in the mountain image. Right: SAPE is able to fit both examples without extra tuning, using the same choice of frequency bandwidth in both cases.
larget Function
emcoding
FEN
progressive encoding
spatially aduptive progressive encrding
Figure 2: ID signal regression. Red: Samples positional coordinates as network input, and signal magnitude labels. Black: Predicted implicit signal at inference time. MLPs with "no encoding" struggle to fit high frequency segments (see appendix for train details). Efforts of static positional encoding models (FFN) to fit high frequency areas of the signal introduce noise the low frequency region. Fitting varying signals with spatial adaptivity allows MLPs to recover all signal frequencies.
of these sinusoidal encodings is the dominant factor in obtaining high quality signal reconstructions. In particular, they present compelling arguments for randomly sampling frequency values from an isotropic Gaussian distribution with a carefully selected scale, providing striking improvement over mapping coordinates directly via standard MLPs.
Despite the success of positional encodings, there are still concerns left unaddressed: (i) Choos- ing the right frequency scale requires manual tuning, oftentimes involving a tedious parameter sweep; (ii) The frequency distribution scale may change between different inputs, and accordingly it becomes harder to tune a "one-fits fone-fits-all" model for signals that are composed of a large range of frequencies (Fig. I (iii) Frequencies are selected for the entire input in global, spatially invariant manner, thus missing an opportunity to better adapt to local high frequencies (Fig. 2).
Our work investigates mitigations to the aforementioned challenges. We study the setting of positional encodings as input to implicit neural networks and present -paapally-Adaative Progressive Encoding (SAPE). SAPE is policy for learning implicit functions, relying on two core ideas: (i) guiding the neural optimization process by gradually unmasking signal components with increasing frequencies over time, and (ii) allowing the mask progression to propagate at different rates among local spatial portions of the signal space. To govern this process, we introduce a feedbuck loop to control the progression of revealed encoding frequencies as bi-variate function of time and space.
Our work enables MLP networks to adaptively fit a varying spectrum of fine details that previous methods struggle to capture in single shot, without involved tuning of parameters or domain specific preprocessing. SAPE excels in learning implicit functions with a large Lipschitz constant, without sacrificing quality of details or optimization stability, in problems that require meticulous configuration to achieve convergence. To highlight the latter, in Section we present the tasks of 2D silhouettes deformation and 3D mesh transfer both require stable optimization from the get-go in order to avoid convergence to sub-optimal local minima.
SAPE is encoding-agnostic: it is not limited to specific positional encoding type. can be easily applied to the learning process of coordinate-basee based neural implicit functions of various domains including images, 2D shapes, 3D occupancy maps and surfaces, showing improvement in all of them.

---

trequence
P
E
feadback
Figure 3: Method overview. Coordinates p are fed into an implicit function network, regressing the signal value ý at position output. An encoder E maps the input coordinates to high dimensional embedding space. Our encoding layer then masks the encoded features. In this example, samples p encoded and progressively exposed during the network training, going from low frequency (top node) to high (bottom node). Node colors indicate the various neuron states: on, off, and partially masked. Finally, the loss between output and target y is fed back to encoder E, which spatially adapts the encoding mask according to the spatial error, indicated by the heatmap on the pufferfish.
Related work
Implicit neural representations. Recent works show the capability of deep neural networks in
representing different functions (e.g.. 2D/3D scenes or objects) as an implicit, memory efficient continuous function
[8]. These networks are used as a signed distance function (SDF) [5] or
occupancy network, either binary 66 28 33 soft volumetric density [26] 30 49].
Several works train such networks to represent collection of 3D shapes via 3D data supervision reconstruct 3D shapes 5
6
16
29
28 31
13 14 18
42 or infer them from 2D images
22
Recent works employ spatial data structures scale the represented shapes size 25 42
Positional encodings (PE), have been suggested as higher dimensional network input for various purposes. Radial basis function (RBF) networks (7] use weighted sums of embedded RBF encodings due to their symmetric property. 47 used random Fourier Features to map time spans. In natural language processing, Transformers 19
46) leverage sinusoidal positional encoding to maintain the order of token sequences. Our work differs by focusing on how to encode the inputs to MLPs in order improve network implicit representations.
24
The closest works to ours are SIREN 41 and Fourier feature networks (FFN to replace the ReLU activations in the network by periodic ones. FFN 4 the network by projecting them to a higher dimensional space using a family of encoding functions. In the appendix provide an illustration of this approach with some examplary used encodings. The analysis of 43 demonstrates that FFN improves the learning of implicit neural representation compared other alternatives. follow-up work in 44 accelerates the training convergence by using a meta-learned initialization of the network weights. The contemporary work of [27] extends SIREN with a modulation network, to allow representing multiple shapes within single network.
SIREN suggests
encodes the inputs to
Concurrently to our work. Park et al. [32] extend NERF to non-rigidly deforming scenes, and BARF 21 extends NERF for cases of imperfect camera poses. Both these works show the advantage of employing coarse-to- to-rse-to-fine annealing linearly over the frequency bandwidth of the positional encoding. The coarse-to- fine approach bears similarity Different from us, these works do not use a feedback loop or spatial encoding, which, as we show in the following, further closes the gap between the regressed function and the ground truth.
progressive frequency approach.
As final remark, SAPE's concept feedback loop resonates with the idea "predictive coding" in computational neuroscience [35] suggesting another angle appreciate the rationale of our method.

---

SAPE
Iteraiinns
Figure 4: Learning progress of SAPE for implicit representation of a 2D image, showing iterations 75, 150, 300 and 3000 (left to right). The beginning of optimization is dominated by low frequencies: spectral bias maintains global optimization stability. As the optimization progresses high frequency mapping takes place, and SAPE fits the finer details of the image. See other methods in the appendix.
3 Preliminaries
The setting of implicit neural representation is commonly formulated as learning mapping function fe. using neural network, from an imput of low dimensional position coordinates p E IR to an output value in the target domain. The network training data are coordinates samples as the input and an expected value of the function, or signal, as the output. Examples of such mappings include pixel locations to intensity values for images, or 3D space coordinates to binary surface occupancy labels.
Achieving such mappings using conventional neural networks with ReLU is hard. It has been shown that ReLU networks exhibit a behaviour of spectral bias [34] 3 36
2 48 4 Networks tend to learn low frequency functions first, as they are characterized by global behaviour, and are therefore more stable to optimize. Spectral bias, however, also prevents networks from properly learning to fit functions that change at high rate, e-g., functions with large Lipschitz constant. When visual domains are concerned, this is mostly evident in delicate details missing from the network output.
To mitigate this deficiency, recent works proposed replace the ReLU activation layers with periodic sine functions [41 or map the input to some higher dimensional space in order to learn mapping with high frequencies 30 43 In the latter approach, the input p is encoded high dimensional embedding by a family of functionals e₁ RR + R, such that:
Epp) (ei (2
where usually n d. Tancik et al.
43 for example, suggest the following encoding:
EFeN (p) = (20,(2 p). sin(2 p),... (2 p). sin(2Tb,,,)))
(2)
where b, are frequency vectors randomly sampled i..d. from Gaussian distribution with standard deviation a. Other examples of positional encodings are included in the appendix.
These embedding schemes facilitate learning of complex functions, at the price of introducing second associated phenomenon: the spectral bias characteristic of the network is reduced. The implication in this case is twofold. Positional encodings can cause neural optimizations processes to become unstable, and consequently converge to a bad local minimum (Fig. D In addition, when fitting functions of varying levels of details/frequencies,, neurons predicting smooth, low frequency areas are still exposed to encoding dimensions high frequency. That, in turn, may complicate the learning process, networks have to learn where to ignore such embedding dimensions.
Indeed, Tancik et al. [43 advocate careful choice the standard deviation of frequency distribution, in their method, showing that using low values yields missing details in the network output, and using values that are very high results in noisy artifacts. Thereupon, for such static encodings the value of requires a parameter sweep per sample.
4 Spatially-Adaptin Progressive Encoding (SAPE)
Our proposed approach, SAPE, is policy for guiding the optimization of implicit neural represen- tations based on input coordinates. SAPE relies on the delicate balance of the two phenomena that govern the learning process: reconciles the effects of the spectral bias and the expressiveness of high frequency encodings in manner that benefits from both. It maintains a stable optimization, without sacrificing the ability to fine signal details. SAPE is composed of two key components: progressive encoding and spatial adaptivity, which allow it to be less sensitive to the encoding frequency bandwidth, i.e., the choice of the standard deviation the case of 43 (Fig. 4. Next, we detail each of the mechanisms that compose SAPE. The full algorithm is included in the appendix.

---

Source Figure 5: Mesh transfer. In this task, we transfer the tessellation of source mesh to target shape (left columns). We estimate an initial transformation using user specified correspondence points (marked in red). Then, we follow with neural optimization tuning process, which brings the source mesh surface to the target shape while keeping the quality of the original tessellation.
Target
No encoding
Sizen
SAPE
Without loss of generality, in our description below we assume the encoding functionals are sorted by their respective Lipschitz constants (LC), i.e. LC((e) I LC(e2) S .... S LC(en). For example, we sort the Fourier features encoding by the frequency value |b:ll- We set the first d dimensions to the identity encodings: (P) = Pt, Vi I d, thereby exposing the network the original input position coordinates p E R as well.
Progressive encoding. A core part of SAPE is progressively fitting the frequencies. The first layer of the network encodes an input position p with set of functionals e,. In order to only part of the functionals at different stages of training. it multiplies the encoding dimensions with a soft mask, which progresses function of the optimization iteration
Epo (p, (eo(p, t), (p, @g( .... (p. a(t))(()).
(3)
where (p, t) = (t) (p) is the progression control for the encoding functional et, regulated by the elements masking vector a+() E [0, 1). During the optimization, we progressively reveal encoding features in manner that allows the encoding neurons to be tuned to different states according to the value of (t), where (t) corresponds to the state on, ax 0 means off and 0 < (t) < denotes partially (Fig.
We define a policy & that controls the progression of the mask vector a(t) by a(t + 1) @ (0)] In this work, we use the following progression rule for the mask of the ith encoding dimension at time step t:
t- T (i d) 0, 1). if d
clamp
Φr [a(()) as(t 11)
(4)
1,
otherwise,
where T/2n represents the number of iterations each encoding dimension takes to progress from 0 to and T' is the maximal number of iterations in the optimization. Essentially, &p performs a linear progression sequentially for each encoding i until it is fully exposed, s.t. (1) 1. When certain encoding dimension mask achieves saturation, we continue to the following one. We allow a technical exception this rule, for proper progression of correlated masks of encoding dimensions sharing the same Lipschitz constant, c.g., pairs of cos, sin in the encoding in Eq. (2) For initialization, the first d positional encoding masks are fully exposed right from the beginning, and in our setting, correspond with the identity encoding functionals.
Notice that previous methods, which introduce all encoding dimensions to the network at once, can now be formulated as a specific case in our framework, where ax (((+1)
Spatial adaptivity. When / > T/2, the progression policy &p achieves global saturation in terms of revealing encoding dimensions, so that the masking vector becomes a(t) = 1. For signals of global- characteristics, c.g., have a low LC', the network is compelled to learn how to cope with undesirable encoding dimensions high frequency to regress a smooth, slowly changing signal. This can lead to sub-optimal outputs, often visible in the form of visual artifacts (Fig. mid-bottom).
Given loss function and convergence threshold E, simple improvement to policy pp may apply early stopping when L < E. However, for signals with varying levels of detail as a function of spatial position p. this improvement not sufficient: early stopping does not oceur due to areas characterized by high frequencies. Relying a global threshold is inherently sub-optimal decision, as when learning an implicit neural function with positional encoding, different spatial segments of the signal in question may converge at different rates (Fig.
5

---

To solve this problem, we set the mask vector to be spatially sensitive: att, p). By design, the new policy PSA progresses the encoding of each spatial location p separately, per encoding dimension i. To simplify our explanation, in the current setting we assume the signal can be discretized to distinct spatial segments on a regular grid, each tracked independently (e.g.. pixels, voxels). The progressive, spatially adaptive policy is controlled by feedback loop, where the independent regression loss of each signal segment is used to control the progression:
(p((a)))) if C(t, 2
SSa (ar(t, p)]: alt +
aa(t. p). otherwise
(5)
For implicit functions, inference time we assume signals can be regressed with "pseudo-continuous"
coordinates p. Therefore, to support coordinates that did not appear during training and have no mask recordings, we extend the estimated parameters of (T, p) continuously over the entire input domain by linear interpolation.
One common example that benefits from spatial adaptivity is natural 2D image containing blurry, out-of-focuu areas in the background together with sharp, detailed foreground objects. To shed more light on the behaviour of our algorithm, Figs. [ and demonstrate examples of spatial heat maps, highlighting the maximal encoding frequency achieved per spatial location, upon convergence.
Sparse grid sampling. We now extend our method to cases where model fo is optimized by coordinates p that do not lie on a regular grid, or when it is simply infeasible to store parameters (t,, p) per sampled coordinate p during training. In this setting, we discretize the input domain and store parameters Qu in a sparse grid G. where for grid of resolution rd C Rd,, denote u the multidimensional grid coordinates. In the forward pass, we ohtain the original encoding mask ax(,, p) per sampled point p by encoding linear interpolation of the masking parameters over the nearest grid nodes (see the inset (a) on the left):
Forward
alt, p) = Σ Wpnn Oxt, u),
(6)
Loss feedback
uEN
where (p) the neighboring grid coordinates in the vicinity of p. and Wpp.u are the interpolation weights for a sample over the multidimensional grid G such that EuENa Wp,u = 1. During
training, the loss L(I, u) at time / for a given grid point u is accumulated over all sampled points affected by (inset (b) above):
C(t, u)
Ep up.u Σ up C(t.
1
Here, L(t, p) is the loss for the original coordinate To obtain the overall training loss, we sum over all the used points for training time t. We note that the linear interpolation used during the forward pass has an added benefit: since only local neighbors participate per update of the original coordinate p. sparse weights allow for efficient accumulations during forward and backward pass updates.
5 Experiments
We evaluate SAPE on a variety of common 2D and 3D regressions tasks. addition, we demon- strate how it may be used to improve the tasks of deforming 2D silhouettes and transferring mesh connectivity between 3D shapes. We demonstrate our results using the Fourier feature encoding 43 (Eq. 2 We emphasize that SAPE is agnostic to the encoding used and applicable to other mapping schemes as well. More examples, as well as the full implementation details appear in the appendix.
5.1 Evaluations
We test SAPE on variety of problems: regression tasks optimized by a direct supervision and geometric tasks optimized by an indirect supervision. All configurations employ 256 unique frequency encodings sampled from a Gaussian distribution. For fair evaluation, the distribution scale and number of neural layers remain the same for all inputs the same problem setting. However, hyperparameter values may vary per problem. For convergence threshold s, we set the values of 1e-3 for regression tasks and le-2 2 for geometric tasks. See the appendixfor full description of implementation details. We compare the settings of 6 MLP configurations:

---

Table 1: Quantitative Evaluation of SAPE against baseline encoded MLPs on various tasks.
Best results displayed in bold, - accounts for deviation among independent experiments. 2D silbooettee (loU)
2D regression (PSNR)
3D occupancy (loU)
Natural images
Text images
Thingi0K
Turbosquid
MPEG7 (loU)
No Encoding
19.72 + 2.77
19.39 2.02
0.899 +: 0.031
0,872 + 0.08
0,81 + 0.177
STREN
27.03
+ 4.28
30.81
1.72
0.964
0.93 0.025
0.779 : 0.253
RBFG
25.98 + 3.98
25.91 + 4.03
0.9.1 + 0.022
0.919 = 0.044
0.851 = 0.15
SAPE RBFG
27.06
3.94
28.52
0,966 0.015
0.94 0.03
0.102
FF
25.57
1 4.19
30.47
1 2.11
0.945 1 0.03
0.964 1 0.017
0.873 1 0.142
Progressive
27.01
+: 3.56
30.2
1.83
0.98 4: 0.018
0.943 0.019
0.86 - 0.133
SAPE FF
28.09
= 4.04
31.84
+ 2.15
0.981 + 0.008
0.969 = 0.013
0.928 = 0.095
1) No encoding: Basic ReLU MLP without en- coding. 2) SIREN: An MLP with sine activa- tions based on the implementation of Sitzmann et al. [4]] 3) RBF-grid: An MLP with repeated radial basis function as a first encoding layer. 4) SAPE + RBF-grid. 5) FFN: An MLP network with Fourier features as the first encoding layer, see Eq. 2 6) SAPE + FFN.
Table 2: Evaluation on the mesh transfer task.
Chamfer (L)
Hausdorff (U)
Dirichlet (L)
No Encoding
1.37
1.36
1.3
SIREN
0.74
1.32
4.75
RBFG
0.97
0.87
1.65
SAPE RBFG
0.55
0.71
1.65
HF
1.27
0.94
2.22
SAPE
0.39
0.49
1.82
The bandwidths of encoding functions in 3) and 5) are optimally selected by a grid search over validation set or taken from a public implementation, depending on the task. For the SAPE variants we use double the a of these bandwidths, SAPE is not sensitive a particular value, as long as allows high frequency encodings. The quantitative results are summarized in Tables 1 and 2 Below an overview of each task.
2D image regression. In this task we optimize the networks to map 2D input pixel coordinates, normalized to [-1, 1]2, their RGB values. We conduct the evaluation on the same test sets as Tancik et al. 43 which contain a dataset of natural images and a synthetic dataset of text images. Similar to them, the network is trained on regularly spaced grid containing 25% of the pixels. We the evaluation metric of PSNR over the entire image, compared to the ground truth image. Quantitative results are reported in Table D on the left.
3D occupancy, In this task, we use a similar setting to occupancy networks 28 : the model is trained to classify input 3D coordinate for being inside or outside the training shape. For training, sample 9 million points divided into 3 equal groups: uniformly sampled points in [-1, 1]3, surface points perturbed with random Gaussian noise vectors using 0.1 and o 0.01. We evaluate the quality of the result by estimating the intersection-over--nioo (IoU) with respect to the ground truth shape by sampling additional 1e6 random points and 1e6 more challenging samples near the surface, and report the average IoU score.
We compare the networks on two test sets The first set is composed of 10 selected models from the Thingi dataset [50). The second, more challenging test set is composed of 4 models from TurboSquid] 2 The quantitative results are summarized in Table I middle section. Qualitative results are shown in the appendix.
2D silhouettes. Here, we optimize an MLP to deform a unit circle represented as a polyline to a target 2D point cloud of silhouette shape. Fig. 9 shows number of examples of such target shapes. We start by calibrating the MIP to learn simple mapping from [o, 1] to the 2D unit circle: (p) = (cos(2 sin (2xp)). We then optimize the mapping function by minimizing the symmetric chamfer distance between the network output and the silhouette.
To evaluate the performance of the different methods, we test the networks on 20 shapes from the MPEF7 dataset (20) and measure the intersection over union between the resulting shape and the target shape; the results are reported in Table [ Fig. 9 shows qualitative results of this task and snapshots from the optimization process. Due to the expressiveness of both FFN and SIREN networks, the chamfer loss causes distortions in the early stage of the optimization that cannot be recovered. leading to undesired results. The coarse-to-fine optimization of SAPE allows both avoiding distortions and matching high frequencies of the target shape.
https:: httpp://www.turbbsuuu...s turbosquid.com
7

---

FFN
SAPE 17 20
20
40
40
Figure 6: 2D Image regression comparing different distributions of bandwidths for Fourier Frequen- cies. SAPE is more robust to the choice of a, consistently producing pleasing results in terms of both PSNR and visual quality.
PSNR
PSNR
SAPE
Stamdard deviation of |bbll
Spatial grid resolution
Figure 7: Left: PSNR as function of the Fourier features' standard deviation of the encoding for the example in Fig. 6 Right: PSNR as a function of spatial grid resolution for the example in Fig. g
3D mesh transfer. In this task, we would like to transfer a 3D source mesh to target shape 45 9
38 39,, which is represented by mesh or a point cloud. The MLP receives the vertices of the source mesh and outputs transformed vertices, such that together with the source tessellation, the optimized mesh fits the target shape while respecting the structure of the source mesh in terms of distortion. In addition, we may utilize a set of marked correspondence points between the input shapes that enable the estimation of an initial affine transformation between the source and the target, followed by biharmonic deformation [17] The optimization loss for this task is composed of two terms: A distance loss that measures the symmetric chamfer distance between the optimized mesh and the target shape, and a structural term that measures the discrete conformal energy between the optimized mesh and the source mesh.
We test the different networks on 6 pairs of meshes and evaluate the results by measuring the chamfer and Hausdorff distances between the target shape and the output mesh. The distortion the transferred meshing is measured by Dirichlet energy with respect to the source mesh. Table shows the quantitative results. Qualitative results are shown Fig.
Standard MLPs struggle to fit the target shape and remain close to the source mesh therefore obtaining low Dirichlet energy but also high Chamfer and Hausdorff distances and thus failing the task. SAPE's feedback specifies the regions on the mesh that are distanced from the target shape. That allows the optimization to gradually increase the frequencies used in high curvature areas while avoiding large distortions at the beginning of the optimization, when global deformations take place. For that reason, contrary other methods such FFN and SIREN, SAPE is able avoid solutions of bad local minima and reconstruct the target signal better.
162
642
256
Figure 8: Influence of the spatial grid resolution output quality. Top row: Output of SAPE with grid resolutions of 12, 42, 162, 64 and 256 Bottom left: Interpolated heatmap of maximal frequency unmasked per grid position. Bottom right: Enlarged results of area with range of frequencies.

---

No encoding
Siren
HIN
Iaget shape
encoding
Figure 9: Iterative deformation of 2D silhouettes. Left: comparison of different network configu-
rations when optimizing for the deformation of unit circle to the target shape, represented by polyline. Right: snapshots of the optimization process. Notice that SAPE (bottom) avoids large distortion at the beginning of the optimization, and can therefore fit the delicate details of the target shape as the optimization progresses.
0
Training scene
PEN
FFN
FEN
SAPE 20
Ground truth
SAPE heat map
16
Figure 10: 3D occupancy scene regression. By spatially adapting the volumetric encoding for the occupancy network, SAPE fit small details such as the wheels without introducing noise in smooth or empty regions. To the right, we display cross sections of the 3D heat map SAPE. Notice that smooth surfaces, like the the trees and the roof, utilize lower frequency encodings compared to detailed objects like the car and the slide.
5.2 Ablation
Progressive non-progressive. Similar to the setting described in Section 5.1 train MLP on randomly sampled pixels of images. We map 2D pixel coordinates to RGB values and measure the PSNR over all pixels compared the ground truth image.
Fig. and Fig. 7 (left) show a comparison between the performance of FFN with and without SAPE when training on 25% of the pixels. Our results show that due to progressive-spatial adjustment of the encoding, SAPE is less sensitive to tuning of the standard deviation (a) of distribution of frequencies b; By contrast, FFN suffers from fragile trade-off between underfitting and overfitting, which results in either blurry pixels in high frequency regions or noise in the smooth background areas.
A similar phenomena is shown in the 3D example in Fig- TO In this task, we train an occupancy network using the setting described in [.11 For regulating the level of encoding across the scene our method based on feedback loss during training. Without SAPE, low frequency encodings cannot represent detailed structures like the car object in the scene. Increasing the frequency bandwidth results in noisy surfaces and undesired blob artifacts in empty spaces. By contrast, SAPE achieves better representation of detailed regions, well as of smooth surfaces.
voxel map of resolution 1283 for spatial encoding. This map is updated
Spatial progressive vs. non-spatial progressive. To demonstrate the advantage of using spatially adaptive encoding, compare it to variant of SAPE that globally progresses the frequency level of encodings for all spatial areas equally, and converges when all training samples fit. Fig. shows the ad advantage of using the spatial encoding in SAPE for ID regression task. In addition, Table

---

shows quantitative improvement gained by SAPE applied on FF, compared to global progression (Progressive FF).
Note that the non-spatial variant is equivalent to SAPE with a spatial grid of resolution one. Fig. B and Fig. 7 (right) show the benefit of the spatial encoding component for 2D image regression task (similar to the one above). We also show how increasing the grid resolution affects the quality of the result 2D image regression task. We observe that the quality stays at its highest when the spatial grid resolution approximates the sampling ratio of coordinates during training.
6 Limitations and Future Work
When SAPE is employed with sampling grid, the chosen resolution provides a trade-off between memory and quality. Choosing a grid of low resolution may yield sub-optimal outputs (see examples in the appendix). As SAPE's varying frequencies sampling grid bears some resemblance to multi- resolution analysis and wavelet transforms, future extensions to SAPE may solve this issue by maintaining the masking parameters in a sparse structure such an Octree 42 GMM [10].
When using very high frequencies, SAPE may enhance inherent output noise, as the maximal frequency encoding gets picked for the noisy portions of the signal.
In this work we investigated linear progression policy for unmasking frequency encodings. Followup works may extend our scheme to a more optimized curriculum, for example, using meta-learning.
7 Conclusions
We presented a policy for improving the quality of signals learned by coordinate based optimizations. We rely on two major contributions: progressive introduction of positional encodings, and spatial adaptivity, which allows different rates of progression per signal location. Our method is simple to implement and improves the implicit neural representation result of MLP networks in various tasks.
Given the surge of applications that use positional encodings, and the ease of deploying SAPE, we believe our approach will be useful for many problems in the vision and graphics domains. Specif- ically, the fact that SAPE is mncoiing-agnostic and insensitive to most encoding hyperparameters provides stable training and facilitates the use of positional encodings in novel tasks.
Broader Impact Statement
SAPE is method for boosting the accuracy of learned implicit neural functions. can be leveraged to enable new applications or improve various downstream tasks that involve precise processing of signals, like segmentation, or reconstruction of complex signals, such as streaming video or signed distanced functions. Beyond the domains of vision and graphics, we conjecture SAPE might also benefit applications in speech, audio, and signal processing general. Depending on their use, all mentioned applications may have positive or negative impact. SAPE can be used to facilitate novel applications that require accurate recovery signals, like 3D reconstruction for AR and VR. can also be misused for the purpose of generating fake media, including audio, video and images.
Acknowledgements
We would like to thank Hao (Richard) Zhang. Hadar Averbuch-Elor and Rinon Gal for their insightful comments, and the anonymous reviewers for their helpful remarks. This work was supported in part by the Israel Science Foundation grant no. 2492/20 and by the ERC-StG grant no. 757497 (SPADE).
References
[1] Matan Atzmon and Yaron Lipman. SAL: sign agnostic learning of shapes from raw data. In 2020 IEEF/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2562-2571. IEEF, 2020. doi: 10. 109/CVP442000 20...022202222
URL https: /oi. org/ 10. CVOR220002220 2020.00264
10

---

[2] Oshrat Bar, Amnon Drory, and Raja Giryes. A spectral perspective of neural networks robustness label noise, 2021.
[3] Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Advances in Neural Information Processing Systems, volume 32, 2019.
[4] Yuan Cao, Zhiying Fang. Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. International Joint Conference on Artificial Intelligence, (IJCAI), pages 2205-2211, 8 2021.
[5] Rohan Chabra, Jan E Lenssen, Eddy lgg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, pages 508-62.. Springer, 2020.
[6] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[7] Ch Dash, Ajit Kumar Behera, Satchidananda Dehuri, and Sung-Bae Cho. Radial basis function neural networks: A topical of-te-of-the-art survey. Open Computer Science, 6, 01 2016. doi: 10.1515/comp-20166
2016-0005.
[8] Thomas Davies, Derek Nowrouzezahrai, and Alee Jacobson. On the effectiveness of weight- encoded neural implicit 3d shapes. 2021.
[9] Zhantao Deng, Jan Bednařík, Mathieu Salzmann, and Pascal Fua. Better patch stitching for parametric surface reconstruction. arXiv preprint arXiv:2010.0702/, 2020.
[10] Benjamin Eckart, Kihwan Kim, Alejandro Troccoli, Alonzo Kelly, and Jan Kautz. Accelerated generative models for 3d point cloud data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5497-5505, 2016.
[11] Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy J Mitra, and Michael Wimmer. Points2surf learning implicit surfaces from point clouds. In European Conference on Computer Vision, pages 108- 124. Springer, 2020.
[12] D. Ezuz, Heeren, O. Azencot, M. Rumpf, and M. Ben-Chen. Elastic correspondence between triangle meshes. Computer Graphics Forum, 18(2):121-134, 2019.
[13] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In International Conference on Computer Vision (ICCV), October 2019.
[14] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857-4866, 2020.
[15] Amos Gropp, Lior Yariy, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In International Conference on Machine Learning, volume 1 119, pages 3789-3799, 2020.
[16] Zekun Hao, Hadar Averbuch-Eloo Noah Snavely, and Serge Belongie. Dualsdf: Semantic shape manipulation using two-level representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
[17] Alec Jacobson, Ilya Baran, Jovan Popovic, and Olga Sorkine. Bounded biharmonic weights for real-time deformation. ACM Trans. Graph., 30(4), 2011.
[18] Chiyu "Max" Jiang. Avneesh Sud, Ameesh Makadia. Jingwei Huang, Matthias Niessner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[19] Guolin Ke, Di He, and Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations, 2021. URL, https::t/::// net/forum?idd09-- F2

---

[20] L.J. Latecki and R. Lakamper. Shape similarity measure based on correspondence of visual parts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(10):1185-1190 2000. doi: 10. 1 179/344478
[21] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. arXiv preprini arXiv:2 2021.
[22] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. arXiv preprint wrXivvI911.00720 2019.
[23] Xuanging Liu, Hsiang-Fu Yu, I. Dhillon, and Cho-Jui Hsich. Learning to encode position for transformer with continuous dynamical model. In JCML, 2020.
[24] Antoine Liutkus, Ondřej Cifka, Shih-Lun Wu, Umut Şimşekli, Yi-Hsuan Yang. and Gaël Richard. Relative positional encoding for transformers with linear complexity, 2021.
[25] Julien N.P. Martel, David B. Lindell, Connor Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural representation. ACM Trans. Graph. (SIGGRAPH), 2021.
[26] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[27] Ishit Mchta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Man- mohan Chandraker. Modulated periodic activations for generalizable local functional represen- tations, 2021.
[28] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[29] Mateusz Michalkiewicz, Jhony Kaesemodel Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations layers in neural networks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4742-4751, 2019. doi:
10.1109//CCV222199000M
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[31] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[32] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Deformable neural radiance fields. arXiv preprins arXiv:2011.12945 2020.
[33] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference Computer Vision (ECCV), 2020.
[34] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310 PMLR, 2019.
[35] Rajesh Rao and Dana Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-fiele effects. Nature neuroscience, 2:79-87, 02 1999. doi: 10. 1038448/4500.
[36] Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In NewrIPS, 2019.
12

---

[37] Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In The IEFE International Conference on Computer Vision (ICCV), October 2019.
[38] Patrick Schmidt, Janis Born, Marcel Campen, and Leif Kobbelt. Distortion-minimizii injective maps between surfaces. ACM Trans. Graph., 38(6), 2019.
[39] Patrick Schmidt, Marcel Campen, Janis Born, and Leif Kobbelt. Inter-surface maps via constant- curvature metrics. ACM Trans. Graph., 39(4), 2020.
[40] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations. In Advances in Neural Information Processing Systems, 2019.
[41] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[42] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[43] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Kan Keil,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
[44] Matthew Tancik, Ben Mildenhall, Terrance Wang. Divi Schmidt, Pratul P. Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate- neural representa- tions. In CVPR, 2021.
[45] Julien Tierny, Joel Daniels, Luis G. Nonato, Valerio Pascucci, and Claudio T. Silva. Inspired quadrangulation. Comput. Aided Des., 43(11):1516-1555 1526, November 2011.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL htttps://proceedinngg www.ediinnnnnnnesneeeeeeeeeeeeee www.neeuriii..m teuurips.cc//pppr/22////11 F Papeeiaaaaaaaaaaaapp..... pdf
[47] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. Advances Neural Information Processing Systems, pages 15889-15899, 2019.
[48] Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In ICONIP, 2019.
[49] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one few images. In CVPR, 2021.
[50] Qingnan Zhou and Alec Jacobson. Thingi10k: A dataset of 10,000 3d-printing models, arXiv preprint 10X1v11600.04797,200 2016.
13


Reviews:
[{"review_id": "s2MjTXveCJ", "reviewer": "Reviewer_cagt", "summary": "The paper presents SAPE, a progressive (gradually unmasking encoding units from low to high frequency during training) and spatially adaptive (each location's unmasking process can be halted based on its local loss to prevent overfitting) encoding of input signals (to be fitted by an MLP), and demonstrates its superior (visually and numerically) results on various regression tasks and 3D mesh transfer.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n+ The experimental results are overall impressive (visually and numerically).\n+ The writing is clear and easy to follow.\n\nCons\n- The loss-conditioned \"linear policy\" (Eq 4 & 5), being the key feature of the paper, is not sufficiently studied or justified. It's unclear if the loss-conditioned linear progression is really a good approximation of some \"optimal policy\" (e.g. learned via RL or meta-learning approaches), which could be critical in this application since the relatively costly training (instead of inference) needs to be run for each new input. In addition, given that the bandwidth requirement for a local region of an input is usually easily estimable (e.g. via DCT/FFT coefficients for 2D images/silhouettes and geometric Laplacian [42] for 3D models), it's unclear how much advantage Eq 5 really provides.\n- The application and usefulness of SAPE are not sufficiently discussed or validated in the paper. For example, without direct comparisons against existing 2D/3D compression (and 3D mesh transfer) algorithms, it's unclear if SAPE is really a strong alternative or replacement (or how its 2D/3D regression capabilities can be used in real life). The paper also should have more validation tasks (e.g. NeRF) given that 3D mesh transfer seems to be the only task closer to real-world applications.\n- The computational and memory complexities (e.g. wall clock time, number of parameters, etc.) of SAPE should be included, given the amount of algorithmic features (including sparse grid sampling) introduced in the paper.\n\n[42] Spectral Compression of Mesh Geometry, ACM SIGGRAPH, 2000\n\nPost rebuttal\n- I'm overall satisfied with the authors' feedback, including the additional details on the storage complexity, the additional results on tiny-Nerf, and the additional discussion of the motivation and applicability of SAPE referencing Davies et al. I've upgraded my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "os5cGTSBe6d", "reviewer": "Reviewer_bcJK", "summary": "The submission presents a way of improving training of deep implicit networks, that take Cartesian coordinates (e.g., 2D or 3D) as inputs. The method uses additional embeddings of the input coordinates, similarly to Positional Encodings and variations (SIREN, FFN), with 2 differences:\n- embeddings corresponding to higher spatial frequencies are only be made available later during the optimization phase, and only if the network does not fit well enough\n- they can be made available selectively on different parts of the input space, based on a spatial grid.\nThis method makes it possible to better represent inputs where the maximal spatial frequency changes a lot between regions (e.g., an image with large smooth regions, and regions of higher detail), and compares favorably to similar methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n---------------\nThe methods are an improvement on a recent set of techniques in this field, with two additions:\n- progressive availability of higher-frequency embeddings, and\n- grid-based local availability of these features.\nThese additions could have been inspired from other fields like image compression: JPEG compression, for instance, also lays a grid on a 2D image, and uses progressively higher-frequency DCT bases.\n\nRelated work is correctly cited, explained, and compared against, the novel improvements are clearly described.\n\nQuality\n-----------\nExperiments are well designed, and support the claim across a variety of tasks: 2D image regression, 2D silhouettes, 3D occupancy, and 3D mesh transfer. Quantitative and qualitative results are shown.\n\nThere's some confusion in Table 1 about intervals and significance: It is not clear what the \"±\" numbers indicate (standard deviation? 95% confidence intervals), what bolding means, and how ties are handled. For instance, on the last column, both \"SAPE + RBFG\" and \"SAPE + FF\" have the same exact mean (0.928), but only the latter is bolded.\n\nClarity\n---------\nThe submission is easy to follow and well written, conveying well the method, and the experiments. With the additional information, I believe someone already familiar with the field of deep implicit networks could reproduce the results.\n\nSignificance\n-----------------\nThese results are part of a series of improvements in a recent and quickly-developing field, while maybe not ground-breaking, I believe this method is straightforward enough and widely applicable, and should improve the state of the art.\n\nSuggestions and minor comments\n-------------------------\n\nL. 17: \"perceptrons\"\nL. 37 is the first time \"implicit neural networks\" are mentioned, I think it should be mentioned earlier, maybe in the abstract\nL.135, Eq (4) seems to have the condition backward, should the first line say \"if i > d\"? It would be consistent with l. 141 which says \"we set the masks of the *first* d identity encoding functionals as 1\"\nL. 143: \"private case\" -> \"special case\"?\nL. 169 : Figure should be numbered\nL. 269, Figure 8: It would be better to have consistent axes for the left / right parts, instead of having each encoding be a column left and a row right\n\nAfter feedback\n============\n\nI found the author feedback to this review satisfactory, and I feel like other reviewers' comments were properly addressed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors seem to only acknowledge limitations, societal impact, and resource use in the Appendix, rather than the main paper, despite the fact that the page limit was increased specifically to make room for these discussions.\n\nI understand that all the items in the checklist are not supposed to fit in the main paper (especially code and details of experiments), but I think that items starting with \"Did you describe\" or \"Did you discuss\" (i.e., 1. (b), (c), 3. (d), 4. (d), (e)) can be answered, at least briefly, in the main paper. If the submission is accepted, I think those discussions should be moved to the main text.\n\nRegarding societal impact, the authors could have mentioned the risk of unfaithful reconstruction of images, as was demonstrated by Xerox incorrectly reproducing characters for instance (https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning), but the statement was overall satisfactory.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "CW9oGuxcSpD", "reviewer": "Reviewer_FBfi", "summary": "This paper highlights a problem with Position Encoding models that use Fourier features: they're very sensitive to a bandwidth parameter that trades-off between reproduction of low- and high-spatial frequency features. Their solution is an iterative feedback algorithm that reconstructs images in a coarse-to-fine manner. They describe incremental improvements across-the-board, and large and impressive improvements in their 2D silhouette reconstruction task.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis paper is great. Very clear and well-motivated, with sensible experiments and nice results. Presentation is solid throughout, and the authors also include an extensive supplement. I learned a lot reading this paper, and I think the authors deserve credit for their bulletproof arguments. This should be accepted to NeurIPS, and depending on the other reviewers' thoughts, selected for a spotlight.\n\nI really like Fig 1 as a motivating example. Super clear.\n\n\nWeaknesses:\n\nThe feedback method is very similar to models/objective functions of \"predictive coding\" in computational neuroscience. It would help to cite those. I think this would make your method even more interesting I think, since it would show that this principle from neuroscience is relevant to the problems you address.\n\nI'm no expert in this field. So my other concerns can be taken with a grain-of-salt. (1) The problem seems like it could be an important one but for most of the experiments the improvements are incremental. There's not much you can do about this, not your fault, just my intuition reading this. (2) I realize the standard in this field is for fitting rather than generalization, but it still seems peculiar to introduce ground-truth data into the inference loop.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "A6-kWXoatfW", "reviewer": "Reviewer_SbWq", "summary": "This paper proposes an improvement of implicit function approximation for image and mesh generation. The main contribution is to use a progressive low resolution to high resolution optimization as well as spatially adapt this `masking`. Performance is shown to be better than FFN by a significant margin.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an improvement of implicit function approximation for image and mesh generation. The main contribution is to use a progressive low resolution to high resolution optimization as well as spatially adapt this `masking`. Performance is shown to be better than FFN by a significant margin.\n\nThe positives of this paper is that it achieves superior results in terms of PSNR, IOU, and qualitatively over different datasets. The technique makes sense and is relatively straightforward. The paper is also clearly written and the figures provide intuition as to why this method works.\n\nThere are a couple of negatives to the paper, which if addressed will help the paper improve.\n- The curriculum of the mask as training progresses is mostly ad hoc and not set according to any objective. Could they be optimized in some way, similar to meta-learning objectives perhaps.\n- Experimental results do not have enough ablative studies. Specifically, the decisions made for Eq. 4 and spatially adapting the \\alpha vector is not justified empirically. Line 273 shows a qualitative and anecdotal image comparing the approaches, quantitative results will make a stronger point. It would be good to quantify the design choices of SAPE.\n- Experimental training parameters and settings are not mentioned in the main paper. (it would be nice to have a brief discussion in the main paper, e.g. how long does it take to train, how many parameters are there.)\n- How does the number of bits required for the network compare to the number of bits for the image itself? In other works, how inefficient/efficient is it to store images via implicit function as compared to just storing a jpeq image.\n\nQuestions:\n- How would this approach work with NERF family of methods.\n- Line 143: private case -> specific case ?\n\nOverall, this is a nice paper with good empirical results. However the lack of ablative experiments and the lack of network details is a negative for the current version of the paper. \n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a coarse-to-fine scheme for learning coordinate-based neural networks that is spatially adaptive to the content of the image. While similar coarse-to-fine schemes have been previously presented for NeRF, the spatial adaptivity is shown to improve results across a variety of applications, and reviewers appreciated the clarity of the writing and presentation. The strong empirical results along with quality of presentation lead me to recommend accepting this paper.
Decision: Accept (Poster)