SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization

Abstract: Multilayer-perceptrons (MLP)  are known to struggle learning functions of high-frequencies, and in particular, instances of wide frequency bands.We present a progressive mapping scheme for input signals of MLP networks, enabling them to better fit a wide range of frequencies without sacrificing training stability or requiring any domain specific preprocessing. We introduce Spatially Adaptive Progressive Encoding (SAPE) layers, which gradually unmask signal components with increasing frequencies as a function of time and space. The progressive exposure of frequencies is monitored by a feedback loop throughout the neural optimization process, allowing changes to propagate at different rates among local spatial portions of the signal space. We demonstrate the advantage of our method on variety of domains and applications: regression of low dimensional signals and images, representation learning of occupancy networks, and a geometric task of mesh transfer between 3D shapes.


Reviews and author responses:

Implicit neural representations are a promising approach to learning compressed representations of data --- for example they can learn to represent an image where the network inputs are the (x,y) coordinates and the outputs are the RGB pixel values. This idea has all sorts of applications in data science and physics. 

This work provides a method for improving the optimization of these implicit neural representations. It addresses the challenge that standard MLPs are biased towards learning low-spatial frequencies, and that a dataset (e.g. an image to compress) can have different frequency distributions in different regions.   This is an excellent, well-written paper. The problem is clearly-defined, supported by nice figures. The approach taken to address the problem is very reasonable. The experiments are well-done and support the conclusions.  Implicit neural representations have a wide range of uses, so this paper could have high impact.  ​

Specific comments:
- The first couple paragraphs of the introduction were a little vague. These could be reworded to clarify the topic of the paper. The second sentence of the introduction could use some citations to clarify which models are being referenced.
-  Line 149: Presumably the loss is computed from the training points. I see why this is reasonable, but one question I had was whether using held-out validation points to perform the early stopping would be better? 

EDIT after author rebuttal: I belive the authors appropriately addressed my comments and those of the other reviewers. The authors have adequately addressed this.

This article presents a way to perform image regression, by proposing a spatial encoding, which is spatially adaptive (it depends on the position in the image) and progressive (it increases the "resolution" we use to encode a point until it does not decrease the loss).

The complexity of the proposed method is acceptable regarding the results, which are significantly better than concurrent methods.  ## Detailed summary

This article presents a way to build a neural network (NN) $f : x \mapsto f(x)$, where $x$ is the vector of coordinates of a point (in an image), and $f(x)$ is the color of the pixel at $x$. This is done by adding an encoding layer before the NN, which encodes each point $x$ by the value of its Fourier transform at different frequencies. The frequencies are chosen and added to the encoding by "increasing order" as long as it decreases the loss, capturing for each $x$ the "resolution" to chose. 

This method is inspired by existing works (FFN), which makes it easy to understand.

## Experiments

The method is tested on various tasks, in 2D and 3D. The main result are given in Table 1, where the proposed method SAPE appears to be the best-performing technique against concurrent ones (including "no encoding"). More information about experiments are in the "limitations" section.

What is the aim of doing regression in a mesh transfer problem?

## Clarity

No issue

## Significance

Image regression is a recent and studied area of medium importance. The proposed method is a significant improvement of existing methods.

## Rebuttal

I am mostly satisfied with authors' comments. I maintain my rating. ## Limitations

Influence of the variance of the norms of "frequency vectors" $\mathbf{b}$: done in Fig. 6. The choice of a precise variance is far less important in SAPE than in FFN. However, in the appendix, the variance of SAPE is always greater than for FFN and RBF. E.g. Section 5.5, why $\sigma = 3$ for FFN and RBFG, but $\sigma = 6$ for SAPE? Section 5.3: SAPE: $\sigma = 20$; FFN: $\sigma = 12$, so the rule of "doubling the bandwidth" (line 204) is not followed.

Influence of the grid resolution of SAPE: done in Fig.7. However, no image reconstructed with several grid sizes is shown. Is it possible to take the image of Fig. 10 (or Fig. 6), and reconstruct it with various grid sizes? It would be interesting to add such figure in the appendix.

2D silhouette transfer: Fig. 8 in the article (example where SAPE performs well) comes with Fig. 9 in the appendix (example where SAPE is worse). 

 My concerns were addressed. I will maintain my score.

 Thank you for your response and addressing my concerns. Assuming the motivation will be expanded and the limitations made more clear, I have increased my score. Due to my unfamiliarity with this domain, I cannot judge what percentile this paper falls into beyond being a "good paper", I hope the AC takes this (and other expert reviews) into account when recommending this paper as poster/spotlight/oral.

The paper proposes a new encoding policy -- termed SAPE -- which enables implicit neural representations to be learned simultaneously for low as well as high frequency portions of an image. SAPE encodes inputs into progressively higher frequencies, guiding its masking of encodings with feedback based on the current prediction mismatch.
  Disclaimer upfront: I am not an expert in implicit neural representations (and am frankly not quite sure how I was assigned to review this paper). As such, I will give a layman's high-level perspective without knowledge of likely many relevant papers and leave it up to the area chair to take this factor into account when making a final decision.

Strengths:
* SAPE demonstrably improves the encoding of images with especially a mix of low (e.g. blurred background) and high frequencies (e.g. detailed foreground object), both in terms of qualitative and quantitative measurements. I especially like that the method is validated on a range of tasks and datasets which speaks to the generality of SAPE.
* the authors visualize and interpret the inner workings of SAPE quite well, e.g. via progressive refiniment of an image (Fig. 4) and heatmaps of frequency encodings
* the paper is generally clearly written and relatively straight-forward to follow, even without expert knowledge of the field. The figures are well chosen and adequately demonstrate improvements of the proposed method

Criticisms:
* marginal quantitative improvements: compared to SIREN, SAPE+FF seems to offer only minor improvements, e.g. 28.09 instead of 27.03 (Table 1). This makes me worried about how hand-picked the images are and what kinds of improvements and shortcomings to expect from SAPE overall. Relatedly, I would have hoped for more discussion (and figures) of the limitations in the main text rather than a minor supplemental section and figure (A7 + Fig. A9). I think you can easily remove/merge one of the many current figures highlighting the method's improvements with one that showcases its limits in ideally more cases than the two supplemental examples (if space becomes an issue, I think you can in the main manuscript compare it against only 1 or 2 other methods rather than all of them).
* lack of background/motivation: especially for someone unfamiliar with the field, the paper insufficiently motivates why implicit neural representations are exciting. This became clear after reading (elsewhere) about e.g. the independence of the representations to spatial resolution, but to make the paper more accessible to a wider audience, I recommend spending a couple more sentences on making the underlying motivation more clear.

I am happy to increase my score if these concerns are addressed and after learning more about the context of this work from other reviewers.

Minor:
* some of the datasets used in especially Table 1 (2D image regression) were not clear to me. References in the table, a more detailed description in the text, and examples next to the table headers (if not too small) would be helpful here
* what are the units in Table 2? What does a difference between 0.5 and 1.0 correspond to, intuitively?
* How necessary is the partial masking compared to a progressive policy switching directly from off to on?
* typos: line 66 "inputs to MLP+s+"; line 169 "our _our_"


EDIT after author rebuttal: thank you for your response and addressing my concerns. Assuming the motivation will be expanded and the limitations made more clear, I have increased my score. Due to my unfamiliarity with this domain, I cannot judge what percentile this paper falls into beyond being a "good paper", I hope the AC takes this (and other expert reviews) into account when recommending this paper as poster/spotlight/oral. Limitations are lacking, see above.


 Thank you for the detailed feedback, and for acknowledging the value of our contributions!

&nbsp;

**[What is the aim of doing regression in a mesh transfer problem?]**

The mesh transfer is formulated as deforming the vertices of a source mesh to fit on top of some target shape, and the regression occurs over the vertex positions of the source mesh.
The optimization loss is controlled by two terms which ensure the vertices fit the target shape, and at the same time, maintain the source mesh connectivity with minimal deformation.
The positional encoding assists stabilizing the optimization at the early, critical iterations, and later facilitates the handling of high frequencies, to fit the finer details of the geometry. 
You can think of it as a more complex extension of the 2d silhouettes task (see Figure 8).
Section 6 in the appendix elaborates further on this setting.

&nbsp;

**[the variance of SAPE is always greater than for FFN and RBF]**

The standard deviation σ of SAPE is higher on purpose: setting a high sigma, allows the network to represent the finer details of the output.
As Figure 7 in the paper demonstrates, the quality of SAPE is high as long as the standard deviation of frequency distribution is not too low. 

The shortcoming of FFN (Tancik et al.) is exactly the sensitivity to this standard deviation: increasing it too much may deteriorate the quality of the results, as too many high frequencies are exposed to the network. That requires manually tuning the sigma parameter to search for this “sweet spot”.
The progressive nature of SAPE ensures that the network learns to use only the dimensions it needs, and accordingly, the method is not sensitive to large choices of sigma.
Consequently, it is enough to choose some arbitrary non-low sigma value without any particular tuning.

&nbsp;

**[Visual results for influence of the grid resolution of SAPE]**

Sure, we can add the actual output images accompanying the quantitative results in Figure 7 to the appendix
(see: https://ibb.co/NNN3kgn)


 Thank you for reviewing our paper thoroughly, and for the thoughtful appreciation of our work!



**[Vague introduction]**

We can definitely rephrase the opening sentences of the introduction to make the exposition clearer.
We will also make sure to back it up with the appropriate citations.

&nbsp;

**[Held out validation set]**

Most of the tasks we target in this paper attempt to overfit a network according to some objective function.
Using held-out validation is not straightforward in this case, since we don't assume anything about the sampling distribution of training points.
By splitting some of the training points for the purpose of validation, we miss on the opportunity of presenting the model more samples of the "ground truth" signal, which results in degradation (see Figure 3 in appendix: sensitivity to sample rate).
Meanwhile, we found that a stop criteria of loss thresholding provides a simple and efficient solution, which is sufficient for the tasks we experimented with in this paper.


 Thank you for your valuable feedback!

**[Marginal quantitative improvements compared to SIREN]**

You’re correct to point out that in more than a few cases, SIREN already performs decently, and in those cases SAPE marginally improves the quantitative metrics.
However, when considering more complex signals (i.e., : spatially varying signals), and more intricate tasks the differences begin to show.

The power of SAPE over SIREN can be clearly observed via the chirp signal presented upfront in Figure 2: non-adaptive positional encoding methods tend to learn an implicit function which overshoots around the smooth areas of the signal (dominated by lower frequencies).
Note that we omitted SIREN from this figure due to space constraints, but it presents similar, or even slightly worse behaviour compared to FFN, which can be seen in the supplementary video (1:05-1:11).
The 1d signal example is actually a good starting point to gain intuition, because the same phenomenon is evident in other media, as we’ll soon illustrate, but is less perceived at first glance.

Next, we advise to review the results in Table 1 & 2 in the context of visual results, as differences in some quantitative metrics like PSNR and IoU, may not perfectly correlate with differences in visual quality.
For example: compare the 3d occupancy IoU in Table 1 with Figures 6,7 & 8 in the appendix.
To appreciate the differences for these high-resolution models, zooming in is highly recommended.
Note the Turbosquid samples of Figure 7, where SIREN fails to capture the side of the house, or the staircase of the pagoda.
You can also notice, i.e, that for the statue sample in Figure 6, both SIREN and FFN exhibit “ripples” at the smooth areas of the shape (i.e: at the cheek right below the eyes, or around the shoulders).
In the revised version, we will include visual insets to better guide readers through these results.

Regarding the test set itself, where possible, we used the same test sets used by previous works. For example, the 2d regression task is evaluated on the same data used by Tancik et al. (compare line 209 in our paper to section E in the appendix of Tancik et al.). 

We emphasize that our goal is to show our method benefits a variety of tasks. You can see, for example, that SAPE outperforms SIREN on geometric tasks which use indirect supervision, e.g: 2d silhouettes deformation (Table 1: 0.928 IoU vs. 0.779 on MPEG7, higher better) and 3d mesh transfer (Table 2: 0.39 vs.0.74 Chamfer, 1.82 vs. 4.75 Dirichlet, lower better).

To provide better intuition to these results, and to answer another one of your questions, the metrics in Table 2 are unitless. 
- District Dirichlet energy between a pair of meshes with the same triangulation is a common metric to quantify the “amount” of deformation between a pair of shapes. It is equal to 1 when the deformation is isometric.
- Chamfer and Hausdorff measure the distance between a pair of shapes, and in order to make this measure agnostic to the scale of the shape, both shapes are scaled such that the diagonal length of the bounding box of the target shape is equal to 1.
Once again, we recommend viewing these results alongside the visualizations in Figure 8 of the paper and Figure 8 of the appendix.

**[Limitations in appendix & lack of background]**

Thank you for the useful suggestions!
We can definitely expand the motivation in the introduction to accommodate readers who are less familiar with the neural implicit functions and positional encoding.
Meanwhile, we feel that the work of Davies et al., though not a positional-encoding paper per-se, does a great job at presenting further motivation to neural implicit functions (see: “On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes”). We recommend the discussion in their introduction for an insightful read.

Regarding the “Limitations” section, we indeed shifted it to the appendix due to space considerations, but we will gladly shift it back to the main paper.


 Thank you for taking the time to review our paper!

**[“Added complexity due to grid of masks]**

Yes, spatial adaptiveness requires keeping track of a single float parameter per spatial location of the signal.
However, the grid of masks was introduced to mitigate this added memory complexity (rather than being the cause of it). Practitioners may configure the resolution of this grid to control the tradeoff of spatial-adaptiveness vs. added complexity.  At the extreme case of resolution of 1, our method has no added complexity, and is progressive, but not spatial adaptive. Figure 7 in the paper discusses this tradeoff.


**[Equation 4 is difficult to parse]**

Thank you for commenting on the readability of this equation, and pointing out the typos.
We will make sure to revise and correct this part to ensure it is clear.

1. The inequality sign of $i <= d$ should be $i > d$.
Therefore, the first d positional encoding masks are fully exposed right from the beginning.
The rest of the dimensions are softly exposed in a linear sequential manner, according to the clamp function, as the text describes. For intuition, i.e: let $i=3$ and $d=2$, then the linear progression region for mask dimension $\alpha_i$ is active when $\tau <= t <= 2\tau$.

2.  $\tau = T/2n$ is correct as $\tau$ represents the number of iterations each encoding dimension takes to progress from 0 to 1.


**[SIREN is only clearly worse in the mesh deformation tasks]**

Indeed, SIREN performs well on the regression task of 2d images, and therefore the margin of improvement for SAPE is small for this task. However, our goal is to show SAPE remains stable on a variety of tasks, and this is actually where things become interesting, as more complex scenarios are dealt with and the advantages of SAPE show.

To appreciate the strength of SAPE over SIREN, we suggest the following walkthrough:
Begin by observing the chirp signal in Figure 2.
You can see that non-adaptive positional encodings tend to learn an implicit function which overshoots around the smooth areas of the signal, which is dominated by lower frequencies.
Note that we omitted SIREN from this figure due to lack of space, but it presents similar, or even inferior behaviour to FFN, as we show in the supplementary video (1:05-1:11).
The 1d signal example is a good starting point to gain intuition, because the same phenomenon is evident in other media, as we’ll soon illustrate, but is less perceived at first glance.

Next, we advise to review the results in Table 1 & 2 in the context of visual results, as differences in some quantitative metrics like PSNR and IoU, may not perfectly correlate with differences in visual quality.
For example: compare the 3d occupancy IoU in Table 1 with Figures 6,7 in the appendix.
These are high-resolution models, and therefore zooming in is highly recommended.
Note the Turbosquid samples of Figure 7, where SIREN fails to capture the side of the house, or the staircase of the pagoda.
You can also notice, i.e, that for the statue sample in Figure 6, both SIREN and FFN exhibit “ripples” at the smooth areas of the shape (i.e: at the cheek right below the eyes, or around the shoulders).

In contrast, the progressive and adaptive nature of SAPE minimizes the effect of such artifacts, and our method is able to provide detail preserving outputs of high quality.
Finally, we reiterate that, as you commented, SAPE provides large improvements over SIREN in geometric tasks which require indirect supervision (e.g: 2d silhouettes and 3d mesh transfer), further supporting the evidence that this method is stable.

Thank you for providing this insightful comment. In our revision, we will make sure to include visual insets to better guide readers through these results.


The paper presents a technique for dealing with the problem of choosing bandwidths for positional encodings in implicit functions. The method presented uses higher frequency positional encodings as training progresses and clips the maximum frequency when a quality criterion has been reached for a position (therefore it is spatially adaptive) in order to avoid the appearance of artifacts.

Results on several domains (images, 3d occupancy, and mesh deformations) are presented with good results in all cases.  The ideas presented are novel and the exposition is clear in most places (see below). The method addresses a well known and important problem (figs 1 and 2). 

However the method adds quite a bit of complexity to the model and training, as it introduces a grid of masks $\alpha(t,u)$ that must be kept track of and interpolated. This adds to the memory requirements when modelling high-dimensional implicit functions (which is acknowledged as a limitation in the conclusions and appendix). One wonders if an implicit function could be used to keep track of $\alpha$, but the possibility is not discussed. 

The paragraph starting in line 133 is difficult to understand and Equation 4 is difficult to parse. I believe there might be a typo, and 1 should be the top line, and the clamp expression "otherwise". Also, I think the rule would make more sense if $\tau$ was $\frac{T}{2n}$. Moreover, $\Phi$ is shown as a function of the previous mask but it actually depends on $t$. What it should do can be guessed from the explanation below, but not easily from the equations. This part of the paper should be double checked and rewritten for clarity.

The results are compelling, however, most images and occupancy functions look just as good (or better, see Appendix) using Siren.  Siren is only clearly worse in the mesh deformation tasks. 

It is an interesting paper that improves results at the cost of some added complexity to the model.
 Limitations a nd societal impact was discussed.

Meta-review: This paper introduces a novel spatially adaptive progressive encoding layer and demonstrates that it performs well on a variety of tasks. The reviewers found to be interesting and relevant for the NeurIPS community, and I recommend acceptance.

Decision: accept