Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes an improvement of implicit function approximation for image and mesh generation. The main contribution is to use a progressive low resolution to high resolution optimization as well as spatially adapt this `masking`. Performance is shown to be better than FFN by a significant margin.
main_review: This paper proposes an improvement of implicit function approximation for image and mesh generation. The main contribution is to use a progressive low resolution to high resolution optimization as well as spatially adapt this `masking`. Performance is shown to be better than FFN by a significant margin.

The positives of this paper is that it achieves superior results in terms of PSNR, IOU, and qualitatively over different datasets. The technique makes sense and is relatively straightforward. The paper is also clearly written and the figures provide intuition as to why this method works.

There are a couple of negatives to the paper, which if addressed will help the paper improve.
- The curriculum of the mask as training progresses is mostly ad hoc and not set according to any objective. Could they be optimized in some way, similar to meta-learning objectives perhaps.
- Experimental results do not have enough ablative studies. Specifically, the decisions made for Eq. 4 and spatially adapting the \alpha vector is not justified empirically. Line 273 shows a qualitative and anecdotal image comparing the approaches, quantitative results will make a stronger point. It would be good to quantify the design choices of SAPE.
- Experimental training parameters and settings are not mentioned in the main paper. (it would be nice to have a brief discussion in the main paper, e.g. how long does it take to train, how many parameters are there.)
- How does the number of bits required for the network compare to the number of bits for the image itself? In other works, how inefficient/efficient is it to store images via implicit function as compared to just storing a jpeq image.

Questions:
- How would this approach work with NERF family of methods.
- Line 143: private case -> specific case ?

Overall, this is a nice paper with good empirical results. However the lack of ablative experiments and the lack of network details is a negative for the current version of the paper. 






limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents SAPE, a progressive (gradually unmasking encoding units from low to high frequency during training) and spatially adaptive (each location's unmasking process can be halted based on its local loss to prevent overfitting) encoding of input signals (to be fitted by an MLP), and demonstrates its superior (visually and numerically) results on various regression tasks and 3D mesh transfer.
main_review: Pros
+ The experimental results are overall impressive (visually and numerically).
+ The writing is clear and easy to follow.

Cons
- The loss-conditioned "linear policy" (Eq 4 & 5), being the key feature of the paper, is not sufficiently studied or justified. It's unclear if the loss-conditioned linear progression is really a good approximation of some "optimal policy" (e.g. learned via RL or meta-learning approaches), which could be critical in this application since the relatively costly training (instead of inference) needs to be run for each new input. In addition, given that the bandwidth requirement for a local region of an input is usually easily estimable (e.g. via DCT/FFT coefficients for 2D images/silhouettes and geometric Laplacian [42] for 3D models), it's unclear how much advantage Eq 5 really provides.
- The application and usefulness of SAPE are not sufficiently discussed or validated in the paper. For example, without direct comparisons against existing 2D/3D compression (and 3D mesh transfer) algorithms, it's unclear if SAPE is really a strong alternative or replacement (or how its 2D/3D regression capabilities can be used in real life). The paper also should have more validation tasks (e.g. NeRF) given that 3D mesh transfer seems to be the only task closer to real-world applications.
- The computational and memory complexities (e.g. wall clock time, number of parameters, etc.) of SAPE should be included, given the amount of algorithmic features (including sparse grid sampling) introduced in the paper.

[42] Spectral Compression of Mesh Geometry, ACM SIGGRAPH, 2000

Post rebuttal
- I'm overall satisfied with the authors' feedback, including the additional details on the storage complexity, the additional results on tiny-Nerf, and the additional discussion of the motivation and applicability of SAPE referencing Davies et al. I've upgraded my score to 7.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper highlights a problem with Position Encoding models that use Fourier features: they're very sensitive to a bandwidth parameter that trades-off between reproduction of low- and high-spatial frequency features. Their solution is an iterative feedback algorithm that reconstructs images in a coarse-to-fine manner. They describe incremental improvements across-the-board, and large and impressive improvements in their 2D silhouette reconstruction task.
main_review: Strengths:

This paper is great. Very clear and well-motivated, with sensible experiments and nice results. Presentation is solid throughout, and the authors also include an extensive supplement. I learned a lot reading this paper, and I think the authors deserve credit for their bulletproof arguments. This should be accepted to NeurIPS, and depending on the other reviewers' thoughts, selected for a spotlight.

I really like Fig 1 as a motivating example. Super clear.


Weaknesses:

The feedback method is very similar to models/objective functions of "predictive coding" in computational neuroscience. It would help to cite those. I think this would make your method even more interesting I think, since it would show that this principle from neuroscience is relevant to the problems you address.

I'm no expert in this field. So my other concerns can be taken with a grain-of-salt. (1) The problem seems like it could be an important one but for most of the experiments the improvements are incremental. There's not much you can do about this, not your fault, just my intuition reading this. (2) I realize the standard in this field is for fitting rather than generalization, but it still seems peculiar to introduce ground-truth data into the inference loop.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The submission presents a way of improving training of deep implicit networks, that take Cartesian coordinates (e.g., 2D or 3D) as inputs. The method uses additional embeddings of the input coordinates, similarly to Positional Encodings and variations (SIREN, FFN), with 2 differences:
- embeddings corresponding to higher spatial frequencies are only be made available later during the optimization phase, and only if the network does not fit well enough
- they can be made available selectively on different parts of the input space, based on a spatial grid.
This method makes it possible to better represent inputs where the maximal spatial frequency changes a lot between regions (e.g., an image with large smooth regions, and regions of higher detail), and compares favorably to similar methods.
main_review: Originality
---------------
The methods are an improvement on a recent set of techniques in this field, with two additions:
- progressive availability of higher-frequency embeddings, and
- grid-based local availability of these features.
These additions could have been inspired from other fields like image compression: JPEG compression, for instance, also lays a grid on a 2D image, and uses progressively higher-frequency DCT bases.

Related work is correctly cited, explained, and compared against, the novel improvements are clearly described.

Quality
-----------
Experiments are well designed, and support the claim across a variety of tasks: 2D image regression, 2D silhouettes, 3D occupancy, and 3D mesh transfer. Quantitative and qualitative results are shown.

There's some confusion in Table 1 about intervals and significance: It is not clear what the "±" numbers indicate (standard deviation? 95% confidence intervals), what bolding means, and how ties are handled. For instance, on the last column, both "SAPE + RBFG" and "SAPE + FF" have the same exact mean (0.928), but only the latter is bolded.

Clarity
---------
The submission is easy to follow and well written, conveying well the method, and the experiments. With the additional information, I believe someone already familiar with the field of deep implicit networks could reproduce the results.

Significance
-----------------
These results are part of a series of improvements in a recent and quickly-developing field, while maybe not ground-breaking, I believe this method is straightforward enough and widely applicable, and should improve the state of the art.

Suggestions and minor comments
-------------------------

L. 17: "perceptrons"
L. 37 is the first time "implicit neural networks" are mentioned, I think it should be mentioned earlier, maybe in the abstract
L.135, Eq (4) seems to have the condition backward, should the first line say "if i > d"? It would be consistent with l. 141 which says "we set the masks of the *first* d identity encoding functionals as 1"
L. 143: "private case" -> "special case"?
L. 169 : Figure should be numbered
L. 269, Figure 8: It would be better to have consistent axes for the left / right parts, instead of having each encoding be a column left and a row right

After feedback
============

I found the author feedback to this review satisfactory, and I feel like other reviewers' comments were properly addressed.

limitations_and_societal_impact: The authors seem to only acknowledge limitations, societal impact, and resource use in the Appendix, rather than the main paper, despite the fact that the page limit was increased specifically to make room for these discussions.

I understand that all the items in the checklist are not supposed to fit in the main paper (especially code and details of experiments), but I think that items starting with "Did you describe" or "Did you discuss" (i.e., 1. (b), (c), 3. (d), 4. (d), (e)) can be answered, at least briefly, in the main paper. If the submission is accepted, I think those discussions should be moved to the main text.

Regarding societal impact, the authors could have mentioned the risk of unfaithful reconstruction of images, as was demonstrated by Xerox incorrectly reproducing characters for instance (https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning), but the statement was overall satisfactory.
review_title: null

Meta-review: This paper presents a coarse-to-fine scheme for learning coordinate-based neural networks that is spatially adaptive to the content of the image. While similar coarse-to-fine schemes have been previously presented for NeRF, the spatial adaptivity is shown to improve results across a variety of applications, and reviewers appreciated the clarity of the writing and presentation. The strong empirical results along with quality of presentation lead me to recommend accepting this paper.
Decision: Accept (Poster)