Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes two modifications to the recently introduced multiplicative filter networks (and their bandlimited variant BACON): 1) by introducing skip connections it enables explicit layerwise bandwidth control, 2) by modifying initialization scale it makes the spectral supports of layers more or less disjoint. The benefit of doing this is that it enables coupling with standard multigrid / frequency marching approaches to solve inverse problems. Thanks to the introduced modifications the low-frequency content of the function implemented by the network need not change when fitting higher frequencies. The authors show how this leads to high-quality reconstructions of molecules in synthetic CryoEM experiments.
strengths_and_weaknesses: The problem addressed by the paper is very clear, the paper is well written, the numerical experiments seem well executed and the results are compelling.

The interventions in the architecture and training are simple. They yield solid (but not staggering) improvements over earlier work. Cryo-EM is an important application but the setting treated here is somewhat simplified (no shifts, no heterogeneity, only moderate noise SNR = 0.1, and a coarse initial orientation estimate). It would be interesting to see how the proposed architecture fares on other problems. problems. 

As far as I can tell the paper is technically correct—there are no formal results, the architecture description is clear and simple, and the successful experiments strongly corroborate the correctness of the derivations.

In terms of the narrative, I have the impression that achieving perfect scale orthogonality is a bit overstated. It can be certainly play a role in inverse problems like Cryo-EM which are linear (or conditionally linear when given the particle orientations) and where the frequencies don’t mix (by the Fourier slice theorem). (I doubt that it need be ideal.) It may however be a liability or at least not as useful in problems with strong scale coupling. 
questions: - What happens with your Cryo-EM reconstructions if you initialize orientations randomly? How important is the current coarse initialization?
- How exactly do you choose the hyperparameters $\lambda$ and epochs / scale count? You state that for PDB4ake you find a better reconstruction if you dedicate more epochs to the first and second scales. Do you do this by comparing the reconstruction to the ground truth or simply eyeing the reconstruction?
- In Cryo-EM, how do other implicit representations (non-marching) perform?
limitations: The discussion of limitations is adequate.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper describes an extension of the recently proposed Multiplicative Filter Networks (MFN) that permits an independent parametrization of different levels of scale (Fourier-bandwidth) by introducing residual connections and a suitable initialization scheme.
MFNs are a simplification of positionally encoded networks / coordinate networks that have a special form of building the result as a linear combination of (potentially) exponentially many basis functions that are controlled non-linearly, achieved by multiplication with Fourier (or Garbor) basis functions in each "level" of the network (in the 2D/3D case considered here, the dimension of the output [function] space does not seem to be exponential though, bringing this closer to a classical linear representation) . The related BACON network extends this idea by providing initialization schemes that yield nested levels of scale (increasing resolution), but the networks are constructed in a way such that changes to higher level "fine" details invalidate the fit of lower layers.
The submission addresses this issue in a simple but effective way by adding short-cut connections that carry the lower-layer function-fits to the next-higher layer unaltered, and building-up a new detail representation on top (which, to my understanding, can show complex scale dependencies). In addition, a suitable initialization is proposed that creates nested levels of detail in the Fourier-spectrum covered (again, to my understanding, targeting the low-dimensional regime of maybe 2D or 3D coordinates).
Results for image fitting are improved over previous methods such as BACON. An example application to CryoEM reconstruction is presented that seems to outperform a well-known approach (characterized as SOTA).

The main contribution is a scale-separated version of MFN/BACON. The applications to CryoEM are also interesting but the rather brief evaluation with only a few example reconstructions move the methodological contribution more in focus.

strengths_and_weaknesses: To explain my view of strength and weaknesses, I would first discuss my understanding of the technical aspects a bit more in depth:

I understand the main goal of the paper as to decouple scales in an MFN-style coordinate network. To my understanding, this is already achieved by the short-cut connections, which yield a superposition of the low-frequency part, solely controlled by low-frequency parameters, with a high-frequency part that depends on both low- and high-frequency parameters. For progressive coarse-to-fine-optimization, this seems to be sufficient, while at the same time being conceptually and technically very easy to understand and implement. A suitable initialization makes sure that progressive parts of the spectrum are covered (by introducing corresponding control parameters); this scheme seems to work only in moderately high dimensions (as it needs to cover Fourier-coefficient space with 2^d or 3^d patches of coefficients). Consequently, it is not fully clear how the decoupling would affect higher-dimensional applications, but they are not the target of the paper (and a lot of previous work, where similar representations are used for example for reconstruction from 3d point clouds).

I think that the concern for scale separation is a valid goal, and results on image fitting in the paper at least indicate subtle advantage of the decoupled representation (although a strong explanation is missing; the case for CryoEM is clearer as multi-scale fitting is commonly used there). It would be useful to understand more clearly how important this is in an overparametrized regime, as multi-scale optimization is often used as a heuristic to counter bad local minima in non-overparametrized, non-convex optimization scenarios.

From a conceptual point of view, I do not fully understand the boundaries (as in representational capacities and ability to find solutions) between linear scale space approaches (mip-maps, wavelets), MFN-approaches (linear in $x$, non-linear in $\theta$), and the current proposal. If - hypothetically - the next-higher frequency band was fully separated by duplicating the required low-res coefficients, would the resulting method be substantially different from an ordinary hierarchy of Fourier-basis functions? How different (how much better) are results in comparison to a simple base-line (say, a linear wavelet or pyramid representation)? It would be useful to see a direct base-line comparison (or maybe arguments for why this is clearly inferior - I do not know this area too well).

Summarized as a list, I would see the following strength...
+ Simple and straightforward idea. Easy to implement.
+ Canonical extension of MFN/BACON (users of those methods probably want to know about this one).
+ Good results with minimal (extra) effort (Seems to yield SOTA on CryoEM)
+ Well written paper.

...and weaknesses:
- Experimental evaluation remains a bit anecdotal (I guess that the CryoEM fitting is very expensive to setup and conduct).
- Very simple, maybe incremental idea (could also be seen as a limitation of the contribution)
- Limited explanation of "why" and "how far" it works.
- Could have more comparisons against base-line

Overall, I would guess that the paper might be a simple but frequently employed addition to MFN-methods and thus potentially impactful (with many citations and applications). For this reason, I tend towards a positive recommendation.
questions: - Does it makes sense to compare against linear base-lines? If so, what results would you get (or expect)? If not, why is it more expressive (and would it still be if the scales were fully separated [if I understood this correctly])?
- Is the method scalable to larger dimensions?
limitations: If my understanding of the limitations in terms of dimensionality were correct (see discussion question above), it would be useful to discuss this more in the paper.
It might also be useful to mention the limitations of multi-scale optimization for avoiding local minima (in my experience with shape matching & reconstruction problems only very vaguely related to the CryoEM example, it helps somewhat but does not eliminate fundamental issues with bad local minima).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors treat the problem of coarse-to-fine optimization for solving inverse problems, which are particularly important for 3D reconstruction. They note that the existing state-of-the-art coordinate networks used to solve these problems are ill-suited for multiscale optimization since they can forget low frequency details learned during earlier stages in the training scheme. The authors propose residual multiplicative filter networks which (1) introduce skip connections between layers with different output scales so that coarse features learned early in the optimization process are explicitly not forgotten and (2) use an initialization scheme that allows for control over the frequency spectra learned by each layer, meaning that the network can avoid learning redundant frequency information in finer-scale layers. Several experiments are presented for 2D and 3D reconstruction problems where the proposed method is shown to be superior to existing approached in capturing details at several scales.   
strengths_and_weaknesses: STRENGTHS:

- I like the intuitive breakdown of the intermediate representations as a sum and shift of frequency sprectra from the previous layers. This provides a straightforward motivation for the proposed method. 

- The visual representation in Figure 6 gives a clear and concrete example of why coarse-to-fine optimization is important for 3D reconstruction problems.

- Detailed background is given on 3D molecular reconstruction problems, which is useful for the larger ML community that may not be familiar with such domains.


WEAKNESSES:

- I do not understand the benefits of having the outputs at each scale "shift" very little (e.g. as in the results in Table 1). I would think that as long as the final output PSNR is satisfactory, then the internal shift in frequency repsresentation does not matter. Looking at Table 1, it seems that even though fair BACON has quite a bit of shift in its frequency representations at each scale before and after optimizing the other scales, the output still has good PSNR.

- I believe that a more thorough background and introduction to coordinate-based networks in general could be given. While some details are given (e.g. that the networks take 2D or 3D coordinates and output some value like RGB), it was not immediately clear to me what the purpose of these networks (e.g. compressing a high-dimensional signal) was compared to other models (e.g. generative models).  
questions: Related to my first point in the weaknesses section, it would be useful if the authors could provide a description of why the multi-scale metrics (e.g. as shown in the FSC plots in Fig. 5) matter in the context of reconstruction problems. My current understanding is that these metrics are provided to demonstrate that the proposed method has stable training during multi-scale optimization and that it successively adds fine-grained detail due to the innovations presented in the paper. However, I am not sure if I am supposed to interpret these metrics as useful for actually solving the reconstruction or inverse problem. My intuition is that as long as the final (i.e. the last scale during training) output is consistently useful, then the intermediate training qualities do not matter nearly as much. 
limitations: The authors provide a small list of limitations in the second paragraph of the conclusion. However, the section reads as a justification for some design and experimental choices rather than a discussion about the shortcomings of the proposed method. I would suggest that the authors make a more explicit limitations (sub)section.
review_title: null

Meta-review: metareview: The paper studies Multiplicative Filter Networks, which are coordinate neural networks in which each layer applies a multiplicative (Hadamard product) filter and a sinusoidal nonlinearity. The paper shows how introducing residual connections and initializing appropriately can lead to networks where the frequency content of the image separates over layers. This leads to a learned version of classical “coarse-to-fine” reconstruction methods, which the paper terms Residual Multiplicative Filter Networks. The paper illustrates its proposals with experiments on image approximation and on cryo-EM reconstruction. 

Reviewers found that the paper presents a simple idea, which can be easily adopted whenever a coarse-to-fine reconstruction is desired, and as such is likely to see followup work. The main questions concerned the necessity of a coarse-to-fine approach in applications where one ultimately seeks a reconstruction at just a single scale, and the cryo-EM experiments, which show good performance compared to a baseline, when the coordinate network model is integrated into a larger system. Overall, the reviewers found that paper presents a natural modification to MFNs which improves both their interpretability and applicability in inverse problems in imaging. 

confidence: Certain
Decision: Accept