Residual Multiplicative Filter Networks for Multiscale Reconstruction
Authors: Shayan Shekarforoush
David B. Lindell
David J. Fleet
Marcus A Brubaker
Venue: NeurIPS 2022 Accept (2022)
Keywords: Coordinate Networks
Implicit Neural Representations
Multiscale Representation Learning
Coarse-to-fine Reconstruction
Cryo-EM

Abstract: Coordinate networks like Multiplicative Filter Networks (MFNs) and BACON offer some control over the frequency spectrum used to represent continuous signals such as images or 3D volumes. Yet, they are not readily applicable to problems for which coarse-to-fine estimation is required, including various inverse problems in which coarse-to-fine optimization plays a key role in avoiding poor local minima. We introduce a new coordinate network architecture and training scheme that enables coarse-to-fine optimization with fine-grained control over the frequency support of learned reconstructions. This is achieved with two key innovations. First, we incorporate skip connections so that structure at one scale is preserved when fitting finer-scale structure. Second, we propose a novel initialization scheme to provide control over the model frequency spectrum at each stage of optimization. We demonstrate how these modifications enable multiscale optimization for coarse-to-fine fitting to natural images. We then evaluate our model on synthetically generated datasets for the the problem of single-particle cryo-EM reconstruction. We learn high resolution multiscale structures, on par with the state-of-the art. Project webpage: https://shekshaa.github.io/ResidualMFN/.


Residual Multiplicative Filter Networks for Multiscale Reconstruction
Shayan Shekarforoush" 1.2 shayan@cs toronto.... edu
David B. Lindell lindell0cs toronto edu
David J. Fleet 3,
Marcus A. Brubaker 2,4
fleetêcs. toronto.co edu
mabčeecs .ereeeccs..rokee yorku. ca
University of Toronto
Vector Institute
3 Google Research
YYrk University
Abstract
Coordinate networks like Multiplicative Filter Networks (MFNs) and BACON offer some control over the frequency spectrum used to represent continuous signals such as images or 3D volumes. Yet, they are not readily applicable problems for which coarse-to-fing to-fine estimation required, including various inverse problems which coarse-to-fin to-fine optimization plays key role in avoiding poor local minima. We introduce new coordinate network architccture and training scheme that enables coarse-to- toenee optimization with fine-grainee control over the frequency support of learned reconstructions. This is achieved with two key innovations. First, incorporate skip connections that structure at scale is preserved when fitting finer-scale structure. Second, we propose a novel initialization scheme provide control over the model frequency spectrum each stage of optimization. We demonstrate how these modifications enable multiscale optimization for coarse- to-fine fitting to natural images. We then evaluate model synthetically generated datasets for the the problem of single-particle cryo-EM reconstruction. We learn high resolution multiscale structures, on par with the state-of-the art. saciiaaa.gttttt github........ Becchuu..ooRReeeee io/ResidualMFV/
Project webpage:
Introduction
Coordinate networks have emerged as a powerful way to represent and reconstruct images, videos, and 3D scenes [13 and for solving challenging inverse problems such 3D molecular reconstruction for cryo-electron microscopy (cryo-EM) A. They typically take as input point defined on continuous, low-dimensional domain (e.g., 2D position for images), and they output the signal value at that point (e.g., the color). recently proposed architectures 7 g. the frequency content of the represented signal can also be explicitly controlled. Such networks motivated in part by the effectiveness of multiscale methods in image processing and 3D reconstruction.
Nevertheless, while current scale-aware coordinate network architectures offer some control over scale X I0), they are not readily compatible with classical multiscale methods used for coarse-to- optimization, like pyramids [1] [12 multigrid solvers [33] -tate-of-the-ee cryo-EM models, for instance, use frequency-marreee marching to progressively estimate 3D density, beginning with a coarse
structure, then adding finer structure each iteration 14 15 Some networks use heuristics to empirically constrain the scale network output for coarse-to-fine optimization, but they have no explicit constraints the represented frequencies 16 17.. Others represent signals multiple
scales at inference time, but do not enable control of the frequency spectrum during training X 10)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

3/
3/3
5/
5n.
us +
Full Spectrum
+
+
+
y1
M2
96
3/
Filter Layer
Linear Leyer
21
Zg
Cuspur Layer
Skip Connactions
(z)
91 (ar)
9: (z)
9b
94
Figure I: Overview of residual multiplicative filter networks. Skip connections efficiently combine low-frequency information from earlier layers with high-frequency information in later layers. With the proposed initialization scheme, each layer learns separate, increasing frequency bands.
Here, we introduce a new coordinate network architecture and training scheme that enables one to effectively control the frequency support of the learned representations during multiscale optimization, building on recent neural networks with an analytical and controllable Fourier spectra 7 coarse-to-fine reconstruction, one divide the training process into stages, each of which learns a single seale. Training starts at the coarsest resolution and then progressively adds finer scales. Naively employing this procedure with existing architectures will damage the signal reconstruction coarser scales when fitting finer ones (e.g., see Fig. 3 To mitigate this, we modify MFNs by adding skip connections (18). reconstruction coarser scales. This alleviates the need to adapt previous layers in favor of fitting the finer grained reconstructions, effectively obviating the need re-learn signal structure captured by coarser scales. Second, we derive an initialization scheme which explicitly introduces hierarchical gaps in the frequency spectrum of a new layer so that, by construction, its frequency support has controllable degree of overlap. This requires the network to use the skip connections in order to fill the holes in the spectrum and ensures lower levels remain faithful coarse scale representations of the signal.
g For
scale naturally incorporates the learned structure
We apply our technique to the inverse problem of single-partich cryo-EM reconstruction to determine the 3D structure of macromolecular complexes, like proteins, from collections of noisy 2D tomographic projections. This is a challenging non-linear inverse problem, closely related to multi- view reconstruction, requiring estimation the pose and 3D structure of bio-molecules. Successful 3D reconstruction relies on coarse-to-fing methods, sometimes called frequency marching [200], to reduce the risk of becoming trapped in poor local minima. Our results demonstrate effective and efficient ab initio reconstruction of 3D structures with cryo-EM to high resolution.
This paper proposes
architecture and training approach to enable powerful multiscale estimation techniques with coordinate- based networks. Specifically, we make the following contributions:
We develop residual multiplicative filter networks, a new coordinate network architecture with tailored initialization scheme for multiscale signal reconstruction and representation.
We design fast and efficient coarse-to-fine training strategies for residual multiplicative filter networks that leverage the multiscale representation.
We apply the proposed architecture and training strategy to cryo-EM reconstruction two synthetic datasets achieve final reconstructions competitive with the cryoSPARC [4] Laae-oof-the-ae method.

---

2 Related Work
Coordinate- based Networks. Coordinate-basee networks (c.g.. 2
B offer memory-cfficien
continuous function parameterization that can be flexibly trained to reconstruct 3D appearance M 21-
25 and structure 32).32), including applications in biomedical imaging 6R) REl and cryo-EM [5 4
G Although originally formulated as fully-con fully-connected MLP architectures, new models have been proposed to improve performance and interpretability. For instance, one use multiple small or combine coordinate-bass based facilitate generalization across shapes or scenes 440443] 29 45 Multiplicative Filter Networks (MFNs) replace the conventional
fully-connected networks to improve representational capacity 35
37),
networks with explicit feature grids to improve efficiency 6
39
MLP architecture with successive layers of Hadamard products and sine non-linearities 77- Closest to our own work, MFNs have an analytical Fourier spectrum whose bandwidth can be explicitly constrained [. improving the controllability and interpretability of the representation. Inspired by this work, we establish Residual MFNs (rMFNs) with specialized control over the Fourier spectrum to enable coarse-to-fine optimization.
Multiscale Reconstruction. Multiscale representations and reconstruction methods are fundamental concepts in signal and image processing. For example, wavelets are fast and efficient multiscale signal representation 46
useful for denoising, compression, communications, and optical Multigrid frameworks are used for many differential equation solvers for
47
flow estimation 48
physics simulations processing and have, in part, inspired modern deep learning architectures [50
Gaussian and Laplacian pyramids 4 11] have broad application to image
network architectures build on these fundamental concepts of multiscale signal representation and reconstruction. Some architectures use explicit feature grids at multiple scales to enable efficient training and inference for representing shape 39 511 or rendering scenes [52] 38. Multiscale fully-connected coordinate network architectures have been realized by progressively increasing the frequencies of positional encodings during optimization, for example, for bundle adjustment and neural scene representation [ 17 It is also possible to optimize separate networks with inductive biases towards low or high frequencies to improve shape representation [[6]. Still, while these methods introduce multiscale training techniques, the architectures rely inductive biases rather than explicit constraints on the scale or Fourier spectrum of the representation.
Recent coordinate-base
Other methods use coordinate networks for multiscale representation rather than efficient optimization. For instance, scale-aware positional encodings are trained at all scales simultaneously so that different output scales can be queried at inference time [IO]. land-limited coordinate networks (BACoN) extend MFNs to control the network bandwidth, but require training on multiple output scales simultaneously for multiscale representation at inference time [8) Our work is inspired by BACON, but significantly modifies the architecture, initialization scheme, and training techniques to make efficient multiscale optimization techniques compatible with coarse-to-fine reconstruction.
Cryo-EM Reconstruction. While coordinate networks have previously been used for cryo-EM reconstruction 15| 41 6). they differ from our method in that they explicitly represent signals in the Fourier domain. Our goal is to demonstrate coarse-to-fine reconstruction methods with coordinate networks in the real domain, and we demonstrate this for cryo-EM reconstruction. The proposed method may also prove promising for advanced cryo-EM methods that resolve intrinsic motion of structures since spatial motion can be directly modeled in the primal domain [53]
3 Background
Coordinate neural networks have emerged as effective tools for approximating complex spatial data (e.g.. see
see [1 2 I.). In the case of images, for example, they provide mapping from continuous positions on the image plane to RGB values. Mapping 3D coordinates to volumetric density allows modeling 3D geometry. In particular, these networks have been shown to be readily trained to accurately approximate diw-dimensional, complex signals in a memory-efficient way. Among this broad family of network architectures, in this paper focus Multiplicative Filter Networks (MFNs) / such as BACON g) as they provide explicit control over the Fourier spectra of the function approximations.
Multiplicative Filter Networks. Most coordinate-based networks, like SIREN 2 and Random Fourier Features
B use MLP architecture consisting of the successive composition of linear
3

---

transformations and element wise non non-linearitis linearities. In contrast, Multiplicative Filter Networks (MFNs) use Hadamard product (i.e.. element- wise multiplication) instead of composition 7 Concretely, in an I--ayer MFN, the din dimensional input coordinates, x E IRdin, are transformed using T. + non-linear filter modules, denoted by (.)= Rddt. + RRdh, i = 0, o....... L, and d, is the hidden layer dimension. In 77. either sinusoidal or Gabor filters are used for this transformation. The sinusoidal case given by (x) sin(w(() + ())), where w(() E RRdxx3 and Φ()) E IRdh are referred to as frequencies and phases. At layer i, the intermediate representation of previous layer, z (i-11), after being linearly transformed, is multiplied by the non-linear filter of input g (x). Formally,
z (0) (0) (x), z ()) g ( (x)
+ bb)), (()))
i= L,
The behaviour of MFNs can be understood by analyzing the frequency of intermediate layers. Using the trigonometric identity
1 2
(a) sin(b) sa + x/22 + sin(a + x x/2).
closed-form expressions for intermediate representations can be derived 7
S Dropping the bias
terms ((0) for simplicity, the individual components of the intermediate representation (+) is equal to a weighted sum of an exponential number of sine terms
Σ
(n) sin (iii n s) x (n.s))
(3)
where (no: 11 1, T2:) is tuple of indices with nj E {l,..., dr} and sj [-1, Each term in the summation comprises an amplitude, frequency and phase shift, given by
(n,s)
-wo (0)
Ei= swi!
(n)
(4)
2:
nn
2no E ss (1 1100--3 -)
(0)
BACON. This analysis of the frequency content of an MFN shows that the bandwidth of z (9) (x) is the sum of the imput bandwidths. This was leveraged in BACON [ to band-limit the representation of each individual layer. Additionally, an output was created for each layer to produce intermediate reconstructions; i.c.
y(
b
where (-( out E Re da and b E Rdd,, However, to encourage y() to preserve its signal represen- tation at scale i, one requires extra scale-specific losses during training, without which, subsequent training will corrupt the lower resolution representation.
spectral bandwidth, when adopted in the context of a coarse-to- to-fee optimization procedure, it fails to preserve and carry the learned coarse-scale representations when one moves to the next finer-scale stage of the optimization. Consequently, at each round, one must re-optimize the entire representations at all previous scales, hindering efficient coarse-to-fine optimization. As example, in Fig. 3 we examine the representation learned by BACON within a coarse-to-fime fine training strategy, where successive output layers are trained fit an image at finer scales. We apply staged training procedure such that loss applied to outputs at each scale in separate rounds of optimization. Unfortunately, when fitting finer scales BACON completely forgets the learned representations of coarse scale outputs from previous optimization rounds (see supplement). For fairer comparison, given any round, we keep all scale-specific losses, but let only the linear output layers for other scales to updated. Still, the results of BACON (Fig. B, top row) show that the representations at coarser scales yet become corrupted while training at finer scales. In fact, reconstructions at different scales are highly coupled in BACON, since new layers can result in new frequencies anywhere within the band limit. In the following, we modify the architecture and its initialization and training scheme (Fig. to make coordinate networks applicable to coarse-to-fine multiscale reconstruction.
Motivating Example. Although BACON does provide control

---

Prequency Support Previous Layer
0]
r- (1,
86
(=
r= (0,
Frequency Support of Current Layer
Figure 2: Illustration of the initialization scheme. (Left) The central red square depicts the base spectrum comprising frequencies of the previous layer (n. Other coloured regions depict copies of the base spectrum, shifted versions by A- in four directions with X = 2. The copies are slightly larger due to the perturbation by the random v Dashed regions are naturally introduced
because of the sign factors 8; = +1. (Right) The new frequeney spectrum of (+)) (n, s) the union of the shaded regions. The initialization controls the overlap with the spectrum of the previous layer.
Residual Multiplicative Filter Networks
From Eq. AI recursion
we can express frequencies in layer i in terms of frequencies in previous layers with the
n s) = :h (n, + s,w!!!
where and are formed from n and s by removing last entry, e.g., ń = (ng... ,7;-1, nzi. Thus (t) contains frequencies from the previous layer, shifted by wi:; and -wn; (1) where w is the frequency hidden unit n, in layer i. More specifically, assuming that frequencies in the previous layer, i.e., ( (ū, $), are band-limited [- B;- + B;-.]. those (+)) lie the union several of its shifted clones. From this perspective, the frequencies at layer i, (f), characterize two important concepts: 1) the growth in bandwidth from the previous layer, and 2) the extent of overlap between the previous layer frequency spectrum and each new individual clone. For instance, given entries of (1) in 1-5,+0], +-), the range of frequencies represented in ( expands to B, 8, B + 5).
Also, if specific (1) is set to relatively large value of 2B, this results in clone with frequency support in the range [B,-1, 3B) and avoids overlap with the original bandwidth [- B B
4.1 Frequency Initialization
Motivated by this interpretation of (+), we propose a new initialization scheme which explicitly controls the overlap and expansion of frequencies in successive layers. For simplicity, we describe this initialization for the 2D case, though can be extended higher dimensions straightforwardly First, we follow a procedure similar to BACON [ to initialize and fix frequencies of filters which allows limiting outputs to specified bandwidths. Then, given the band-limited support [-B,-1, + +B i] of
frequencies in layer - 1, we initialize the 2D vectors w/ as follows:
w = >2B (3) (1) 1 UR)),
+V3
(0) vi U(-MBLI.
where R = {0,, 1), (1, 0), (1, and U' denotes either continuous or discrete uniform distributions. Here. ( is composed of two terms. The first term mainly specifies the direction (r()) and the proportional amount (,- by which frequencies of layer 1 will be shifted. The second term is a relatively small random perturbation to break the symmetry. An intuitive illustration of these operations is provided in Fig 2 In 2D, there are 8 different directions to expand, 11 entails two shifts in opposite directions ((r,,,,r),2) Thus we only consider 4 possible values R. Using this initialization z()) is thus band-limited to (9)) E [1 + A B,, (1 + a B-1).
half of which are redundant since
5

---

Reconstructions
Fourier Spectrum Difference
Scale
Scale
Scale
Scale
Scale
Scale
Vanilla (fair)
Residual
Residual New Initializaatton
Figure 3: Coarse-to-fine image fitting experiments. We compare fair BACON (Vanilla), residual MFN with the BACON initialization scheme, and residual MFN with the proposed initialization. The proposed method performs best at retaining image quality at each scale during the staged training regiment. We also compare the difference between the Fourier spectra at each output scale immediately after the training stage and at the end of training (Right). In the proposed method, residual connections allow information at each scale to be re-used while other methods overwrite lower frequency information during coarse-to-fine training.
4.2 Residual Connections
Consider setting λ2 = 2 A during initialization. In this case, by design, the frequency spectrum for sine terms in layer would thus not overlap at all with that of layer 1; in at least one direction, the frequencies would either end up left or right of interval [- BBB B Thus the output layer would include no sine terms with frequencies in the central interval B B. To obtain the output of layer i, we sum it with the output of layer
That is,
( = ((-1)) + z + (1)
This can be interpreted as residual connection that ensures we keep identical copy of frequencies from the previous layer; i.e., y (i-1) will fill the gap in the frequency space. In practice, rather than the setting A2 Å₁ one can use other less strict combinations which allow for different amounts of overlap and expansion. We found empirically setting x2 2, a1 = 0.3 to be effective.
2D Image Fitting
We first evaluate each of the proposed modifications to band-limitee limited networks in the context of 2D image fitting. This task useful for didactic purposes, allowing one clearly the benefits of skip connections and our new initialization scheme in the context of coarse-to-fing image reconstruction. We adopt a staged training procedure and fit series of band-limited coordinate networks natural images provided by M. The dataset consists of 32 RGB images, each of size 512 x 512. We first downsample the images by factor of obtain images of size 256 x 256. These images are fit during training using Mean Squared Error (MSE) training objective, applied to the output at the scale corresponding to the stage of training. As previously described, for the fairest comparison, the "Vanilla" BACON baseline is optimized with losses at all scales, though, at the stage of given scale, only the linear output layers of other scales can be updated. Still, this keeps the optimization of the internal representations staged. All networks use four-layer architecture and are trained 10, 000 iterations, divided into chunks of 500,500, 1000, 8000 for optimizing output scales {1, 2, 4), respectively. The frequencies for all methods are initialized to have the same band-limit consistency. All methods were implemented in PyTorch and trained on single NVIDIA GeForce RTX 2080.

---

Table 1: Coarse-to- image fitting results. We show the mean absolute difference of Fourier spectrum of outputs at each scale immediately after optimization and after the entire training regiment and peak signal-to-noise ratio for the image generalization task. Residual connections and the new initialization help to preserve the spectrum during training, especially for the second and third scales. The final reconstruction achieves competitive generalization results in terms PSNR metries.
Method
Mean Abs. Difference
PSNR (dB)
Scale
Scale 2
Scale 3
Scale 4
SAPE
9
27.02 + 3,29
BACON S
28.39 = 3,66
(Staged) BACON (fair)
1.61 + 0.45
7.36 + 1.84
17.97 + 4.27
28.46 = 3.61
(Staged + Res) BACON
4.33 + 0.97
5.16 = 1.22
9.23 + 2.73
27.90 = 3.27
(Staged + Res + Init) BACON
3.60 + 0.95
3,42 = 0.85
4.32 + 1.35
28.33 = 4.42
Figure 3 visualizes the outputs at each scale for BACON, the residual MFN with the band-limited BACON initialization, and the residual MFN with our proposed initialization. Since the coarse-to-fine training scheme can potentially disturb previously trained outputs when applying the loss at the finer scales, we visualize the differences between the Fourier spectra at each output scale after the corresponding training iterations and after the full training regiment (Fig. B right). The proposed method shows the least disturbance after training. For the entire image dataset, we report the Fourier spectrum mean absolute difference and standard deviation in Table !
Finally, we examine all networks the image generalization task by querying them finer grid of 512 x 512 and computing the Peak Signal-to- -ooal-to-Noise Ratio (PSNR) between reconstructed signal and the original ground truth. The average PSNR values over all images is summarized in the last column of Table 1 We also benchmark SAPE 9 as a baseline for coarse-to-fine reconstruction, on the image generalization task. SAPE gradually exposes higher frequencies as function of time and space, while providing only a single reconstruction at a time, so only their full-scale reconstruction can be evaluated for comparison. While our approach improves interpretability of the reconstructions, by enabling reuse of previous learned representations, it achieves empirical performance competitive with baselines. Additional experiments on a textual dataset are provided in Appendix A
6 Cryo-EM 3D Reconstruction
Single particle cryo-EM is an increasingly popular experimental technique to recover the 3D structure of macromolecular complexes, such as proteins and viruses. Cryo-EM has gained widespread attention in the last decade with pioneering advances in hardware and data processing techniques. This has enabled atomic or near-atomic resolution reconstruction of challenging structures (55). Multiscale, coarse-to-fing fine reconstructions has long played critical role in cryo-EM estimation, e.g.. 20 making it an ideal testbed for our proposed methods.
Image Formation Model. Based on simplified model (56]. the protein structure is represented as function V: 3 -> R2 which maps points in 3D space to non-negative real values, expressing the Coulomb potential induced by the atoms. Images (T. / are captured with transmission electron microscope which are 2D orthogonal projections of randomly oriented potential maps. Each image I, has a corresponding latent pose (orientation) R; E SO(3) representing the 3D rotation of the molecule the image. practice, to ensure sufficient contrast, the microscope defocused when images are captured, resulting in point-spread function (PSF) gs. To minimize radiation damage to delicate biological molecules, samples are exposed to low dosages electrons resulting notoriously noisy images. Taken together, the image formation model can be expressed
(x, = gi * V xz +e, x 2))
where the projection is, by convention, assumed to be along the z-direction after rotation, corre- sponds to convolution and is Gaussian noise [14 4 The PSF can be estimated in a preprocessing step and is assumed here to known; for more details see [57]. (A more detailed formulation for
cryoEM reconstruction is given in Appendix B
7

---

PDB101S
PDB4ake
(a) 1o15
(b) PDB4ake.
(c) Sample images.
Figure 4: Ground truth potential maps and cryo-EM measurements. (Left) Potential maps generated using the corresponding atomic models. (Right) Noisy example images from a synthetically generated particle stack after applying the PSF and adding white Gaussian noise.
3D Reconstruction Problem. The goal of a cryo-EM experiment is to estimate the unknown 3D potential map of the biomolecule present in the observed images {1))-11 This task has some intrinsic challenges. First, the images are extremely noisy (see Fig. 4c Second, the ground-truth poses are typically unknown, or only a rough estimate is available, so it is not possible to perform 3D reconstruction using the forward model of Eq. directly. Previous work 6 4l [9] estimate pose parameters during reconstruction with various techniques including amortized inference, variational bounds or direct marginalization. Here use stochastic gradient- based method, which optimizes the representation of the 3D structure V. parameterized by our proposed residual MFN, together with the unknown pose variables (R; ) The loss function is the squared error between the observed image I; and that generated by the model, i.e.,
L= Σ (3,,,)
(10)
where I7 (z. y) is given above in Eq.
g See Appendix
D for more detail the pose parameterization
and optimization.
In practice, naively optimizing for both the poses and potential map suffers from poor local minima. To avoid this, coarse-to-fing fine reconstruction approaches are commonly employed. At early stages, low frequency content of the map is estimated as the SNR is higher so estimation is casier, which then allows for more reliable subsequent pose estimation as higher frequencies are included. Previous approaches [4] [4] employ this technique, called Frequency Marching (20), as they model the map in the Fourier domain where individual frequencies are readily accessible. In contrast, we address the reconstruction problem in Searse-to-fine manner while expressing the potential map in real space.
6.1 Experiments
We evaluated our approach on two synthetic datasets which allow us to directly evaluate against ground truth structures. The datasets are based on atomic models of 10L.5 htttp:://s/t :/www. /www. rcsb and 4AKE (https://wwil rcsb. org. wtt.ccss.......ccccoooeeeeeeeeee
wttg//stuutuureeeaeee from Protein Data Bank (PDB). We first convert atomic models to 64 x 64 x 64 discrete potential maps, with voxel sizes Á and 1.2 Á respectively, using pdb2mrr.py module from EMAN2 (58). Then we simulate the image formation model as in Equation at 50, 000 rotations uniformly sampled on 50(3). To generate each image, we rotate the map V. resample it on the original grid, project it along canonical z-direction and apply the PSF Zero-mean white Gaussian noise is added to these simulated clean images produce realistic SNR of 10%. The structures and sample images visualized in Fig. 4
Experimental Settings. We use 3-layer network with 128 hidden units to represent the 3D structure. The network outputs the structure in 3 hierarchical scales and, accordingly, the training process divided into stages, one for each scale. We set the hyperparameters as X 0.3. X 2.0 and train the network for a total of 100 epochs. For PDB lol5, we spend respectively 15, 15, and 70 epochs optimizing the first to the third scales. For PDB4ake, we find a better reconstruction if we dedicate more epochs on the first and second scales, namely 25 epochs on each of the first and second scales. When computing the loss, images are band-limited to the corresponding scale using Gaussian filter whose scale parameter is set to match the bandwidth of the current stage
8

---

CryaSPARC
Proposed
plats
Sodee AMA
2.13
POBToIS
244
10c
PDB4ake
1.
10
13 XIS
STIA 4NZA SRME EOIA 2AM
Higher Scale
Figure 5: (Left) CryoSPARC reconstructions. (Middle) Our reconstruction at multiple scales. (Right) Fourier Shell Correlation plot measuring correlation in Fourier spectrum between reconstructions and ground-truth structure. Threshold 0.5 is the criterion to compute the resolutions in Angstroms.
0.X 5.5
RE
Figure 6: (Left) Histogram geodesic distances between estimated and ground-truth poses. Some example reconstructed images are illustrated well. The first row for each dataset contains particle with the estimated pose within most 4 degrees away from ground-truth while the second row corresponds to another with totally wrong pose estimate. (Right) Getting stuck at local minima when optimizing the full-scale reconstruction from scratch (not coarse-to-fime Although, the overall shape is well captured, significant improvement is achieved during several iterations and poses stuck at local minima, mostly concentrated around 45 degrees away ground-truth,
scale. Mini-batches of images are used update poses and weights of the network representing the structure. For each mini-batch, we alternate between optimizing the structure and poses for 5 and 20 iterations, respectively. We use Adam optimizer parameters and higher learning rate of 0.01 for pose parameters. Each pose parameter is initialized to random value uniformly distribution over a geodesic distance of to 90 degrees from the ground truth. This can be considered
[59] with learning rate 0.001 for the network
very crude initial estimate for poses. In Appendix E we consider even coarser initial pose values, showing that they achieve competitive qualitative and quantitative performance to those here. Experiments are implemented in Py Torch [54], models are trained on NVIDIA TITAN Xp GPUs, and ChimeraX 60 is used to visualize the 3D maps.
Results. Qualitative comparison of reconstructions with those obtained by homogeneous refinement of cryoSPARC (as the baseline) are illustrated Fig. 5 In the middle, we observe that, from low to high scales, increasingly more structural features are resolved. The low-scale reconstructions capture only the global and low-resolution features, whereas the full-scale ones contain fine- one-grained details. Our approach is able well-resolve some peripheral parts of the structure, especially for the more

---

complex structure of PDB4ake. Also there are some structural features learned by our reconstruction yet totally missed by cryoSPARC. All in all, our final reconstruction visually looks competitive or better when compared with the baseline.
As shown on the right side of Fig. [5] to quantitatively evaluate reconstructions, we compute the Fourier Shell Correlations (FSCs)
between the reconstructed map and the ground-truth. FSC
the gold-standard metric in cryo-EM area measuring the normalised cross-correlation coefficient between two 3D volumes in Fourier domain along shells of increasing radius (see Appendix C for more background on FSC). We measure FSC between ground-truth and reconstructions at all scales, following the convention of the threshold 0.5 to compute final resolution, as ground-truth is known priori in our experiments. We also evaluate how well the poses are optimized and converge to the ground-truth. To this end, the histogram showing number of images with various geodesic distances between estimated and ground-truth poses are visualized as in Fig. G For both datasets, there is a concentration of images on left hand side corresponding to slight error in estimation of pose parameters. Thus, for most of the images, predicted pose parameters are within few degrees of the corresponding ground truth, verifying that poses are optimized well along with the reconstruction. Furthermore, per dataset, we select two sample images, one with almost the correct pose estimate, and the other with one that is far from ground truth. Then we demonstrate their reconstruction at all scales as in Fig. g Finally. we conduct an experiment in which we supervise the network only at the full-scale reconstruction throughout the entire training. The output structures are depicted in the right panel of Fig. 6 realizing a situation where poses are trapped in local minima, and we thus observe no substantial improvement in the structure for many iterations.
Conclusions, Limitations and Future Work
In this paper, we build on and training strategy providing fine-grained control over spectral characteristics of network outputs, yielding a multiscale representation suitable for coarse-to-fine reconstruction. The combination of skip connections and a consistent initialization scheme allows the proposed network to ensure lower levels of the network represent the coarser scales of the signal which are then reused at finer scales. We empirically show that these modifications enable coordinate networks to fit natural images in coarse-to-fine fashion. They are also shown to be effective for cryo-EM reconstruction.
Coordinate Network (BACON) and propose a new architecture
Limitations. We find that the proposed multi-scale optimization procedure is effective in avoiding local minima in the Cryo-EM reconstruction problem. We have yet to explore whether residual MFNs are also effective on other problems with local minima due to the existence of nuisance variables. Although beyond the scope of this paper, it is also possible that minor modifications of the architectural and hyper-parameter settings would improve the robustness of the optimization; e.g., increasing the number of layers for more fine-graine scales, or a more diverse inspection of hyperparameters. Moreover, the cryo-EM experiments in this paper used synthetic data, to enable more thorough the evaluation of the method. Real datasets are likely to require the use of other bases as well, e.g., Gabor, which are better able to represent the spatial variability of resolution in cryo-EM reconstructions. We have also focused on homogeneous and rigid structures. An advantages of our approach is that the efficient multiscale representation of the signal in the real domain opens promising doors for modeling flexible structures, a major open problem in cryo-EM 4 (3
Societal Impact. Cryo-EM for macromolecular structure determination is one of the foremost exciting areas in molecular biology with enormous social impact. Cryo-EM tools were key to determining the structure of the S SARS-COV2 spike protein, the discovery of its pre-fusine pre-fusion conformation, and the assessment of potential medical countermeasures. Advances in this field enable the discovery and exploration of large protein complexes and viruses more generally (62). On the other hand, we strongly condemn any usage of our proposed coordinate networks for generating malicious representations, improperly modifying signals, or spreading misinformation.
Acknowledgments and Disclosure of Funding
We thank Ali Punjani for numerous valuable discussions and for use of the cryoSPARC software package. This research was supported in part by the Province of Ontario, the Government of through NSERC, CIFAR, and the Canada First Research Excellence Fund for the Vision: Science to Applications (VISTA) programme, and by companies sponsoring the Vector Institute.
Canada,
10

---

References
[1] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In Proc. ECCV, 2020.
[2] Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
[3] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Proc. NeurIPS, 2020.
[4] Ellen D Zhong, Adam Lerer, Joseph H Davis, and Bonnie Berger. CryoDRGN2: Ab initio neural reconstruction of 3D protein structures from real cryo-EM images. In Proc. ICCV, 2021.
[5] Ellen D Zhong, Tristan Bepler, Bonnie Berger, and Joseph H Davis. Cryodrgn: reconstruction of heterogeneous cryo-EM structures using neural networks. Nature Methods, 1882):176-185, 2021.
[6] Axel Levy, Frédéric Poitevin, Julien Martel, Youssef Nashed, Ariana Peck, Nina Miolane, Daniel Ratner, Mike Dunne, and Gordon Wetzstein. CryoAl: Amortized inference of poses for ab initio reconstruction of 3D molecular volumes from real cryo-EM images. In Proc. CVPR, 2022.
[7] Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and Zico Kolter. Multiplicative filter networks. In Proc. ICLR, 2021.
[8] David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proc. CVPR, 2022.
[9] Amir Hertz, Or Perel, Raja Giryes, Olga Sorkine- Sorkine-Hornung, and Daniel Cohen-Or. SAPE: apatially-adaptive progressive encoding for neural optimization. Proc. NeurIPS, 2021.
[10] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In Proc. ICCV, 2021.
[11] Edward H Adelson, Charles Anderson, James R Bergen, Peter J Burt, and Joan M Ogden. Pyramid methods in image processing. RCA Engineer. 29(6):33 41, 1984.
[12] Eero P. Simoncelli and William T. Freeman. The steerable pyramid: A flexible architecture for multi-scale derivative computation. In Proc. ICIP, 1995.
[13] James H Bramble. Multigrid methods. Chapman and Hal//CRC, 2019.
[14] Ali Punjani, John Rubinstein, David J Fleet, and Marcus Brubaker. cryosparc: algorithms for rapid unsupervised cryo-EM structure determination. Nature Methods, 14(3):290-296, 2017.
[15] Sjors HW Scheres. Relion: implementation of bayesian approach cryo-EM structure determination. Journal of Structural Biology, 180(3):51--530, 2012.
[16] Wang Yifan, Lukas Rahmann, and Olga Sorkine-Hornung. Geometry--onsistent neural shape representation with implicit displacement fields. In Proc. ICLR, 2022.
[17] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: Bundle-adjusting neural radiance fields. In Proc. ICCV, 2021.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016.

---

[19] Youssef SG Nashed, Frédéric Poitevin, Harshit Gupta, Geoffrey Woollard, Michael Kagan, Chun Hong Yoon, and Daniel Ratner. CryoPoseNet: End-to-end simultaneous learning of single-particle orientation and 3D map reconstruction from cryo-electron microscopy data. In Proc. ICCV, 2021.
[20] Alex Barnett, Leslie Greengard, Andras Pataki, and Marina Spivak. Rapid solution of the cryo-em reconstruction problem by frequency marching. SIAM Journal on Imaging Sciences, 100))11100-115, 2017.
[21] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In Proc. IROS, 2021.
[22] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. Proc. CVPR, 2021.
[23] David B. Lindell, Julien N. P. Martel, and Gordon Wetzstein. Autolnt: Automatic integration for fast neural volume rendering. In Proc. CVPR, 2021.
[24] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Proc. NeurlPS, 2021.
[25] Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proc. ICCV, 2021.
[26] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In Proc. CVPR, 2019.
[27] Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In Proc. ECCV, 2020.
[28] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In Proc. CVPR, 2020.
[29] Eric R. Chan, Connor Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. Proc. CVPR, 2022.
[30] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proc. ICCV, 2019.
[31] Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In Proc. CVPR, 2021.
[32] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In Proc. CVPR, 2019.
[33] Albert W Reed, Hyojin Kim, Rushil Anirudh, K Aditya Mohan, Kyle Champley, Jingu Kang, and Suren Jayasuriya. Dynamic CT reconstruction from limited views with implicit neural representations and parametric motion fields. In Proc. ICCV, 2021.
[34] Yu Sun, Jiaming Liu, Mingyang Xie, Brendt Wohlberg, and Ulugbek S Kamilov, Coil: Coordinate-bal based internal learning for imaging inverse problems. arXiv preprint arXiv:2102.05181, 2021.
[35] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3D shape. In Proc. CVPR, 2020.
[36] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3D scenes. In Proc. CVPR, 2020.
12

---

[37] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands tiny mlps. In Proc. ICCV, 2021.
[38] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Proc. NeurIPS, 2020.
[39] Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. ACORN: Adaptive coordinate networks for neural scene representation. ACM Trans. Graph. (SIGGRAPH), 40(4), 2021.
[40] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Proc. ECCV, 2020.
[41] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proc. ICCV, 2019.
[42] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one few images. In Proc. CVPR, 2021.
[43] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In Proc. CVPR, 2021.
[44] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3D-aware image generation. In Proc. CVPR, 2022.
[45] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. Stylesdf: High-resolution 3D-consistent image and geometry generation. In Proc. CVPR, 2022.
[46] Martin Veterli and Cormac Herley. Wavelets and filter banks: Theory and design. IEEE Trans. Signal Process., 4009):207--2332, 1992.
[47] Olivier Rioul and Martin Vetterli. Wavelets and signal processing. IEEE Signal Process. Mag., 844):14-38, 1991.
[48] Stéphane Mallat. A wavelet tour of signal processing. 1999.
[49] Peter Burt and Edward Adelson. The laplacian pyramid as a compact image code. In Readings in Computer Vision, pages 671-679. 1987.
[50] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. MICCAI, 2015.
[51] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In Proc. CVPR, 2021.
[52] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In Proc. ICCV, 2021.
[53] Ali Punjani and David J Fleet. 3D flexible refinement: structure and motion of flexible proteins from cryo-EM. BioRxiv, 2021.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-peeformance deep learning library. Proc. NeurIPS, 2019.
[55] Werner Kühhbrandt.. The resolution revolution. Science,
2014444,,44
[56] Earl J Kirkland. Theory calculation of images of thick specimens. Advanced Computing in Electron Microscopy, pages 143-195. 2020.
13

---

[57] Joseph A Mindell and Nikolaus Grigorieff. Accurate determination of local defocus and specimen tilt in electron microscopy. Journal of Structural Biology, 142(3):334-347, 2003,
[58] Guang Tang, Liwei Peng, Philip Baldwin, Deepinder S Mann, Wen Jiang, lan Rees, and Steven J Ludtke. EMAN2: extensible image processing suite for electron microscopy. Journal of Structural Biology, 157(1):38 46, 2007.
[59] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proc. ICLR, 2015.
[60] Thomas D Goddard, Conrad C Huang, Elaine C Meng, Eric F Pettersen, Gregory Couch, John H Morris, and Thomas E Ferrin. Ucsf chimerax: Meeting modern challenges in visualiza- tion and analysis. Protein Science, 24(1): 2018.
[61] Marin Van Heel and Michael Schatz. Fourier shell correlation threshold criteria. Journal of Structural Biology, 151(3):550-200 2005.
[62] John H Van Drie and Liang Tong. Cryo-EM powerful tool for drug discovery. Bioorganic medicinal chemistry letters, 30(22) 2020.
[63] Pawel A Penczek. ciree-dimensional spectral signal-to to-ioee ratio for a class of reconstruction algorithms. Journal of Structural Biology, 138(1-2):34-46, 2002.
[64] Ali Punjani, Haowei Zhang, and David Fleet. Kni-Umiform Refinement: Adaptive regu- larization improves single particle cryo-em reconstruction. Nature Methods, 17:1214-1221, 2020.
14


Reviews:
[{"review_id": "KidkbyFBzZM", "reviewer": "Reviewer_Vy34", "summary": "The authors treat the problem of coarse-to-fine optimization for solving inverse problems, which are particularly important for 3D reconstruction. They note that the existing state-of-the-art coordinate networks used to solve these problems are ill-suited for multiscale optimization since they can forget low frequency details learned during earlier stages in the training scheme. The authors propose residual multiplicative filter networks which (1) introduce skip connections between layers with different output scales so that coarse features learned early in the optimization process are explicitly not forgotten and (2) use an initialization scheme that allows for control over the frequency spectra learned by each layer, meaning that the network can avoid learning redundant frequency information in finer-scale layers. Several experiments are presented for 2D and 3D reconstruction problems where the proposed method is shown to be superior to existing approached in capturing details at several scales.   ", "questions": "Related to my first point in the weaknesses section, it would be useful if the authors could provide a description of why the multi-scale metrics (e.g. as shown in the FSC plots in Fig. 5) matter in the context of reconstruction problems. My current understanding is that these metrics are provided to demonstrate that the proposed method has stable training during multi-scale optimization and that it successively adds fine-grained detail due to the innovations presented in the paper. However, I am not sure if I am supposed to interpret these metrics as useful for actually solving the reconstruction or inverse problem. My intuition is that as long as the final (i.e. the last scale during training) output is consistently useful, then the intermediate training qualities do not matter nearly as much. ", "limitations": "The authors provide a small list of limitations in the second paragraph of the conclusion. However, the section reads as a justification for some design and experimental choices rather than a discussion about the shortcomings of the proposed method. I would suggest that the authors make a more explicit limitations (sub)section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\n- I like the intuitive breakdown of the intermediate representations as a sum and shift of frequency sprectra from the previous layers. This provides a straightforward motivation for the proposed method. \n\n- The visual representation in Figure 6 gives a clear and concrete example of why coarse-to-fine optimization is important for 3D reconstruction problems.\n\n- Detailed background is given on 3D molecular reconstruction problems, which is useful for the larger ML community that may not be familiar with such domains.\n\n\nWEAKNESSES:\n\n- I do not understand the benefits of having the outputs at each scale \"shift\" very little (e.g. as in the results in Table 1). I would think that as long as the final output PSNR is satisfactory, then the internal shift in frequency repsresentation does not matter. Looking at Table 1, it seems that even though fair BACON has quite a bit of shift in its frequency representations at each scale before and after optimizing the other scales, the output still has good PSNR.\n\n- I believe that a more thorough background and introduction to coordinate-based networks in general could be given. While some details are given (e.g. that the networks take 2D or 3D coordinates and output some value like RGB), it was not immediately clear to me what the purpose of these networks (e.g. compressing a high-dimensional signal) was compared to other models (e.g. generative models).  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EgLvuk7mB8A", "reviewer": "Reviewer_9iUQ", "summary": "The paper describes an extension of the recently proposed Multiplicative Filter Networks (MFN) that permits an independent parametrization of different levels of scale (Fourier-bandwidth) by introducing residual connections and a suitable initialization scheme.\nMFNs are a simplification of positionally encoded networks / coordinate networks that have a special form of building the result as a linear combination of (potentially) exponentially many basis functions that are controlled non-linearly, achieved by multiplication with Fourier (or Garbor) basis functions in each \"level\" of the network (in the 2D/3D case considered here, the dimension of the output [function] space does not seem to be exponential though, bringing this closer to a classical linear representation) . The related BACON network extends this idea by providing initialization schemes that yield nested levels of scale (increasing resolution), but the networks are constructed in a way such that changes to higher level \"fine\" details invalidate the fit of lower layers.\nThe submission addresses this issue in a simple but effective way by adding short-cut connections that carry the lower-layer function-fits to the next-higher layer unaltered, and building-up a new detail representation on top (which, to my understanding, can show complex scale dependencies). In addition, a suitable initialization is proposed that creates nested levels of detail in the Fourier-spectrum covered (again, to my understanding, targeting the low-dimensional regime of maybe 2D or 3D coordinates).\nResults for image fitting are improved over previous methods such as BACON. An example application to CryoEM reconstruction is presented that seems to outperform a well-known approach (characterized as SOTA).\n\nThe main contribution is a scale-separated version of MFN/BACON. The applications to CryoEM are also interesting but the rather brief evaluation with only a few example reconstructions move the methodological contribution more in focus.\n", "questions": "- Does it makes sense to compare against linear base-lines? If so, what results would you get (or expect)? If not, why is it more expressive (and would it still be if the scales were fully separated [if I understood this correctly])?\n- Is the method scalable to larger dimensions?", "limitations": "If my understanding of the limitations in terms of dimensionality were correct (see discussion question above), it would be useful to discuss this more in the paper.\nIt might also be useful to mention the limitations of multi-scale optimization for avoiding local minima (in my experience with shape matching & reconstruction problems only very vaguely related to the CryoEM example, it helps somewhat but does not eliminate fundamental issues with bad local minima).", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To explain my view of strength and weaknesses, I would first discuss my understanding of the technical aspects a bit more in depth:\n\nI understand the main goal of the paper as to decouple scales in an MFN-style coordinate network. To my understanding, this is already achieved by the short-cut connections, which yield a superposition of the low-frequency part, solely controlled by low-frequency parameters, with a high-frequency part that depends on both low- and high-frequency parameters. For progressive coarse-to-fine-optimization, this seems to be sufficient, while at the same time being conceptually and technically very easy to understand and implement. A suitable initialization makes sure that progressive parts of the spectrum are covered (by introducing corresponding control parameters); this scheme seems to work only in moderately high dimensions (as it needs to cover Fourier-coefficient space with 2^d or 3^d patches of coefficients). Consequently, it is not fully clear how the decoupling would affect higher-dimensional applications, but they are not the target of the paper (and a lot of previous work, where similar representations are used for example for reconstruction from 3d point clouds).\n\nI think that the concern for scale separation is a valid goal, and results on image fitting in the paper at least indicate subtle advantage of the decoupled representation (although a strong explanation is missing; the case for CryoEM is clearer as multi-scale fitting is commonly used there). It would be useful to understand more clearly how important this is in an overparametrized regime, as multi-scale optimization is often used as a heuristic to counter bad local minima in non-overparametrized, non-convex optimization scenarios.\n\nFrom a conceptual point of view, I do not fully understand the boundaries (as in representational capacities and ability to find solutions) between linear scale space approaches (mip-maps, wavelets), MFN-approaches (linear in $x$, non-linear in $\\theta$), and the current proposal. If - hypothetically - the next-higher frequency band was fully separated by duplicating the required low-res coefficients, would the resulting method be substantially different from an ordinary hierarchy of Fourier-basis functions? How different (how much better) are results in comparison to a simple base-line (say, a linear wavelet or pyramid representation)? It would be useful to see a direct base-line comparison (or maybe arguments for why this is clearly inferior - I do not know this area too well).\n\nSummarized as a list, I would see the following strength...\n+ Simple and straightforward idea. Easy to implement.\n+ Canonical extension of MFN/BACON (users of those methods probably want to know about this one).\n+ Good results with minimal (extra) effort (Seems to yield SOTA on CryoEM)\n+ Well written paper.\n\n...and weaknesses:\n- Experimental evaluation remains a bit anecdotal (I guess that the CryoEM fitting is very expensive to setup and conduct).\n- Very simple, maybe incremental idea (could also be seen as a limitation of the contribution)\n- Limited explanation of \"why\" and \"how far\" it works.\n- Could have more comparisons against base-line\n\nOverall, I would guess that the paper might be a simple but frequently employed addition to MFN-methods and thus potentially impactful (with many citations and applications). For this reason, I tend towards a positive recommendation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AqFnG3g-UT9", "reviewer": "Reviewer_bz3B", "summary": "This paper proposes two modifications to the recently introduced multiplicative filter networks (and their bandlimited variant BACON): 1) by introducing skip connections it enables explicit layerwise bandwidth control, 2) by modifying initialization scale it makes the spectral supports of layers more or less disjoint. The benefit of doing this is that it enables coupling with standard multigrid / frequency marching approaches to solve inverse problems. Thanks to the introduced modifications the low-frequency content of the function implemented by the network need not change when fitting higher frequencies. The authors show how this leads to high-quality reconstructions of molecules in synthetic CryoEM experiments.", "questions": "- What happens with your Cryo-EM reconstructions if you initialize orientations randomly? How important is the current coarse initialization?\n- How exactly do you choose the hyperparameters $\\lambda$ and epochs / scale count? You state that for PDB4ake you find a better reconstruction if you dedicate more epochs to the first and second scales. Do you do this by comparing the reconstruction to the ground truth or simply eyeing the reconstruction?\n- In Cryo-EM, how do other implicit representations (non-marching) perform?", "limitations": "The discussion of limitations is adequate.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem addressed by the paper is very clear, the paper is well written, the numerical experiments seem well executed and the results are compelling.\n\nThe interventions in the architecture and training are simple. They yield solid (but not staggering) improvements over earlier work. Cryo-EM is an important application but the setting treated here is somewhat simplified (no shifts, no heterogeneity, only moderate noise SNR = 0.1, and a coarse initial orientation estimate). It would be interesting to see how the proposed architecture fares on other problems. problems. \n\nAs far as I can tell the paper is technically correct—there are no formal results, the architecture description is clear and simple, and the successful experiments strongly corroborate the correctness of the derivations.\n\nIn terms of the narrative, I have the impression that achieving perfect scale orthogonality is a bit overstated. It can be certainly play a role in inverse problems like Cryo-EM which are linear (or conditionally linear when given the particle orientations) and where the frequencies don’t mix (by the Fourier slice theorem). (I doubt that it need be ideal.) It may however be a liability or at least not as useful in problems with strong scale coupling. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept