Learning Adaptive Tensorial Density Fields for Clean Cryo-ET Reconstruction
Authors: YUANHAO WANG
Ramzi Idoughi
Wolfgang Heidrich
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural density fields
Coordinate-based representations
Quadtree structure
Cryo-electron microscope

Abstract: We present a novel learning-based framework for reconstructing 3D structures from tilt-series cryo-Electron Tomography (cryo-ET) data. Cryo-ET is a powerful imaging technique that can achieve near-atomic resolutions. Still, it suffers from challenges such as missing-wedge acquisition, large data size, and high noise levels. Our framework addresses these challenges by using an adaptive tensorial-based representation for the 3D density field of the scanned sample. First, we optimize a quadtree structure to partition the volume of interest. Then, we learn a vector-matrix factorization of the tensor representing the density field in each node. Moreover, we use a loss function that combines a differentiable tomographic formation model with three regularization terms: total variation, boundary consistency constraint, and an isotropic Fourier prior. Our framework allows us to query the density at any location using the learned representation and obtain a high-quality 3D tomogram. We demonstrate the superiority of our framework over existing methods using synthetic and real data. Thus, our framework boosts the quality of the reconstruction while reducing the computation time and the memory footprint. The code is available at https://github.com/yuanhaowang1213/adaptivetensordf.

Learning Adaptive Tensorial Density Fields for Clean Cryo-ET Reconstruction
Yuanhao Wang Ramzi Idough Wolfgang Heidrich
King Abdullah University of Science and Technology (KAUST)
fyuanhao. sang, ramzi. idoughi, volggang... heidrichh @kaust edu.sa
Abstract
We present novel learning-basee framework for reconstructing 3D structures from tilt-series cryo-Electron Tomography (cryo-ET) data. Cryo-ET is a powerful imaging technique that can achieve near-atomic resolutions. Still, it suffers from challenges such as missing- wedge acquisition, large data size, and high noise levels. Our framework addresses these challenges by using adaptive tensorial-based representation for the 3D density field of the scanned sample. First, we optimize a quadtree structure partition the volume of interest. Then, learn vector-matrix factorization of the tensor representing the density field each node. Moreover, we use a loss function that combines a differentiable tomographic formation model with three regularization terms: total variation, boundary consistency constraint, and isotropic Fourier prior. Our framework allows us query the density at any location using the learned representation and obtain high-quality 3D tomogram. We demonstrate the superiority of our framework over existing methods using synthetic and real data. Thus, our framework boosts the quality of the reconstruction while reducing the computation time and the memory footprint. The code is available at httpgghhww..oo//uuuaaaaaaaa 3/adaptivetennsordf.
Introduction
Tilt-series Cryo-ET is an important imaging tool used in a variety of fields, such as structural biology [5] and material science [48]. It consists of reconstructing a 3D tomogram from captured 2D projections of the scanned sample. In cryo-ET, due to hardware constraints, the scanning process performed from a limited angular range, typically less than 120". The tomographic reconstruction with missing-wedges is known to be challenging task. In addition, sample motion and deformation that occurs during the acquisition step, induce misalignment between the projections, which deteriorates the reconstruction quality. Also, cryo-ET requires a large amount of data, which makes the reconstruction a resource-intee ieouurceintenssvee task, both terms of computational time and memory. Last but not least, to prevent the sample from getting damaged, the scanning is performed with low-intensity electron beams. This results in high level noise in the captured projections.
In the literature, several approaches have been proposed to overcome some of these challenges. The misalignment and the motion can be corrected using marker tracking [28, 35, 49], even by tracking specific features between the projections [10, 19]. On the other hand, the high level noise was initially reduced by applying image-based based volume- based denoising approaches on the projections or the tomogram respectively. Recently, Bepler et al. [7] proposed to reconstruct two copies of the tomogram using odd/even projections only. Then, using Noise2Noise [29] technique they learn a denoised version of the tomogram. Kniesel et al. [26] combines noise model for Scanning Transmission Electron Microscopy (STEM) and an implicit 3D shape representation a differentiable framework to denoise the reconstructed data.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this work, we propose a new framework for cryo-ET that jointly reconstructs and denoises the tomograms, This framework is based on a quadtree structure that we use to define an adaptive Tensorial Density Fields (TensorDF) representation. Fach node of the quadtree is featured with tensorial density fields to represent the tomogram's 's density at associated region. The loss function that guides the learning process is composed of four terms: differentiable tomographic image formation model, a total variation term that encourages smoothness in the reconstruction, a boundary consistency constraint that enforces agreement between the adjacent nodes, and an isotropic Fourier prior that penalizes directional artifacts and helps in denoising the reconstructed tomogram.
The evaluation of our approach on synthetic and real datasets shows considerable improvements comparing to state-of sfute-of-the-an reconstruction methods, including some recent approaches based on neural representations. The main contributions of our work are summarized as follows:
We propose an adaptive tensorial density fields representation, based on a quadtree structure, well-suited to large datasets. This allows us to achieve breakthrough in 4K-resolution tomography reconstruction in less than day.
We propose a novel isotropic Fourier prior to remove the directional artifacts, and reduce the noise level.
2
Related Work
Computed tomography (CT). Cryo-ET is one of the various types of tomographic inverse problem, which reconstructs a density volume of the scanned object from captured projections from different views. Analytical reconstruction algorithms, such as Filtered Back-Projection (FBP) [13] and Weighted Filtered Back-Projectis approaches produce fast and accurate reconstructions but require a considerable number of projections that are uniformly sampled in the angular space. Algebraic Reconstruction Technique (ART) [18] and its variants, like the Simultaneous Algebraic Reconstruction Technique (SART) [2], solve the tomography problem in an iterative way. These methods are well-suited to challenging CT reconstructions (sparse view, missing wedges, noisy projections, etc.), since they can be combined with different regularizers like the total variation into optimization frameworks [21,39, 23]. However, these approaches suffer from high computational requirements and fastidious hyper-paramett tuning. I Learning- basedd approaches have been introduced in CT to improve reconstruction quality through pre-processing projections [3, 52]. post-processing the reconstructed tomogram [40, 32], or using neural networks to learn a differentiable reconstruction operator [1, 25, 20J. More recently, combining Deep Image Prior [4, 6] and implicit representations [53, 46, 45] with traditional reconstruction approaches has produced unprecedented reconstruction results. Nevertheless, these approaches are not well-designed to overcome the several challenges of cryo-ET, mentioned in Section 1, especially the high level noise.
(WFBP) [43] are commonly used to solve this problem. These
Cryo-ET data Denoising. Cryo-ET acquisition uses low-dose beams to avoid damaging the sam- ples, but this results in capturing noisy projections [42]. In the literature, several models have been proposed for the cryo-ET noise, such as the additive white Gaussian noise [7, 56), or a Poisson-Gaussian noise [55]. Most existing denoising approaches
applied either before or
after the reconstruction step [14], which is performed using classical algorithms like WFBP. The pre-reconstruction approaches aim denoise the 2D projections using denoising algorithms such as the bilateral filter [24], the non-local means filetering [51], the wavelet shrinkage filter [22], and deep learning-based techniques [9]. On the other hand, post-reconstruccion denoising is applied directly to the tomograms to save the linearity between captured projections and reconstructed volume. This approach unlikely introduce new artifacts or suppress existing features. Total variation [54], and BM4D [33] were the state-of-the-art in 3D denoising, for a decade. However, these approaches require considerable computational resources. In structural biology, subtomogram averaging [8] is a specific denoising technique for data containing several copies of the same molecule. Recently, shaming-basee methods gained great success in denoising tasks. Specifically, unsupervised methods like Noise2Noise [29], Noise2 Void [27], present great potential for tomograms denoising, given the absence of ground truth data in cryo-ET field. Thus, the Topaz algorithm [7] produces clean tomograms by leveraging the Noise2Noise concept to train pairs of noisy tomograms reconstructed from odd/even projections. Kniesel et al. [26] proposed to jointly learn a model for 2D sensor noise and 3D implicit representation of the scanned sample. However, this approach depends highly on
2

---

the noise level. In our approach, the denoising task is performed during the reconstruction process through the proposed loss function and the parametrization of our adaptive representation.
3D Neural representation. Neural representation has seen impressive and rapid development in the last few years [47]. The Neural Radiance Fields (NeRF) [36] use Multi-Layer Perceptron (MLP) networks to learn mapping between spatial coordinates and physical scene properties (e.g., density field, color). However, NeRF-like approaches, also known as neural representations, suffer from long training times and slow rendering. Techniques such octree structures [31, 15, 45, 50], multi-scale network architecture [34], network factorization [44], caching [16], multi-resolution hash encoding [37], and tensorial-based representation [11] have been applied to handle this issue. Some works adapted these neural representations to solve CT reconstruction with impressive results [46, 53, 45, 26]. In this work, we leverage a quadtree structure to build an adaptive tensorial-ber -based representation. We also incorporate several regularizers in our loss function to handle the high level of noise in cryo-ET data. In addition, our method can deal with large datasets (4K resolution) in an unprecedented computational time.
3 Methodology
3.1 Overview
In this section, we introduce our approach, Adaptive TensorDF, which leverages a quadtree structure to create a multi-scale and effective tensor-based representation for reconstructing noisy tilt-series cryo-ET tomograms. Our framework represents the scanned sample's s density field as 3D continuous neural field. Since the cryo-ET tomograms have a low extension along the z-axis, use adaptive quadtree structure to partition the volume instead of an octree. Inside each node of the quadtree, the density field is represented using tensorial representation. Our framework is composed of three steps: quadtree update, optimization of the tensorial representation, and 3D volume reconstruction. These steps are depicted in Figure 1. The quadtree update step involves optimizing the quadtree structure to achieve the best partition of the reconstructed volume. This is done by encouraging uniform nodes while limiting their total number. In the second step. optimize loss function to build a tensor-based representation of the density field inside each node. The proposed loss function incorporates data-fidelity term and three regularizer terms, which will detail in the following. The reconstruction step involves uniform sampling of the volume in the region of interest (ROI) and querying the density values at the sampled positions.
Tropetioes
Extrention
Aupyy AsceCacc
1ip0012..
Quad one optintatiin
(a) Adaptive tensor donsity fields optimization.
(b) Volume querying.
Figure 1: Our framework is composed of two steps: During the training step (a) we first update the quadtree structure using downsampled projections. Then, we freeze that structure and update the tensor representation in each node using the original noisy projections. The second step is the volume querying (b), where we uniformly sample the ROI, and use the learned representation to estimate the densities at the selected positions.
3.2 Image formation model
Cryo-electron tomography is a classical tomographic reconstruction problem, where the projection image captured by the sensor corresponds the integration of density along the rays between the source and the detectors in the log space. After discretization, the image formation model for a given ray, can be written as follows:
b, AAx+ n
(I)
3

---

where A, represents the Radon transform operation along the ray i. b; and n. correspond to the intensity measured by the detector and its associated noise, respectively. x is the 3D density vector that we would like to reconstruct. As discussed in Section 1, the noise in real Cryo-ETT acquisitions complex and difficult to model, especially after applying the different pre-processing steps such as the intensity correction and the motion correction. For the sake of simplicity, we assume in this work a Gaussian white noise in the pre-processed projections. By regrouping all the captured rays, we deduce the Sata-fidelity loss from the Equation 1:
Ldata (x)
K (Ax-b) 12
(2)
where K is a binary mask to disable rays intersecting with the fiducial markers. This mask is introduced to suppress the projection artifacts caused by those markers.
3.3 Coordinate-based representation (CBR)
In tomography applications, the coordinate-based networks have been proposed map the 3D spatial coordinates inside the volume of interest to the density field. This mapping is given by:
fo :b:Po P: + x; with D; E IR3 , E
(3)
where p; and x; are respectively the 3D coordinate in the volume to be reconstructed and the corresponding density. fo corresponds to the mapping function that should be optimized. With the CBR, the captured projections are estimated by sampling positions along rays, applying the mapping to each sample, and then summing their contribution using the Radon coefficients. In common implicit representation [36, 26), So chosen to a fully connected MLP, as illustrated in Figure 2-(a). However, this representation is unsuitable for large volume sizes, as encountered in cryo-ET. To address this limitation, works like KiloNeRF [44], ACORN [34], and NeAT [45] suggested partitioning the volume of interest using uniform blocks multi-scale octree-basee based structures, and assigning a smaller MLP or decoder to each partition (block or octree node) for a local density field representation. In our approach, shown in Figure 2-(b), we partition the ROI using quadtree structure. Indeed, cryo-ET data have limited extension in the z-axis. Therefore, we base our representation
adaptive quadtree structure of the x-y plan that extend in the z-axis. For each quadtree node, we use a censorial-based representation. Thus, for a given 3D point in the region of interest Pi, represent the density field using sum of vector-matio outer products, follows:
(p:) D
Σ V (p.M
Y. (p.) + V. (P.) (p.) V/2 (.))) (p:)
X;
(4)
where DD is a one-layer decoder network, that converts encoded features to the output density. VAx V., and V correspond to factorized vectors of the three modes (X. and Z) for the th component of the tensor decomposition. Similarly, M. M.X.Z. and M, are matrix factors for the two denoted modes: (Y, 2), (X, 2), and (X, Y), respectively. R is the rank of the representation, which is hyperparameter be tuned according to the complexity of the reconstructed sample. Indeed, selecting a small rank can help denoise the tomogram, but it may lose some detailed features of the scanned sample.
lete
(b)
Figure 2: In (a), we show the application of neural radiance field to tomography reconstruction, while in (b), we illustrate the general idea of our approach. Where replaced the MLP by an adaptive differentiable tensor fields for faster optimization and better feature recovery.
The main advantage of the tensor representation is the huge reduction of the number of parameters needed to represent the volume. Consequently, this representation well-suited to reconstruct the large-size volume that commonly encounter in cryo-ET datasets.

---

Furthermore, it is important to take into account the contribution of the regions outside the ROI in the projection process, to ensure a more accurate output. However. there no need have high-quality representation in such regions. We propose to use the same representation detailed in Eq. (4), but with smaller size tensors.
3.4 Regularizations
Total Variation. The Total Variation (TV) loss is commonly used traditional tomographic reconstruction spatial regularizer to improve the reconstruction quality. The use of TV in neural fields representation approaches is not always straightforward. and could increase the computational complexity of the reconstruction. [45] proposed compute the TVV loss in the features space, before applying the decoder to get the density. [11] used this loss on the vector and matrix factors, to handle noise and outliers issues in regions with fewer observations. In implementation,
also apply the TV prior on the vector and matrix factors of enabled quadtree node in the ROI. Then we average all the contributions of those nodes compute the TV loss, which could be expressed as follows:
(5) where is scaling factor. We find empirically that 0.1 produced the best results. (p))) and V.M(p) represent the gradient the vector and matrix factors respectively. NEN is the number of enabled nodes of the quadtree.
LTV
NEN Σ a mean (IIVV()))I + mean (1 (\V)()))l)
Boundary Consistency Constraint. In our proposed framework, each node of the quadtree has its own feature representation and is optimized separately. This will inevitably introduce discontinuity artifacts in the final reconstruction. To address this issue, we propose a Boundary Consistency Constraint (BCC) that penalizes the discrepancy between the features obtained from the tensor representation of neighboring nodes for sampling points on their shared edge. Our proposed BCC loss is given by:
LBC
Σ
mean
Σ [If(l)m - fo(()ull PE
(n.m)E O.
where On refers to all pairs of neighboring quadtree nodes, Nn.m is the set of sampling points on the boundary surface between nodes and m, [P(P) and (P)n are the densities evaluated at the point p using the tensor representation of the nodes m and nt respectively.
Isotropic Fourier Prior. By combining the quadtree structure with the tensor density fields rep- resentation, we reduced the number of parameters needed to represent the 3D volume, as well as the reconstruction time. However, a downside of the low rank tensor representation is that it favors reconstructions with structured artifacts in the form of axis aligned streaks.
In the Fourier space, these artifacts manifest themselves as high peaks along the vertical and horizontal directions. To solve this issue, we introduced in our loss Isotropic Fourier Prior (IFP), to penalize such peaks in Fourier space. This constraint aims to limit the difference between the horizontal and vertical frequencies and the other frequencies in the Fourier domain. To do this, we first calculate the mean amplitude for each ring of the Fourier domain, which represents a given spatial frequency for all possible directions. Then, we apply penalty the horizontal and vertical frequencies that are much higher than this mean amplitude. Furthermore, we apply a weighting coefficient to penalize more the high frequencies that correspond mainly the noise. In our implementation, we sample the ROI coarsely, and we query the volume at those samples. Then, we compute the Fourier transform slice by slice. Our proposed IFP loss could be expressed as follows:
LIFP Σ w({) [F((0, <) mean (Fo
|+
3.5
Fo 0) mean (F& (u, e)((n(
where F& is the Fourier transform of fo computed on the slice s. u, e and Ç are spatial frequencics. w(C) is the weighting coefficient given by:
w(C) exp (Co () +
(8)
5

---

So is frequency parameter, to control the weight of the loss according to the frequency. Empirically, we found that Co 0.2 provides an optimal weighting for the isotropic Fourier prior.
3.5
Adaptive tensor density field optimization
We have defined our tensor density representation and the different losses used for our optimization. By combining all those terms, our loss is given by:
L == datta + Xrv Lrv + ÁBcc LBCC + AIPP LIFP
(9)
where λrv. ABCC, and APFP are the weights of the three regularizers defined previously. In the following, we describe the key elements of our framework's training step: updating the quadtree and sampling along the rays.
Quadtree update. As mentioned previously, we build our tensor rep- resentation using quadtree structure, illustrated in Figure 3. First, we ini- tialize our quadtree structure from the ROI, and we disable the nodes out- side this region. Then, sample uni- formly the volume of each node, and compute the standard deviation (STD) of the density values of the sampled positions, to define the quadtree update loss. high STD in a given node indicates that it contains fine details. Therefore, this node is more likely to be split into four child nodes. On the other hand, a node with low STD probably contains less features and represents uniform region of the scanned sample. Such node has higher chance to be merged or kept the same. We use a mixed-integer program (MIP) [41] to optimize this process, where, in each iteration, the nodes are either merged, split, or kept the same according to the update loss. During this optimization, the total number of nodes should stay lower than a fixed limit.
Inital tree
Quactroe update
intersection
sampling
nesr
near
(8)
(b)
Figure 3: Illustration of (a) the quadtree updating process, and (b) the ray sampling strategy.
Furthermore, we use a coarse-to- fine strategy to speed up the quadtree update and deal with the high noise level. We first update the quadtree using down-same wwwn-ed projections. During this step. we also update the tensor representation inside each node. This strategy reduces the impact of noise on the quadtree update. After some epochs, we keep the quadtree structure fixed and continue optimizing the tensor representation of each node using the original projections.
Ray sampling. During the optimization, we sample each ray to get list of 3D positions to be used for the density integration and the loss evaluation. The applied sampling is not uniform, but follows a stratified random sampling strategy that takes into account on the quadtree structure. For each node that the ray goes through, we pick N, 3D positions on the ray, randomly sampled from N,, uniform segments of the ray. We define N, as follows:
1/ dg
Ng
N max
(10)
where Nmax is hyperparameter corresponding to the maximum number of samples per node, ly is the length of the ray inside the quadtree node and d, is the length of the diagonal of q.
Experiments
We fully implemented framework in C++, which helps in speeding-up the reconstruction process. Implementation details could be found in the Supplement.
We designed a series of experiments to showcase the effectiveness of our framework on both synthetic and real captured datasets. We compare the performances of our approach to different baseline methods: (1) SART+TV, well established iterative reconstruction technique SART combined with a total variation prior. (2) Kniesel et al., an implicit neural representation for cryo-ET proposed by [26]. (3) I-NGP, reimplementation of Instant-NGP [37] for cryo-ET reconstruction. (4) TensoRF, a reimplementation of Tensor Radiance Fields [11] for cryo-ET reconstruction. For fair comparison, all the output densities are normalized into [0, 1].

---

4.1 Experiments on synthetic dataset
In our experiments, we used the synthetic dataset introduced in [26]. This dataset consists of randomly distributed ellipsoidal shells with random densities that mimic the density model of the ZIKV (i.e., Zika) virion at 15A. The simulated lomogram is projected in the angular range of [-70%, 69.5%), with an angular step of 1.5%. Gaussian noise is then added to the projections and several noise levels have been tested. When the standard deviation of the noise exceeds 0.05, the simulated projections look visually close the captured ones, terms of noise. This synthetic dataset is used for parameter tuning and robustness evaluation of our approach to the noise level.
Parameter tuning After several experiments on all the hyperparameters of our approach, it appears that the most important parameters are the tensor dimensions (the dimension of the matrix-vector factors) and the feature size. The choice of tensor dimensions affects the speed and quality of the training and reconstruction. Smaller dimensions lead to faster training and smoother reconstruction, but they may miss some fine details of the sample. Larger dimensions capture more details, but they may also introduce overfitting noise, as the network tends to learn the noise after learning the structures (See the Supplement for visual comparison). The feature size also influences the training speed and the reconstruction quality. Smaller features can speed up the training, but they may lose some details. Larger features can preserve more details, but they may increase the computation cost. A balance between these trade-offs should be sought for a better recovery. To study the impact of these two parameters, we measured the 3D Peak-Se Pcak-Signal- Hignal-to-Noise Ratio (PSNR) and 3D Structural Similarity Index Measure (SSIM) of the reconstructed volume using different values of these parameters. The results are shown in Table and Table 2.
Table 1: Impact of the tensor dimensions.
Table 2: Impact of the feature size.
22
26
30
12
16
20
PSNR 35.86 36.82 37.15 37.45 37.50
37.12
PSNR 37.21 37.45 37.47 37.86 37.42 37.68 0.822 0.827
SSIM 0.769 0.797 0.803 0.820 0.747 0.729
SSIM 0.788 0.820 0.819 0.837
From the analysis of Table and Table 2, the best performances are obtained with a tensor dimensions and feature size equals to 22 and 16, respectively.
SART + TV
SART+ TV
Inlestt
Kaleed
I-NGP
I-NGP
TensiRU
08
TensoR
W/O Car
WIO
PSNR
Ours
SSIM
Ours
0394
0.66
0.10 0.12
0.00
0.10
D122
Noise Level
Noise Level
Figure 4: PSNR and SSIM comparison for different noise levels.
Robustness to the noise level In this experiment, simulate different noise levels, and compare our results with baseline methods. Figure 4 illustrates the PSNR and SSIM computed on the results of each methods for different level of noise in the range [0.01, 0.12] for the standard deviation. Our approach consistently outperforms the other methods in terms of both PSNR and SSIM, regardless of the noise level. The neural representation based approaches are quite robust to the noise, since the performance do not drop consequently for larger level of noise. However, Kniesel et al. is not adapted to high noise situations, because relies on the noise levels that was learned from. In Figure 5, we show slice visualization representing the reconstruction results using the different methods, from simulated noisy projections. The noise level have been selected to be 0.08, to illustrate the robustness to noise level of each approach. Two regions in the dataset are zoomed in to illustrate the detail recovery (red frame) and the denoising effect in uniform regions (blue frame). SART+TV shows poor detail recovery and maintain high noise level. Kniesel et al. approach is relatively
7

---

Cypp
Ground
General View
Recovery
Denoising
Figure 5: Reconstruction results with different methods from simulated noisy projections (Noise level=0.08). Zoomed regions show details recovery (red box) and denoising effect (blue box).
good in denoising uniform regions, but the details not recover correctly. I-NGP and TensorRF approaches perform relatively good both preserving details and reducing the noise. TensorRF introduces, however, some artifacts that visually has the best performances in both tasks. By comparing Ours and Ours W/O Lirr, we can the impact of the Isotropic Fourier Prior in reducing artifacts and improving the denoising effect (More comparisons can be find in the Supplement).
notice in some ellipses in the General View. Ours
4.2 Experiments on real dataset
We evaluated our method on tilt-series datasets from EMPIAR (Electron Microscopy Pilot Image Archive): EMPIAR 10643 [38], and 10761 [12]. The EMPIAR 10643 dataset a cryo-ET acquisition of the HIV-1 GagdeltaMASPI A assemblies with an angular range of 1-600,6000000000 660) and an increment of 3 From this dataset, we reconstructed two different series (40 and 51) independently. The EMPIAR 10751 dataset corresponds to the cryo-ET acquisition of a HEK cell, with an angular range of [-60,, 60%). The three used datasets are pre-processed using IMOD [35] for the projection alignment and contrast transfer function (CTF) correction. For fair comparison, we normalize all the output densities to the range [0, 1). Figure 6, we show a comparison of the reconstruction results obtained with our approach and the four baselines methods, on EMPIAR 10643-40 and EMPIAR 10751 datasets. More results could found the Supplement.
SART+TV
Kniesel et al.
I-NGP
TensoRF
Ours
EMPIAR 10643-40
EMPIAR 10751
Figure 6: Reconstruction results of the HIV-1 (EMPIAR 10643) and a HEK cell (EMPIAR 10751).

---

10-
W
+V I-NGP
SART+TV
I-NGP
TensoRF
TensoRF
Ours
Ours
Intensities
Intensities
0.3
0.-
100
100
150
200
250
Sampling Point
Sampling Point
Figure 7: Intensity profile along the red line in the zoomed regions of the reconstruction of EMPIAR 10643-40 (Left), and EMPIAR 10761 (right) datasets.
Evaluation the denoising power In Figure 6, the first dataset shows our approach has the best de- noising in uniform regions. Despite using the TV regularization, SART+TV still exhibit noise within uniform regions. Increasing the weight of the TV constraint to enhance the SART+TV's denoising capabilities would result in the loss of main features. Kniesel et al. yields struction, it could be caused by the difference between the datasets used for their learning of the noise statistic and our used dataset. I-NGP and TensoRF have relatively good denoising. Some direc- tional artifacts degrade the quality of TensoRF. They are due to the impact of dummy regions (outside the ROD) which are not well modeled with this approach. Besides its best denoising power, our approach is also the most effective in improving the contrast between the features and the background.
oversmoothed recon-
To assess quantitatively the denoising effectiv- ness of each approach, we performed a compari- son using two statistical metrics: the Contrast-to- Noise Ratio (CNR), and the Equivalent Number of Look (ENL) [30]. The CNR metric measures the contrast between the region of interest and uniform background. While, the ENI. metric evaluates the smoothness in the uniform areas. We did not include the Kniesel et al. apprach in this comparison, since it does not perform well in the reconstruction of the real captured data (See Figure 6). The results shown in Table confirm our qualitative observations by showing higher CNR and ENL values. Our approach en- hances the contrast between feature regions and the background while producing smoother uniform regions. The table also indicates that the SART-TV result is the noisiest.
Table 3: Evaluation of the contrast enhancement (CNR) and the smoothing effect (ENL), (higher is better).
Metric Dutaset
SART+TV I-NGP TensoRF
Ours
10643-40
0.0618
0.5455
0.4123
0.6808
CNR
10643-51
0.0578
0.4981
0.4801
0.5367
10643-40
8.143
52.231
48.293
114.400
ENL1
10643-51
9.033
21.467
24.419
66.015
Quadtree regularization The quadtree structure has more benefits than just speeding up the computation. It allows us to have more local matrix representations in the XZ. and YZ planes, compared to TensoRF which can only have global planes for the whole scene. This can reduce the noise and misrepresentation in the reconstruction at a sacrifice of only a 50 percent reduction in parameter size. We compare our approach to: TensoRF W/O LtV, TensoRF, and TensoRF W LIFP in Table 4 and Figure 8. Our method has better detail preservation and denosing. Our method achieves significant improvement in quality.
Table Comparison of the different method in terms of the contrast enhancement (CNR) and the smoothing effect (ENL).
Metric Dataset
RensoRF W/O Lyv TensoRF TensoRF W LiFP
Ours
EMPIAR 10643-40
0.2884
0.4123
0.4470
0.6808
CNR
EMPLAR 10643-51
0.4530
0.4801
0.5218
0.5367
EMPLAR 106443-40
54.287
48.293
69.483
114.400
ENL
FMPIAR 10643-51
21.120
24.419
25.256
66.015

---

TensoRF W/O Lrv
TensoRF
TensoRF W LIFE
Ours
EMPIAR 10643 40
EMPIAR 10751
Figure 8: Reconstruction results of the HIV-1 (EMPIAR 10643) and a HEK cell (EMPIAR 10751).
Detailed feature analysis Figure 6 shows that our approach has the best feature recovery. Thus, for the EMPIAR 10643-40 dataset, reconstruction allows better resolving of the spike proteins. In addition, our reconstruction of the EMPIAR 10751 results in less blurry and more contrasted structures of the HEK cell. These observations are confirmed by the intensity profile in Figure 7, where our approach yields a more regular profile with an important difference between peaks (background regions) and valleys (features). On the other hand, all other methods show more intermediate peaks and valleys due to the residual noise in their reconstructions.
Conclusion and future work
In this paper, we introduced Adaptive TensorDF as a novel technique for fast and high-quality Cyro-ET reconstruction and denoising. Our technique leverages quadtree structure to represent the density field using a vector-matris factorized tensors representation. We optimize the quadtree structure and the tensor representation in two-stage process, using the down- down-sampled and then the original projections. We also combine three priors with the tomographic formation model into the loss function: total variation term, boundary consistency constraint, and an isotropic Fourier prior.
Extensive experiments demonstrate that our technique outperforms the existing methods terms of reconstruction quality and speed. It also involves reasonable number of parameters (three times less than TensoRF). Besides, Adaptive TensorDF is scalable and efficient, and can handle 4k resolution projections, shown in the Supplement, which is significant for high resolution cyro-ET dataset.
possible direction for future work is develop a joint alignment and reconstruction approach that can account for the misalignment between the projections and the volumes. This could improve the accuracy and robustness of our framework.
The authors would like to thank the NeurlPS'23 reviewers for their constructive suggestions. This work was supported by King Abdullah University of Science and Technology as part of the VCC Competitive Funding as well as the CRG program.
Acknowledgement
10

---

References
[1] Jonas Adler and Ozan Öktem. Learned primal-dual reconstruction. JEEE Trans. Med. Img., 37(6):1322-15 13322 2018.
[2] Anders Andersen and Avinash Kak. Simultancous algebraic reconstruction technique (sart); superior implementation of the art algorithm. Ultrasonic imaging, 6(1):81-94, 1984.
[3] Rushil Anirudh, Hyojin Kim, Jayaraman Thiagarajan, K Aditya Mohan, Kyle Champley, and Timo Bremer. Lose the views: Limited angle CT reconstruction via implicit sinogram completion. In Proc. CVPR, pages 6343-6352, 2018.
[4] Daniel Otero Baguer, Johannes Leuschner, and Maximilian Schmidt. Computed tomography reconstruction using deep image prior and learned reconstruction methods. Inverse Problems, 36(9):094004, 2020.
[5] Xiao-Chen Bai, Greg McMullan, and Sjors HW Scheres. How cryo-EM is revolutionizing structural biology. Trends in biochemical sciences, 40(1):49-57) 2015.
[6] Semih Barutcu. Selin Aslan, Aggelos K Katsaggelos, and Doğa Gürsoy. Limited-angle com- puted tomography with deep image and physics priors. Scientific reports, 11(1):1-12, 2021.
[7] Tristan Bepler, Kotaro Kelley, Alex Noble, and Bonnie Berger. Topaz-denoise: general deep denoising models for cryoEM and cryoET. Nature Methods, 11(1):1-12, 2020.
[8] John AG Briggs. Structural biology in situ-the the potential of subtomogram averaging. Current opinion in structural biology, (022):6612277) 2013.
[9] Tim-Oliver Buchholz, Alexander Krull, Réza Shahidi, Gaia Pigino, Gáspár Jékely, and Florian Jug. Content-aware image restoration for electron microscopy. Methods in cell biology. 152:277-289, 2019.
[10] Daniel Castañoo-Diez, Margot Scheffer, Ashraf Al-Amoudi, and Achilleas S Frangakis. Aligna- tor: GPU powered software package for robust fiducial-less alignment of cryo tilt-series. J. structural biology, 177-11 2010.
[11] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In Proc. ECCV, 2022.
[12] Matthew Croxford, Michael Elbaum. Muthuvel Arigovindan, Zvi Kam, David Agard, Elizabeth Villa, and John Sedat. renruiyyregularrieed deconvolution of cellular cryotransmission electron tomograms. Proceedings of the National Academy of Sciences, 118(50), 2021.
[13] Lee A Feldkamp. Lloyd C Davis, and James W Kress. Practical cone-beam algorithm. JOSA A. 1(6):612-619, 1984.
[14] Achilleas S Frangakis. It's noisy there! A review of denoising techniques in cryo-electron tomography. J. structural biology, 213(4): 107804, 2021.
[15] Sara Fridovich-Keil Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proc. CVPR, pages 5501- 5510, 2022.
[16] Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelin neural rendering at 200fps. In Proc. ICCV, pages 13355-14355, 2021.
[17] Muhammad Usman Ghani and W Clem Karl. Deep learning- based sinogram completion for low-dose CT. In IEEE 13th Image, Video. and Multidimensional Signal Processing Workshop (IVMSP), pages 1-5, IEEE, 2018.
[18] Richard Gordon, Robert Bender, and Gabor T Herman. Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. theoretical biology, 29(3):471-493 1970.

---

[19] Renmin Han, Fa Zhang, Xiaohua Wan, Jose-Jesus Fernández, Fei Sun, and Zhiyong Liu. A marker-free free automatic alignment method based on scale-invariant features, structural biology. 186(1):167-180, 2014.
[20] He, Yongbo Wang, and Jianhua Ma. Radon inversion via deep learning. IEEE Trans. Med. Img., 3966):006-2087,, 2020.
[21] Jing Huang, Yunwan Zhang, Jianhua Ma, Dong Zeng, Zhaoying Bian, Shanzhou Niu, Qianjin Feng, Zhengrong Liang, and Wufan Chen. Iterative image reconstruction for sparse-view CT using normal-dose image induced total variation prior. PloS one, 8(1 8(11):e79709,2)) 2013.
[22] Xinrui Huang, Sha Li, and Song Gao. Applying modified wavelet shrinkage filter to improve cryo-electron microscopy imaging. Journal of Computational Biology, 25(9):1050-1058 2018.
[23] Yixing Huang, Oliver Taubmann, Xiaolin Huang, Viktor Haase, Guenter Lauritsch, and Andreas Maier. Scale-space anisotropic total variation for limited angle tomography. IEEE Transactions on Radiation and Plasma Medical Sciences, 2(4):307-314, 2018.
[24] Wen Jiang, Matthew I. Baker, Qiu Wu, Chandrajit Bajaj, and Wah Chiu. Applications of a bilateral denoising filter in biological electron microscopy. J. structural biology, 144(1-2): 14- 122, 2003.
[25] Eunhee Kang, Won Chang, Jaejun Yoo, and Jong Chul Ye. Deep convolutional framelet denosing for low-dose CT via wavelet residual network. IEEE Trans. Med. Img., 3766):138--136, 2018.
[26] Hannah Kniesel, Timo Ropinski, Tim Bergner, Kavitha Shaga Devan, Clarissa Read, Paul Walther, Tobias Ritschel, and Pedro Hermosilla. Clean implicit 3d structure from noisy 2d stem images. In Proc. CVPR, pages 00762-20772, 2022.
[27] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void - learning denoising from single noisy images. Proc. CVPR, pages 2124 2132, 2019.
[28] Michael C Lawrence. Least-squares method of alignment using markers. Springer, 1992.
[29] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In Proc. ICML, pages 2965-2974, 2018.
[30] Muxingzi Li, Ramzi Idoughi, Biswarup Choudhury, and Wolfgang Heidrich. Statistical model for oct image denoising. Biomedical optics express, 8(9):3903-39) 3917, 2017.
[31] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Proc. NIPS, volume 33, pages 15661-1566,, 2020.
[32] Alice Lucas, Michael Iliadis, Rafael Molina, and Aggelos K Katsaggelos. Using deep neural networks for inverse problems in imaging: beyond analytical methods. IEEE Signal Processing Magazine, 35(1):20-36, 2018.
[33] Matteo Maggioni, Vladimir Katkovnik, Karen Egiazarian, and Alessandro Foi. Nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE Trans. Image Proc., 22(1):119-133 2012.
[34] Julien N. Martel, David B. Lindell, Connor Z. Lin, Eric Chan, Marco Monteiro, and Gordon Wetzstein. Acorn Adaptive coordinate networks for neural scene representation. ACM Trans. Graph., 40(4), 2021.
[35] David N Mastronarde and Susannah R Held. Automated tilt series alignment and tomographic reconstruction in imod. structural biology, 102-111 2017.
[36] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, pages 405-421. Springer, 2020.
12

---

[37] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 1104):102:11102:15, July 2022.
[38] Tao Ni, Thomas Frosio, Luiza Mendonça, Yuewen Sheng, Daniel Clare, Benjamin Himes, and Peijun Zhang. High-resolution in situ structure determination by cryo-electron tomography and subtomogram averaging using emClarity. Nature protocols, pages 1-24, 2022.
[39] Shanzhou Niu, Yang Gao, Zhaoying Bian, Jing Huang, Wufan Chen, Gaohang Yu, Zhengrong Liang, and Jianhua Ma. Sparse-view x-ray CT reconstruction via total generalized variation regularization. Physics in Medicine & Biology, 59(12)29997, 2014.
[40] Daniël M Pelt, Kees Joost Batenburg, and James A Sethian. Improving tomographic recon- struction from limited data using mixed-scale dense convolutional neural networks. Journal of Imaging, 4(1 1)21 2018.
[41] Laurent Perron and Vincent Furnon. Or-tools.
[42] M Radermacher: Electron tomography: three-dimensional imaging with the transmission electron microscope, 1992.
[43] Michael Radermacher. Weighted back-projection methods. In Electron tomography, pages
245-273, Springer, 2007.
[44] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In Proc. CVPR, pages 14335-14345, 2021.
[45] Darius Rüccert, Yuanhao Wang, Rui Li, Ramzi Idoughi, and Wolfgang Heidrich. NeAT: Neural Adaptive Tomography. ACM Trans. Graph., 41(4), July 2022.
[46] Yu Sun, Jiaming Liu, Mingyang Xie, Brendt Wohlberg, and Ulugbek S Kamilov. Coil: Coordinate- internal learning for tomographic imaging. IEEE Trans. Com. Img., 7:1400-
1412, 2021.
[47] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, W Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla Stephen Lombardi, al. Ad- vances in neural rendering. In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library, 2022.
[48] Ugo Valdrè. Electron microscopy in material science. Elsevier, 2012.
[49] Yuanhao Wang, Ramzi Idoughi, and Wolfgang Heidrich. Joint motion-correction and recon- struction in cryo-EM tomography. Proc. ICIP, 2022.
[50] Yuanhao Wang. Ramzi Idoughi, Darius Rückert, Rui Li, and Wolfgang Heidrich. Adaptive differentiable grids for cryo-electron tomography reconstruction and denoising. Bioinformatics Advances(to appear), 2023.
[51] Dai- Yu Wei and Chang-Cheng Yin. An optimized locally adaptive non-local means denoising filter for cryo-electron microscopy data. J. structural biology. 172(3):21122) 2010.
[52] Seunghwan Yoo, Xiaogang Yang. Mark Wolfman, Doga Gursoy, and Aggelos K Katsaggelos. Sinogram image completion for limited angle tomography with generative adversarial networks. In Proc. ICIP, pages 1252-1255 IEEE, 2019.
[53] Guangming Zang, Ramzi Idoughi, Rui Li, Peter Wonka, and Wolfgang Heidrich. IntraTomo: self-supervised Searning-based tomography via sinogram synthesis and prediction. In Proc. CVPR, pages 1960-1970, 2021.
[54] Qin Zhang and Chandrajit I. Bajaj. Cryo-electron microscopy data denoising based on the generalized digitized total variation method. Far east journal of applied mathematics, 45(2):83, 2010.
13

---

[55] Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang. Siyuan Zhang, Cody Smith, and Scott Howard. A gaassonngauusian denoising dataset with real fluorescence microscopy images. In Proc. CVPR, pages 11710-11718, 2019.
[56] Ellen D Zhong, Adam Lerer, Joseph H Davis, and Bonnie Berger. CryoDRGN2: Ab initio neural reconstruction of 3D protein structures from real cryo-em images. In Proc. CVPR, pages 4066-4075, 2021.
14


Reviews:
[{"review_id": "NaCoPspJY9", "reviewer": "Reviewer_bANQ", "summary": "The work combines quad-tree structure with low-rank tensorial representation to adaptively model cryo-EM volumetric density for reconstruction problem. The quad-tree structure is updated by merging or splitting to encourage uniformity in the area of each node, while the feature tensors for each node are obtained by outer product of learnable vectors and matrices. For the optimization, three regularization terms are used: total variation on vector and matrix features, boundary constraints to ensure points on the edge of two neighboring nodes have similar features, and Isotropic Fourier Prior. The latter penalizes outliers in horizontal/vertical Fourier coefficients and thus mitigates axis-aligned artifacts. A two-stage, coarse-to-fine optimization with downsampled projects helps avoid overfitting to noise. \nThrough experiments on synthetic data, effect of tensor and feature dimensions as well as robustness to noise is studied. On real data, the proposed method is compared with SOTA and related implicit representation learning.", "questions": "1. According to L229-233, the optimization is divided into two stages. In the first stage, both the quad-tree and tensorial representations inside each node are updated. The question is: when a node subdivided, how are tensorial representations of the new nodes initialized? Are they related to the tensorial representation of the parent node? What about the case when some nodes are merged together?\n2. How much of the optimization time in the first stage is spent on quad-tree updates vs. tensorial representation updates? How expensive is this discrete optimization problem? How do you set the hyper-parameters, such as the threshold for STD, for this step?", "limitations": "Apart from above questions and concerns, I have a suggestion: you might be able to replace the Isotropic Fourier Prior with another one that can be computed more efficiently in real space (rather than Fourier space).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Introduction to the problem is well written. It perfectly provides necessary background to understand the challenges in tilt-series tomography.\n2. The work combines spatial partitioning idea with tensorial based representation to obtain a compact representation of the density field.\n3. Two simple yet effective regularizers of Boundary constraint and Isotropic Fourier are cleverly used to mitigate artifacts and their impact is supported with evidence.", "weaknesses": "1. Based on my understanding, the proposed method is building upon TensoRF by locally (rather than globally) defining tensorial representation in an adaptive quad-tree structure, while adding regularizations to avoid artifacts and impose smoothness. Some ablation studies are missing examining the sole effect of each of these additions, e.g. qualitative and quantitative results on TensoRF (global representation) with all the new regularization terms. Currently, it is not clear how much of the improvement in denoising metrics (or qualitative results) is because of quad-tree vs. regularization terms. In other words, quad-tree is claimed to be important contribution but its effect is not fully studied.\n2. The isotropic Fourier regularization term requires Fourier Transform which can be expensive and slows down the computation, although a coarse sampling helps reducing the computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "a1: In our implementation, we start with a fixed number of nodes (for example, 70, as mentioned in the paper), we initialize each tensor representation with random values between -0.5 and 0.5. After each iteration, some nodes may become inactive, because they are subdivided or merged with others. Then, we map the currently active nodes to the initial 70 nodes. This mapping has no geometrical considerations. Thus, the tensor representations from one iteration to another (of the quadtree update) are not related. However, after only a few iterations of the tensor representations optimization we converge effectively, demonstrating our representation's robustness to initialization.\n\na2: In the first stage of optimization, around 10% of the total optimization time is dedicated to quadtree optimization. Then, we fix the quadtree structure and optimize the tensorial representation. This step accounts for around 85% of the total optimization time. The STD calculation takes around 5 minutes and represents less than 5% of the overall computation time. The discrete optimization takes less than 5 seconds. We do not use a threshold for STD, and the updating takes into account the STD for all current nodes to decide which should be divided, merged or kept the same. However, we impose a limit on the total number of nodes (we set 70 in the paper). We also add discrete constraints similar to [1]:\n\nWe define three binary variables for each node: ws, wk, and wm, which indicate whether the node should be subdivided, kept, or merged.\nWe ensure that the sum of these variables is one for each node.\nWe enforce the total number of nodes to not exceed a fixed limit.\nThen, we minimize the objective function, which is the sum of the products of the binary variables (ws, wk, and wm) and the STD of each node. We solve this problem using mixed integer programming with or-tools [2].\n[1]Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. ACM Trans. Graph., 40(4), 2021\n[2]Laurent Perron and Vincent Furnon. Or-tools.\n\n\nanswer to limitations: We appreciate your insight and will keep it in mind for our future work. We agree that using a feature extractor or a filter in real space could be beneficial for our design. We tried using a pre-trained VGG model as a feature extractor, but it did not work well because it was trained on ImageNet, which differs from our dataset and feature space. The Isotropic Fourier Prior does not consume much memory after our optimization. We use some techniques to reduce memory usage, such as releasing the memory of the intermediate computations and sampling the strategy. The computation time is still relatively high, but we think it is worth it for improving the quality of the results."}, {"review_id": "wUL0lf0hte", "reviewer": "Reviewer_tcrC", "summary": "In their paper, the authors present a learning-based framework that tackles the challenges faced in reconstructing 3D structures from tilt-series cryo-Electron Microscopy (cryo-EM) data. Cryo-EM is a powerful imaging technique known for its ability to achieve near-atomic resolutions. However, it is not without its drawbacks, including missing-wedge acquisition, large data size, and high noise levels.\n\nTo address these challenges, the authors introduce an innovative approach that utilizes an adaptive tensorial-based representation for the 3D density field of the scanned sample. The framework consists of several key components. First, a quadtree structure is optimized to partition the volume of interest effectively. Then, a vector-matrix factorization technique is employed to learn the tensor representation of the density field within each node.\n\nTo further enhance the reconstruction quality, the authors incorporate a loss function that combines a differentiable tomographic formation model with three regularization terms: total variation, boundary consistency constraint, and an isotropic Fourier prior. This allows the authors to generate high-quality 3D tomograms and query density information at any location using the learned representation.\n\nThe authors demonstrate the superiority of their framework over existing methods using both one synthetic and one experimental dataset. They claim that their framework not only enhances the quality of reconstructions but also reduces computation time and memory footprint. Overall, the paper presents a novel framework that addresses critical challenges and provides potential improvements in cryo-EM reconstruction.\n", "questions": "1. What are the motivations behind the specific formulation of the Coordinate-based representation (CBR) in this study? What are the data efficiency advantages or invariances exhibited by CBR that could benefit the reconstruction process?\n2. Could the authors present more examples beyond one simulated and one experimental dataset to enhance the comprehensiveness and generalizability of their findings?\n3. How does the method address challenges like the missing-wedge problem, colored noise, and the point-spread function?\n4. Can the authors provide benchmarks or evidence demonstrating the claimed faster optimization? \n5. What is the reason behind representing the \"one-layer decoder network\" (\\mathcal{D}) as a function instead of a matrix multiplication? Could you provide clarity on its parameter size?\n6. Could you please explain the definitions of the Vs and Ms variables in equation (4)? Additionally, what is the output dimensionality of these variables?\n7. Can you clearify which parameters are trained?\n8. Can you provide information on the memory footprint of the proposed framework? How does it impact computational resources and efficiency?", "limitations": "1. Lack of clarity and exploration of the motivations and effects of the Coordinate-based representation (CBR).\n2. Absence of access to code and example reconstructions, affecting reproducibility.\n3. Insufficient number of examples, limiting the comprehensiveness and generalizability of the findings.\n4. Potential limitation in the suitability of the chosen simulated data, emphasizing the need for evaluation on more relevant datasets with ground truth information.\n5. Failure to directly address challenges like the missing-wedge problem, colored noise, and the point-spread function in cryo-EM data.\n6. Lack of benchmarks or evidence supporting the claim of faster optimization in comparison to existing methods.\n6. Need for specific benchmark comparisons and a focus on computational efficiency to substantiate claims and understand the advantages and limitations of the proposed approach.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "*Adaptive Tensorial Representation:* The framework utilizes an adaptive tensorial-based representation for the 3D density field, which should in theory wllow for efficient partitioning of the volume of interest. This adaptive approach ensures that the representation is tailored to the specific characteristics of the sample, enhancing the accuracy of the reconstruction.\n\n*Comprehensive Regularization:* The authors incorporate multiple regularization terms, including total variation, boundary consistency constraint, and an isotropic Fourier prior. This comprehensive regularization scheme should help mitigate the challenges associated with missing-wedge acquisition, large data size, and high noise levels. It should promote smoothness in the reconstructions and improve the overall quality.\n\n*Querying Flexibility:* The learned representation enables querying of the density at any location within the reconstructed volume. This flexibility is valuable for further analysis and examination of specific regions of interest, providing researchers with a detailed understanding of the 3D structure.", "weaknesses": "The motivations behind the Coordinate-based representation (CBR) in this study remain unclear. While the compressed nature of CBR suggests it may possess desirable regularizing properties for this application, the specific effects of this representation have not been thoroughly explored by the authors. It remains uncertain whether CBR offers any data efficiency advantages or exhibits any invariances that could benefit the reconstruction process. To gain more insights into the robustness of the CBR representation, it would be beneficial for the authors to conduct ablation studies that combine alternative representations with the proposed loss functions. Such experiments would shed light on the effectiveness and reliability of CBR, further elucidating its potential contributions to the overall framework.\n\nReproducibility is of concern, since the authors have not provided access to their code or shared example reconstructions. In addition to expanding on the theoretical details of the method (see above), It would be beneficial for them to present more examples beyond just one simulated and one experimental dataset. Including a wider range of examples would enhance the comprehensiveness and generalizability of their findings, providing a more robust evaluation of their method's capabilities.\n\nAs an application paper, it is worth considering the suitability of the chosen simulated data in this study. The authors should aim to evaluate their method on a more relevant dataset that provides ground truth information. It is crucial to assess the performance of the proposed method in a scenario where accurate and relevant reference data is available for comparison. \n\nThe paper acknowledges challenges such as the missing-wedge problem, but it lacks results that directly address this issue. Furthermore, there is no mention of colored noise or the point-spread function, which are important factors in cryo-EM reconstruction.\n\nThe authors assert that their method offers \"faster optimization and better feature recovery.\" However, there is a lack of benchmarks or evidence showcasing the claimed faster optimization. The statement that reconstructions take \"less than a day\" sounds excessively slow when compared to several existing methods that achieve reconstruction within a few minutes.\n\nTo strengthen their claims, the authors should provide specific benchmark comparisons demonstrating the improved optimization speed of their method. Additionally, it would be beneficial for them to address the computational efficiency aspect and compare their results with other methods that achieve faster reconstruction times. This would provide a clearer understanding of the advantages and limitations of their proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "a1: Recently, several works [1,2,3,4] have demonstrated the superiority of Coordinate-based representations in solving tomographic problems, especially in missing-wedges scenarios, compared to traditional methods. So there is excellent evidence for the superiority of CBNs in that regard. However, existing approaches are aimed at X-ray projection data, and do not deal with the high level of noise that we encounter in Cryo-ET reconstruction, which is the main issue we want to address in this paper. In our comparison, we selected SART+TV as the baseline of traditional approaches since it performs better than the widely used Weighted Filtered Backpropagation (WFBP) [5]. Our comparison shows better reconstruction results for our approach than SART+TV. Moreover, this traditional approach requires at least 128G (no matter CPU or GPU) for reconstructing a 4K size dataset, while we can run our CBR-based method on a 48G GPU with less CPU memory usage.\n\na2: We conducted experiments on one simulated data and two real datasets in the main paper: EMPIAR 10643-40 (HIV-1 viruses) and EMPIAR 10751 (HEK cell). In the supplement, we added more experiments on another series of HIV-1 viruses (EMPIAR 10643-51). These datasets are representative of the typical data encountered in structural biology, with respect to noise distribution and feature structures. With our approach, learning is performed for each single dataset, so we do not face overfitting or generalization issues in this case. However, we will include more experiments with new datasets in the supplement to further demonstrate our method. Moreover, it is important to note that cryo-ET does not provide data with ground truth information, which limits our validation options.\n\na3: [3,4] prove that CBR-based methods could deal with the missing-wedge problem quite well compared with SART+TV and FBP. The point-spread function (PSF), or contrast transfer function (the Fourier transform of the PSF), is corrected during the preprocessing step using IMOD. For some datasets in EMPIAR databank, the CTF is already corrected and the projections are aligned. In the revised version we will make it clear that the CTF should be corrected as a preprocessing step. We also performed some experiments using the approach of [6] to compensate somehow the CTF, but we did not get any clear improvement with our datasets. This means that the preprocessing resulted in a good CTF correction. By running our approach on real captured data from EMPIAR databank, we have shown that the reconstructed tomogram is less noisy than the results from the baseline approaches. Specifically we computed two metrics (CNR and ENL), that evaluate the contrast improvement and the denoising effect. We also evaluate the intensity profiles along a line that contains virus spikes, to show that our approach has a better distinction between spikes and background. In summary, these experiments show that our approach does a better job dealing with the colored noise in the real data.\n\na4: We test the optimization time using different methods. Ours and I-NGP converge in a similar time (around 2 hours for the 1K dataset). [6] takes several days. SART+TV needs around 4 hours, with a Tomosipo [7] based implementation. Please refer to Part C of the supplementary.\n\na5: The decoder network consists of a single layer that applies a non-linearity function to the model. This non-linearity enhances the performance of the model in comparison to simple matrix multiplication, as demonstrated by [8] for similar tasks. The decoder has only 0.005 M parameters, which is much smaller than the total number of 28.2 M parameters in the model.\n\na6: Vs and Ms correspond respectively to rank-one (vector) and rank-two (matrix) tensor components. The output dimensionality of these variables depends on the dimension decomposition. For example, for a scene with the following dimensions: n * m * p, Ms^(X,Y) has a dimensionality of n * m, while Vs^(Y) has a dimensionality of m. We will add the dimensionality information in the revised version.\n\na7: In our framework, we optimize the vector and matrix factors Vs and Ms for all nodes, as well as the decoder D.\n\na8: We only consider the GPU memory here. It depends on the batch size setting (the number of samples processed before the model is updated). A larger batch size usually requires more memory, but also speeds up the computation. Therefore, we try to use a relatively large batch size to fully utilize the GPU memory capacity, such as 44GB out of 48GB available on our cards.\n\n[1]Sun, Yu, et al. \"Coil: Coordinate-based internal learning for tomographic imaging.\" IEEE Transactions on Computational Imaging 7 (2021): 1400-1412.\n\n[2]Liu, Renhao, et al. \"Recovery of continuous 3D refractive index maps from discrete intensity-only measurements using neural fields. Nat Mach Intell 4, 781–791 (2022).\n\n[3]Rückert, Darius, et al. \"Neat: Neural adaptive tomography.\" ACM Transactions on Graphics (TOG) 41.4 (2022): 1-13.\n\n[4]Zang, Guangming, et al. \"IntraTomo: self-supervised learning-based tomography via sinogram synthesis and prediction.\" ICCV 2021.\n\n[5]Li, Lun, et al. \"Compressed sensing improved iterative reconstruction-reprojection algorithm for electron tomography.\" BMC Bioinformatics 21.6 (2020): 1-19.\n\n[6]Kniesel, Hannah, et al. \"Clean implicit 3d structure from noisy 2d stem images.\" CVPR 2022.\n\n[7]Hendriksen, Allard A., et al. \"Tomosipo: fast, flexible, and convenient 3D tomography for complex scanning geometries in Python.\" Optics Express 29.24 (2021): 40494-40513.\n\n[8]Karnewar, Animesh, et al. \"Relu fields: The little non-linearity that could.\" ACM SIGGRAPH 2022 Conference Proceedings. 2022."}, {"review_id": "yTuU5KWpMV", "reviewer": "Reviewer_9Yvf", "summary": "This paper proposed a field-based method for 3D image reconstruction in Cryo-EM, which is a challenging task due to strong measurement noise and ill-posedness. The proposed method, **TensorDF**, has the following features: 1) it combines implicit representation and a quadtree, where each node corresponds to a feature tensor, 2) it can automatically update the quadtree structure during training, and 3) it includes three regularizers, that is, total variation, boundary consistency, and penalty in the Fourier space, to improve the reconstruction accuracy.\n\nExperimental validations on both simulated and real datasets are presented. **TensorDF** is compared with several baseline methods and yields superior performance. The authors also discuss how the hyperparameters of TensorDF are selected, and demonstrate the robustness of **TensorDF** under different noise levels.", "questions": "1. Could the author further explain the factorization of each feature map $p_i^m$? namely $V_x$, $V_y$, $V_z$, and $M_{xy}$, $M_{yz}$, $M_{xz}$. I am not clear on what is trainable and what is not.\n\n2. Although MLP is not scalable to large volumes, the network itself can help regularize the final reconstruction (see [1]). Perhaps not using an MLP makes the additional regularizers necessary?\n\n3. A followed-up question is how good/bad will TensorDF w/o all regularizers perform? This can show if a quadtree structure can impose some regularization. \n\n[1] Recovery of continuous 3D refractive index maps from discrete intensity-only measurements using neural fields.", "limitations": "The limitation of the proposed method is not discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear presentation of the proposed method.\n2. Solid experiments with good performance.\n3. Thorough supplementary material (ablation study, more visual examples, and discussion).", "weaknesses": "I do not see any obvious weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "a1: All the Vs and Ms are trainable variables. Vs and Ms work like volume factorization. Our target is to optimize the volume, and it is represented as a product of Vs, and Ms. Decoder is also optimized by default.\n\na2: The reviewer is correct that the absence of the MLP necessitates the additional regularizer. If we use direct explicit expression, we will encounter many local minima, which is why we need to regularize the model. However, explicit representation has the advantage of being fast. Moreover, even with random initialization, it can produce stable reconstructions.\n\na3: We appreciate the reviewer’s suggestion to perform comparisons to illustrate the quadtree regularization effect in our approach. Indeed, the quadtree structure has more benefits than just speeding up the computation. It also allows us to have more local matrix representations in the XY and XZ planes, compared to TensorDF which can only have global planes for the whole scene. This can reduce the noise and misrepresentation in the reconstruction. That is why our method achieves a significant improvement in quality with only a 50 percent reduction in parameter size. We provide in the rebuttal.pdf file some additional experiments to illustrate the impact of the quadtree as a regularizer. Since TensorDF (TensorRF in the original paper) already incorporates TV prior and does not require boundary consistency prior, we compare our approach to: TensorDF W/O TV prior, TensorDF, and TensorDF W the Isotropic Fourier Prior."}, {"review_id": "NJ4jCbKw0X", "reviewer": "Reviewer_6xkQ", "summary": "The authors address the problems of denoising and tomographic reconstruction, specifically in the context of cryoelectron tomography (cryoET), in which 3D structures (e.g. of proteins) are reconstructed using a tilt-series of tomograms. CryoET holds much promise for the elucidation of biological structures with atomic or near-atomic resolution in situ but faces significant challenges in denoising and reconstruction. While a good deal of work exists to address both of these challenges, the authors here introduce a novel method using tensor density fields that jointly solves the reconstruction and denoising problems, a first. Another innovation is the introduction of a isotropic Fourier regularization term in their loss function, which serves to ameliorate issues with streaking artifacts. In sum, the authors make a significant contribution to cryoET reconstruction with applications to structural biology and other fields in which cryoET is used.", "questions": "Figures 4 and 7: Increase the font sizes of the axes for better readability. Adjust some of the lighter line colors for better readability.\n\n", "limitations": "Yes, limitations are clearly addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality: The authors here present the first (to my knowledge) method in which tomographic reconstruction and denoising are jointly solved for cryoET. The authors employ an innovative architecture and pipeline that enables unprecedented computational speedup for large, data-intensive cryoET datasets. Finally, the authors incorporate a novel regularization term (isotropic Fourier prior) that helps deal with artifacts and yields better results than state-of-the-art approaches for cryoET.\n\nQuality: The authors deliver significant improvements on existing methods for denoising, reconstruction, and computation time for cryoET datasets. Their method is validated on both synthetic and real data, demonstrating applicability. The code is efficiently implemented and results in significant improvements to computational efficiency.\n\nClarity: The paper is written exceptionally clearly. The authors provide extensive information on existing methods for denoising and clearly states the ways in which their method differs and innovates on previous ones. The authors also delineate conditions under which their methods performs better or worse.\n\nSignificance: New methods for the effective reconstruction and denoising of cryoET data promises to make this powerful method more accessible, with the potential for making new discoveries in structural biology and other fields. As a method that jointly addresses reconstruction and denoising efficiently, the author's method makes a significant contribution. The method or representational architecture might also be applied to other tomography modalities.", "weaknesses": "Clarity: Although minor, some of the figures could be presented more clearly.\n\nQuality: The authors mention that the issues they solve in the context of cryoET are general to many tomographic reconstruction problems. While true in principle, the authors do not demonstrate that their method work well in other modalities compared to state-of-the-art.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for you recommending. We thank Reviewer 4 for their valuable feedback, and we will make the necessary changes to the font size and color of the graphs."}]

Author rebuttal:
We include in the rebuttal.pdf file some additional experiments to answer question 3 of Reviewer 3.

Meta-review: This paper presents a learning-based method for reconstruction of cryo-ET tomograms. The method combines quad-tree structure with low-rank tensorial representation to adaptively model cryo-EM volumetric density. Several regularization terms are introduced to cope with the high amount of noise in cryo-ET data. The method is innovative and the manuscript shows good empirical results on both synthetic and real data. Acceptance is therefore recommended.
Decision: Accept (poster)