SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We fully implemented our framework in  $\mathrm{C}++$ , which helps in speeding-up the reconstruction process. Implementation details could be found in the Supplement.
We designed a series of experiments to showcase the effectiveness of our framework on both synthetic and real captured datasets. We compare the performances of our approach to different baseline methods: (1) SART+TV, a well established iterative reconstruction technique SART combined with a total variation prior. (2) Kniesel et al., an implicit neural representation for cryo-ET proposed by [26]. (3) I-NGP, a reimplementation of Instant-NGP [37] for cryo-ET reconstruction. (4) TensoRF, a reimplementation of Tensor Radiance Fields [11] for cryo-ET reconstruction. For a fair comparison, all the output densities are normalized into  $[0, 1]$ .
4.1 Experiments on synthetic dataset
In our experiments, we used the synthetic dataset introduced in [26]. This dataset consists of randomly distributed ellipsoidal shells with random densities that mimic the density model of the ZIKV (i.e., Zika) virion at  $15\AA$ . The simulated tomogram is projected in the angular range of  $[-70^{\circ}, 69.5^{\circ}]$ , with an angular step of  $1.5^{\circ}$ . Gaussian noise is then added to the projections and several noise levels have been tested. When the standard deviation of the noise exceeds 0.05, the simulated projections look visually close to the captured ones, in terms of noise. This synthetic dataset is used for parameter tuning and robustness evaluation of our approach to the noise level.
Parameter tuning After several experiments on all the hyperparameters of our approach, it appears that the most important parameters are the tensor dimensions (the dimension of the matrix-vector factors) and the feature size. The choice of tensor dimensions affects the speed and quality of the training and reconstruction. Smaller dimensions lead to faster training and smoother reconstruction, but they may miss some fine details of the sample. Larger dimensions capture more details, but they may also introduce overfitting or noise, as the network tends to learn the noise after learning the structures (See the Supplement for visual comparison). The feature size also influences the training speed and the reconstruction quality. Smaller features can speed up the training, but they may lose some details. Larger features can preserve more details, but they may increase the computation cost. A balance between these trade-offs should be sought for a better recovery. To study the impact of these two parameters, we measured the 3D Peak-Signal-to-Noise Ratio (PSNR) and 3D Structural Similarity Index Measure (SSIM) of the reconstructed volume using different values of these parameters. The results are shown in Table 1 and Table 2.
Table 1: Impact of the tensor dimensions.
10	14	18	22	26	30
PSNR	35.86	36.82	37.15	37.45	37.50	37.12
SSIM	0.769	0.797	0.803	0.820	0.747	0.729
Table 2: Impact of the feature size.
4	8	12	16	20	24
PSNR	37.21	37.45	37.47	37.86	37.42	37.68
SSIM	0.788	0.820	0.819	0.837	0.822	0.827
From the analysis of Table 1 and Table 2, the best performances are obtained with a tensor dimensions and feature size equals to 22 and 16, respectively.
Figure 4: PSNR and SSIM comparison for different noise levels.
Robustness to the noise level In this experiment, we simulate different noise levels, and compare our results with baseline methods. Figure 4 illustrates the PSNR and SSIM computed on the results of each methods for different level of noise in the range [0.01, 0.12] for the standard deviation. Our approach consistently outperforms the other methods in terms of both PSNR and SSIM, regardless of the noise level. The neural representation based approaches are quite robust to the noise, since the performance do not drop consequently for larger level of noise. However, Kniesel et al. is not adapted to high noise situations, because it relies on the noise levels that it was learned from. In Figure 5, we show a slice visualization representing the reconstruction results using the different methods, from simulated noisy projections. The noise level have been selected to be 0.08, to illustrate the robustness to noise level of each approach. Two regions in the dataset are zoomed in to illustrate the detail recovery (red frame) and the denoising effect in uniform regions (blue frame). SART+TV shows a poor detail recovery and maintain a high noise level. Kniesel et al. approach is relatively
Figure 5: Reconstruction results with different methods from simulated noisy projections (Noise level=0.08). Zoomed regions show details recovery (red box) and denoising effect (blue box).
good in denoising uniform regions, but the details are not recover correctly. I-NGP and TensorRF approaches perform relatively good in both preserving details and reducing the noise. TensorRF introduces, however, some artifacts that we can notice in some ellipses in the General View. Ours visually has the best performances in both tasks. By comparing Ours and Ours W/O  $\mathcal{L}_{IFP}$ , we can see the impact of the Isotropic Fourier Prior in reducing artifacts and improving the denoising effect (More comparisons can be find in the Supplement).
4.2 Experiments on real dataset
We evaluated our method on tilt-series datasets from EMPIAR (Electron Microscopy Pilot Image Archive): EMPIAR 10643 [38], and 10761 [12]. The EMPIAR 10643 dataset is a cryo-ET acquisition of the HIV-1 GagdeltaMASP1T8I assemblies with an angular range of  $[-60^{\circ}, 60^{\circ}]$  and an increment of  $3^{\circ}$ . From this dataset, we reconstructed two different series (40 and 51) independently. The EMPIAR 10751 dataset corresponds to the cryo-ET acquisition of a HEK cell, with an angular range of  $[-60^{\circ}, 60^{\circ}]$ . The three used datasets are pre-processed using IMOD [35] for the projection alignment and contrast transfer function (CTF) correction. For a fair comparison, we normalize all the output densities to the range [0, 1]. In Figure 6, we show a comparison of the reconstruction results obtained with our approach and the four baselines methods, on EMPIAR 10643-40 and EMPIAR 10751 datasets. More results could be found in the Supplement.
Figure 6: Reconstruction results of the HIV-1 (EMPIAR 10643) and a HEK cell (EMPIAR 10751).
Figure 7: Intensity profile along the red line in the zoomed regions of the reconstruction of EMPIAR 10643-40 (Left), and EMPIAR 10761 (right) datasets.
Evaluation of the denoising power In Figure 6, the first dataset shows our approach has the best denoising in uniform regions. Despite using the TV regularization, SART+TV still exhibit noise within uniform regions. Increasing the weight of the TV constraint to enhance the SART+TV's denoising capabilities would result in the loss of main features. Kniesel et al. yields to an oversmoothed reconstruction, it could be caused by the difference between the datasets used for their learning of the noise statistic and our used dataset. I-NGP and TensoRF have a relatively good denoising. Some directional artifacts degrade the quality of TensoRF. They are due to the impact of dummy regions (outside the ROI), which are not well modeled with this approach. Besides its best denoising power, our approach is also the most effective in improving the contrast between the features and the background.
Table 3: Evaluation of the contrast enhancement (CNR) and the smoothing effect (ENL), (higher is better).
Metric	Dataset	SART+TV	I-NGP	TensoRF	Ours
CNR↑	10643-40	0.0618	0.5455	0.4123	0.6808
10643-51	0.0578	0.4981	0.4801	0.5367
ENL↑	10643-40	8.143	52.231	48.293	114.400
10643-51	9.033	24.467	24.419	66.015
To assess quantitatively the denoising effectiveness of each approach, we performed a comparison using two statistical metrics: the Contrast-to-Noise Ratio (CNR), and the Equivalent Number of Look (ENL) [30]. The CNR metric measures the contrast between the region of interest and the uniform background. While, the ENL metric evaluates the smoothness in the uniform areas. We did not include the Kniesel et al. approach in this comparison, since it does not perform well in the reconstruction of the real captured data (See Figure 6). The results shown in Table 3
confirm our qualitative observations by showing higher CNR and ENL values. Our approach enhances the contrast between feature regions and the background while producing smoother uniform regions. The table also indicates that the SART-TV result is the noisiest.
Quadtree regularization The quadrtree structure has more benefits than just speeding up the computation. It allows us to have more local matrix representations in the XZ and YZ planes, compared to TensoRF which can only have global planes for the whole scene. This can reduce the noise and misrepresentation in the reconstruction at a sacrifice of only a 50 percent reduction in parameter size. We compare our approach to: TensoRF W/O  $\mathcal{L}_{TV}$ , TensoRF, and TensoRF W  $\mathcal{L}_{IFP}$  in Table 4 and Figure 8. Our method has better detail preservation and denosing. Our method achieves a significant improvement in quality.
Table 4: Comparison of the different method in terms of the contrast enhancement (CNR) and the smoothing effect (ENL).
Metric	Dataset	TensoRF W/O LTV	TensoRF	TensoRF W LIFP	Ours
CNR↑	EMPIAR 10643-40	0.2884	0.4123	0.4470	0.6808
EMPIAR 10643-51	0.4530	0.4801	0.5218	0.5367
ENL↑	EMPIAR 10643-40	54.287	48.293	69.483	114.400
EMPIAR 10643-51	21.120	24.419	25.256	66.015
Figure 8: Reconstruction results of the HIV-1 (EMPIAR 10643) and a HEK cell (EMPIAR 10751).
Detailed feature analysis Figure 6 shows that our approach has the best feature recovery. Thus, for the EMPIAR 10643-40 dataset, our reconstruction allows a better resolving of the spike proteins. In addition, our reconstruction of the EMPIAR 10751 results in less blurry and more contrasted structures of the HEK cell. These observations are confirmed by the intensity profile in Figure 7, where our approach yields a more regular profile with an important difference between peaks (background regions) and valleys (features). On the other hand, all other methods show more intermediate peaks and valleys due to the residual noise in their reconstructions.
5 Conclusion and future work
In this paper, we introduced Adaptive TensorDF as a novel technique for fast and high-quality Cyro-ET reconstruction and denoising. Our technique leverages a quadtree structure to represent the density field using a vector-matrix factorized tensors representation. We optimize the quadtree structure and the tensor representation in a two-stage process, using the down-sampled and then the original projections. We also combine three priors with the tomographic formation model into the loss function: a total variation term, a boundary consistency constraint, and an isotropic Fourier prior.
Extensive experiments demonstrate that our technique outperforms the existing methods in terms of reconstruction quality and speed. It also involves a reasonable number of parameters (three times less than TensorRF). Besides, Adaptive TensorDF is scalable and efficient, and can handle 4k resolution projections, as shown in the Supplement, which is significant for high resolution cyro-ET dataset.
A possible direction for future work is to develop a joint alignment and reconstruction approach that can account for the misalignment between the projections and the volumes. This could improve the accuracy and robustness of our framework.
Acknowledgement The authors would like to thank the NeurIPS'23 reviewers for their constructive suggestions. This work was supported by King Abdullah University of Science and Technology as part of the VCC Competitive Funding as well as the CRG program.
[55] Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. Aoisson-gaussian denoising dataset with real fluorescence microscopy images. In Proc. CVPR, pages 11710-11718, 2019.
[56] Ellen D Zhong, Adam Lerer, Joseph H Davis, and Bonnie Berger. CryoDRGN2: Ab initio neural reconstruction of 3D protein structures from real cryo-em images. In Proc. CVPR, pages 4066-4075, 2021.