DRACO: A Denoising-Reconstruction Autoencoder for Cryo-EM

Authors: yingjun  shen
haizhao  dai
qihe  chen
yan  zeng
jiakai  zhang
yuan  pei
jingyi  yu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 09:11:16

Abstract: Foundation models in computer vision have demonstrated exceptional performance in zero-shot and few-shot tasks by extracting multi-purpose features from large-scale datasets through self-supervised pre-training methods. However, these models often overlook the severe corruption in cryogenic electron microscopy (cryo-EM) images by high-level noises. We introduce DRACO, a Denoising-Reconstruction Autoencoder for CryO-EM, inspired by the Noise2Noise (N2N) approach. By processing cryo-EM movies into odd and even images and treating them as independent noisy observations, we apply a denoising-reconstruction hybrid training scheme. We mask both images to create denoising and reconstruction tasks. For DRACO's pre-training, the quality of the dataset is essential, we hence build a high-quality, diverse dataset from an uncurated public database, including over 270,000 movies or micrographs. After pre-training, DRACO naturally serves as a generalizable cryo-EM image denoiser and a foundation model for various cryo-EM downstream tasks. DRACO demonstrates the best performance in denoising, micrograph curation, and particle picking tasks compared to state-of-the-art baselines.

Main paper: 
 introduction: 
 Foundation models in computer vision have demonstrated remarkable capabilities in zero-shot and few-shot tasks. These models learn to extract multi-purpose visual features from large-scale, diverse datasets through text-guided [1, 2, 3] or self-supervised [4, 5] pre-training methods such as masked image modeling (MIM) [6] . The features can then be applied to various downstream tasks. For instance, DINOv2 [5] is trained on a large-scale curated dataset and shows significant performance improvements in classification, retrieval, segmentation, etc. The success of vision foundation models has stimulated advances across various scientific disciplines. Due to the diverse modalities of scientific imaging, training domain-specific foundation models [7, 8, 9, 10] is essential to meet specific demands. For example, the UNI [7] foundation model for tissue imaging is pre-trained on 100 million images for 34 representative clinical downstream tasks.In structural biology, cryogenic electron microscopy (cryo-EM) stands as a pivotal bio-imaging technique [11] . Unlike optical imaging methods, cryo-EM possesses several distinctive characteristics: first, cryo-EM utilizes high-energy electron beams as its illumination source [12] and direct detector device (DDD) captures a continuous multi-frame sequence, often called a movie [13] . To mitigate specimen damage during exposure, the electron dose per frame is restricted, leading to extremely low signal-to-noise ratios (SNR) in the captured images. Second, cryo-EM employs motion correction to counteract blurring induced by specimen drift during exposure to ultimately obtain a sharper single micrograph [13] . Last, the acquired images comprise hundreds of thousands of target protein particles with diverse poses. To resolve 3D structures, researchers utilize a pipeline composed of multiple downstream tasks including micrograph curation, micrograph denoising, particle picking, pose estimation, and ultimately high-resolution reconstruction.In line with denoising autoencoders [14] , which consider "robustness to partial destruction of the input" a good representation criterion, existing self-supervised learning methods, such as masked autoencoders (MAE) [6] , have been successful in learning expressive representations by reconstructing the missing patches from a partially masked image. However, in cryo-EM, these methods overlook the severe corruption caused by pixel-level random noise, leading to degraded performance. To be more robust to noises, DMAE [15] reconstructs the clean image from the masked one that is further corrupted by synthetic Gaussian noise. Nevertheless, cryo-EM clean reference images are impossible to obtain due to the fragile biological specimens, posing a significant challenge.In this paper, we present DRACO, a Denoising-Reconstruction Autoencoder for CryO-EM, as shown in Figure 1 . Inspired by Noise2Noise (N2N) [16] , which learns to denoise images using only paired noisy images, we divide the original movie into two sub-movies based on odd and even frame numbers, processing them into odd and even images. We treat them as two independent, noisy observations of the underlying true signal, thus the idea of N2N can apply. During training, we partially mask both images, creating masked regions and unmasked regions corresponding to denoising and reconstruction tasks: in the unmasked region, the odd noisy patch learns to recover the even noisy patch, and vice versa. In the masked region, we introduce relatively low-noise images from the complete movie to supervise the reconstructed results. This denoising-reconstruction hybrid training scheme achieves the robust feature extraction of noisy cryo-EM images.The quality of training samples is crucial for the general-purpose feature extraction of cryo-EM images. Direct access to the public database leads to varying data quality, inconsistent data formats, or missing annotations. Therefore, we construct a large-scale, high-quality, and diverse single-particle cryo-EM image dataset by curating and manually processing 529 sets of data from EMPIAR [17] , obtaining over 270,000 cryo-EM movies or micrographs in total. After pre-training, DRACO naturally serves as a generalizable cryo-EM image denoiser and a foundation for various downstream model adaptions. We hence explore the performance of DRACO on three downstream tasks: micrograph curation, denoising, and particle picking. Extensive experiments show that DRACO outperforms the state-of-the-art baselines in all downstream tasks. We will release code, pre-trained/fine-tuned models, and the large-scale curated dataset. 
related work: 
 Our work aims to extend the vision foundation model to the field of cryo-EM. We therefore only discuss the most relevant works in respective fields.Vision foundation models in computer vision. Vision foundation models are pre-trained on largescale image datasets [18, 19] using self-supervised learning methods [20] , aimed at extracting general visual signals rapidly adaptable to various downstream visual tasks [21, 22, 23, 24] . Techniques for pre-training vision foundation models, such as contrastive learning [25, 26, 1, 27] and self-distillation [28, 5] , focus on aligning features across different models or modalities, while another method, masked image modeling [6, 29, 30, 4] , reconstructs features from masked images to capture highlevel visual semantics. However, these existing vision foundation models are not directly applicable to cryo-EM imaging. In particular, their application to cryo-EM imaging is limited by the high noise levels in micrographs, which degrade signal capture. Therefore, we propose a denoisingreconstruction pre-training framework that is robust to highly noisy cryo-EM micrographs, making it suitable for specific downstream tasks in cryo-EM.Vision foundation models in life science. The remarkable success of foundation models has extended to various life science imaging domains, including applications in retinal [10] , fluorescence microscopy [9] , histopathology [7, 8] , and radiology imaging [31] . These models have shown considerable effectiveness in tasks such as disease diagnosis, lesion detection, and image restoration within these fields. In contrast to these domains, which benefit from extensive and well-curated datasets supporting model training, the cryo-EM field lacks such resources. To fill this gap, we have developed a well-curated, large-scale dataset specifically designed to support the training of cryo-EM foundation models, ensuring that they can be effectively applied in this specialized field.Cryo-EM image denoising. To tackle the issues of low SNR and complex noise patterns in cryo-EM images, traditional denoising techniques often employ noise models like the Poisson-Gaussian model [12, 32] and rely on filtering methods [33, 34, 35] to denoise. However, these methods oversimplify the noise patterns, which can lead to the loss of high-frequency signal details. Recently, NT2C [36] uses Generative Adversarial Network to learn the noise patterns for denoising, but it requires simulated datasets as the clean references. Another series of learning-based methods [37, 38] make full use of multi-frame data by generating odd and even images for denoising based on Noise2Noise (N2N) [16, 39] framework. These methods do not require clean images for denoising, but they still suffer from small-scale datasets and network architectures, which limit their generalizability. In this paper, we propose DRACO, pre-trained on a large-scale curated dataset, that can naturally serve as a generalizable denoiser for cryo-EM micrographs.Downstream tasks in single particle analysis. An effective foundational model can benefit downstream tasks in cryo-EM, including micrograph curation and particle picking. Micrograph curation aims to ensure that only high-quality images are selected for further analysis, yet current methods rely heavily on manual inspection [40, 41] . Particle picking involves identifying and extracting representative particles from micrographs, which is a critical task in the cryo-EM single particle analysis (SPA) reconstruction pipeline. Traditional methods [42, 43, 44] , such as template matching [45, 46] and difference of Gaussians (DoG) method [47] , heavily rely on prior information and require substantial ad hoc post-processing. Learning-based models [48, 49, 50] , such as Topaz-Picking [51] , crYOLO [52] , and CryoTransformer [53] , offer more streamlined processes but still face challenges in generalizability due to the limited data scale. Our DRACO, pre-trained on large-scale cryo-EM image datasets, can effectively adapt to these tasks and demonstrate strong generalization capabilities. 
preliminary: imaging formation model: 
 Cryo-EM uses a Direct Detector Device (DDD) camera [32] for their notably higher detective quantum efficiency (DQE) compared to traditional cameras. This allows recording the micrograph as a multi-frame movie rather than a single integrated exposure. In this setup, a movie is a series of continuous multi-frame images, denoted as M = { Îi } M i=1 , where each frame Îi is an independent observation of the true signal I.Ideally, the imaging process in cryo-EM involves two main steps: 1) projecting the 3D density volume of the region of interest V (x, y, z) : R 3 → R along the z-axis via weak-phase object approximation [54] , and 2) modulating the projection image with the Point Spread Function (PSF) g of the cryo-EM optical lens, expressed as:EQUATION)where I is considered as the true signal.However, in practice, the captured frames suffer from extremely low SNR due to the limited electron dosage and the high sensitivity of DDD. The main noise source is Poisson (shot) noise from the detector, denoted as Poisson(I), arising from the inherent uncertainty of the electron measurement [12] . We assume that the additional noise types like heat, readout, and dark current noise are collectively modeled as a zero-mean Gaussian noise G with an unknown variance σ 2 [42] :EQUATIONAs the number of observations increases, the average of these movie frames converges to the true signal:E[ Î] = I ≈ 1 M M i=1Îi .We define the image noise ϵ as the difference between the captured and clean images for each frame. Thus, the expectation (mean value) of noise distribution is:EQUATIONThus, we derive that the cryo-EM image noise is zero-mean. This conclusion is also aligned with existing cryo-EM reconstruction methods [42, 41] , which directly assume that the noise distribution is an additive zero-mean Gaussian yet can still achieve high-resolution reconstruction. We are well aware that this derivation is relatively trivial compared to an actual theoretical analysis [12] , but this gives us intuitive guidance to integrate the N2N [16] idea: learning to denoise images from solely noisy image pairs, into our pre-training framework.Movie to micrograph triplets. Given an M -frame movie M, we divide it into odd framesM o = {I 2i-1 } ⌈M/2⌉ i=1and even frames M e = {I 2i } ⌊M/2⌋ i=1. An off-the-shelf motion correction method [13] is then applied to correct cross-frame drifts in M, M o , and M e . By summing up the frames within each subset, we generate three micrographs with the same shape: the original micrograph Î, odd micrograph Îo , and even micrograph Îe , As aforementioned, all these micrographs are expected to reflect the true signal I but corrupted by noises. 
denoising-reconstruction autoencoder: 
 We introduce DRACO, a denoising-reconstruction autoencoder for cryo-EM, as illustrated in Figure 2 . Different from existing masked imaging modeling methods, our model uses paired odd-even micrographs as inputs for the denoising target on visible patches. Further, we utilize the original micrographs as the additional supervision signal for reconstruction on masked patches.Masking. Following the standard scheme in Vision Transformer (ViT) [55] , each micrograph in a triplet, consisting of one original, one odd, and one even micrograph from the same movie, is divided into regular non-overlapping patches. We create patch sets {x o i } N i=1 for the odd, {x e i } N i=1 for the even, and {x i } N i=1 for the original micrograph, where N represents the number of patches. For the odd and the even micrographs used as inputs in our model, we generate two sets of binary masks,{m o i } N i=1and {m e i } N i=1 , with a mask ratio γ. Here, m i = 1 means the i-th patch is masked, and 0 means unmasked. Additionally, we ensure that a patch can be 1) only visible in one of them, or 2) masked in both. This strategy ensures that each visible patch has no information sharing of its corresponding patch on the other micrograph. Notably, this requires γ ≥ 0.5 for each input micrograph.Network architecture. For DRACO's pre-training, we employ a ViT-based encoder-decoder architecture following the MAE framework [6] . Positional embeddings are first added to the input patches, which are then masked to select only visible patches, denoted as {v i } γN i=1 . The encoder, G enc , is a ViT that transforms these visible patches from either odd or even micrographs into latent features. To The pipeline of DRACO. Given a pair of partially masked odd and even micrographs, the encoder takes odd-visible patches and even-visible patches as inputs. The unmasked latent patches are combined with masked latent patches together to generate the latent representation z i . Then the latent representation passes through the decoder to generate predicted patches. The N2N loss is applied to odd-visible predicted patches with corresponding even input patches, and vice versa. The reconstruction loss is applied to both invisible predicted patches with higher SNR input patches.align with the original unmasked size, zeros are padded to the latent features based on the positions indicated by the corresponding masks before the encoder outputs them:EQUATIONEach of the latent features from odd and even micrographs retain part of the original micrograph's information. To reconstruct the micrograph, we generate the latent representation {z i } N i=1 for further processing by the decoder:EQUATIONwhere the [MASK] token is a shared learnable embedding representing the masked patch to be predicted. Finally, the decoder G dec takes the latent representation {z i } N i=1 with another positional embedding added as input and predicts all masked and visible patches to reconstruct the complete micrograph:EQUATIONDenoising target. Inspired by N2N and Topaz-Denoise [37] , which predict denoised images only from paired noisy images, we introduce an image-denoising target on visible patches. For any patch x i visible in only the odd micrographs, the model predicts its counterpart in the even micrographs, and vice versa, as illustrated in Figure 2 . As mentioned earlier, the expectation (mean) values of the odd and even micrographs are the true signal. Therefore, following Topaz-Denoise, we employ a patch-wise L2 loss function for each visible patch, aiming for regressing the true signal:EQUATIONReconstruction target. For masked patches, we let the decoder predict the pixel value of patches from the original micrograph with higher SNR compared to odd and even micrographs for better reconstruction quality. Thus, the reconstruction loss is:EQUATIONTraining objective. During training, we combine the N2N loss with the reconstruction loss:EQUATION)where λ is a hyper-parameter set to 1.0 in all our experiments. 
experiments: 
 Large-scale curated dataset for pre-training. The effectiveness and robustness of DRACO depend heavily on the quality and scale of the cryo-EM pre-training dataset. However, direct access to public databases like the Electron Microscopy Public Image Archive (EMPIAR) [17] results in variations in data quality, inconsistent data formats, and inaccurate or even missing annotations. To overcome these challenges, we have developed a data generation workflow. First, we select datasets with reported resolutions better than 10 Å, ensuring high-quality data acquisition. Next, we collect the raw data, including metadata, movies, and micrographs, from pre-defined high-quality datasets available on EMPIAR. Finally, we re-process the raw data using cryoSPARC [41] through a custom processing pipeline designed to exclude low-quality micrographs and movies, generate annotations for downstream tasks, and verify the resolutions of the reconstructed results. This workflow has allowed us to compile a large-scale, curated cryo-EM dataset containing over 270,000 raw micrographs and more than 50,000 raw movies from 529 verified single-particle cryo-EM datasets, occupying approximately 25 TB of disk storage in total. Details of the cryoSPARC processing pipeline are provided in Appendix C.1.Data augmentation. Each micrograph in a triplet goes through the same data augmentation process: randomly cropping to between 1/16 and 1/4 of the original size (typically 4096 × 4096), resizing to 256 × 256, applying random horizontal and vertical flips, and finally normalizing based on the mean and standard deviation computed from the original micrograph. We randomly crop each micrograph 16 times within a single epoch for fully utilization.Pre-training details. We explore the performance of DRACO using two ViT architectures for the encoder, ViT-B and ViT-L, denoted as DRACO-B and DRACO-L, respectively. The decoder of DRACO uses 8 Transformer blocks with embedding dimension 512, followed by a three-layer convolution neck and a linear projection layer with an output dimension 16 × 16, which is also the patch size of the input. The mask ratio for the one input micrograph is 0.75 by default. To fully utilize our large-scale curated dataset, we warm up DRACO on the original 270,000 micrographs based on the MAE training scheme for 200 epochs. Then we adopt our novel denoising-reconstruction pre-training for 400 epochs. The warm-up stage takes 6 hours, and the pre-training stage takes 16 hours on a GPU cluster with 64 NVIDIA A800 GPUs, requiring approximately 80 GB of memory for a batch size of 4096.Particle picking aims to accurately locate particles in highly noisy micrographs, which is directly related to the resolution of the final reconstructed result. For adaption, we conduct supervised finetuning based on the ViT-based object detection framework Detectron2 [21] on our curated dataset. The results show that DRACO is capable of accurately detecting particles of various shapes and sizes across three challenging datasets.Dataset. To create the annotated dataset for adaptation, we employ the standard workflow of cryoSPARC to generate a high-quality annotated dataset containing over 80,000 full micrographs and approximately 8 million particles across 46 types of protein. Detailed descriptions of the annotation workflow can be found in Appendix C.2. The test dataset includes three full micrograph sets with EMPIAR ID 1) 10081: Human HCN1 channel protein [56] , the protein structure has a similar shape to ice, which could lead to false positive picking, 2) 10350: LetB transport protein [57] , this kind of protein tends to aggregate together, posing challenges in accurate picking in crowded area, and 3) 10407: 70S ribosome [58] , the micrographs are in the extremely low SNR.Baseline and metrics. We compare DRACO with existing state-of-the-art learning-based methods for generalized particle picking, including Topaz [51] , crYOLO [52] , and CryoTransformer [53] .For ViT-based baselines, including MAE, DRACO-B and DRACO-L as aforementioned. We integrate them into Detectron2 framework by loading pre-trained weights of DRACO's encoder into Detectron2's encoder. Additionally, to show the effectiveness of pre-trained encoder, we compare with Detectron2 trained from scratch. More configuration details can be found in Appendix D.1. We evaluate baselines in terms of conventional metrics including precision, recall, and F1 score, and the resolution of the resolved 3D structure from the picked particles, which is also a crucial metric. We process the particles selected by each method with the standard cryoSPARC workflow and finally Table 1 : Particle picking results. We report the precision, recall, F1 score, and resolution on each test dataset among all baselines. The resolution is obtained from the default cryoSPARC workflow. We compare DRACO with existing state-of-the-art methods and consistently achieve the best F1 score and resolution of reconstructed results. produce 3D reconstruction density maps and the corresponding resolution, as described in Appendix C.3.Results. As illustrated in Figure 3 , both Topaz and CryoTransformer tend to pick a larger number of particles, but this often results in many false positives. In contrast, crYOLO achieves higher precision in picking, yet exhibits a higher number of false negatives. Detectron2 trained from scratch and pre-trained MAE both have difficulties in distiguishing signal and noise, leading to a lack of generalizability. In contrast, DRACO effectively identifies correct particles, surpassing the performance of all baselines, as demonstrated in Table 1 .Once pre-trained, our model can naturally serve as a generalizable denoiser by directly predicting every patch of input noisy micrograph without any further fine-tuning.Baseline and metrics. To evaluate the effectiveness of DRACO on the denoising task, we first compare DRACO with the standard MAE trained on the 270,000 micrographs from our large-scale curated dataset with ViT-B as the backbone, denoted as MAE. We further compare with a popular denoising method Topaz-Denoise [37] in cryo-EM. For a fair comparison, we train Topaz-Denoise on our odd-even micrograph dataset for 100 epochs with default settings. Last, we compare with the traditional method Low-pass filtering that has already been integrated into commercial software cryoSPARC [41] . We utilize the same protocol used in cryoSPARC and set the low-pass cutoff resolution to 20 Å. As cryo-EM micrographs lack clean ground truth, following Topaz-Denoise, we employ an SNR calculation method that involves 20 manually annotated signal-background region pairs as references. For each i-th pair, we calculate the mean and variance for both the signal region r s i and the background region r b i , yielding µ s i , v s i for the signal, and µ b i , v b i for the background. The average SNR is then computed in dB as follows:SNR = 10 N N i=1 log 10 (µ s i -µ b i ) 2 v b i (11)Dataset. To evaluate the denoising capabilities of the baseline models, we select four original micrograph datasets as test datasets, which are excluded from the training set. These sets are Human Apoferritin (EMPIAR-10421) [59] , HA Trimer (EMPIAR-10096) [60] , Phage MS2 (EMPIAR-10075) [61] , and RNA polymerase (EMPIAR-11521) [62] . For each dataset, 5 micrographs are selected and 20 signal-background pair regions are labeled in total. Specifically, these signal and background regions are chosen close together to ensure similar background signals across both regions.Results. The quantitative experiments show that DRACO achieves significant performance improvements in terms of SNR after denoising compared with state-of-the-art methods, as shown in Table 2 . In Figure 4 , the standard MAE can only recover smooth contours of the particle with severe artifacts. Low-pass filtering smooths both signal and background noise, but the background noise is still relatively high and the structure information of particles is corrupted. Topaz sometimes fails to effectively denoise micrographs but over-smooth them instead, which affects the generalizability. In contrast, DRACO outperforms all baselines in retaining the original particle signals with the lowest background noises, showing the best generalizability. We additionally reconstruct 3D density map and corresponding resolution using the denoised particles generated by each method. To ensure that comparisons reflect only the impact of denoising quality, we fix the locations and poses of the picked particles, which were determined using the cryoSPARC workflow. The results are shown in Table 3 . DRACO consistently achieves the highest resolution in most cases, demonstrating its effectiveness in preserving more high-frequency signals while effectively reducing background noise. We demonstrate additional denoising results in Figure 5 .A modern cryo-EM can capture thousands of micrographs in a day. However, the quality of captured micrographs is unverified. Low-quality micrographs may arise from artifacts such as empty sample, ice crystals, ethane contamination, severe drifting, etc. [40] . Low-quality micrographs can negatively contribute to the final reconstruction results. A reliable automated micrograph curation method can significantly improve the efficiency of the data processing pipeline, resulting in shorter processing time and improved final resolution. We show that DRACO can easily adapt to this 2-class classification task by linear probing, and achieving the best performance compared to the state-of-the-art method.Similar to [6] , we freeze the encoder backbone and train an extra linear classification head.Dataset. We manually annotate 1,194 micrographs from original micrograph datasets, assigning a binary label (accept or reject) to each to indicate quality. The dataset comprises 617 high-quality and 577 low-quality micrographs. We divided these micrographs into training and evaluation datasets using an 80%/20% split ratio.Baseline and metric. We compare DRACO-B and DRACO-L with linear probing against several baselines: a small ResNet18 [63] ; an existing supervised method Miffi [40] , which has been trained on 45,000 annotated data; and the standard MAE with linear probing both pre-trained and adapted on our datasets. The ResNet18 is trained from scratch to show the effectiveness of our pre-training strategy. More configuration details is provided in Appendix D.2. We report the widely used metrics for classification including precision, recall, F1 score, and accuracy on our test dataset. As Miffi predicts multi-labels on low-quality micrographs, we consider them all as rejections for our metric calculations. We compare the performance of networks with different parameter sizes in particle picking and micrograph curation tasks, as shown in Table 1 and Table 4 . The results demonstrate that our method can effectively scale up. Additionally, we evaluate the impact of different mask ratios on denoising and micrograph curation performance, as shown in Table 5 . DRACO achieves the highest SNR and curation metric at a 0.75 mask ratio, thus we choose 0.75 as our default mask ratio. We have also conducted an additional ablation study of loss design. Specifically, we remove either the N2N loss (w/o N2N) or the reconstruction loss (w/o recon) from the training and evaluate the resulting models on particle picking and denoising tasks, as shown in Table 6 . The result shows that both N2N and reconstruction losses improve performance. Without the N2N loss, DRACO struggles to distinguish between signal and noise in micrographs. Without the reconstruction loss, DRACO loses its ability to extract general features. We show additional visualization of their denoising results (Figure 6 ) in the Appendix A.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We do not provide code and data in submission. However, the code and data will be released upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
discussion: 
 Limitations. As the first attempt to achieve robust feature extraction for cryo-EM via a novel denoising-reconstruction autoencoder, our work presents opportunities for future enhancements. First, our method relies heavily on the performance of motion correction algorithms. This can be improved by designing a more comprehensive denoising task for raw noisy movies. Second, although we have collected what we believe to be the largest curated dataset for cryo-EM, it focuses primarily on mainstream single-particle datasets. To enhance dataset diversity, other types of cryo-EM datasets, such as cryo-electron tomography (cryo-ET) [64] datasets, should also be included. Finally, our current approach only supports various micrograph-level downstream tasks. For particle-level tasks, such as pose estimation, a more fine-grained yet robust feature extraction is required. This can be achieved by developing a particle-level version of DRACO.We have introduced DRACO, a foundation model designed specifically for cryo-EM image processing, supported by a unique denoising-reconstruction pre-training framework to enable robust feature extraction for cryo-EM micrographs. We have constructed a diverse and high-quality cryo-EM image dataset from the uncurated public database, comprising over 270,000 movies and micrographs. After pre-training, our model's versatility is evidenced by its superior adaptation performance across multiple downstream tasks, including denoising, micrograph curation, and particle picking. All code, pre-trained model weights, and datasets will be made publicly available for further research and model development.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have stated the pre-training details in Section 5 and the adaptation details in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
acknowledgement: 
 This work was supported by HPC Platform of ShanghaiTech University. We thank Zhenyang Xu for shaping the paper.We demonstrate that DRACO achieves the highest visual denoising quality in terms of both signal preservation and noise removal, as shown in Figure 5 . This figure serves as an extension to Figure 4 in our main paper. Furthermore, we visualize the denoising results from our ablation study on different mask ratios in Figure 6 . We observe that at a 0.75 mask ratio, DRACO achieves the best denoising results, as supported by Table 2 in our main paper. Lastly, we visualize the reconstruction ability at a 0.75 mask ratio in Figure 7 . DRACO demonstrates comparable reconstruction capability to MAE, while significantly better denoising results on visible patches. Figure 7 : Additional results on image reconstruction. We present the reconstruction results at various image resolutions while maintaining a consistent mask ratio of 0.75. DRACO demonstrates enhanced detail preservation on the visible patches compared to MAE.Though DRACO has not been pre-trained on cryo-ET datasets, we found that DRACO can be directly applied on cryo-ET tilt series. Here, we demonstrate that DRACO is capable of denoising the unseen HIV tilt series [65] , as shown in Figure 8 . Specifically, we evaluate DRACO on both the tilt series and the volume slices, showing that DRACO effectively removes background noise and achieves higher contrast in the volume. Furthermore, we assess DRACO's performance in the context of reconstruction. We compare the slices from both tomograms reconstructed using original and denoised tilt series. Additionally, we compare these with the denoised slice from the original tomogram. The result shows that DRACO can improve the contrast of slices before or after the reconstruction. EMPIAR [17] is a public archive for storing raw cryo-EM images and 3D reconstructions from vEM and XT experiments. It currently contains over 2,000 entries, totaling more than 2 PB of data. The EMDB [66] is an archive of 3D reconstructions derived from cryo-EM experiments, many of which supplement the experimental information of the EMDB-related EMPIAR entries, such as sample preparation and reconstruction processes.In the field of 3D electron microscopy, cryo-EM micrographs undergo complex image processing to achieve 3D reconstructions with specified resolutions. Experts can model protein molecules accurately on 3D reconstructions with resolutions better than 3 Å. We define the quality of datasets from the perspective of structural biology; hence, high-quality datasets should produce high-resolution 3D reconstructions suitable for detailed analysis.When constructing the pre-training dataset, we utilize the REST API provided by EMPIAR to obtain metadata for each dataset, such as experiment type, EMDB ID, and image classification. We specifically filter for EMPIAR entries that are experimentally linked to EMDB, prioritizing those with reconstruction resolutions better than 10 Å. Subsequently, we collect as many datasets as possible that contain single-frame micrographs. For datasets that include multiple frames, we further process them by separating the frames into odd and even micrographs.Cryo-EM raw data consists of multi-frame recordings known as movies, which capture the number of electrons. After motion correction [13] , these movies yield single frames referred to as micrographs. We download as many single-frame micrographs as possible, categorized by image type. Additionally,• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We do not provide error bars or other appropriate information about the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix:  


Decision: This work introduces DRACO, a foundation model for cryo-EM tasks. The model uses a hybrid denoising-reconstruction training strategy, with a Noise2Noise loss function paired with a reconstruction loss, to enhance the quality of cryo-EM images. The learned representations are fine-tuned for various downstream tasks, such as denoising, particle picking, and micrograph curation. Experimental results show that DRACO outperforms state-of-the-art techniques across these tasks, achieving higher signal-to-noise ratios (SNRs) and providing a robust solution for cryo-EM image processing. Although the reviewers pointed out some minor weaknesses (such as the motivation of choices, ablation studies), the rebuttal addresses most of them.

Reviews: review: paper_summary: paper_summary: This work proposes a foundation model for Cryo-EM tasks. The technique consists of pretraining a masked autoencoder on a curated dataset, with a Noise2Noise-like self-supervision scheme. The learned representations are then fine-tuned for various tasks, such as denoising, particle picking and micrograph curation. Experimental results show promise across all tasks compared to existing techniques.
strength_weakness: strength_weakness: - Justification for the training scheme is somewhat unclear. Why is the supervision signal from the original micrographs not used in every patch. By only utilizing the original micrograph on the masked area we discard some information. Alternatively, odd/even micrographs could be used to predict the original averaged micrograph.

- It is unclear how the method generalizes to unseen real data, a common scenario where foundation models are deployed. It would be important to understand how well the technique can be adapted in such cases.

- It seems like authors introduce a noise model (Gaussian + Poisson), but does not leverage the model further, beyond the zero-mean assumption.
questions: questions: - How well would the method perform outside of the single-particle setting, for instance slices of cryo-ET?
- Can the proposed technique be adapted to 3D data such as that in cryo-ET?
- Typo: line 134 theological should be theoretical
- Vector notation would be better in Eq (8) and (9) (using norms)
limitations: limitations: Limitations are addressed.
paper_summary: paper_summary: In this paper, the authors introduce an autoencoder-based model to denoise cryo-EM micrographs. By splitting aligned movies into even and odd images, the authors obtain two images with similar signal and two different realizations of noise. Following Noise2Noise and Topaz-Denoise, the denoising target is to predict the "even" realization from the "odd" on and vice versa. The authors show that their method outputs micrographs with higher SNRs (as defined by Eq 11). The representation learned by the encoder can be used to fine-tune an object detection algorithm for particle picking or a linear layer for filtering micrographs.
strength_weakness: strength_weakness: **Denoising section.** Section 5.1 describes the method as a denoising method. I found this section slightly misleading. To me, the main contribution of DRACO is its ability to learn a compact representation of cryo-EM patches, which can be used for downstream tasks. When the method is first described as a denoiser, the reader can be misled and think that downstream tasks (e.g particle picking) use the denoised images. Furthermore, I find that the evaluation of the denoising performances lacks quantitative analysis. Table 1 only reports the SNR obtained with a heuristic definition of the SNR, but reporting the resolution obtained after homogeneous reconstruction on the denoised images would be more compelling. If the authors think the denoising capability is *not* the main contribution of the paper, my recommandation would be to move down Section 5.1 to avoid misunderstandings. Otherwise, Section 5.1 should contain further quantitative evaluations.
questions: questions: Do the authors plan to grant access to their pretraining datasets?
limitations: limitations: The limitations are well described at the end of the paper and the authors suggest interesting avenues for future work.
paper_summary: paper_summary: In this paper, the authors introduce DRACO, a Denoising-Reconstruction Autoencoder for Cryogenic Electron Microscopy (cryo-EM) inspired by the Noise2Noise (N2N) approach. DRACO aims to address the high-level noise corruption in cryo-EM images, which are often overlooked by other foundation models in computer vision. The method involves processing cryo-EM movies into odd and even images, treating them as independent noisy observations, and applying a denoising-reconstruction hybrid training scheme. By masking both images, DRACO creates denoising and reconstruction tasks to enhance the quality of the output.

For the pre-training phase, the authors build a high-quality and diverse dataset from an uncurated public database, comprising over 270,000 movies or micrographs. This dataset is essential for ensuring the effectiveness of DRACO as a generalizable cryo-EM image denoiser and a foundation model for various downstream tasks. The experimental results demonstrate that DRACO outperforms state-of-the-art baselines in denoising, micrograph curation, and particle picking tasks.
strength_weakness: strength_weakness: Weaknesses
While the paper makes a valuable contribution, there are several areas for improvement:

Training Method Justification: The authors use a Noise2Noise (N2N) approach for training but do not justify why a Noise2Clean (N2C) approach was not considered, given the Poisson-Gaussian noise model in electron microscopy images.

Comparison with Blind-Spot Networks: The paper lacks a comparison with existing blind-spot network-based denoising methods. Including such a comparison would provide a clearer benchmark for evaluating the effectiveness of the proposed method.

Evaluation Metrics: The use of SNR as the sole evaluation metric may not fully capture the quality of restored images, as SNR tends to favor smoother results and may overlook important texture details.
questions: questions: [Why Not Use N2C for Network Training]
The authors mention in the paper that the noise model for electron microscopy images is a Poisson-Gaussian noise model. Given this, why not use a Noise2Clean (N2C) approach by inputting simulated noisy images and learning the corresponding clean images for network training?

[Comparison with Blind-Spot Network-Based Methods]
There are existing denoising methods for real noisy images based on blind-spot networks. Why can't these methods be applied to electron microscopy image denoising? Furthermore, in the comparison with other methods, the authors only compare with low-pass filtering and MAE methods, making it difficult to clearly see the effectiveness of the proposed method.

[Is SNR Sufficient to Evaluate Restored Image Performance?]
The authors compare the SNR of restoration results from different methods. However, considering the calculation method of SNR, it may favor smoother restoration results and potentially overlook some texture details.
limitations: limitations: Can the authors construct a more novel paired electron microscopy image dataset?
paper_summary: paper_summary: This manuscript introduces a new denoising autoencoder for cryo-EM micrographs based on vision transformers (ViTs) which is trained on masked pairs of images with a noise2noise loss function paired combined a reconstruction loss. In addition to the model itself, an important contribution of the work is the curation of a large dataset of experimental micrographs used to train the model, containing a total of 270 000 micrographs from publicly available sources. The resulting model is shown to achieve state-of-the-art performance on denoising, particle picking, and micrograph curation.
strength_weakness: strength_weakness: The manuscript is lacking in details in several important areas. First, the description of the method is sometimes at a very superficial level, making it hard to understand how it is actually implemented. For example, the mask is given as input to the decoder in eq. (5), but it is not clear how this is given as an input to the ViT. Furthermore, the [MASK] token is simply described as a “shared learnable embedding” without any more details. Clearly spelling out the details of the architecture in the main text of in an appendix would significantly strengthen the work. This also goes for the adaptation of the network to the other two downstream tasks (particle picking and micrograph curation), where the authors simply state that they “conduct supervised fine-tuning based on Detectron2” (for particle picking) and that they “freeze the encoder backbone and train an extra linear classification head” (for micrograph curation).

Second, the motivation for the various choices made is sometimes missing. This goes for the choice of architecture but also for the construction of the loss function, the choice of patch size, data augmentation strategy, and so on.

Third, the evaluation methods for the first two task need to be better motivated. For example, the SNR calculation method in eq. (7) is simply stated (with a reference to the Topaz-denoise paper) without further explanation. Similarly, the definition of true and false positives and false negatives for the particle picking task needs to be spelled out.

The final weakness is Section 5.4, which purports to be an ablation study. It refers to different results based on parameter sizes in Tables 2 and 3, but those results do not seem to be there. Furthermore, this section seems to be more of a hyperparameter study, rather than an ablation study (no steps of the pipeline are removed or replaced by others here).

There are also some issues with language, such as the reference to “theological analysis” on page 4 and “a modern cryo-EM” and “artifacts such as empty” on page 8.
questions: questions: – Why use a ViT as opposed to a simpler convolutional network?
– How does masking an image make sense on images such as micrograph which typically lack long-range structure?
– Please define γ on page 4.
– Why is the reconstruction target necessary for the proper training of the network? How does the dependence of the noise realization between input and target affect training here.
– How do we expect the mask ratio to affect the results? Why does this trade off between “signal preservation and background noise removal”?
limitations: limitations: The authors have largely addressed the limitations of the work in the manuscript.

Comments: title: Response to rebuttal
comment: Thank you for your rebuttal. Given the changes you propose, I will raise my score to 6.
I acknowledge that I have read the authors' rebuttal and the other reviewers's comment. My concerns have been addressed and I have decided to raise my score.
I thank the authors for addressing my questions. I maintain my opinion that this work is an interesting contribution and keep my score.
Dear reviewers,

Could you please respond to the rebuttal and discuss with authors?