Title: DRACO: A Denoising-Reconstruction Autoencoder for Cryo-EM
Venue: NeurIPS 2024
Accepted: yes
Keywords: masked image modeling

foundation model

cryo-EM

denoising autoencoder
Abstract: Foundation models in computer vision have demonstrated exceptional performance in zero-shot and few-shot tasks by extracting multi-purpose features from large-scale datasets through self-supervised pre-training methods. However, these models often overlook the severe corruption in cryogenic electron microscopy (cryo-EM) images by high-level noises. We introduce DRACO, a Denoising-Reconstruction Autoencoder for CryO-EM, inspired by the Noise2Noise (N2N) approach. By processing cryo-EM movies into odd and even images and treating them as independent noisy observations, we apply a denoising-reconstruction hybrid training scheme. We mask both images to create denoising and reconstruction tasks. For DRACO's pre-training, the quality of the dataset is essential, we hence build a high-quality, diverse dataset from an uncurated public database, including over 270,000 movies or micrographs. After pre-training, DRACO naturally serves as a generalizable cryo-EM image denoiser and a foundation model for various cryo-EM downstream tasks. DRACO demonstrates the best performance in denoising, micrograph curation, and particle picking tasks compared to state-of-the-art baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces a new denoising autoencoder for cryo-EM micrographs based on vision transformers (ViTs) which is trained on masked pairs of images with a noise2noise loss function paired combined a reconstruction loss. In addition to the model itself, an important contribution of the work is the curation of a large dataset of experimental micrographs used to train the model, containing a total of 270 000 micrographs from publicly available sources. The resulting model is shown to achieve state-of-the-art performance on denoising, particle picking, and micrograph curation.\n\nSTRENGTHS:\nThe manuscript describes the problem to be solved accurately and outlines why traditional methods for training autoencoders do not perform well. The overall description of the proposed model is also well illustrated through the use of figures and well-defined notation. The considerable effort expended to ensure that the training and testing datasets are of sufficient quality is also a major strength of the work. Finally, the results on the three evaluation tasks are quite impressive, even though some questions arise regarding their calculation.\n\nWEAKNESSES:\nThe manuscript is lacking in details in several important areas. First, the description of the method is sometimes at a very superficial level, making it hard to understand how it is actually implemented. For example, the mask is given as input to the decoder in eq. (5), but it is not clear how this is given as an input to the ViT. Furthermore, the [MASK] token is simply described as a “shared learnable embedding” without any more details. Clearly spelling out the details of the architecture in the main text of in an appendix would significantly strengthen the work. This also goes for the adaptation of the network to the other two downstream tasks (particle picking and micrograph curation), where the authors simply state that they “conduct supervised fine-tuning based on Detectron2” (for particle picking) and that they “freeze the encoder backbone and train an extra linear classification head” (for micrograph curation).\n\nSecond, the motivation for the various choices made is sometimes missing. This goes for the choice of architecture but also for the construction of the loss function, the choice of patch size, data augmentation strategy, and so on.\n\nThird, the evaluation methods for the first two task need to be better motivated. For example, the SNR calculation method in eq. (7) is simply stated (with a reference to the Topaz-denoise paper) without further explanation. Similarly, the definition of true and false positives and false negatives for the particle picking task needs to be spelled out.\n\nThe final weakness is Section 5.4, which purports to be an ablation study. It refers to different results based on parameter sizes in Tables 2 and 3, but those results do not seem to be there. Furthermore, this section seems to be more of a hyperparameter study, rather than an ablation study (no steps of the pipeline are removed or replaced by others here).\n\nThere are also some issues with language, such as the reference to “theological analysis” on page 4 and “a modern cryo-EM” and “artifacts such as empty” on page 8.\n\nQUESTIONS:\n– Why use a ViT as opposed to a simpler convolutional network?\n– How does masking an image make sense on images such as micrograph which typically lack long-range structure?\n– Please define γ on page 4.\n– Why is the reconstruction target necessary for the proper training of the network? How does the dependence of the noise realization between input and target affect training here.\n– How do we expect the mask ratio to affect the results? Why does this trade off between “signal preservation and background noise removal”?\n\nLIMITATIONS:\nThe authors have largely addressed the limitations of the work in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce DRACO, a Denoising-Reconstruction Autoencoder for Cryogenic Electron Microscopy (cryo-EM) inspired by the Noise2Noise (N2N) approach. DRACO aims to address the high-level noise corruption in cryo-EM images, which are often overlooked by other foundation models in computer vision. The method involves processing cryo-EM movies into odd and even images, treating them as independent noisy observations, and applying a denoising-reconstruction hybrid training scheme. By masking both images, DRACO creates denoising and reconstruction tasks to enhance the quality of the output.\n\nFor the pre-training phase, the authors build a high-quality and diverse dataset from an uncurated public database, comprising over 270,000 movies or micrographs. This dataset is essential for ensuring the effectiveness of DRACO as a generalizable cryo-EM image denoiser and a foundation model for various downstream tasks. The experimental results demonstrate that DRACO outperforms state-of-the-art baselines in denoising, micrograph curation, and particle picking tasks.\n\nSTRENGTHS:\nThe paper is well-organized and clearly written, with a logical flow of information from the problem statement to the proposed solution and experimental validation. The authors provide a clear explanation of their denoising-reconstruction hybrid training scheme, including the masking of images to create denoising and reconstruction tasks, which enhances the reader’s understanding of the methodology.\n\nWEAKNESSES:\nWeaknesses\nWhile the paper makes a valuable contribution, there are several areas for improvement:\n\nTraining Method Justification: The authors use a Noise2Noise (N2N) approach for training but do not justify why a Noise2Clean (N2C) approach was not considered, given the Poisson-Gaussian noise model in electron microscopy images.\n\nComparison with Blind-Spot Networks: The paper lacks a comparison with existing blind-spot network-based denoising methods. Including such a comparison would provide a clearer benchmark for evaluating the effectiveness of the proposed method.\n\nEvaluation Metrics: The use of SNR as the sole evaluation metric may not fully capture the quality of restored images, as SNR tends to favor smoother results and may overlook important texture details.\n\nQUESTIONS:\n[Why Not Use N2C for Network Training]\nThe authors mention in the paper that the noise model for electron microscopy images is a Poisson-Gaussian noise model. Given this, why not use a Noise2Clean (N2C) approach by inputting simulated noisy images and learning the corresponding clean images for network training?\n\n[Comparison with Blind-Spot Network-Based Methods]\nThere are existing denoising methods for real noisy images based on blind-spot networks. Why can't these methods be applied to electron microscopy image denoising? Furthermore, in the comparison with other methods, the authors only compare with low-pass filtering and MAE methods, making it difficult to clearly see the effectiveness of the proposed method.\n\n[Is SNR Sufficient to Evaluate Restored Image Performance?]\nThe authors compare the SNR of restoration results from different methods. However, considering the calculation method of SNR, it may favor smoother restoration results and potentially overlook some texture details.\n\nLIMITATIONS:\nCan the authors construct a more novel paired electron microscopy image dataset?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce an autoencoder-based model to denoise cryo-EM micrographs. By splitting aligned movies into even and odd images, the authors obtain two images with similar signal and two different realizations of noise. Following Noise2Noise and Topaz-Denoise, the denoising target is to predict the \"even\" realization from the \"odd\" on and vice versa. The authors show that their method outputs micrographs with higher SNRs (as defined by Eq 11). The representation learned by the encoder can be used to fine-tune an object detection algorithm for particle picking or a linear layer for filtering micrographs.\n\nSTRENGTHS:\n**Large-scale curated dataset.** The authors give a detailed description of the protocol they followed to preprocess the EMPIAR datasets and pretrain their model.\n\n**Description of the model.** The authors give a detailed description of the architecture of the model.\n\n**Downstream model adaptation.** I find the possibility of using DRACO for model adaptation particularly exciting and potentially impactful. The authors illustrate this capability with two examples: particle picking and micrograph cleaning.\n\nWEAKNESSES:\n**Denoising section.** Section 5.1 describes the method as a denoising method. I found this section slightly misleading. To me, the main contribution of DRACO is its ability to learn a compact representation of cryo-EM patches, which can be used for downstream tasks. When the method is first described as a denoiser, the reader can be misled and think that downstream tasks (e.g particle picking) use the denoised images. Furthermore, I find that the evaluation of the denoising performances lacks quantitative analysis. Table 1 only reports the SNR obtained with a heuristic definition of the SNR, but reporting the resolution obtained after homogeneous reconstruction on the denoised images would be more compelling. If the authors think the denoising capability is *not* the main contribution of the paper, my recommandation would be to move down Section 5.1 to avoid misunderstandings. Otherwise, Section 5.1 should contain further quantitative evaluations.\n\nQUESTIONS:\nDo the authors plan to grant access to their pretraining datasets?\n\nLIMITATIONS:\nThe limitations are well described at the end of the paper and the authors suggest interesting avenues for future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a foundation model for Cryo-EM tasks. The technique consists of pretraining a masked autoencoder on a curated dataset, with a Noise2Noise-like self-supervision scheme. The learned representations are then fine-tuned for various tasks, such as denoising, particle picking and micrograph curation. Experimental results show promise across all tasks compared to existing techniques.\n\nSTRENGTHS:\n- A foundation model for cryo-EM is an interesting contribution that has potential in a wide range of downstream applications in structural biology.  \n- The proposed training scheme is interesting and novel to the best of my knowledge.\n- Authors curate a pretraining dataset which, if released to the community, can greatly benefit data-driven research for cryo-EM.\n- The experimental results are promising.\n\nWEAKNESSES:\n- Justification for the training scheme is somewhat unclear. Why is the supervision signal from the original micrographs not used in every patch. By only utilizing the original micrograph on the masked area we discard some information. Alternatively, odd/even micrographs could be used to predict the original averaged micrograph.\n\n- It is unclear how the method generalizes to unseen real data, a common scenario where foundation models are deployed. It would be important to understand how well the technique can be adapted in such cases.\n\n- It seems like authors introduce a noise model (Gaussian + Poisson), but does not leverage the model further, beyond the zero-mean assumption.\n\nQUESTIONS:\n- How well would the method perform outside of the single-particle setting, for instance slices of cryo-ET?\n- Can the proposed technique be adapted to 3D data such as that in cryo-ET?\n- Typo: line 134 theological should be theoretical\n- Vector notation would be better in Eq (8) and (9) (using norms)\n\nLIMITATIONS:\nLimitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}