CryoGEM: Physics-Informed Generative Cryo-Electron Microscopy
Authors: Jiakai Zhang
Qihe Chen
Yan Zeng
Wenyuan Gao
Xuming He
Zhijie Liu
Jingyi Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Synthesis
Contrastive Learning
Cryo-EM
Primary area: generative_models

Abstract: In the past decade, deep conditional generative models have revolutionized the generation of realistic images, extending their application from entertainment to scientific domains. Single-particle cryo-electron microscopy (cryo-EM) is crucial in resolving near-atomic resolution 3D structures of proteins, such as the SARS-COV-2 spike protein. To achieve high-resolution reconstruction, a comprehensive data processing pipeline has been adopted. However, its performance is still limited as it lacks high-quality annotated datasets for training. To address this, we introduce physics-informed generative cryo-electron microscopy (CryoGEM), which for the first time integrates physics-based cryo-EM simulation with a generative unpaired noise translation to generate physically correct synthetic cryo-EM datasets with realistic noises. Initially, CryoGEM simulates the cryo-EM imaging process based on a virtual specimen. To generate realistic noises, we leverage an unpaired noise translation via contrastive learning with a novel mask-guided sampling scheme. Extensive experiments show that CryoGEM is capable of generating authentic cryo-EM images. The generated dataset can be used as training data for particle picking and pose estimation models, eventually improving the reconstruction resolution.

CryoGEM: Physics-Informet Generative Cryo-Electron Microscopy
Jiakai Zhang
Qihe Chen 1,2"
Yan Zeng
Wenyuan Gao
Xuming He-
Zhijie Liu
Jingyi Yu-
Shanghai Tech University
Cellverse iHHman Institute
Schangik,cheng22200 zengyan2024, Sangyyan024,gaoryy gaowy, hexm, liuzhj ywwjangril/ssaanghattet edu.
Abstract
In the past decade, deep conditional generative models have revolutionized the generation of realistic images, extending their application from entertainment to scientific domains. Single-particla cryo-electron microscopy (cryo-EM) is crucial in resolving near-atomic resolution 3D structures of proteins, such as the SARS- COV-2 spike protein. To achieve high nigh-resolution reconstruction, a comprehensive data processing pipeline has been adopted. However, its performance is still limited as it lacks high-quality annotated datasets training. To address this, we introduce physics-informed generative cryo-electron microscopy (CryoGEM)), which for the first time integrates physics-basee cryo-EM simulation with a generative umpaired noise translation to generate physically correct synthetic cryo-EM datasets with realistic noises. Initially, CryoGEM simulates the cryo-EM imaging process based virtual specimen. To generate realistic noises, we leverage unpaired noise translation via contrastive learning with a novel mask-guided sampling scheme. Extensive experiments show that CryoGEM is capable of generating authentic cryo- EM images. The generated dataset can used as training data for particle picking and pose estimation models, eventually improving the reconstruction resolution.
Introduction
In the past decade, deep generative models like VAES 24 GANs 13,
and diffusion models 16
have achieved significant success in conditional image generation. Recently, Stable Diffusion
43
can produce high-quality images given simple textual descriptions. Additional controls [62 can be
imposed produce tailored visual effects, e.g., theatrical lighting (39) and specifie perspectives [3]
In fact, the successes of image generation have gone way beyond visual pleasantness, stimulating significant advances in scientific explorations. Examples include brain magnetic resonance imaging
computational tomography (MRI-to-CT) translation [58). X-ray image generation [49]. etc. Different
from entertainment applications, generation techniques for scientific imaging should faithfully follow the physical process: the generated results would eventually be applied to real-world downstream tasks such medical image diagnosis. In this work, we extend image generation biomolecular imaging technique, single-particle cryo-electron microscopy (cryo-EM) to improve the performance of its downstream tasks. Cryo-EM aims recover the near-atomic resolution 3D structure of proteins, with its latest application in recovering the SARS-COV spike protein [59 structure for drug development.
specific
'The authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

fleetonn beam
Villecue imag ni
Pregrecanaing
Oyyrailiee Ir
Friten aggrepiinn
jodan
Cryatilia Syrtheii booue
(+) Partide
Clavifiotion
Partide Curation
(4)
[o
Heteroo Relinement
ano
IP, Horm Refimeeennt
improved by CryoGEM
CryodEM)
(d*)
Figure I: CryoGEM improves cryo-EM data analysis. Cryo-EM captures images of molecules in vitrified ice via electron beams. Data is processed for a high-resolution 3D reconstruction by comprehensive pipeline. However, some modules like (a) particle picking and (d) ab-initio 31) reconstruction still lack high-quality training datasets. Given coarse result as an input, CryoGEM can synthesize authentic single-particle micrographs as training dataset augmentation.
As shown in Figure a comprehensive data processing pipeline of single-particle cryo-EM starts with capturing transmission images of flash-frozen purified specimens, termed as micrographs, using high-energy electron beams. The complete dataset contains hundreds of thousands of images of target particles with unknown locations, poses, and shapes. The main challenge in cryo-EM is accurately estimate the locations and orientations of particles in the extremely low signal-to -ggaaltt-noise ratio (SNR), mainly caused by detector shot noise in limited dosage conditions and other structural noises such as global ice gradients. Thus, the performance of existing methods for particle picking [4] and pose estimation (28] are limited due to the lack of high-quality annotated training datasets, which are labor-intensive for human experts.
In this paper, we present novel physics-informed generative cryo-electron microscopy (CryoGEM) technique, which is capable of generating authentic annotated synthetic datasets using just 100 unannotated micrographs from a real dataset. To achieve highly controllable generation results, we introduced simple yet highly controllable physical simulation process. Based on the coarse density volume, we achieve control at both the particle level and the micrograph level. However, this physical simulation still lacks critical authentic noise modeling. We thus adopt unpaired image-to-image translation to generate authentic noises. Existing methods like CycleGAN source and target domains are bijective; however, real cryo-EM micrographs have random noises, existing contrastive learning methods, such as contrastive unpaired translation (CUT) B7], relax these assumptions by using random sampling strategy for positive and negative samples. However, cryo-EM, the randomly sampled positives and negatives can be semantically similar since particles are densely located in a micrograph. To address these challenges, we employ novel contrast noise translation to transform the simulated micrographs into authentic cryo-EM micrographs. Furthermore, segmentation paired with the simulated result as a guide for positive and negative sample selection. We show that precise guidance on sample selection can significantly improve the quality of image generation.
68 assume that the
We validate CryoGEM on five diverse and challenging real cryo-EM datasets. Extensive experiments show that CryoGEM achieves significantly better visual quality compared with state-of-ther the-th--rt art methods. Also, we demonstrate that the performance of existing deep models in downstream tasks, including particle picking and particle pose estimation, can be significantly improved by training on our synthetic dataset. Notably, we achieve 44% picking performance improvements, leading to 22% better resolution of final reconstruction on average.

---

2
Related Works
Our work aims to extend generation methods the field of cryo-EM. We therefore only discuss the most relevant works in respective fields.
Cryo-EM Pipeline. In recent decades, the CryoEM pipeline has rapidly evolved. Popular software such as CryoSPARC [41], Relion [70], and Warp [48] has been widely used for high-resolution 3D reconstruction of macromolecules. Two critical steps are particle picking and pose estimation. To achieve accurate particle picking, recent neural methods [4] 10 51 14 [66 have built upon several manually annotated real datasets incorporated with few-shot learning techniques. However, they still lack generalization capability as the training dataset only covers a small portion of real scenarios. In pose estimation, traditional methods 770 41 propose a Maximum-A A-Posterooi (MAP) optimization by Expectation-Maxmnizaae been adopted for ab-initio reconstruction, i.e., 3D reconstruction from particle images with unknown poses. However, their performance is still limited without further fine-gred grained refinement processes. We improve the performance of particle picking and pose estimation by generating high-quality annotated synthetic datasets as training datasets.
(EM). Recent self-supervised models (28] 27 67 have
Cryo-EM Simulations. Theoretical simulation techniques, based on physical priors, combine atomic- level simulations with global projection to accurately compute electron scattering during the imaging process 63
50
45
30
32
15]. Traditional simulation methods such as InsilicoTEM (3) model
the interaction between electrons by taking specimens as multi-slices to improve the algorithm's ' performance. However, they typically require expensive computational resources and may require complex adjustments of parameters to achieve ideal results. Our work also includes an efficient physics-based simulation module to present the structural information of particles. Additionally, we generate realistic noises via novel unpaired noise translation technique.
Unpaired Image-to-image Translation. Deep generative models including GAN-based meth- ods [68, 61 23 29 17 and diffusion models [46] 55 are widely used in unpaired image-to-image
translation tasks. CycleGAN (68] introduces generative adversarial networks calculate cycle
consistency losses, allowing for training on unpaired data (20) 26 38 22 TraVel 2). Distance-
GAN [3]. and GeGAN 12 achieve one-way translation while avoiding traditional cycle consistency.
57 40) have been recently introduced to unpaired image-to-image translation. But they often demonstrate results in low resolution and they often rely on a large-scale training dataset. Recently, Contrastive Unpaired Translation (CUT) 337 introduces a new generative framework via contrastive learning [2] 52 65 to propose a more efficient training framework. However, existing methods do not combine the physical process as additional constraints. In contrast, our method includes physical simulation during authentic cryo-EM micrograph generation.
Diffusion models 46 34 55 64
3 Physics-informed Generative Cryo-EM
We propose CryoGEM, the first method to combine a physics-based simulation process with a novel contrastive noise generation technique (Figure 2) Our approach begins with preparing a virtual specimen containing numerous uniformly distributed target protein structures (Section 3.1). We then emulate the cryo-EM imaging process to introduce physical constraints (Section 3.2 To generate authentic cryo-EM noise, we use novel unpaired noise translation technique via contrastive learning guided by a crttile-bbckground mask and detail the final objective during training. (Section
3.1 Virtual Specimen Preparation
To simulate the cryo-EM imaging process, we first prepare the virtual specimen S(x, y, 2), a large 3D density volume that contains multiple copies of the target molecule's coarse result with randomly gen- erated locations, orientations, and conformations. To obtain the coarse result, we employ cryoSPARC 41 for ab-initio reconstruction, followed by cryoDRGN (66) for continuous heterogeneous recon- struction to obtain neural volume
Vip, w)
R,
where p (r,y, x))² =)T E 3 represents the spatial coordinates, E R denotes a high-dimensional conformational embedding. This neural volume can generate a density volume given a learned conformational embedding following CryoDRGN (66]. Notably, such a coarse result can be easily
3

---

500 o*)
Projection
Synthedic Mizrogrphh
los gradient
Binanzation
Guido
Mu Mutioiiaaioon Canvolation
Sumpling
@ PSF
Extolance
Physical Simulation
Parkile-Backgruund
Mask-Gulder Patch-wise Contrastive
Figure 2: Pipeline of CryoGEM. We begin by creating virtual specimen containing various initial reconstruction results. We then simulate the imaging process of cryo-EM, incorporating physical priors such ice gradient and point spread function (PSF) generate physical simulation. By adding simple Gaussian noise to the physically simulated results, we introduce randomness within a contrastive learning framework. To enhance training efficiency and performance, background mask as guide for patch sampling. The sampled positive and negative instances are then encoded into multi-scale features for contrastive learning. Additionally, we introduce an adversarial loss to ensure realistic cryo-EM image synthesis.
the particle
resolved, but the final result requires further iterative optimizations of selected particles, estimated particle poses, 3D templates, etc.
During the specimen preparation, we sequentially add N particles Vp, w) into an empty virtual specimen. During every addition, we randomly sample orientation matrices R; e 333x3 from SO(3) space, random conformations w, from the learned conformational space, as well as translations t, E R3 from unoccupied areas of the virtual specimen. Both sampled orientations and locations can be further used as ground-truie annotations. Thus, the virtual specimen can expressed as:
N
S(x,8,2)
v,
Σ V(Rp ts:
where the number of particles N ~ Nuu v.), with the mean μo and standard deviation ON derived from the actual distribution of particle count real micrographs. We ensure that the minimum N and maximum N Nmax values are within two standard deviations from the mean.
3.2 Emulating the Imaging Process
We emulate the physical imaging process of cryo-EM including electron interaction, ice gradient, and cryo-EM optical aberration based on virtual specimen. For simplicity, we consider the complex electron-specimen interaction an orthogonal projection of the specimen by applying the weak phase object approximation (53) (WPOA). To obtain the true ice gradients, we estimate them on real micrographs by IceBreaker (36], which can generate weight map of ice, (( y), whose every pixel represents the attenuation ratio compared to the maximum value of intensity. Also, we model the optical aberrations
point spread function (PSF) g, which is the Fourier transform of the contrast transfer function (CTF), by off-the-shelf software CTFFIND4 42 physics-bane based result Tph (Ir, y) can be expressed as:
To sum up, the 2D
Jpny (r. y) = * (((rv)) S(x, s(a, 2,2)
During the simulation process, we randomly select pair of weight maps and PSF estimated from real micrographs.
4 Contrastive Noise Generation
Real cryo-EM micrographs contain high-level noises related to specimen-electron interaction and detector [50]. Thus, accurate noise modeling is essential for authentic synthetic cryo-EM micrograph

---

Real Image & Queries
Background-Partide Similarity Maps
CUT's Similarity Maps
Figure 3: Visualization of the learned similarity. Given query patches (red for particle and blue for background) on the input real micrograph, we visualize the learned similarity maps of our and CUT's encoders Gene by calculating exp(Gene(() Gene (eene(" )/T, where denotes the query and v denotes the patches of real micrograph. The results imply that our encoder can recognize particles and backgrounds in real cryo-EM micrographs. However, CUT fails to learn that without the guidance of particle-backgroum maps during training.
generation. We model the noise generation as an unpaired image-to-image translation task. The input, 4phy X, learns the actual noise distribution from real cryo-EM images Leal Y, generate synthetic cryo-EM images Lyn ~ Y. The generative model includes a discriminator D, and a generator G = Geec Genes where Gene is encoder and Geec is decoder.
Non-deterministic noise translation. Existing contrastive learning methods such as CUT have shown significant promise for efficient training and generating high-quality natural images. Nonetheless, they do not account for generating random noise in cryo-EM, i.e., they can only generate synthetic micrographs in deterministic way, against the nature of the random noise generation process, leading to an unstable training process and degraded performance. To address this, we introduce random process by adding zero-mean Gaussian random noise G N o²) into the physical simulations, where o² = var(4phy)/ //SR (SNR equals to 0.1 in our experiments). We
introduce an intermediate result linter phy + G ~ X, where X = X + +((,00))) Thus, the synthetic image can be defined as:
(4) This simple strategy significantly improves our noise generation quality by constructing mapping of single physical simulation result to infinite synthetic noisy results with varied noise patterns.
Lsyn = G(Imnter)
Mask-Guided sampling scheme. A fundamental assumption of patch-wise contrastive learning is that the randomly chosen negatively sampled patches are semantically different from the positive samples [8) But in single cryo-EM micrograph, hundreds of particles against an almost pure yet noisy background, which means that randomly chosen negative and positive patches are likely semantically similar. Taking advantage of our highly controllable physical simulation process, we introduce a mask mask-guudded sampling scheme by generating a binary particle- indicates particle. Guided by this mask, our selection process can choose positive and negative samples from locations with contrasting mask labels. This significantly improves the encoder's ''s performance, as shown in Figure 3 our encoder can generate accurate similarity maps given particles or backgrounds as references on real images, but CUT's encoder fails to recognize them.
densely distributed
background mask paired with Iphy- denoted as (x,, y). where M M y)
Mutual information extraction. Guided by in 4phy and Lyyn as the positive samples and queries on particle region, respectively, We then choose K patches in Lphy as negative samples on the background region. The encoder Gene maps these to normalized vectors v = { w = { 1 and v = {o/ | respectively. Our objective maximize the likelihood of selecting positive sample by minimizing the cross-entropy loss:
mask, we select Q paired patches
q=1
J0=1+
Q
exp(e. ot /T)
f(v. at v
Σ
log
(5)
exp(1,
vut /T) + Σk exp(n, : /T)
where 7 is a temperature factor that scales the distance between the query and samples.
Mask-guided patch-wise contrastive learning. We leverages multiple intermediate layers of Gene to extract multi-scale features from input patches. Notably, we align the receptive field with particle size by selecting the Gone's 's first L layers. Feature map from each layer is then passed through
5

---

Smibee
Particle image generation with varied orientations
Integgrin
Particle image generation with varied conformations
Pheyas
Particle image gemeration with varied defocus values
Figure Result gallery. The first row showcases the diverse synthetic contents generated by CryoGEM (from left to right for Proteasome, Ribosome, Integrin, PhageMS2, and HumanBAF). The second, third, and fourth rows demonstrate CryoGEM's ability to control the particle's pose, conformation, and defocus during the image generation. In every row, the leftmost column is the clean particle image, and the controlled variable is smoothly adjusted from left to right.
compact two-layer MLP networks Hi (1 E (1,, to generate feature z; = Hi (G (Hinter)) where G is the 1-th layer. We define those on particles as p P. and those on the background as b E B. The feature of (-th layer at particle position is denoted as zf, and the set represented as pep- Similarly, for background positions, we denote the 1-th layer feature as 2/, with the set represented as 2 {xf [=f]bEB: where 2; = Hi(G (4sy)). We propose a novel mask-guided Noise Contrastive Estimation (NCE) loss for efficient contrastive learning in cryo-EM:
zi {z()) ppp-
be Features encoded from the output 4ynn are represented as
LMaRNNCE H, X) = EE
Σ I=1 PPEP
Σ 2f)) + Σ (2f
(6)
ama X
BEB
By minimizing LMaRNCE, we effectively differentiate particle and background features (Figure 3 Final objective. We introduce an adversarial loss function, LGAN, to encourage the network to produce more realistic simulated cryo-EM images, Isyn as follows:
LGAN(G: (G, D. Y) E flog D(Trea)) E E [log(1 D ([mmer))))
Combining this with the contrastive learning loss LMMANCE, the overall training loss function is:
LGAN (G, D, x) + XLMmANCE H, H, X),
we set the hyper-parameter À to 10.0 in all our experiments.
5 Experiments
Datasets.
We evaluate CryoGEM across five challenging cryo-EM datasets. Each includes expert- curated particle-picking annotations for high-resolution reconstruction of target molecules. Note that the particle annotations in the dataset are solely used for validation of downstream tasks and

---

Real
Ours
Gaussian
Poísson
Poi-Gau
CycleGAN
CUT
CycleDiffusion
Ribosone Protessone
Integrin
Phage
MS2
Huma
BAF
Figure 5: Qualitative comparison results. Our approach achieves the most authentic noise generation across all datasets. The traditional noise models succeed preserving the structural information while lacking realistic noise patterns. CycleGAN, CUT, and CycleDiffusion introduce severe artifacts on generated results.
Table 1: Quantitative comparison of visual quality. Our approach consistently achieves the best performance FID metric.
Dotaset
Protcasome
Ribosome
Iniegrin
PhageMS2
HomannBAF
Avg.
Goissinn
80.87
177.01
174.4
162.81
46.48
130.22
Poisson
112.40
86.88
173.94
343.16
44.17
152.11
173.64
311.50
CycleGAN
46.61
44.23
49.96
88.65
74.76
60.84
CyclePiflusinn
470.37
173.62
396.83
468.83
$77.46
415.42
Ours
42.96
6.34
42.46
63.11
34.30
37.91
not utilized during the training phase of our CryoGEM model. 1) T20S Proteasome (EMPIAR- 10025) [6]' micrographs exhibit a high density of particles, Icading to occlusions. posing significant challenges in particle picking. Also, the 3D structure is D7 symmetric, which brings ambiguities in the pose estimation task. 2) 80S Ribosome (EMPIAR- 10028) (54) has a complex 3D structure, making difficult to estimate the poses of particle images. 3) Asymmetric a 38 Integrin (EMPIAR- 10345) [5] contains molecules with varied conformations, requiring a heterogeneous reconstruction. 's micrographs contain enormous spherically-shapee virus particles, suitable for testing the generalizability of the particle picking model. 5) Endogenous HumanBAF complex CAMPIAR-10599
4) PhageMM2 (EMPIAR- 10075) 25 II
333 [1] has significant ice gradient artifacts. We kindly
refer to Appendix for more detailed dataset description and illustration.
Baselines. We evaluate the performance of CryoGEM compared to several traditional noise baselines and deep generative models. We use Poisson noise to simulate the high-level detector shot noises in cryo-EM images, Gaussian noise for general simplified noise modeling in cryo-EM, and Poisson- Gaussian mixed noise (Pos-Gau) as traditional baselines. We choose CycleGAN, CUT, and recent CycleDiffusion as deep generative baselines. In Appendix C detail their specific settings.
Implementation details. All experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU. As lightweight model, CryoGEM trains 100 epochs on single dataset within an hour using less than GB of memory. The training dataset (all held out for the evaluation) is comprised of 100 real images alongside 100 physics-based simulated results. We enhance the dataset through

---

Topaz
Ours
Poi-Gau
CycleGAN
CUT
CycleDiffusion
Ribosone
Interri
PhageMS2
Figure 6: Qualitative comparison results of particle picking. The blue circles indicate matches with manual picking results, while the red circles represent misses or excess picks by the model.
Table 2: Quantitative comparison of particle picking. Our approach consistently achieves the best in AUPRC and Res(A) metrics.
Mosic
AUPRCI
RedAI
Dhtrect
Prosstomme
Riboneme
Intera
PhigaMS
Asg
Kihosame
Imiignn
PlegAMS:
Alp:
11433
01335
7.57
CydleGAN
0.200
0.308
0.459
CUT
0.244
0.335
0.513
0.421
2.77
CycleDiftatoo
0.565
0.305
0.235
0.669
0.99
0.329
290
0.35
Topaz
0.00
0.79
0.336
0.59
0.45
0.466
311
3.54
6.5
10.67
9.59
Cure
0.498
0.997
0.466
8.915
0.562
0.04
168
3.25
3354
7.16
7.34
827
data augmentation, specifically by rotating the images by 90, 180, and 270 degrees. Please see Appendix B for more details. We choose PatchGAN [8] as our discriminator and UNet implemented by [37 as our generator. We fix the image resolution to 1024 1024 during training. Guided by the mask, we evenly sample 256 queries on particles and 256 on the background, ensuring that their corresponding negative samples located where the labels are opposite.
5.1 Visual Quality
In Figure we demonstrate CryoGEM's ability generate diverse content through multi-level controls, including adjustments to the ice gradient and defocus values of micrographs, as well as the particles' positions, orientations, and conformations. We then evaluate CryoGEM's s visual quality compared to baselines. As illustrated in Figure our approach consistently achieves the best qualitative results across all datasets terms of overall noise patterns and the preservation of particle structural information. Traditional methods fail to mimic authentic noise distributions. CycleGAN struggles with convergence due the breakdown of bijection assumption. CUT can achieve better visual quality but its performance is still limited due to its random sampling scheme. CycleDiffusion cannot learn the the authentic noise patterns. As shown Table we also employed Frechet Inception Distance (FID) which is widely used in generative tasks to measure the similarity between generated and real data. Our results significantly outperform existing baseline methods in all five datasets. For additional conditional generated and zero-shot results, please refer to Figure 10 in the appendix.
5.2 Particle Picking
To evaluate how CryoGEM can enhance downstream particle picking task, we finetune the popular particle picking model Topaz 4 using synthetic annotated datasets from different baselines. Our

---

Rendbed Stantarr
Coass
PhapeMis
CmuGEM
Tlai
CmoG231
Tead
Figure 7: Evaluation of CryoGEM's picked results on 3D Reconstruction. We present both the initial coarse inputs from CryoGEM and the final refined structures, with particles selected using CryoGEM's refined particle picking model. Additionally, we provide a quantitative comparison of the reconstruction results across various baselines. The resolution of each reconstruction is evaluated using the Fourier Shell Correlation (FSC) criterion (threshold-O.1 143 for real datasets).
Table 3: Quantitative comparison of pose estimation. Our approach achieves the best performance in Res(px) and (rad metrics.
Motne
KeipaL
RoLiralr
Proteasceme
PtageMS2
HummBAF
Protzssome
Ribosomne
PhageM52
HumiaBAF
1.20
CycleGAN
2.66
CyclDDiflasinn
3.61
CysETRRE
1632
13877
17:23
6.88
12.18
1.55
0.51
0.3
0.73
1.33
108
Ouis
133
4.27
4.85
5.54
0.50
4.29
0.31
0.3
038
1133
1.42
009
approach outperforms the original Topaz and other baseline methods in terms of the quality of visual picking results. As illustrated in Figure the fine-tuned Topaz model is employed to select particles across five datasets. The original Topaz often mistakenly picks false particles from contaminants, particle aggregations, and ice patches. Our fine-tuned Topaz significantly improves accuracy by reducing the incidence of false positives and redundant particle picks. presenss the quantitative particle picking results, where we utilize the Area Under the Precision-Recee Curve (AUPRC) metric that widely used as particle picking metric 55. For the calculation of the AUPRC score, we retain the original particle-picking results for comparison against the ground truth labels, without any thresholding. The quantitative comparison shows that the finetuned Topaz using our data achieves the best performance across all datasets. We also show that our final reconstruction resolution (Res(A)) is significantly improved using cryoSPARC [41)'s default reconstruction pipeline. Finally, we present the coarse volumes used for CryoGEM training, the final resolved structures from CryoGEM picked particles, and the FSC curves from different baselines in Figure
5.3 Pose Estimation
In the cryo-EM reconstruction pipeline, the accuracy of pose estimation is crucial for the resolution of the final reconstruction. Existing cryo-EM ab-initio methods, such as CryoFIRE (88. leverage self-supervised learning framework predict particle poses and reconstruct 3D volume from images simultancously. Given synthetic datasets generated from different baselines, we directly supervise its pose estimation module by minimizing the quadratic erтor between the matrix entries in the prediction and ground truth. The details of direct supervision and the evaluation metrics as well as the visualization of improved structures (Figure [4] can found in Appendix C.3 We evaluate the pose estimation module on real datasets to obtain estimated poses as ground truths. We use traditional reconstruction algorithm, filter back- projeerojettion (FBP), to obtain final reconstruction results. As demonstrated in Table the pre-trained pose estimation module using our data consistently achieves the best performance terms of pose accuracy (rotation error in radian) and reconstruction resolution (in pixel).

---

Purnecalion
Panteie Pecting
Fout PaHieRInE
Renipesl
Fand
Finth Layer
21
0.777
5756
6.46
Second Chaznel Noise
663
0.012
9.76
4.29
0.34
SN8-201
22
0.313
3.80
7:11
1.50
RD
SNR-L0 10
17:00
0:349
3.40
4.11
0.25
Win Garreian Neine
50.24
0.735
3.31
4.27
0.55
Wit Paracle
6.55
0.785
341
6.3
0350
w/o Backkgonnd
0.779
3.39
6:3
0.34
w/o Gurds
12
Number training
Figure 8: Quantitative evaluation of ablation study. Our complete model consistently achieves the best or second performance across all metrics, as shown in the left table. It demonstrates stable and efficient training compared to CycleGAN, CUT, and the ablation without Mask Guide (w/o Mask Guide), as illustrated in the right figure.
5.4 Ablation Study
We validate the effectiveness of our several key designs on the Ribosome dataset.
Physical priors
We denote the variation of our model without physical priors as w/o Physical Priors. Results imply that the physical priors narrow the domain gap between simulated results and real micrographs.
Introducing randomness. We introduce randomness by different kinds of noises, such as introducing feature-level noise in the encoder's third layer (Third Layer Noise) and fifth layer (Fifth Layer Noise). Additionally, we treat random noise as a second input channel, which may potentially preserve the content of physical simulation better (Second Channel Noise). Also, we demonstrate the impact of noise scaling relative to the signal with SNR=0.01 and SNR=1.0. Additionally, we remove the Gaussian noise (w/o Gaussian Noise) validate the effectiveness of this randomness.
Sampling scheme. We use particle background mask to guide sampling in the contrastive learning process. For accurate particle position and shape control, the generator must preserve the input image's content. we sample positives only in the background, the mutual information between particles can only be optimized indirectly. leading to inaccurate particle positions and shapes (w/o Particle). Conversely, sampling positives only in particles results in less realistic noise patterns (w/o Background). We also present the results without the mask -guiedd sampling scheme (w/o Mask Guide), which includes only the physics-basee bassed module and input domain randomness.
Training efficiency ablation. In the right of Figure X we validate that our method can generate realistic cryo-EM images with fewer samples compared baselines in terms of FID. Notably, some high FID outliers indicate that removing the mask-guided sampling scheme or ignoring the physics- module can decrease training stability.
6 Conclusion
Limitations. As the first trial to achieve authentic cryo-EM image generation with novel combina- tion of physics-based simulation and unpaired noise translation via contrastive learning, CryoGEM does have room for further improvements. First, our physies-based simulation relies on a coarse result as an input, which may be hard to obtain in some challenging cases, e.g., when the target molecule is very small or dynamic. The latest AlphaFold [ can help predict the structure directly. Furthermore, we sacrifice the generalization capability for a lightweight training framework. In the future, we will train generalized version of CryoGEM to unlock more real applications.
Conclusion. We have presented the first generative approach in cryo-EM -Mo image synthesis, CryoGEM, that marries physics-based bassed simulation with a contrastive noise generation, to enhance downstream deep models, finally improving cryo-EM reconstruction results. Extensive experiments have shown that CryoGEM's generated dataset with ground-truth annotations can effectively improve particle picking and pose estimation models, eventually improving reconstruction results. We believe that CryoGEM serves as a critical step for high-fidelity cryo-EM data generation using deep generative models, with diverse applications of structure discovery in in cryo-EM.
10

---

7 Acknowledgement
This work supported by ShanghaiTech University's HPC Platform. We would like to thank the Cellverse team for their valuable discussions. We also extend our gratitude Yutong Liu for contributing to the design Figure I.
References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
[2] Matthew Amodio and Smita Krishnaswamy. Travelgan: Image-to-inage translation by transformation vector learning. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 1983-8992. 2019.
[3] Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. Advances neural information processing systems, 30, 2017.
[4] Tristan Bepler, Andrew Morin, Micah Rapp. Julia Brasch, Lawrence Shapiro, Alex J. Noble, and Bonnic Berger. Positive-unllabled convolutional neural networks for particle picking in cryo-clectron micrographs. Nature methods, 16: 1::3 1160, 2018.
[5] Melody G. Campbell, Anthony Cormier. Saburo Ito, Robert Seed, and Stephen L. Nishimura, Cryo-em reveals mediated tgf-3 activation without release from latent tgf-B. Cell. 180:490-501. 501.-001.16.....1 2020.
[6] Melody G. Campbell, David Veesler, Anchi Cheng, Clinton S. Potter, and Bridget Carragher. 2.8 à resolution reconstruction of thermoplasma acidophilum 20s proteasome using cryo-electron microscopy. eLife, 4, 2015.
[7] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.0222 2021.
[8] Ching Yao Chuang, Joshua Robinson, een- Lin, Antonio Torralba, and Stefanic Jegelka, Debiased contrastive learning. Advances in newral information processing systems, 33:8765-8775, 2020.
[9] U Demir and Unal. Patch-based image inpainting with generative adversarial networks. arxiv 2018. arXiv preprint arXiv:1803.074 07422
[10] Ashwin Dhakal, Rajan Gyawali, Liguo Wang, and Jianlin Cheng. Cryotransfomeer: A transformer model for picking protein particles from cryo-em micrographs. bioRxiv, 2023.
[11] Ashwin Dhakal, Rajan Gyawali, Liguo Wang, and Jianlin Cheng. A large expert-curated cryo-em image dataset for machine learning protein particle picking. Scientific Data, 10(1):392, 2023.
[12] Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. Geometry-consisten generative adversarial networks for one-sided unsupervised domain mapping. In Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recognition, pages 2427-2436. 2019.
[13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, vol- ume Curran Associates, Inc., 2014.
[14] Rajan Gyawali, Ashwin Dhakal, Liguo Wang, and Jianlin Cheng. Accurate cryo-em protein particle picking by integrating the foundational ai image segmentation model specialized u-net. bioRaiv. 2023.
[15] Benjamin Himes and Nikolaus Grigorieff. Cryo- Cryo-TEM simulations of amorphous iddiation-sensitive samples using multislice wave propagation. IUCrJ, 8(6):943 953, Nov 2021.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840 6851. Curran Associates, Inc., 2020.
[17] Xun Huang, Ming Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to to-iage translation. In Proceedings of the European conference on computer vision (ECCV), pages 172-189, 2018.
11

---

[18] Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros, Image-to-imagg translation with conditional adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[19] Andrii Iudin, Paul K Korir, Sriram Somasundharam, Simone Weyand, Cesare Cattavitello, Neli Fonseca, Osman Salih, Gerard Kleywegt, and Ardan Patwardhan. Empiar: the electron microscopy public image archive. Nucleic Acids Research, 51(D1))D1503-D1511 2023.
[20] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. ArXiv, abs/1603.08 155, 2016.
[21] Chanyong Jung, Gihyun Kwon, and Jong- Chgg Ye. Exploring patch-wise semantic relation for contrastive learning in image-lo-image translation tasks. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 18239-18248, 2022.
[22] Junho Kim, Minjae Kim. Hyeonwoo Kang. and Kwanghee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. ArXiv, abs/1907.10830, 2019.
[23] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross cross-domain relations with generative adversarial networks. In International conference on machine learning, pages 1857-1865. PMI.R, 2017.
[24] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114 2013.
[25] Roman Koning, Josue Komez-Blance Inara Akopjana, Javier Vargas, Andris Kazaks, Kaspars Tars, José María Carazo, and Abraham J Koster. Asymmetric cryo-em reconstruction of phage ms2 reveals genome structure in situ. Nature communicutions, 7(1):12524, 2016.
[26] Hsin- Ying Lee, Hung- Yu Tseng, Jia-Bin Huang. Maneesh Kumar Singh, and Ming Yang. Diverse image-to-image translation via disentangled representations. ArXiv, awss/s00.000000000 2018.
[27] Axel Levy, Frédéric Poitevin, Julien N. P. Martel, Youssef S. G. Nashed, Ariana Pock, Nina Miolane, Daniel Ratner, Mike Dunne, and Gordon Wetzstein. Cryoai: Amortized inference of poses for ab initio reconstruction of 3d molecular volumes from real cryo-em images. Computer vision ECCV European Conference on Computer Vision proceedings. European Conference on Computer Vision, 13681:540-557, 2022.
[28] Axel Levy, Gordon Wetzstein, Julien N. Martel, Frédéric Poitevin, and Ellen D. Zhong. Amortized inference for heterogeneous reconstruction in cryo-em. Advances in neural information processing systems. 35: 35:13038-1 10003--13440 13049, 2022.
[29] Ming- Yu Liu, Thomas Breuel. and Jan Kautz. Unsupervised image-to-imane translation networks, Ad- vances neural information processing systems, 30, 2017.
[30] Lobato and Van Dyck. Multem: A new multislice program to perform accurate and fast electron diffraction and imaging simulations using graphics processing units with cuda. Utramicroscopy, 156:9-17, 2015.
[31] Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-IHai Zhang, Mare Habermann, Christian Theobalt, and Wenping Wang. Wonder3d: Single image to 3d using cross- doms-n diffusion, 2023.
[32] Jacob Madsen and Toma Susi. abtem: Ab initio transmission electron microscopy image simulation. Microscopy and Microanaysiss, 26($2):448-455 2020.
[33] Nazar Mashtalir, Hiroshi Suzuki, Daniel P Farrell, Akshay Sankar, Jie Luo, Martin Filipovski, Andrew D' Avino, Roodolph St Pierre, Alfredo M Valnciia, Takashi Onikubo, et al. A structural model of the endogenous human haf complex informs disease mechanisms. Cell, 1 183(3)8002-817, 2020.
[34] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0100 2021.
[35] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising difffusion probabilistic models. In International conference machine learning, pages 8162-8171 8171. PMLR, 2021.
[36] Mateusz Olek, Kevin Cowtan, Donovan Webb, Yuriy Chaban, and Peijun Zhang. Icebreaker: Software for high-resolution single particle cryo-em with non-uniform ice. Structure, 30(4):522-53 531, 2022.
12

---

[37] Taesung Park. Alexei A Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for unpaired image-to-image translation. In Computer Vision-ECCV 2020: 16th European Conference, Glaxgow, UK. August 23-28, 2020, Proceedings, Part IX 16, pages 319-345. Springer, 2020.
[38] Micha Pfeiffer, Isabel Funke, Maria Ruxandra Robu, Sehastian Bodenstedt, Leon Strenger, Sandy Engel- hardt, Tohias Ross, Matthew J. Clarkson, Kurinchi Gurusamy, Brian R. Davidson, Lena Maier-Hein, Carina Riediger, Thilo Welsch, Jürgen Weitz, and Stefanie Speidel. Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation. In International Conference on Medical Image Computing and Compuder-Asiiseed Intervention, 2019.
[39] Puntawat Ponglertnapakorn, Nontawat Tritrong. and Supasorn Suwajanakom. Difareli: Diffusion face relighting. 2023.
[40] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasor Suwajanakom. Diffusion autoencoders: Toward meaningful and decodable representation. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10609-10619, 2021.
[41] Ali Punjani, John L Rubinstein, David J Fleet, and Marcus A Brubaker. cryospare: algorithms for rapid unsupervised cryo-em structure determination, Nature methods, 14(3):290-29,, 2017.
[42] Alexis Rohou and Nikolaus Grigorieff. Ctffind4: Fast and accurate defocus estimation from electron micrographs. Journal of structural biology, 192(2):216-211, 2015.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10669 10695, 2022.
[44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
intervention MICCAI 2015; 18th
[45] Hans Rullgård. L-G Öfverstedt, Sergey Masich, Bertil Daneholt, and Ozan Öktem. Simulation of trans- mission electron microscope images of biological specimens. Journal of microxcopy, 243(3):23342555 2011.
[46] Hiroshi Sasaki, Chris G Willeocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358, 2021.
[47] Shayan Shekarforoush, David B Lindell, David I Fleet, and Marcus A Brubaker. Residual multiplicative filter networks for multiscale reconstruction. arXiv preprint arXiv:2206.00746 2022.
[48] Dimitry Tegunov and Patrick Cramer. Real-time cryo-em data pre-processing with warp. Nature methods. 16: 1411 1152, 2018
[49] Brian Teixeira, Vivek Singh, Terrence Chen, Kai Ma, Birgi Tamersoy, Yifan Wu, Flena Balashova, and Dorin Comaniciu. Generating synthetic x-ray images of a person from the surface geometry. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018
[50] Miloš Vulović, Raimond BG Ravelli, Lucas J van Vliet, Abraham J Koster, Ivan Lazič, Uwe Lücken, Hans Rullgård, Ozan Öktem, and Bernd Rieger. Image formation modeling in cryo-electron mieroscopy. Journal of structural biology, 183(1):19 32, 2013.
[51] Thorsten Wagner, Felipe Merino, Markus Stabrin, Toshio Moriya, Claudia Antoni, Amir Apelbaum, Philine Hagel, Oleg Sitsel, Tobias Raisch, Daniel Prumbaum, Dennis Quentin, Daniel Roderer, Sebastian Tacke, Birte Sieholds, Evelyn Schubert, Tanvir R. Shaikh, Pascal Lill, Christos Gatsogiannis, and Stefan Raunser. Sphire-cryolo is a fast and accurate fully automated partiele picker for cryo-em. Communications Biology, 2, 2019.
[52] Weilun Wang, Wen gang Zhou, Jianmin Bao, Dong Chen, and Houqiang Li. Instance-wise hard negative example generation for contrastive learning in unpaired image-to-image translation. 2021 IEEE/CVF International Conference Computer Vision (ICCV), pages 18000-14000 2021.
[53] David B Williams, C Barry Carter, David B Williams, and C Barry Carter. The transmission electron microscope. Springer, 1996.
[54] Wilson W. Wong, Xiao chen Bai, Alan Brown, Israel S. Fernández, Eric Hanssen, Melanie M Condron, Yan hong Tan, Jake Baum, and Sjors H. W. Scheres. Cryo-em structure of the plasmodium falciparum 80s ribosome bound the anti anti-protozoan drug emetine. eLife, 2014.
13

---

[55] Chen Henry Wu and Fernando De la Tore. Unifying diffusion models" latent space. with applications to cyclediffusion and guidance. arXiv preprint urXiv:2210.05559, 2022.
[56] Chentianye Xu, Xueying Zhan, and Min Xu, Cryomae: Few-shot cryo-em particle picking with masked autoencoders. arXiv preprint arXiv:2404. 10178, 2024.
[57] Sihan Xu, Ziqiao Ma, Yidong Huang, Honglak Lee, and Joyce Chai. Cyclenet: Rethinking cycle consistency in text-guided diffusion for image manipulation. ArXiv, abs/2310.13165 2023.
[58] Heran Yang. Jian Sun, Aaron Carass, Can Zhao, Junghoon Lee, Jerry L.. Prince, and Zonghen Xu. Unsupervised mr-to-ct synthesis using structure-constraiieed cyclegan. IEEE Transactions Medical Imaging, 39(1))4249 1261. 2020.
[59] Hangping Yao, Yutong Song, Yong Chen, Nanping Wu, Jialu Xu, Chujie Sun, Jiaxing Zhang, Tianhao Weng, Zheyuan Zhang, Zhigang Wu, al. Molecular architecture of the sars-cov-2 virus. Cell, 183(3):730-735 2020.
[60] Lin Yao, Ruihan Xu, Zhifeng Gao, Guolin Ke, and Yuhang Wang. Boosted ab initio eryo-em 3d recon- struction with ace-em. ArXiv, abs/2302.06091, 2023.
[61] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision, pages 2849-2857, 2017.
[62] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1847-3847, 2023.
[63] Yue Zhang, R Tammaro, Peter J Peters, and RBG Ravelli. Could egg white lysozyme be solved by single particle cryo-em? Jourmal of Chemical Information and Modeling, 60(5):2605 26113, 2020.
[64] Min Zhao, Fan Bao, Chonexuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623, 2022.
[65] Chuanxia Zheng. Tat-Jen Cham, and Jianfei Cai. The spatially-correlan correlative loss for various image translation tasks, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16402- 16412, 2021.
[66] Ellen Zhong. Tristan Bepler, Bonnie Berger, and Joseph H Davis. Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks. Nature methods, 18(2):166-18) 2021.
[67] Ellen Zhong, Adam Lerer, Joseph H Davis, and Bonnie Berger. Cryodrgn2: Ab initio neural reconstruc- tion of 3d protein structures from real cryo-em images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4066-4075, 2021.
[68] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to to to-image translation using conessconsistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[69] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, In IEEE International Conference on Computer Vision (ICCV), 2017.
[70] Jasenko Zivanov, Takanori Nakane, Bjöm O Forsberg, Dari Kimanius, Wim J. H. Hagen, Erik Lindahl, and Sjors H. W. Scheres. Relion-3: new tools for automated high-resolution cryo-em structure determination. bioRxiv, 2018.
14

---

Supplementary Material-
CryoGEM: Physics-Informed Generative
Cryo-Electron Microscopy
A Additional results.
We include supplementary results in Figure These serve as an extension to Figure from our main paper. Here, we demonstrate CryoGEM's ability to generate synthetic micrographs with control over the particle mask, defocus value, and ice gradient. Additionally, we explore CryoGEM's zero-shot capability, showcasing its proficiency in generating authentic noise patterns on unseen datasets. These results show CryoGEM's potential in creating an extensive synthetic dataset in a wide range of particle and noise pattern combinations.
B Dataset Details
We evaluate CryoGEM across five diverse cryo-EM datasets. We source the Proteasome, Ribosome, and Integrin datasets from the Electron Microscopy Public Image Archive (EMPIAR) [1) a global public resource offering raw cryo-EM mierographs and selected particle stacks for constructing high-resolution 3D molecular maps. The PhageMS2 and HumanBAF datasets were downloaded from the cryoPPP dataset with provided manual particle picking annotations.
6 (EMPAIR-10025) contains 196 real micrographs and 49,954 manually filtered particles to achieve high-resolution result at 2.8 Á. The T20S proteasome forms 700kDa complex that contains 14 a-helices and 14 3-sheets organized by D7 symmetry. The captured micrographs comprise particles with high symmetry, random rotational orientations, and mutual occlusions.
Proteasome Thermoplasma acidophilum 20S proteasome
Ribosome Human 80S ribosome [54
(EMPIAR- 10028) contains 1,081 1 real micrographs and 105,417 manually filtered particles to achieve a high-resolution result at 3.6 Å. It is large assembly. incorporating numerous RNA chains and proteins.
(EMPAIR- 10345) contains 1,644 real micrographs and 84,266 manually filtered particles to achieve a high-resolution result at 3.3 Å. The V,888 integrin is complex of the human a V (88 ectodomain with porcine I.-TGF-81, showing significant movement in its leg in the captured micrographs.
Integrin Asymmetric a V integrin 5
PSageMS2 Bacteriophage MS2 25 (EMPAIR-10075) contains 300 real micrographs and 12682 manually filtered particles [1] to achieve high-resolution result at 8.7 Å. PhageMS2 consists of 178 copies of the coat protein, single copy of the A-protein, and the RNA genome. Note that its size is significantly larger than others, making a challenge for the particle picking model.
HumanBAF Endogenous human BAF complex 33 (EMPAIR- 10590) contains 300 real micro- graphs and 62493 manually filtered particles |1I to achieve high-resolution result at 7.8 Á.
In Figure 10 we demonstrate datasets in a wide variety of noise patterns, particle scales, and types. These characteristics present significant challenges to the accuracy of particle picking and pose estimation, which are critical for automated high-resolution cryo-EM reconstruction. Notably, CryoGEM can generate similar visual effects such as the ring of defocus (Ribosome) and crystalline ice(Integrin). We analyze the distribution of particle counts and defocus values across micrographs for each dataset, calculating their mean and standard deviation. Based on these analyses, we generate two Gaussian distributions to simulate particle count and defocus value accurately CryoGEM, as illustrated in Figures 11 and 12 During the training phase, we select 100 micrographs from each real dataset as training samples for CryoGEMM, alongside other generative models like CUT 37 and CycleGAN (68). To ensure a fair comparison, exclude these micrographs from our evaluation.
15

---

Particle Mask
Our Resalts
(a) Micrograph generation with particle location control
Ice Gradient
Our Resallt
(b) Micrograph generation with ice gradient control
Zero-shot Results
(c) Zero-shot particle transfer from PhageMS2 to other datasets
(d) Zero-shot noise transfer from Ribosome to other datasets
Figure 9: (a) Location-controlled generation. CryoGEM uses a particle mask to guide the sampling process in contrastive noise modeling. This method allows for precise control over the placement of particles in synthetic micrographs. (b) Ice-gradient generation. CryoGEM calculates the ice gradient in real micrographs, as indicated in the lower right corner of each ice gradient image. This calculated ice gradient is then incorporated into the final images. (c, d) Zero-shot transfer between particle and noise. We show that CryoGEM datasets without the need for additional training. This demonstrates CryoGEM's ability for zero-shot transfer between different types of particles and noise.
generate convincing results previously unseen
16

---

Real
Our Results
Profssson
Ribosome
Integrin
PhageMs2
HummBAF
Figure 10: Additional visual comparison with real datasets. Our evaluation of five complex real datasets demonstrates that CryoGEM accurately reproduce noise patterns, preserve structural details, and replicate specific anomalies like crystalline ice in Integrin. For each dataset, we highlight the visual characteristics adeptly captured by CryoGEM with red circles arrows.
Dataset preprocessing. To improve the quality of the training datasets, we meticulously remove "dirty" micrographs, which are defined as micrographs captured outside the intended target area, those exhibiting substantial jittering artifacts, and those only containing background ice. To improve the training efficiency, downsample the images to 1024 x 1024 resolution. We enhance the contrast of the electron microscope images by adjusting the mean and standard deviation of the image intensity values 150 and 40 (with maximum value of 255), respectively. All these reepprccssing steps crucial for a stable and high-performant training process.
Initial 3D result. We reconstruct low-resolution 3D volume and simultaneously estimate the poses of input particle images using an ab-initio reconstruction of cryoSPARC (41) with its default settings. To generate an initial neural volume with conformational changes, we employ an advanced neural method, cryoDRGN (66 for continuous heterogeneous (dynamic) reconstruction, given the particle stack with estimated poses, We train the cryoDRGN 50 epochs by particles at a resolution aligned with the low-resolution 3D volume. Note that for homogeneous (static) datasets in our experiments, we directly utilize the 3D initial volume from cryoSPARC, as these datasets exhibit only negligible motions.
gradient estimation. of the main paper, we show that ice gradients result in intensity variations across micrographs, leading signal-to-noise ratio (SNR) disparities in local areas. Images from thicker specimen regions often exhibit lower SNR than those from thinner regions. To mimic this physical effect, we employ IceBreaker (36), established method that addresses the challenge
17

---

Proseemee to 345.33 2330
174
Pexssaee 1,5
imegrn 11 1800 4.h
Rugeeee James 43.wm3 48
0.
Wu Doand
Awargee beboc
Figure 11: Probability curve of particle count in cryo-EM datasets.
Figure 12: Probability curve of defocus value in cryo-EM datasets.
by treating the estimation of uneven ice thickness as a clustering problem. We have developed a GPU-accelerated version of IceBreaker for efficient estimation of average brightness per class. By dividing the outeome by its maximum value, we achieve a normalized weight map. Then, to replicate the gradual variations in ice thickness, apply broad Gaussian kernel.
Optical aberration estimation. In cryo-EM, a high-energy electron beam interacts with the specimen, goes through sophisticated optical imaging process, and is then captured by the detector as projection images. This optical modulation can be described by the Contrast Transfer Function (CTF), with the objective defocus being particularly crucial parameter. Experimentally, biologists can modify the defocus to enhance contrast or capture higher-frequency signals. We directly incorporate
CTF by applying it in our physical simulation process. Similar to CTFFIND4 42 expressed as:
CTF can be
CTF( A, gf C., Ap) =
2 sin (A, gl. C,, Ap)]
- coos[x (g). Af, C., Ap)],
(8)
where
x (A, AJ, = AS - - 11 A.p.
(9)
is relative phase contrast factor, X is the deepende- phase shift function with the inputs including electron wavelength X, the spatial frequency vector g, the objective defocus Af. the spherical aberration C₂ and the phase shift App. W, X, C.,, Aip are the Eryo-EM hardware parameters.
We utilize CTFFIND4 estimate the range of the objective defocus, Af. Subsequently, Resid- 47 is used to generate specific 2D CTF image for any given defocus value. As depicted in Figure 12 we estimate the defocus value distribution in real cryo-EM datasets. For the physical simulation phase, defocus values are randomly chosen from simplified Gaussian distributions, based on previously calculated means and standard deviations.
ualMFN
C Evaluation Details
C.1 Baseline Details
Traditional noise simulations. These baselines represent traditional simulation methods. The Poisson noise emulates the high-level noise caused by the stochastic nature of the detector under limited dosage conditions. Zero-mean Gaussian noise, as suggested by Vulovic al. (50). used to simulate complex noise distribution, including readout noise, dark current noise, shot noise, and structural noise. We also combine these to create Poi-Gau noise generation baseline. In all our experiments, set the SNR to 0.1 for images generated by these basclines.
18

---

We use UNet 44 as the generator and PatchGAN 9 as the discriminator. The complete CycleGAN objective comprises five components: synthetic-to-rea GAN loss, real-to- synthetic GAN loss, two-cycle consistency losses, and identity mapping loss. To maintain training stability, we assign their relative importance as 1, 1, 10, 10, and 0.5, respectively.
CycleGAN 669 18
We employ UNet as the generator and PatchGAN 9 as the discriminator. The complete objective of CUT includes three components: synthetic-to-real GAN loss, NCE loss on synthetic and generated images, and NCE loss on real and real-identity images. We follow the original paper, setting their relative importance to and
CUT (37..
CycleDiffusion (55, 35
CycleDiffusion necessitates the use of diffusion models for both the source and target domains. Consequently, we train two Denoising Diffusion Probabilistic Models (DDPMs) for each dataset: one for the synthetic image domain and one for the real image domain. Each model is trained on 100 images at resolution of 512x512, with batch size of 2, over 10,000 steps. For CycleDiffusion inference, we use the default settings. The associated code and models will be made available.
C.2 Particle Picking Details
Topaz [4] is a cryo-EM particle picking model that outputs the probability for each pixel belonging to particle. With threshold, it can filter out detection results. The training data for Topaz consists of complete micrographs and the central coordinates of particles within them. In practice, we select pre-trained rreseet__u __3 as baseline Topaz model without fine-tuning. When we fine-tune Topaz by synthetic datasets from different baselines, we use 10 annotated micrographs to fine-tune the pre- pre-trained Topaz for 20 epochs and then test on the real evaluation datasets. Note that Topaz has already been pre- pre-trained on the Proteasome and Ribosome datasets, which have been utilized in our experiments. Therefore, for these datasets, we infer the picking results the pre- trained resolution of 512 x 512. For other datasets, we evaluate it at a resolution of 1024 x 1024, aligning with the fine-tuned resolution.
Evaluation in terms of metric AUPRC. We retain the original particle picking results for compari- against the ground truth labels, without any thresholding. For fair comparison of particle picking results, we prioritize our particle picks based on their confidence scores. Specifically, we select the highest-ranking 50,000 picks from each method for further filtering. For PhageMS2, however, due to the limited number of particles present in the micrographs, we only select the top 10,000 picks. By applying different thresholds divided into n intervals by {Te)-=] we can filter out varying prediction results and calculate corresponding precision and recall metrics against the true labels. Finally, can compute the area under the precision-recall curve (AUPRC) as expressed by
Pr(k)(Re(k) Re(k - 1)),
(10)
where the precision Pr(k) represents the proportion of true positives in the prediction results with a probability greater than or equal to Tke the recall Re(k) means the proportion true positives in the prediction results with probability greater than or equal to т out of the total number of true positives.
Evaluation in terms of metric Res(Å). We further assess the particle picking results by evaluating the final reconstruction resolution achieved using cryoSPARC [41]. The filtered picked results are fed into cryoSPARC for the subsequent reconstruction process. For static proteins such as Proteasome, Ribosome, PhageMS2,, and BumanBAF,, the reconstruction process involves a 1-class ab initio reconstruction followed by homogeneous refinement. For dynamic proteins like Integrin, the reconstruction process includes a 5-class ab initio reconstruction, heterogeneous refinement, and homogeneous refinement for each class. We select the best final resolution value as the reconstruction resolution. Importantly, since no 2D classification or particle filtering operations are performed before reconstruction, the resolution of the resolved structures directly reflects the precision of particle picking and the quality of the particles. This method ensures that human bias is introduced.
19

---

Profeosome
Ribosome
Integrit
PhageMS2
HmmanBAF
Figure 13: Quantitative comparison of pose estimation using ESC curves. Our approach demon- strates superior performance the Res(px) metric. Notably, we present CryoFIRE, the sole neural- based reconstruction method that operates without ground truth particle poses.
Ours
Ours
Ous
Profeasome
Ribosume
Integrin
PhageMS2
HumanBAF
Figure 14: Qualitative evaluation of performance improvement on pre-trained CryoFIRE. The figure illustrates the detailed improvements in ab initio volumes. (Left: original CryoFIRE, Right: pre-trained CryoFIRE using CryoGEM's data).
C.3 Pose Estimation Details
To evaluate whether CryoGEM can help with pose estimation tasks, we utilize the state-of ooae--of-thaattt ab-initio reconstruction algorithm, cryoFIRE (28), from its official Github page, We employ synthetic datasets that provide direct supervision mechanism through pairs of synthetic particle images and their corresponding true poses. In all experiments, we generate 100,000 particle images per baseline, each with ground-truth orientation J.W and translation p= ((R,, T;)); N To train the pose prediction module, we adopt the loss function for direct pose supervision by the loss introduced from ACE-EM [60].
B B
Lpsse
R
1
1 + " -7- 7,p0d 11
(11)
where represents the training batch size, while R. R, THth and T. respectively represent the ground truth and network-predictte For various baselines, train the pose estimation module for 200 epochs using only particles and their ground-truth poses. We then test the fine-tuned cryoFIRE's pose estimation module on the real datasets. Notably, we include the original cryoFIRE as an additional baseline in the pose estimation task. To determine the final reconstruction resolution, we randomly split the real data into two equal parts for separate reconstructions. The final reconstruction resolution is calculated by thresholding the Fourier Shell Curve (FSC) of the two reconstructions to 0.5.
rotation and translation. We set B-256 in our experiments.
Evaluation in terms of metric Res(px). We evaluate the pose estimation module on real datasets by using the estimated poses as training labels. For this purpose, we reconstruct two independent volumes from equally split particle stacks using the filter back-projection (FBP) method. We choose this traditional reconstruction algorithm over contemporary neural methods to directly assess pose accuracy without the bias of reconstruction algorithms. Due to the characteristics of FBP, some reconstructions' Fourier Shell Curves (FSC) do not meet the standard threshold Therefore, use correlation coefficient of 0.5 for the Fourier shells the metric Res(px) for pose estimation reconstruction resolution.
ISC=0.143.
We showcase the performance of the original CryoFIRE, the only neural-based reconstruction method the FSC curves
our comparison that does not rely on ground truth particle poses. for structures resolved during pose estimation. Additionally, we present the enhanced performance of pre-trained CryoFIRE when supplemented with CryoGEM's pose-labeled particle images, illustrating these improvements in Figure [4 The formula for the Fourier Shell Correlation (FSC) is provided
20

---

below:
FSC(r) Enerr Fi(ri) F2(ri)
(12)
11):e)l En.rr (rr))l1
where F1, F2 are the Fourier transforms of the FBP reconstructed volumes on equally split particle stacks, respectively. represents all three-dimensional frequency components shown in one- dimensional form.
Evaluation in terms of metric Rot(rad). In addition to evaluating the resolution of resolved structures pose estimation, we also report posing errors for all real images. For each dataset, obtain the estimated ground truth pose from the cryoSPARC reconstruction pipeline. Due to coordinate system misalignment issue, a rigid 6D-body alignment is applied to the poses predicted by the pose estimation module trained generated data. This alignment ensures that the resolved structure in the same coordinate system as the ground truth coarse volume. We then compute the mean angular error between the aligned estimated rotations Riprod and the ground truth rotations R
in radians.
Rot(rad) 180
Pines
R [
Apeed U Sprd
arccOs
(13)
vll
Ull
where represents a unit vector (0, 0, 1).
D Societal Impacts
The CryoGEM technique, a novel tool, has significant societal implications. outline the potential positive and negative impacts, acknowledge uncertainties, and emphasize the broader implications this technology.
generative cryo-electron microscopy (cryo-EM)
D.1 Positive Impacts
By improving particle picking and pose estimation, CryoGEM enhances 3D reconstructions of proteins, potentially accelerating the discovery and understanding of biomolecular structures, crucial for drug development and disease treatment.
Improvement in biomedical fields:
Reduction in labor-intensive tasks: CryoGEM automates the generation of annotated datasets. reducing the burden on human experts.
D.2 Negative Impacts
The ability generate realistic synthetic data raises ethical concerns about misuse, such as manipulating research outcomes or creating misleading scientific evidence. Establishing guidelines for ethical use is crucial. There are several ways to prevent it from misusing including 1) developing and enforcing comprehensive ethical guidelines, 2) ensuring transparency in data generation, and forcing the users claim the usage of CryoGEM in their research projects.
Ethical and Misuse Concerns:
D.3 Conclusion
CryoGEM offers transformative opportunities for cryo-EM and related fields, with significant benefits for biomedical research. However, careful consideration of ethical implications, potential misuse, and long-term societal impacts is essential. By fostering a balanced and reflective approach, we can maximize the positive outcomes of CryoGEM and mitigate potential risks, steering the technology in a beneficial direction for society.
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe that the main contributions of our method are appropriately de- scribed in our abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
We clearly discuss the limitations of our method in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no theory assumptions or proofs in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have clearly described the implementation details including network architecture, loss functions and training details in Section 5 and Appendix B, C.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not provide open-source code and dataset in this submission. However, we will release our code and dataset upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public//uust http://///giiiee//eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
24

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe the implementation details in Section and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do not provide error bars for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computing resources in Section 5 (a single RTX 3090 GPU).
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neuripp: wwtprrips.ccpppppppee httpppplicc//hhcssueeeeelll
Answer: [Yes]
Justification: Our research are all conducted in line with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact of our method in Section I and Appendix D.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
If the authors
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
26

---

Justification:
We do not describle safeguards in our paper, but we will consider to prevent our model and data from misuse when we release our code and dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All of code and datasets we built upon are in public with proper licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi www.we.hhde...m caree.comdaie com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not provide new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
27

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not have crowdsourcing experiments in our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do have crowdsourcing experiments in our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "FLTdjDgKp5", "reviewer": "Reviewer_w8uj", "summary": "In this paper, the authors introduce Physics-Informed Generative Cryo-Electron Microscopy (CryoGEM), a novel generative model for cryo-electron microscopy (Cryo-EM) micrographs. CryoGEM is trained to produce micrographs that accurately replicate the ice gradient, point spread function (PSF), and noise characteristics of experimental micrographs. The model offers two main applications in Cryo-EM analysis through the annotations provided by the generated micrographs: a) The generated micrographs include precise positional annotations for particles (2D projections of proteins in the micrograph). b) In addition to positional information, CryoGEM provides data on particle orientations and conformations. This information can be utilized with methodologies such as CryoFIRE to train models that distinguish particle orientations and conformations in experimental micrographs.\nThe authors propose this innovative generative model to assist Cryo-EM data analysts in a) Fine-tuning particle picking models, and b) Training deep learning models for heterogeneous 3D reconstruction. This approach has the potential to significantly enhance the particle-picking process in experimental micrographs, ultimately leading to improved 3D reconstructed volumes of proteins.", "questions": "Originality: CryoGEM presents an innovative approach by integrating physics-informed modeling with generative techniques, addressing a significant gap in cryo-EM analysis.\nQuality: The study features well-designed experiments that effectively demonstrate the model's capabilities. The application of CryoFIRE provides additional validation of CryoGEM's practical utility.\nClarity: The paper is articulated with precision, offering detailed explanations and a well-structured layout that enhances reader comprehension.\nSignificance: CryoGEM shows considerable potential to impact cryo-EM analysis significantly. It provides valuable tools that can improve both particle picking accuracy and the quality of 3D reconstruction processes, potentially advancing structural biology research.", "limitations": "The authors have adequately addressed the limitations of their work, discussing potential areas for future improvements, such as generalizing the model to different experimental conditions and the need for a coarse cryo-EM map as an input, which may be hard to produce. However, a more explicit discussion on the accessibility of CryoGEM to the wider research community and the efficiency of the proposed pipeline would strengthen the paper, as well as the code availability.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: CryoGEM presents an innovative approach by integrating physics-informed modeling with generative techniques, addressing a significant gap in cryo-EM analysis.\nQuality: The study features well-designed experiments that demonstrate the model's capabilities. The application of CryoFIRE provides additional validation of CryoGEM's practical utility.\nClarity: The paper is articulated precisely, offering detailed explanations and a well-structured layout that enhances reader comprehension.\nSignificance: CryoGEM shows considerable potential to impact cryo-EM analysis significantly. It provides valuable tools that can improve particle picking accuracy and the quality of 3D reconstruction processes, potentially advancing structural biology research.", "weaknesses": "Code Availability: Currently there is no code availability, which may limit the accessibility of CryoGEM to the broader research community.\n\nResolution and Time Requirements: There is a lack of detailed discussion on the resolution of the initial coarse cryo-EM density map obtained from the ab-initio reconstruction of CryoSPARC, as well as the time required for this process.\n\nPipeline Efficiency: The time consumption for fine-tuning Topaz through CryoGEM raises concerns about the practicality of the proposed pipeline compared to manually picking a small number of micrographs.\n\nComparison with Template Matching: The particle picking approach of CryoGEM is not compared with template matching from the coarse cryo-EM input map, which could provide a more comprehensive evaluation of its advantages.\n\nIncomplete Quantitative Comparisons: The quantitative comparisons for pose estimation are incomplete without the fine-tuning of CryoFIRE using pose estimations of particles from the coarse cryo-EM map input from CryoSPARC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. We appreciate the opportunity to address these concerns.\n\n## On the Pipeline Efficiency\n\nWe acknowledge that CryoGEM indeed takes longer than manual annotating, as shown in the following table. However, particle picking is a tedious, labor-intensive, and time-consuming task for technicians. Even with a blob picker, identifying target particles from the candidates requires excessive effort. CryoGEM enhances productivity by automatically generating annotated data for particle picking with high precision, although at the expense of speed.\n\nThe bottleneck in CryoGEM's pipeline is the particle labeling (generation) time, which includes ab-initio reconstruction to get the coarse volume, as well as the training and inference time. Currently, CryoGEM's training and inference are conducted on a single RTX 3090 GPU. To improve speed, we are developing a more parallelized GPU version, which could potentially accelerate the process by an order of magnitude.  Additionally,  AlphaFold3 serves as a potential alternative to CryoSPARC, which could further speed up ab-initio reconstruction.\n\n|     **Method**    | **Labeling Time** | **Topaz Fine-tune Time** | **Reconstruction Time** |  **Total Time** | **AUPRC (↑)** | **Res. ($ \\unicode{x212B} $)** |\n|:------------------------|:--------------------------|:-------------------------|:---------------------------------|:---------------|:---------|:---------------|\n| **Manual** |      1h26m41s     |       16m27s      |          **1h41m7s**         | **3h24m15s** |   0.776   |       3.59      |\n|     **Blob Picker**    |       **14m31s**       |       12m12s      |         2h26m33s         | 2h53m16s |   0.684   |       4.57      |\n|        **Ours**        |      2h56m39s    |       **10m0s**       |         2h28m13s        | 5h34m52s | **0.796** |     **3.25**    |\n\n\n## Comparison with Template Matching\n\nThe particle picking approach of CryoGEM is not compared with template matching from the coarse cryo-EM input map, which could provide a more comprehensive evaluation of its advantages.\nThank you for the suggestion. It is indeed an excellent point. The following table shows the suggested quantitative comparison of our finetuned Topaz (**Ours**, by CryoGEM's synthetic annotated datasets) with cryoSPARC’s template-based matching method (**Template Picker**, using the coarse volume as an input). CryoGEM consistently outperforms the Template Picker in nearly all examples in both AUPRC and resolution, except for the AUPRC of Proteasome and the resolution of Integrin. We will add Template Picker as a picking baseline in the revision.\n\n|      **Metric**     |  **AUPRC (↑)**  |            |            |            |            |           | **Res ($ \\unicode{x212B} $, ↓)** |          |          |          |          |          |\n|:-------------------|:-----------:|:----------:|:----------:|:----------:|:----------:|:---------:|:-----------------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|  **Method/Dataset** | Proteasome  |  Ribosome  |  Integrin  |  PhageMS2  |  HumanBAF  |    Avg.   |    Proteasome     | Ribosome | Integrin | PhageMS2 | HumanBAF |   Avg.   |\n| **Template Picker** |  **0.547** |    0.742   |    0.585   |    0.873   |    0.393   |   0.628   |        2.71       |   3.58   | **4.95** |   9.63   |   10.91  |   6.36   |\n|       **Ours**      |    0.490    | **0.797** | **0.606** | **0.915** | **0.562** | **0.674** |      **2.68**     | **3.25** |   5.54   | **7.16** | **7.74** | **5.27** |\n\n## Comparisons with Pre-trained CryoFIRE\n\nThank you for your insightful comments. We did not include this comparison because we focused on evaluating the performance improvement between our method (**Ours**) and the original **CryoFIRE**. However, it is indeed more convincing to compare our method with the pre-trained cryoFIRE (**CryoFIRE\\***) using the particle images whose poses are estimated at the ab-initio reconstruction stage. As shown in the following table, the pre-trained cryoFIRE exhibits a reasonable performance improvement compared to the original CryoFIRE. However, our method still outperforms both schemes by a large margin in most cases.\n\n\n| **Metric**    | **Res. (px. ↓)** |          |          |          |          |          | **Rot. (rad. ↓)** |          |          |          |          |          |\n|---------------|:---------------------:|:--------:|:--------:|:--------:|:--------:|:-------------:|:---------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n| **Method**    | Proteasome            | Ribosome | Integrin | PhageMS2 | HumanBAF | Average  | Proteasome            | Ribosome | Integrin | PhageMS2 | HumanBAF | Average  |\n| **CryoFIRE**  | 5.94                  | 16.92    | 13.87    | 17.23    | 6.98     | 12.18    | 1.55                  | 0.64     | 0.93     | 0.75     | 1.53     | 1.08     |\n| **CryoFIRE*** | 2.96                  | 8.09     | 7.95     | **3.89** | 9.04     | 6.386    | 1.43                  | 0.52     | **0.73** | 0.69     | 1.54     | 0.98     |\n| **Ours**      | **2.59**              | **4.27** | **4.88** | 5.54     | **6.55** | **4.29** | **0.41**              | **0.32** | 0.88     | **0.43** | **1.42** | **0.69** |\n\n## On the Code Availability\n\nAll of our code and datasets will be released to the public upon acceptance for further evaluation of CryoGEM. In the meantime, we will provide an anonymous version of our code for training and evaluating CryoGEM to the area chair.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vz1BiqLSVf", "reviewer": "Reviewer_D477", "summary": "The paper introduces CryoGEM, an innovative method combining physics-based cryo-EM simulation with unpaired noise translation via contrastive learning to generate high-quality synthetic cryo-EM datasets. The approach significantly improves the visual quality of generated images and enhances downstream tasks like particle picking and pose estimation, leading to better 3D reconstructions.", "questions": "Is there any reference indicating whether the Gaussian noise distribution accurately represents the actual physical noise?\n\nIn practice, it is relatively easy to obtain a large number of observed samples of the target image in transmission images. Even if the proposed approach enhances the results, can we still easily access more samples of the target particle with minimal effort?", "limitations": "See Weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1 Extensive experiments demonstrate that CryoGEM produces high-quality synthetic cryo-EM images that significantly outperform existing methods like CycleGAN and CUT. The visual quality of the generated images is notably superior, preserving structural details and realistic noise patterns.\n\n2. The synthetic datasets generated by CryoGEM improve the performance of downstream tasks, such as particle picking and pose estimation. The paper reports substantial improvements in these tasks, leading to better resolution in the final 3D reconstructions.", "weaknesses": "The physics-based simulation in CryoGEM relies on a coarse result as an input. This requirement can be a significant limitation in scenarios where obtaining a reliable coarse result is challenging, such as with very small or highly dynamic molecules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful suggestions. We will improve the paper based on them.\n\n## On the Relationship between Gaussian and Actual Physical Noise\n\nWe follow the common practice in the literature of using Gaussian noise to model the reconstruction problem in cryo-EM. For example, cryoDRGN [1] and RELION [2] model image noise in the Fourier domain as zero-mean, independent Gaussian distributed noise. The inverse Fourier transform of this noise in the real domain is also i.i.d. Gaussian noise.\n\nWe'd like to stress that CryoGEM can also accommodate other noise models, such as signal-dependent Poisson noise [3], by replacing the current Gaussian noise model with the new one during training. If space permits, we will include an example of this.\n\n[1] Zhong E D, Bepler T, Davis J H, et al. RECONSTRUCTING CONTINUOUS DISTRIBUTIONS OF 3D PROTEIN STRUCTURE FROM CRYO-EM IMAGES[C]//8th International Conference on Learning Representations, ICLR 2020. 2020.\n\n[2] Scheres S H W. RELION: implementation of a Bayesian approach to cryo-EM structure determination[J]. Journal of structural biology, 2012, 180(3): 519-530.\n\n[3] Vulović M, Ravelli R B G, van Vliet L J, et al. The Supplementary Material of Image formation modeling in cryo-electron microscopy[J]. Journal of structural biology, 2013, 183(1): 19-32.\n\n## On the Effort to Capture More Samples\n\nWe agree that capturing more micrographs can improve the resolution of the final results. However, this approach leads to longer capture time on expensive cryo-EM equipment and demands substantial computational resources for iterative optimizations, often taking several days for a human expert.\n\nComplementarily, CryoGEM aims to enhance downstream results without the need for a large amount of raw data, thereby improving the final resolution of resolved structures. This aligns with the recent generation-based reconstruction approaches [1], where sparse view reconstruction is achieved by generative models.\n\nIn terms of resource consumption, CryoGEM is a lightweight generative model that can be trained on 100 real micrographs using a single NVIDIA RTX 3090 GPU in just two hours. After training, it can rapidly generate annotated synthetic datasets with minimal additional cost. Therefore, the best practice should combine efficient data capture with advanced data processing, such as inputting as many as possible samples into a CryoGEM-improved pipeline for optimal reconstruction results.\n\n[1] Wang S, Leroy V, Cabon Y, et al. Dust3r: Geometric 3d vision made easy[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20697-20709.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E9UHOvjMEM", "reviewer": "Reviewer_B6fS", "summary": "In this paper, the authors introduce a method to generate large annotated cryo-EM datasets from a small number (100) of real micrographs. The method combines a physics-based model of the image formation model with a contrastive learning strategy. The authors show that their method can be used to improve the quality obtained with downstream tasks such as particle picking, homogeneous reconstruction and pose estimation.", "questions": "**Mutual information extraction.** This paragraph of the method was unclear to me. Why are $\\mathbf{v}$ and $\\mathbf{v}^+$ not indexed by $q$ while $\\mathbf{v}^-$ is indexed by $k$ in (5)? Why does (5) correspond to the probability of \"selecting\" a positive sample?\n\n**Origin of coarse models.** CryoGEM needs to a coarse 3D model to generate synthetic images. For the experiments conducted in this paper, where do these models come from? I did not find this information in the paragraph \"Datasets\".\n\n**Resolution of coarse models.** What is the resolution of the coarse models used in this paper? What is the influence of the resolution of the initial model on the accuracy obtained on downstream tasks?\n\n**Pose accuracy** What does $v$ correspond to in Eq (15)?", "limitations": "Yes, limitations and potential negative impacts are discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Although acquiring more data in cryo-EM is usually easy, getting access to annotated data is hard. CryoGEM combines a physics-based model with a contrastive learning strategy to generate realistic annotated data. This is particularly interesting because, due to the low SNR in cryo-EM images, most downstream tasks often significantly benefit from pretraining (pose estimation) or finetuning (particle picking). The method introduced in this paper holds the potential to improve the accuracy of cryo-EM reconstruction pipelines. Notably, the authors explicity showed that cryoGEN can improve the performances obtained on downstream tasks.\n\nThe particularly appreciated the following points:\n- the method is described in a clear way ;\n- the experiments are fully described and seem reproducible ;\n- all the contributions claimed are illustrated with an experiment -- I appreciated the effort made by the authors to evaluate the quality of particle picking, pose estimation and homogeneous reconstruction after cryoGEM ;", "weaknesses": "I find that some parts of Section 3 (description of the method) lack clarity and that information on the 3D models used by the simulator of cryoGEM are missing (see \"Questions\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation and insightful comments. We will improve the clarity of the paper based on them.\n\n## Clarification of Notations\n\nThank you for thoroughly reading our paper and the supplementary material. We truly appreciate your suggestions.\n\nFor mutual information extraction (Equation 5), we followed CUT [1], where $\\boldsymbol{v}, \\boldsymbol{v}^+ \\in \\mathbb{R}^{N}$ represent an arbitrary pair of feature vectors of the query and the positive sample from the whole set. However, in our case, we have $Q$ pairs, and it will be clearer to rewrite $\\boldsymbol{v}, \\boldsymbol{v}^+ \\in \\mathbb{R}^{N}$ to $\\boldsymbol{v}_q, \\boldsymbol{v}_q^+ \\in \\mathbb{R}^{N}$. We will fix this in the revised version.\n\nThe \"selecting\" operation in Equation 5 corresponds to a multi-class classification task, aiming to maximize the probability of correctly matching the corresponding positive $\\boldsymbol{v}_q^+$ from a set of it and $K$ negatives, given the query $\\boldsymbol{v}_q$. Therefore, we formulate Equation 5 to minimize a $K+1$-class cross-entropy loss.\n\nIn Equation 13, $v$ is a unit vector $(0,0,1)$.\n\nWe will clarify all of these points in the revision.\n\n[1]  Park T, Efros A A, Zhang R, et al. Contrastive learning for unpaired image-to-image translation[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16. Springer International Publishing, 2020: 319-345.\n\n## The Origin of Coarse Models\n\nThank you for your attention. We will add this description to the Datasets section for better clarity. The coarse 3D model is obtained by running an ab-initio reconstruction of CryoSPARC with its default setting, followed by cryoDRGN to handle heterogeneous cases. We train cryoDRGN for 50 epochs using particles at a resolution aligned with the low-resolution 3D volume.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We are grateful that all reviewers recognize that CryoGEM showcases the usefulness of generative AI in structural biology. We will soon release the code and the data for the community to experiment with and improve. The reviewers have provided many insightful suggestions that we will incorporate into the revision. As follows, we first address the common question raised by the reviewers and then their comments.

## On the resolution of the coarse model (w8uj, D477, B6fS)

The reviewers are correct that CryoGEM uses a coarse model as an input, which can be obtained by ab-initio reconstruction, e.g., CryoSPARC. In our experiments, the resolution of coarse models ranges from 7.05 to 21.22. We have shown that even using such low resolutions as initializations, CryoGEM can still achieve an accurate location, pose, and conformation-controlled data generation (we kindly refer to Figure 8 in our paper for visual illustrations). The generated data significantly benefits downstream tasks such as particle picking and pose estimation, subsequently improving the reconstruction's final resolution. The following table shows the ab-initio and final resolutions of several examples, as well as the time required for ab-initio reconstruction.

It is worth mentioning that ab-initio reconstruction may fail on the first attempt. A common practice is to conduct another round of data collection or cleaning to obtain a better ab-initio initialization. In extreme cases where it completely fails, a potential solution is to leverage structure prediction models, such as AlphaFold3[1], to initialize a density volume as a de facto ab-initio reconstruction. This approach is part of our immediate future work. We will clarify these points in the revision.

| Dataset | Res. of the coarse model ($ \unicode{xC5}$, ↓) | Time of ab-initio reconstruction | Final Resolution ($ \unicode{x212B}$, ↓) |   |
|:-----------:|:-----------------------------------------------:|:--------------------------------:|:---------------------------:|---|
|  Proteasome |                       7.79                      |              1h0m27s             |             2.68            |   |
|   Ribosome  |                       7.05                      |             1h56m39s             |             3.25            |   |
|   Integrin  |                       9.65                      |             1h24m21s             |             5.54            |   |
|   PhageMS2  |                      11.78                      |              1h6m28s             |             7.16            |   |
|   HumanBAF  |                      21.22                      |              42m49s              |             7.74            |   |


Note that the resolution of the coarse model is calculated by splitting the real particle dataset into halves and conducting ab-initio reconstruction independently [2]. The coarse models are then aligned to calculate the spatial resolution in CryoSPARC. For the heterogeneous dataset, Integrin, we utilized cryoDRGN to obtain the neural volume. This process took 8 hours, 5 minutes, and 24 seconds after the ab-initio reconstruction.

[1] Abramson, J., Adler, J., Dunger, J., et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature 630, 493–500 (2024). https://doi.org/10.1038/s41586-024-07487-w

[2] Van Heel M, Schatz M. Fourier shell correlation threshold criteria[J]. Journal of Structural Biology, 2005, 151(3): 250-262.

Meta-review: (5,5,7) In this paper, the authors introduce a method to generate synthetic cryo-EM images from a small number (100) of real micrographs. The method combines a physical simulation of the image formation model with a contrastive learning approach for learning noise statistics. The authors show that their method can be used to improve the quality obtained with downstream tasks such as particle picking, homogeneous reconstruction, and pose estimation. Reviewers were positive, noting the novelty of the task.
Decision: Accept (poster)