cryoSPHERE: Single-Particle HEterogeneous REconstruction from cryo EM

Authors: Gabriel Ducrocq, Lukas Grunewald, Sebastian Westenhoff, Fredrik Lindsten
Primary area: generative models
Keywords: cryoEM; protein structure; Deep Learning; Machine Learning; generative modelling

Abstract:
The three-dimensional structure of proteins plays a crucial role in determining their function. Protein structure prediction methods, like AlphaFold, offer rapid access to a protein’s structure. However, large protein complexes cannot be reliably predicted, and proteins are dynamic, making it important to resolve their full conformational distribution. Single-particle cryo-electron microscopy (cryo-EM) is a powerful tool for determining the structures of large protein complexes. Importantly, the numerous images of a given protein contain underutilized information about conformational heterogeneity. These images are very noisy projections of the protein, and traditional methods for cryo-EM reconstruction are limited to recovering only one or a few consensus conformations.

In this paper, we introduce cryoSPHERE, which is a deep learning method that uses a nominal protein structure (e.g., from AlphaFold) as input, learns how to divide it into segments, and moves these segments as approximately rigid bodies to fit the different conformations present in the cryo-EM dataset. This approach provides enough constraints to enable meaningful reconstructions of single protein structural ensembles. We demonstrate this with two synthetic datasets featuring varying levels of noise, as well as two real dataset. We show that cryoSPHERE is very resilient to the high levels of noise typically encountered in experiments, where we see consistent improvements over the current state-of-the-art for heterogeneous reconstruction.

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730263000404

thread_type:
official_review

<thread_object>
creator: reviewer_jga5

<content>
summary:
This paper introduces cryoSPHERE, a deep learning model that uses nominal protein structures (e.g., from AlphaFold) to break proteins into segments and fit these segments to various conformations in cryo-EM data.

strengths:
1. The approach of learning to decompose the protein’s amino acid chain into segments to represent various conformations is novel.
2. The interpretability of moving part is highly beneficial for practitioners, offering a clearer understanding of conformational dynamics.

weaknesses:
1. No meaningful improvements were observed over CryoStar
    - Figures 3, 12, 13: It’s unclear what meaningful differences exist compared to CryoStar.
    - Figures 14, 21: There doesn’t appear to be a significant difference between CryoStar and CryoSphere (CryoStar may even appear better).
    - Figure 29: It would be great to compare CryoStar with CryoSphere rather than with CryoDRGN.
2. Should change Figure 6:
    - Instead of showing only CryoDRGN results and G.T, a comparison with CryoSTAR or Dynamight would be more appropriate (perhaps include Figure 32 and the CryoSphere result, as there are no CryoSphere results with SNR 0.001).

questions:
Could you include additional comparisons with methods like CryoSTAR, or Dynamight (e.g., computational cost, novelty, etc)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730407482244

thread_type:
official_review

<thread_object>
creator: reviewer_nciv

<content>
summary:
- The authors propose a VAE and GMM-based approach to atomistic heterogeneous Cryo-EM reconstruction
- The GMM part learns how to divide the amino acid chan in segments
- The VAE learns how to encode images into latent variables and decode these variables along with the GMM output into segment-wise conformational changes
- The authors claim that their method overcomes current challenges that atomistic models face, while also achieving improved interpretability by learning these segments (that ideally correspond to domains in the biological sense)

strengths:
The work is of high quality and overall very well-written. I’d like to highlight the following strong points of the paper:
- The authors seem very well aware of the difficulties atomistic models have in terms of the optimisation landscape and address the problem adequately by regularising while keeping sequential information intact
- The authors give useful biological remarks such as that their methods seems to recover domains, which is useful to know as a practitioner
- The authors show that their method works on real data, which is actually the first time I’ve read this for atomistic models

weaknesses:
There is one main weak point in the paper in my regard:
- The authors do not address limitations of their method very thoroughly:
    - whether or not their method is able to overcome poor initialisation of the initial structure S_0. This is one of the major risks of working with (pseudo)-atomistic models as shown in works such as DynaMight (Schwab et al. 2024) 
    - Whether or not local physical information such as bond lengths between atoms at the boundaries of the segments is preserved. This might be challenging

questions:
Questions for clarification 
- In section 3.6 the authors say “Similar to Li et al. (2023), instead of using a mean squared error loss between the predicted and ground truth image, we use a correlation loss between the true and predicted image”. However, they do not motivate this choice. Would you please care to elaborate?
- In section 5.1 the authors write “Testing the segment decomposition, we then run cryoSPHERE by requesting division into Nsegm = 4. The program learnt a first and third segment with 0 residues, a second segment with 1353 residues and a fourth segment with 157 residues (Figure 5). Thus, cryoSPHERE learnt segments according to the ground truth.” Is there a reason why we should expect that the model prefers to learn two segments?

Additional feedback
- It reads as if equation 7 is the only loss that is being used. I assume that the VAE latent distribution is also trainable? It would benefit the readability if the authors clarify whether the loss is just the decoder loss or the whole loss.“
- In section 5.3 the authors write “The first two principal components explain more than 96 percent of the variance.” In general, I would be very reluctant to make any claims about PCA in the latent space as you could come up with a transformation that scrambles the latent space and still gives the same decoding, but covers much less variance. I can imagine that Figure 7 is a nice illustration, but I would recommend to leave it with that.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694072131

thread_type:
official_review

<thread_object>
creator: reviewer_rjxo

<content>
summary:
The authors introduced a new method, cryoSPHERE, that resolves the continuous heterogeneity problem in cryo-EM reconstruction. Similar to some other methods, cryoSPHERE uses an atomic model as the input, makes segmentations, and uses the results as the regularization in finding the flexible movement in the cryo-EM dataset.

strengths:
Instead of finding the deformation for each residue, cryoSPHERE learns a segmentation of the amino acid or nucleotide chains and deforms the segment to fit the heterogenous cryo-EM data. This is indeed a valid assumption in many cases especially for large complexes  like the spliceosome (EMPIAR-10180).

weaknesses:
Major:
1. CryoSPHERE uses an atomic model as the reference and formulate the conformational heterogeneity in cryo-EM dataset as the deformation of the segments of the reference model. In finding the optimal segments, the approach is similar to what used in e2gmm, where a GMM with N_{segm} components is fitted. This segmentation result is then used in a cryoSTAR/DynaMight-like setting, replacing the regularization losses and compute predicted projections to compare with the particle images. The author should also compare the result with other similar methods, including DynaMight, e2gmm, and maybe 3DFlex, and discuss the outcomes.

2. In cryo-EM, density maps are extremely important in evaluating the validity of the results. However, the authors only presented the density maps for one of the synthetic dataset. Especially in the case of under the hypothesis that the heterogeneity is modeled as the deformations of a canonical density, it is very important that the reconstructed canonical density is free from any obvious artifact.

3. The FSC comparison in Sec. 5.2 between cryoSPHERE and cryoDRGN and cryoSTAR may not be very fair, since cryoSPHERE's density is reconstructed under the assumption that there exist a canonical density, while cryoDRGN does not hold the same hypothesis, and cryoSTAR explicitly avoids this to reduce the unwanted bias from the reference model. A fair comparison would be made against DynaMight and 3DFlex.

4. Lack of ablation study. The authors should discuss how N_{segm}, which is a key hyperparameter that is related to the degree of freedom in the segmentation, affects the results in both synthetic and real data.

5. The authors compare cryoSPHERE to cryoDRGN and cryoSTAR, but from the experiments, it is unclear what the advantage that cryoSPHERE brings to the field. The only advantage seems to be the increase FSC resolution, but it is only shown in one synthetic dataset.

Minor:
1. The sequence of Fig.6 and Fig.7 in the paper is reversed.
2. In the related works, e2gmm should also be considered as a method using deep learning.
3. In the FSC comparison, if not comparing two half maps, the cutoff of 0.5 should be used and 0.143 is not meaningful. See [1] appendix for the reason (and also why 0.143 is the number widely used for half maps comprison). Therefore the discussion at the end of Sec. 5.2 about 0.5 and 0.143 cutoff improvement is not very correct.
4. The writing could be better polished.
5. I do not think the "Ethics Statement" content is proper as it lacks evidence.


[1] Rosenthal, Peter B., and Richard Henderson. "Optimal determination of particle orientation, absolute hand, and contrast loss in single-particle electron cryomicroscopy." Journal of molecular biology 333.4 (2003): 721-745.

questions:
Can the authors elaborate how the density maps (volumes) are reconstructed in cryoSPHERE (like in Fig. 6)? I understand how backprojection in cryo-EM works, but still find it difficult to understand how this is performed exactly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732102427905

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General answer

comment:
Here, we provide brief & general answers. We have submitted a new version with major revision and a new experimental data set, which directly shows that the approach used in cryoSPHERE outperforms other methods, in particular cryoSTAR, for experimental data of samples with high conformational heterogeneity. 

Given the importance of devising robust and usable methods to analyze dynamics from cryo EM data, and the unique advantages of CryoSPHERE to cope with the high noise levels in real experimental data, we hope that the reviewers can agree that this is an important contribution to the conference. 

None of the links we provide in this rebuttal refer to any accounts or identity of the authors of cryoSPHERE.

1/ We agree that comparison to other methods is important. In the revised version we planned to extend comparisons to e2gmm and dynaMight. However, dynaMight only produces bugs when run with a reference structure a often even without reference structure, thus it limits comparability, and we still cannot install eman2. We keep trying.

2/ Regarding comparison with CryoSTAR: cryoSTAR is concurrent work that has been developed independently from cryoSPHERE. In the new version of this paper we show more conclusively that CryoSPHERE is more resilient to noise compared to cryoSTAR (see fig. 4, see reviewer JGA5, point 1 for a detailed reply). This is expected because the amount of free structural parameters is much reduced in cryoSPHERE, which is highly relevant when interpreting experimental data sets.

3/ To demonstrate the applicability of cryoSPHERE to high levels of noise, we included in the revised version  the analysis of an experimental cryo EM data set (medium size protein recorded by a state-of-the-art experimental facility) in the revised version of the paper. Here, cryoSTAR leads to overfitting/physically unrealistic deformations with clashes of the protein for the “disordered state”, while cryoSPHERE gives plausible results. This is highly relevant, because the most interesting data sets typically have low SNR and high degrees of structural heterogeneity.

4/ Regarding the debiasing of the results: All methods that directly fit structures to the images face this challenge. cryoSTAR has implemented this as a phase II training (the volume method), however, we prefer to run cryoDRGN/DRGN-AI in a second, separate step of training. See our answer to reviewer NciV weakness 3. Figures 15, 22, 29 show this approach gives similar or better results to the implementation that cryoStar has chosen. In any case, when dealing with highly heterogeneous data sets (like the experimental data that we analyze), post computation of the volumes is only of limited value as the consensus densities are very low amplitude. We do agree that for all methods, more rigid methods for debiasing of the results should be developed. 




Explanation of point 1/

We genuinely tried to compare cryoSPHERE to DynaMight. However, this comparison only makes sense if we can run it using a base structure, as in cryoSPHERE. This is one of the two modes in which DynaMight can be run. Unfortunately, the DynaMight documentation does not mention how to run it with a base structure, as you can see on the github page

 https://github.com/3dem/DynaMight
or on the official documentation

 https://relion.readthedocs.io/en/release-5.0/SPA_tutorial/Flexibility.html.

From the source code, we figured out how to use this option, see line 59:
https://github.com/3dem/DynaMight/blob/main/dynamight/deformations/optimize_deformations.py

However, when running this we only encountered bugs that led to the termination of the DynaMight training. This problem has been noted by others as you can see with this unanswered github issue (not raised by us!):
https://github.com/3dem/DynaMight/issues/11

In addition, neither the official documentation nor the github page explain how to obtain one structure/volume per image. They both describe how to display volumes based on a GUI. This is another obstacle to quantitatively compare DynaMight with cryoSPHERE.
We feel that we have genuinely tried to run DynaMight as a structural method and that it is very unfortunate that it was not working properly. 
We thought it would be considered unfair to compare its results with cryoSPHERE without starting with the same base structure. But since the reviewers insist on DynaMight, we are currently trying to run it without base structure and will try to modify its source code to obtain one structure/volume per image, even though we also encounter bugs in this mode on many datasets. This will be used to make a DynaMight/cryoSPHERE comparison if we succeed in running it.
Regarding e2gmm: we have been consistently unable to install the latest version of eman2, in spite of following the documentation. We are still trying to the best of our efforts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732105869314

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
What is new in the revised version of our draft:

comment:
We want to thank the reviewers for their thoughtful comments and we feel that their suggestions have improved our paper. Here is a summary of what has changed:

1/ We have added a new experimental dataset, featuring a high level of noise and heterogeneity. We show that cryoSPHERE is able to recover physically plausible motion, while cryoStar overfits.

2/We change the run of cryoSPHERE with N_segm = 20 to N_segm = 25. This way we perform very marginally worse than cryoStar on SNR 0.1, marginally better than cryoStar on SNR 0.01 and one standard deviation better than cryoStar on SNR 0.001

3/ We provide a discussion of the impact of the choice of N_segm in appendix B.2.5.

4/ We also have change more minor points raised by the reviewers, see specific answers to the reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732535169269

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Addition to the rebuttal

comment:
We would like to thank the reviewers again for their suggestions. They have improved our paper a lot.
 
We upload a new revised version of our paper, this time adding Appendix B.2.4 showing that the Phase II of training cryoStar, to assess the bias, can also be performed with cryoSPHERE using DRGN-AI, at no extra cost.
 
We have been trying the best we could to run DynaMight and e2gmm. However:

1/ As explained in the previous general answer, we have not been able to run DynaMight with a base structure. (There seems to be some bug in the available implementation that prevents this. This has been observed by others as well, as indicated by the previously mentioned github issue.)
 
2/ We therefore ran the method without base structure on the synthetic datasets. Unfortunately, as far as we can tell, the only way to obtain the volumes is through a GUI which makes it impossible to obtain the 1k volumes that would be needed to compare DynaMight and cryoSPHERE quantitatively. In addition, the volumes recovered are low resolution, since the high resolution reconstruction is done in a second step, on a single volume for DynaMight. This prevents any meaningful comparison in terms of FSC.
 
3/ We only encountered errors when trying to run DynaMight, with and without base structure, on EMPIAR 12093.
 
4/After much efforts, we could finally install eman2. Unfortunately, when running e2gmm on any of our datasets, we encountered Out Of Memory errors terminating the training. We have tried many unsuccessful solutions: to reduce the box size, to use only the backbone of the base structure instead of the full pdb, to reduce the batch dimension to 8, to increase the number of GPUs to get more RAM, to run it on only 50k images, but nothing has worked.
 
 
We have genuinely tried to compare cryoSPHERE to e2gmm and DynaMight and we regret that we could not, due to issues with the publicly available code for these methods. We hope that the reviewers can acknowledge that providing new and stable solutions to the problem of extracting conformational heterogeneity is highly interesting.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732716698040

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New rebuttal

comment:
We thanks the reviewers again for this new rebuttal version of our paper. We feel that we have a much stronger case now and that their suggestions helped us precise the strengths and shortcomings of cryoSPHERE.

We now summarize the changes made :

1/ We now demonstrate two ways to debias cryoSPHERE: using DRGN-AI for EMPIAR 10180 and using cryoStar volume method for EMPIAR 12093 (sections 5.2 and 5.3 respectively), using cryoSPHERE's latent space.

2/ We have added examples of volumes showing this debiasing procedure, see Figures 5 and 6.

3/ We have added more examples of volumes and movies of the motions recovered by the volumes methods on EMPIAR 12093, see Appendix B.4.

4/ We now conclude that we can debias cryoSPHERE successfully on real data, but that for the most noisy and heterogeneous datasets, volume method debiasing is still impossible (for cryoSPHERE and cryoStar). We leave that problem to future work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes cryoSPHERE, a method for reconstructing proteins that can occur in different configurations. 

The paper received three reviews, and after the rebuttal all three reviewers are supportive of publication.

* Reviewer RJxo was initially not supportive of publication; the reviewer noted that that some comparisons are not fair, and pointed out that it is unclear what the advantage of the method is over methods like cryoSTAR and others. In the rebuttal, the authors highlight that cryoSPHERE can provide physically plausible reconstructions for examples where cryoSTAR cannnot. After the rebuttal, the reviewer still has concerns about the performance of the method, but finds the novelty acceptable to recommend weak accept.

* Reviewer NciV is supportive of publication and highlights that the method is evaluated on real data, contrarily to other works in the area that are based on atomistic models. 

* Reviewer JGA5 finds elements of the approach to be novel and the interpretability of the moving part beneficial for understanding conformational dynamics of proteins. However, the reviewer also notet that there are no meaningful improvements over CryoSTAR. In the rebuttal, the authors provided new figures that partially addressed the reviewers concerns, and while the reviewer still has concerns about the novelty relative to CryoSTAR, the reviewer raised the score to weak accept.

The paper provides a new method for reconstructing proteins in different configuration, and demonstrates that the method has, in some cases, a slight benefit over existing methods, in particular over cryoSPHERE. I agree with the reviewers that while the method uses many elements from prior methods (such as the way the VAE is used, the atomistic representations, etc) and while the benefit over existing methods is relativly marginal, the method and evaluations contribute to the important problem of reconstructing a single protein in different configurations, and I therefore recommend to accept the paper.