Package: cryosieve
CryoSieve OverviewCryoSieve is an advanced software solution designed for particle sorting/sieving in single particle analysis (SPA) for Cryogenic Electron Microscopy (cryo-EM). Supported by extensive experimental results, CryoSieve has demonstrated superior performance and efficiency compared to other cryo-EM particle sorting algorithms.Its unique ability to eliminate unnecessary particles from final stacks significantly optimizes the data analysis process. The refined selection of particles that remain contribute to a notably higher resolution output in reconstructed density maps.For certain datasets, the precision of CryoSieve's particle subset selection is so refined that it approaches the theoretical limit, delivering unprecedented detail and accuracy.Video TutorialA video tutorial is available within this repository. You can directly download ithere.PublicationsFor more details, please refer to the paper"A minority of final stacks yields superior amplitude in single-particle cryo-EM". If you find that CryoSieve contributes to your work, we kindly request you to cite this paper.The List of Available Demo Casesdatasetnumber of particlesmolecular weight (kDa)EMPIAR linkexpected result linkTRPA143,585688EMPIAR-10024linkhemagglutinin130,000150EMPIAR-10097linkLAT1250,712172EMPIAR-10264linkpfCRT16,905102EMPIAR-10330linkTSHR-Gs41,054125EMPIAR-11120linkTRPM842,040513EMPIAR-11233linkapoferritin382,391440EMPIAR-10200linkstreptavidin23,99152EMPIAR-10269linkInstallationCryoSieve is an open-source software, developed using Python, and is available as a Python package. You can access our source codeon GitHub.PrerequisitesPython version 3.7 or later.NVIDIA CUDA library installed in the user's environment.DependenciesThe CryoSieve package depends on the following libraries:numpy>=1.18
mrcfile>=1.2
starfile>=0.4,<0.5
cupy>=10
torch>=1.10Preparation of CUDA EnvironmentWe recommend you install CuPy and PyTorch initially, as their installation largely depends on your CUDA environment. Please note, your PyTorch package should be CUDA-capable.To streamline this process, we suggest preparing a conda environment with the following command:conda create -n CRYOSIEVE_ENV python=3.10 cupy=10.2 cudatoolkit=10.2 pytorch=1.12.1=py3.10_cuda10.2_cudnn7.6.5_0 -c conda-forge -c pytorchThis command is specifically for a CUDA environment version 10.2. If your CUDA environment is higher than 10.2, adjust the command based on the suitable variants and versions recommended by theCuPyandPyTorchdevelopers for your specific CUDA environment.Installing CryoSieveAfter preparing CuPy and PyTorch according to your CUDA environment, it is crucial to activate it before proceeding with the CryoSieve installation.We recommend using the following command to activate it directly. (replace CRYOSIEVE_ENV with the name of your custom environment).conda activate CRYOSIEVE_ENVThen, we turn to the step of installing CryoSieve. CryoSieve can be installed either viapiporconda.To install CryoSieve usingpip, execute the following command:pip install cryosieveAlternatively, to install CryoSieve usingconda, execute the following command:conda install -c mxhulab cryosieveVerifying InstallationYou can verify whether CryoSieve has been installed successfully by running the following command:cryosieve -hThis should display the help information for CryoSieve, indicating a successful installation.TutorialQuickstart: A Toy ExampleTo validate your successful installation of CryoSieve and familiarize yourself with its functionalities, we highly recommend trying CryoSieve on thistoy example. Please follow the steps below:Download the dataset and place it into any directory of your choice, e.g.,~/toy/.Navigate to this directory by executing the following command:cd ~/toy/Initiate CryoSieve with the following command:cryosieve-core --i CNG.star --o my_CNG_1.star --angpix 1.32 --volume CNG_A.mrc --volume CNG_B.mrc --mask CNG_mask.mrc --retention_ratio 0.8 --frequency 40You may find explanation for each option ofcryosieve-corein the following section.When the--num_gpusparameter is used with a value larger than 1, CryoSieve's core program will leverage multiple GPUs to expedite the sieving process. It accomplishes this by using PyTorch'selastic launchfeature to initiate multiple processes. Each of these processes will use exactly one GPU.For instance, on a machine equipped with 4 GPUs, you can use the following command to run the toy example:cryosieve-core --i CNG.star --o my_CNG_1.star --angpix 1.32 --volume CNG_A.mrc --volume CNG_B.mrc --mask CNG_mask.mrc --retention_ratio 0.8 --frequency 40 --num_gpus 4Upon successful execution, the command will generate two star files,my_CNG_1.starandmy_CNG_1_sieved.star. These files contain the information of the remaining particles and the sieved particles, respectively. You can compare them with the providedCNG_1.starandCNG_1_sieved.starfiles. If executed correctly, they should contain the same particles.Processing Real-World DatasetIn this section, we provide a hands-on example of how to utilize CryoSieve for processing the final stack in a real-world experimental dataset.Download the DatasetFor this tutorial, we'll be using the final particle stack from theEMPIAR-11233dataset. This dataset includes a final particle stack of TRPM8 bound to calcium, collected on a 300 kV FEI Titan Krios microscope.To download the final particle stack, navigate to your desired working directory and execute the following command:wget -nH -m ftp://ftp.ebi.ac.uk/empiar/world_availability/11233/data/Final_Particle_Stack/Upon completion, you'll find a new directory namedXXX/data/Final_Particle_Stackin your working directory. This directory contains a star file with all particle information and an mrcs file representing the final stack.Additionally, you'll need a mask file. You can generate a mask file using any cryo-EM software, based on the reconstructed volume. If you prefer not to generate a mask file, we've provided one used in our experiments which you can download from thislink. Once you have the mask file, move it into theFinal_Particle_Stackdirectory.You can find additional demonstration data, along with expected results, inthis repository.Iterative Reconstruction and SievingTo achieve optimal results with real-world datasets, the sieving process generally involves several iterations. In each iteration, we perform 3D reconstruction (and perhaps postprocessing to derive the Fourier Shell Correlation (FSC) curve and resolution). We then apply CryoSieve to sieve a fraction of the particles based on the reconstructed map. The highpass cut-off frequency typically increases with each round.For your convenience, we've developed an automatic commandcryosievewhich performs all these steps in a single run. To use it, please follow these steps:Change the working directory toXXX/data/Final_Particle_Stack:cd XXX/data/Final_Particle_StackOur automatic program currently uses Relion for 3D reconstruction and postprocessing. Therefore, make sure thatrelion_reconstructorrelion_reconstruct_mpiandrelion_postprocessare accessible. Once confirmed, run the following command:cryosieve --reconstruct_software relion_reconstruct --postprocess_software relion_postprocess --i diver2019_pmTRPM8_calcium_Krios_6Feb18_finalParticleStack_EMPIAR_composite.star --o output/ --mask mask.mrc --angpix 1.059 --num_iters 10 --frequency_start 40 --frequency_end 3 --retention_ratio 0.8 --sym C4For a detailed explanation of eachcryosieveoption, please refer to the following sectionOptions/Arguments ofcryosieve.The entire process may take over an hour, depending on your system resources. Multiple result files will be generated and saved in theoutput/directory. For instance, the_iter{n}.starfile contains particles that remain after the n-th sieving iteration, and the_postprocess_iter{n}folder houses the postprocessing result after the n-th iteration.[Recommended] Re-estimate poses by CryoSPARCThe objective of re-estimating poses using CryoSPARC is to prevent the unintentional transfer of information from the discarded particles to those that are retained. This process of re-estimating poses with CryoSPARC can be conducted either manually or through automation using a Python script. The script, namedcryosieve_auto_cryosparc.py, is provided by us in this repository.For manually re-estimating poses with CryoSPARC, particles must be imported using CryoSPARC’simport particle stackjob. This importation is from the_iter{n}.starfile, which contains particles remaining after the n-th sieving iteration. Subsequently, the process involves conducting sequentialab-initiojobs, followed by eitherhomogeneous refinementornon-uniform refinementjobs.Alternatively, users can utilize thecryosieve_auto_cryosparc.pyscript, available in this repository, to circumvent the labor-intensive manual operations in CryoSPARC. The prerequisite for executing thecryosieve_auto_cryosparc.pyscript is the loading of CryoSPARC’s environment variables. This can be accomplished by executing the commandeval $(cryosparcm env)in the shell. The successful loading of this environment can be verified by using thewhich pythoncommand. Upon successful execution, this command should point to a Python interpreter located within CryoSPARC’s installation path. Utilizing this Python interpreter, users can execute thecryosieve_auto_cryosparc.pyscript to automatically and sequentially carry outimport particle stack,ab-initio,homogenous refinement, andnon-uniform refinementjobs. These operations are performed for each particle stack listed in a sheet, which is provided as an argument to thecryosieve_auto_cryosparc.pyscript. For a detailed explanation of eachcryosieve_auto_cryosparc.pyoption, please refer to the following sectionOptions/Arguments ofcryosieve_auto_cryosparc.py.Options/Arguments ofcryosive-core,cryosieveandcryosieve_auto_cryosparc.pyOptions/Arguments ofcryosieve-coreThe programcryosieve-coreis the core particle sieving module.$ cryosieve-core -h
usage: cryosieve-core [-h] --i I --o O [--directory DIRECTORY] --angpix ANGPIX --volume VOLUME [--mask MASK] [--retention_ratio RETENTION_RATIO] --frequency
                      FREQUENCY [--balance] [--num_gpus NUM_GPUS]

CryoSieve core.

options:
  -h, --help            show this help message and exit
  --i I                 input star file path.
  --o O                 output star file path.
  --directory DIRECTORY
                        directory of particles, empty (current directory) by default.
  --angpix ANGPIX       pixelsize in Angstrom.
  --volume VOLUME       list of volume file paths.
  --mask MASK           mask file path.
  --retention_ratio RETENTION_RATIO
                        fraction of retained particles, 0.8 by default.
  --frequency FREQUENCY
                        cut-off highpass frequency.
  --balance             make retained particles in different subsets in same size.
  --num_gpus NUM_GPUS   number of GPUs to execute the cryosieve program, 1 by default.Options/Arguments ofcryosieveThe programcryosieveis an integreted program iteratively calling relion andcryosieve-coreto do sieving process.$ cryosieve -h
usage: cryosieve [-h] --reconstruct_software RECONSTRUCT_SOFTWARE [--postprocess_software POSTPROCESS_SOFTWARE] --i I --o O --angpix ANGPIX [--sym SYM]
                 [--num_iters NUM_ITERS] [--frequency_start FREQUENCY_START] [--frequency_end FREQUENCY_END] [--retention_ratio RETENTION_RATIO] --mask MASK
                 [--balance] [--num_gpus NUM_GPUS]

CryoSieve: a particle sorting and sieving software for single particle analysis in cryo-EM.

options:
  -h, --help            show this help message and exit
  --reconstruct_software RECONSTRUCT_SOFTWARE
                        command for reconstruction.
  --postprocess_software POSTPROCESS_SOFTWARE
                        command for postprocessing.
  --i I                 input star file path.
  --o O                 output path prefix.
  --angpix ANGPIX       pixelsize in Angstrom.
  --sym SYM             molecular symmetry, c1 by default.
  --num_iters NUM_ITERS
                        number of iterations for applying CryoSieve, 10 by default.
  --frequency_start FREQUENCY_START
                        starting threshold frquency, in Angstrom, 50A by default.
  --frequency_end FREQUENCY_END
                        ending threshold frquency, in Angstrom, 3A by default.
  --retention_ratio RETENTION_RATIO
                        fraction of retained particles in each iteration, 0.8 by default.
  --mask MASK           mask file path.
  --balance             make remaining particles in different subsets in same size.
  --num_gpus NUM_GPUS   number of gpus to execute CryoSieve core program, 1 by default.There are several useful remarks:CryoSieve utilizes theRECONSTRUCT_SOFTWAREin its reconstruction command. This enables you to enhance the speed of the reconstruction step through multiprocessing by using the option--reconstruct_software "mpirun -n 5 relion_reconstruct_mpi". Additionally, you can further boost the reconstruction speed by using the option--reconstruct_software "mpirun -n 5 relion_reconstruct_mpi --j 20", leveraging multi-threading.If POSTPROCESS_SOFTWARE is not given, CryoSieve will skip the postprocessing step. Notice that postprocessing is not necessary for the sieving procedure.Sincerelion_reconstructuse current directory as its default working directory, user should ensure thatrelion_reconstructcan correctly access the particles.Options/Arguments ofcryosieve_auto_cryosparc.pyThecryosieve_auto_cryosparc.pyis a Python script designed to automatically and sequentially execute a series of operations in CryoSPARC, namelyimport particle stack,ab-initio,homogenous refinement, andnon-uniform refinementjobs.usage: cryosieve_auto_cryosparc.py [-h] [--particles_sheet PARTICLES_SHEET] [--cryosparc_user_id CRYOSPARC_USER_ID]
                                   [--cryosparc_project_uid CRYOSPARC_PROJECT_UID] [--cryosparc_workspace_uid CRYOSPARC_WORKSPACE_UID]
                                   [--cryosparc_lane CRYOSPARC_LANE] [--molecular_symmetry MOLECULAR_SYMMETRY]

The cryosieve_auto_cryosparc.py is a Python script designed to automate CryoSPARC operations via the command line. Its purpose is to
bypass the labor-intensive manual processes.

options:
  -h, --help            show this help message and exit
  --particles_sheet PARTICLES_SHEET
                        a file containing a list of starfiles; each starfile corresponds to a single-particle dataset; NOTE, absolute
                        directory is mandatory
  --cryosparc_user_id CRYOSPARC_USER_ID
                        the E-mail address of the user of CryoSPARC
  --cryosparc_project_uid CRYOSPARC_PROJECT_UID
                        the project UID in cryoSPARC
  --cryosparc_workspace_uid CRYOSPARC_WORKSPACE_UID
                        the workspace UID in cryoSPARC
  --cryosparc_lane CRYOSPARC_LANE
                        the lane for computing resource in cryoSPARC
  --molecular_symmetry MOLECULAR_SYMMETRY
                        molecular symmetryRelease NoteVersion 1.2.4:Fix the bug occurring with starfile >=0.5. CryoSieve now requires starfile >=0.4, <0.5.Support gloo backend for multi-GPU version of cryosieve-core.Fix the bug of SequentialSampler occurring with pytorch >=2.Version 1.2.3: Add new video tutorial.Version 1.2.1: fix the bug occurring when multiple optic groups are present in a star file.Version 1.2: Initial stable release.