Package: cesped
CESPED: Utilities for the Cryo-EM Supervised Pose Estimation DatasetCESPED, is a new dataset specifically designed for Supervised Pose Estimation in Cryo-EM. You can check of manuscript athttps://arxiv.org/abs/2311.06194.Installationcesped has been tested on python 3.11. Installation should be automatic using pippip install cesped
#Or directy from the master branch
pip install git+https://github.com/rsanchezgarc/cespedor cloning the repositorygit clone https://github.com/rsanchezgarc/cesped
cd cesped
pip install .Basic usageParticlesDataset classIt is used to load the images and poses.Get the list of downloadable entriesfrom cesped.particlesDataset import ParticlesDataset
listOfEntries = ParticlesDataset.getCESPEDEntries()Load a given entrytargetName, halfset = listOfEntries[0] #We will work with the first entry only

dataset = ParticlesDataset(targetName, halfset)For a rapid test, usetargetName="TEST"andhalfset=0Use it as a regular datasetdl = DataLoader(dataset, batch_size=32)
for batch in dl:
  iid, img, (rotMat, xyShiftAngs, confidence), metadata = batch
  
  #iid is the list of ids of the particles (string)
  #img is a batch of Bx1xNxN images
  #rotMat is a batch of rotation matrices Bx3x3
  #xyShiftAngs is a batch of image shifts in Angstroms Bx2
  #confidence is a batch of numbers, between 0 and 1, Bx1
  #metata is a dictionary of names:values for all the information about the particle
  
  #YOUR PYTORCH CODE HERE
  predRot = model(img)
  loss = loss_function(predRot, rotMat)
  loss.backward()
  optimizer.step()
  optimizer.zero_grad()Once your model is trained, you can update the metadata of the ParticlesDataset and save it so that it can be used in cryo-EM softwarefor iid, pred_rotmats, maxprob in predictions:
    #iid is the list of ids of the particles (string)
    #pred_rotmats is a batch of predicted rotation matrices Bx3x3
    #maxprob is a batch of numbers, between 0 and 1, Bx1, that indicates the confidence in the prediction (e.g. softmax values)

    particlesDataset.updateMd(ids=iid, angles=pred_rotmats,
                              shifts=torch.zeros(pred_rotmats.shape[0],2, device=pred_rotmats.device), #Or actual predictions if you have them
                              confidence=maxprob,
                              angles_format="rotmat")
particlesDataset.saveMd(outFname) #Save the metadata as an starfile, a common cryo-EM formatFinally, evaluation can be computed if the predictions for the halfset 0 and halfset 1 were saved using the evaluateEntry script.python -m cesped.evaluateEntry  --predictionType SO3 --targetName 11120  \
--half0PredsFname particles_preds_0.star  --half1PredsFname particles_preds_1.star \
--n_cpus 12 --outdir evaluation/evaluateEntry usesRelionfor reconstruction, so you will need to install it and
edit the config filedefaultRelionConfig.yamlor provide, via command
line arguments, where Relion is installed--mpirun /path/to/mpirun  --relionBinDir /path/to/relion/binAlternatively, you can build asingularityimage, using the
definition file we providerelionSingularity.defsingularity build relionSingularity.sif relionSingularity.defand edit the config file to point where the singularity image file is located, or use the command line argument--singularityImgFile /path/to/relionSingularity.sifImage2Sphere experimentsThe experiments have been implemented usinglightningand lightingCLI. You can find the configuration files
located at :YOUR_DIR/cesped/configs/You can also find it as:import cesped
cesped.default_configs_dirTrainIn order to train the model on one target, you runpython -m cesped.trainEntry --data.halfset <HALFSET> --data.targetName <TARGETNAME> --trainer.default_root_dir <OUTDIR>with<HALFSET>0 or 1 and<TARGETNAME>one of the list that can be found usingParticlesDataset.getCESPEDEntries()Some available targets includeTEST. A small subset of EMPIAR-10166The EMPIAR-10166The EMPIAR-11120The EMPIAR-10280The EMPIAR-10409Do not forget to change the configuration files or to provide different values via the command line or environmental
variables. In addition,[--config CONFIG_NAME.yaml]also allows overwriting the default values using (a/several) custom
yaml file(s). Use-hto see the list of configurable parameters. Some of the most important ones are.trainer.default_root_dir. Directory where the checkpoints and the logs will be saved,
fromdefaultTrainerConfig.yamloptimizer.lr. The learning rate, fromdefaultOptimizerConfig.yamldata.benchmarkDir. Directory where the benchmark entries are saved, fromdefaultDataConfig.yaml. It is recommended
to change this in the config file.data.num_data_workers. Number of workers for data loading, fromdefaultDataConfig.yamldata.batch_size. fromdefaultDataConfig.yamlInferenceBy default, when usingpython -m cesped.trainEntry, inference on the complementary halfset is done on a single GPU
after training finishes, and the starfile with the predictions can be found at<OUTDIR>/lightning_logs/version_<\d>/predictions_[0,1].star. In order to manually run the pose prediction
code (and to make use of all GPUs) you can runpython -m cesped.inferEntry --data.halfset <HALFSET> --data.targetName <TARGETNAME> --ckpt_path <PATH_TO_CHECKPOINT> \
--outFname /path/to/output/starfile.starEvaluationAs before, evaluation can be computed if the predictions for the halfset 0 and halfset 1 were saved using the evaluateEntry script.python -m cesped.evaluateEntry  --predictionType SO3 --targetName 11120  \
--half0PredsFname particles_preds_0.star  --half1PredsFname particles_preds_1.star \
--n_cpus 12 --outdir evaluation/##API
For API documentation run
pdoc --http : .Relion SingularityA singularity container for relion_reconstruct with MPI support can be built with the following command.singularity build relionSingulary.sif relionSingulary.defThen, Relion reconstruction can be computed with the following command:singularity exec relionSingulary.sif mpirun -np 4 relion_reconstruct_mpi --ctf --pad 2 --i input_particles.star --o output_map.mrc
#Or the following command
./relionSingulary.sif  4 --ctf --pad 2 --i input_particles.star --o output_map.mrc #This uses 4 mpisHowever, typical users will not need to execute the container manually. Everything happens transparently within the evaluateEntry.py script