Package: cryoemservices
cryoem-servicesServices and configuration for cryo-EM pipelines.This package consists of a number of services to process cryo-EM micrographs,
both for single particle analysis and tomography,
using a range of commonly used cryo-EM processing software.
These services can be run independently to process data,
or as part of a wider structure for performing live analysis during microscope collection.
For live analysis, this package integrates with a package
for transferring and monitoring collected data,Murfey,
and a database for storing processing outcomes,ISPyB.To run these services the software executables being called must be installed.
These do not come with this package.Tomography processingThe tomography processing pipeline consists of:Motion correctionCTF estimationTomogram alignmentTomogram denoising usingTopazSingle particle analysisThe single particle analysis pipeline produces a project
that can be opened and continued usingCCP-EM doppioorRelion.The processing pipeline consists of:Motion correctionCTF estimationParticle picking(Optionally) Ice thickness estimationParticle extraction and rebatching2D classification using RelionAutomated 2D class selection using Relion3D classification using RelionServices currently availableThe following services are provided for running the pipelines:Utility services:ClusterSubmission: Submits zocalo wrappers to an HPC clusterDispatcher: Converts recipes into messages suitable for processing servicesImages: Creates thumbnail images for viewing processing outcomesISPyB: Inserts results into an ISPyB databaseNodeCreator: Creates Relion project files for the services runProcessing services:CrYOLO: Particle picking on micrographs usingcrYOLOCTFFind: CTF estimation on micrographs usingCTFFIND4Extract: Extracts picked particles from micrographsIceBreaker: Ice thickness estimation withIceBreakerMotionCorr: Motion correction of micrographs usingMotionCor2orRelionMotionCorrSlurm: MotionCor2 processing submitted to a slurm HPC clusterSelectClasses: Runs automated 2D class selection usingRelionand re-batches the particles from these classesSelectParticles: Creates files listing batches of extracted particlesTomoAlign: Tomogram reconstruction from a list of micrographs usingimodandAreTomoThere are also two zocalo wrapper scripts that can be run on an HPC cluster.
These perform 2D and 3D classification usingRelion.Running services using zocaloThe services in this package are run usingzocaloandpython-workflows.
To start a service run thezocalo.servicecommand and specify the service name.
For example, to start a motion correction service:$zocalo.service-sMotionCorrOnce started, these services will initialise and then wait for messages to be sent to them.
Messages are sent through a message broker,
currentlyRabbitMQis supported using pika transport inpython-workflows.
Individual processing stages can be run by sending a dictionary of the parameters,
but the processing pipelines are designed to run through recipes.A recipe is a specication of a series of steps to carry out,
and how these steps interact with each other.
Recipes for the current processing pipelines are provided in therecipesfolder.To run a recipe in python a dictionary needs to be provided consisting of
the recipe name and the parameters expected by the recipe.
The following snippet shows an example of the setup needed.
This will send a message to a runningDispatcherservice which
prepares the recipe for the processing services.importworkflows.transport.pika_transportasptexample_message={"recipes":["em-tomo-align"],"parameters":{"path_pattern":"micrograph_*.mrc","pix_size":"1",...},}transport=pt.PikaTransport()transport.connect()transport.send("processing_recipe",example_message)