Package: limes-x
Workflows on demand!For the impatientDependenciesPythonSingularityGit (to setup existing modules)Snakemake (to setup existing modules)Setup#!/bin/bashpipinstalllimes-x

gitclonehttps://github.com/hallamlab/Limes-compute-modules.git
python./Limes-compute-module/setup_modules.py./lx_refRun#!/bin/python3.10importlimes_xaslxmodules=lx.LoadComputeModules("./Limes-compute-modules/metagenomics")wf=lx.Workflow(compute_modules=modules,reference_folder="./lx_ref",)wf.Run(workspace="./test_workspace",targets=[Item('metagenomic gzipped reads'),Item('metagenomic assembly'),Item("metagenomic bin"),Item("checkm stats"),Item('bin taxonomy table'),Item('assembly taxonomy table'),Item('genomic annotation'),],given=[lx.InputGroup(group_by=(Item("sra accession"),"SRR19573024"),children={Item("username"):"Steven"},# use "whoami" in bash)],executor=lx.Executor(),)DependenciesAnaconda (optional, but recommended)faster version (Mamba); Use "mamba" instead of "conda" belowplain AnacondaPythonNOTE:Most compute servers and linux distributions will have python installed alreadyinstall with Anacondaconda create -n test_env python=3.10feel free to pick a better name than "test_env"SingularityNOTE:Most compute servers will have singularity installed alreadyinstall with Anacondaactivate the conda environment:conda activate test_envconda install singularitymanual installGityou may already have gitusegit --versionin the console to find outotherwise, here's a tutorialSetupNOTE:We are working on a conda package. Meanwhile, this will work in a conda environment with python installed.pip install limes-xRunning workflowsCompute ModulesLimes-x encapsulates the complexity of workflows by surfacing a declarative syntax that allows you to focus onwhatyou want and worry less abouthowto achieve it. This is made possible by compute modules that provide conversions between datatypes such as changing the format of an image or assembling a metagenome from Illumina sequences.Limes-x finds the set of compute modules required to convert the given inputs to the desired inputs. This set of modules is then joined together into an execution-ready workflow.A list of available compute modules can be found at this repoUse thesetup_modulesscript to install each module's dependencies and reference databases using Singularity and Snakemake.gitclonehttps://github.com/hallamlab/Limes-compute-modules.git
python./Limes-compute-module/setup_modules.py./lx_refMinimal execution exampleCreate a workflow with the compute modules found in./Limes-compute-modules/metagenomics#!/bin/python3.10importlimes_xaslxmodules=lx.LoadComputeModules("./Limes-compute-modules/metagenomics")wf=lx.Workflow(compute_modules=modules,reference_folder="./lx_ref",)Run the workflow by indicating the desired data products and giving an SRA accession string as the input. Asequence read archive(SRA) accession points to DNA sequnces hosted by the National Center for Biotechnology Information.NOTE:While multipleInputGroupscan be provided, each must have identical formats (sameItems). This is a bug.wf.Run(workspace="./test_workspace",targets=[Item('metagenomic gzipped reads'),Item('metagenomic assembly'),Item("metagenomic bin"),Item("checkm stats"),Item('bin taxonomy table'),Item('assembly taxonomy table'),Item('genomic annotation'),],given=[lx.InputGroup(group_by=(Item("sra accession"),"SRR19573024"),children={Item("username"):"Steven"},)],executor=lx.Executor(),)Workspace format:├── ./test_workspace
    ├── comms.json
    ├── comms.lock
    ├── limesx_src.tgz
    ├── input_paths.tsv
    ├── workflow_state.json

    ├── <module name>--######
        ├── context.json
        ├── result.json
        ├── <module ouputs>

    ├── inputs
        ├── <soft links to each input file/folder>
        
    ├── outputs
        ├── <data type (Item)>
            ├── <each instance of Item produced>Different execution environmentsThe default executor will run modules locally.wf.Run(...executor=lx.Executor(),)We can use theHpcExecutorto interface with high performance compute clusters (HPC) by specifying how to interact with the cluster's scheduler. Here, we write the callback function,schedule_job, which will be called when a compute module needs to be executed on the cluster. The executor will pass in ajobobject to our function that provides ashell, therun_commandto execute the compute module.fromlimes_ximportJobdefschedule_job(job:Job)->tuple[bool,str]:returnjob.Shell(f"""\<schedule a job with the following command>{job.run_command}""")ex=lx.HpcExecutor(hpc_procedure=schedule_job,tmp_dir_name="TMP")wf.Run(...executor=ex,)tmp_dir_nameis the environment variable that stores the path to the temporary directory on the worker node. TheHpcExecutorwill transfer all required files/folders there before running the job.Below is an example withslurm, the scheduler used by the Digital Alliance of Canada's Cedar cluster.defget_res(job:str,manifest:dict,cores,mem):_cores,_hrs,_mem={"download_sra":lambda:(cores,4,mem),"extract_mg-reads":lambda:(cores,4,mem),"metagenomic_assembly":lambda:(cores,12,mem),"metagenomic_binning":lambda:(cores,24,mem),"taxonomy_bin":lambda:(cores,4,mem),"taxonomy_assembly":lambda:(cores,4,mem),"checkm_on_bin":lambda:(cores,1,mem),"annotation_metapathways":lambda:(cores,8,mem),}.get(job,lambda:(cores,4,mem))()return(_cores,_hrs,_mem)defslurm(job:lx.Job)->tuple[bool,str]:p=job.context.paramstime.sleep(2*random.random())job_name=job.instance.step.namejob_id=job.instance.GetID()cores,hrs,mem=get_res(job_name,job.context.manifest,p.threads,p.mem_gb)returnjob.Shell(f"""\sbatch --wait --account={ALLOC}\--job-name="lx-{job_name}:{job_id}"\--nodes=1 --ntasks=1\--cpus-per-task={cores}--mem={mem}G --time={hrs}:00:00\--wrap="{job.run_command}"\""")ex=lx.HpcExecutor(hpc_procedure=slurm,tmp_dir_name="SLURM_TMPDIR")wf.Run(...executor=ex,)Making new modulesFirst, use Limes to generate a template in the folder where you want to keep all of your compute modules.importlimes_xaslxlx.ModuleBuilder.GenerateTemplate(modules_folder="./compute_modules",name="a descriptive name",)├── ./compute_modules
    ├── <module name>
        ├── lib
            ├── definition.py
        ├── setup
            ├── setup.smk

    ├── <module name>
        ├── lib
        ├── setup
    .
    .
    .Thesetupfolder contains the snakemake workflow required to install the module. Thelibfolder must contain (or link to) all scripts required by the compute module. Limes will invoke the module by loadingdefinition.pyand looking for aMODULEvariable that holds the compute module.# template definition.pyfrompathlibimportPathfromlimes_ximportModuleBuilder,Item,JobContext,JobResultA=Item('a')B=Item('b')DEPENDENCY="image.sif"defprocedure(context:JobContext)->JobResult:input_path=context.manifest[A]output_path=context.output_folder.joinpath('copied_file')context.shell(f"cp{input_path}{output_path}")returnJobResult(manifest={B:Path(output_path)},)MODULE=ModuleBuilder()\.SetProcedure(procedure)\.AddInput(A,groupby=None)\.PromiseOutput(B)\.Requires({DEPENDENCY})\.SuggestedResources(threads=1,memory_gb=4)\.SetHome(__file__,name=None)\.Build()For some examples, take a look at this repo.