Package: gardnersnake
gardnersnakeUtilities for writing concise snakemake workflowsTable of ContentsIntroductionRequirementsInstallationThe Configuration ClassCommand Line Tools1. IntroductionSnakemakeis an incredibly powerful workflow manager that enables computational biologists to produce clear, reproducible, and modular analysis pipelines using a familiar Python-based grammar.
Unfortunately, the bioinformatics tools that we'd like to utilize inside of our Snakemake workflows are often a bit less well-behaved.
Gardnersnake is a small package built on the python standard library that aims to make handling this wide variety of tools easier and more compact, especially when working on cluster-based systems.2. RequirementsThe gardnersnake package requires Python >= 3.7.0. Additionally, gardnersnake depends onpyyaml 6.03. InstallationGardnersnake can be installed most conveniently via pip and thePython Package Index(PyPi). The most recent version isgardnersnake==0.4.1pipinstallgardnersnakeThis repo can also be cloned and built from source using thebuildpackage.4. The Configuration ClassBasic Usage of ConfigurationThe basis for much of gardnersnake is theConfigurationclass object implemented as part ofgardnersnake.core. To initialize the object, we simply pass as its only argument, the name of a yaml configuration file for the snakemake workflow. This can be a string or apathlib.Path object.fromgardnersnake.coreimportConfiguration# import the Configuration classfrompathlibimportPath# provides Path objects from the python standard libraryconfig_filepath=Path("~/path/to/my/config.yaml").resolve()# build a pathlib.Path object and resolve the absolute pathcfg=Configuration(filepath=config_filepath)# instantiate the Configuration objectcfg.load()# load the configuration dataThe YAML configuration fileTo interface cleanly withgardnersnake, We have implemented a configuration parsing system inConfiguration.load()which require YAML files of a certain format. As currently implemented,gardnersnakeexpects to recieve a YAML file comprising one or more documents separated by---. Documents may be in any order, but should be of one of two types: aRULE_CONFIGor aGLOBAL_CONFIG, specified at the top level byDOC_TYPE:field (see below). Configuration expects at most oneGLOBAL_CONFIGdocument, and will only remember the last of document of this type defined in the yaml file. There may be as manyRULE_CONFIGdocuments as desired (typically one per rule in the snakemake file, although not enforced). Each document of typeRULE_CONFIGmust have a unique fieldrule_name:. Repeats will cause the Configuration to overwrite previously read documents from earlier in the file.Below is an example of a very basic configuration file with both aGLOBAL_CONFIGdocument, and twoRULE_CONFIGdocuments.---# File Name: basic-config.yaml# Created On: 2022-07-21DOC_TYPE:"GLOBAL_CONFIG"analysis_name:"my-snakemake-workflow"workding_dir:"/my/working/directory"files:{some_reference_file:"ref.txt",some_metadata:"meta.txt"}workflow_log:"my-snakemake-workflow.log"---DOC_TYPE:"RULE_CONFIG"rule_name:"RuleA"parameters:{par1:100,par2:"--verbose"}resources:{walltime:"2:00:00",nodes:1,...}---DOC_TYPE:"RULE_CONFIG"rule_name:"RuleB"parameters:{}resources:{walltime:"0:30:00",nodes:1,...}For each document type, there are a number of required fields. These are outlined in the table below.FieldRequired in GLOBAL_CONFIGRequired in RULE_CONFIGShort DescriptionDOC_TYPEYes (DOC_TYPE: "GLOBAL_CONFIG" )Yes (DOC_TYPE: "RULE_CONFIG" )Specifies the type of document to be read into Configurationanalysis_nameYes (String, empty = "" )NoHigh level workflow name for logs/ reports/ etc/ reproducibility and clarity/.working_directoryYes (String, empty = "" )NoSpecifies the working directory for the snakemake workflow. to be used withworkdir:in a Snakefile.filesYes (Dict, empty = {} )NoA dictionary of files and their metadata to be used in the analysis.rule_nameNoYes (String, empty="", must be unique )A unique identifier for each rule configuration document. Should generally correspond to a rule in the Snakefile file.parametersNoYes (Dict, empty = {} )A dictionary to contain the non-file arguments to a given Snakefile rule.resourcesNoYes (Dict, empty = {} )A dictionary to contain the resource requirements for a given Snakefile rule.Interfacing Configuration with SnakefilesIf we take the above configuration yaml file as an example, we can illustrate its integration with a given snakemake Snakefile. One of the main features of theConfigurationclass beyond abstraction and portability is the implementation of dot-style attribute access so that we don't have to chain together long dictionary access calls to get deep into the structure of our workflow config data object.A psuedo-Snakefile that interfaces with the abovebasic-config.yaml:fromgardnersnake.coreimportConfigurationfrompathlibimportPathconfig_filepath=Path("basic-config.yaml").resolve()cfg=Configuration(filepath=config_filepath)cfg.loadGLOBALS=cfg.global_params# Rule A:ruleA_params=cfg.get_rule_params("RuleA")ruleA:input:GLOBALS.files.some_reference,GLOBALS.files.some_metadataoutput:"some-output.txt"params:**(ruleA_params.parameters),log=GLOBALS.misc.workflow_logresources:**(ruleA_params.resources)shell:"mycommand{input}-o{output}--trys={params.par1}{params.par2}2>{params.log}"# Rule B:ruleB_params=cfg.get_rule_params("RuleB")ruleBfiles=[Path("plots/plot-out-B")/extforextin(".png",".pdf",".rData")]ruleB:input:"some-output.txt",GLOBALS.files.some_metadataoutput:"returncode.out"params:**(ruleB_params.parameters),filestocheck=ruleBfiles,targetdir=str(Path("plots/")),log=GLOBALS.misc.workflow_logresources:**(ruleB_params.resources)shell:"Rscipt plotting.R -i{input}2>{params.log}\&& check_directory -o{output}{params.filestocheck}{params.targetdir}2>{params.log}"One important note about the above Snakefile. You can see that the field in the configuration fileworkflow_logis now accessed not at the top level but under the attributemisc. In the global configuration, every field which is not one of the three required fields is refiled under this miscillaneous attributes field with thier structure otherwise preserved. This is not the case in rule configurations because every non-file field specific to the rule is likely to fit well under parameters or resources. Because of this, all other fields in theRULE_CONFIGdocuments other than the required ones are ignored by theConfiguration.load()routine.5. Command Line Toolscheck_directoryMany bioinformatics tools produce directories of various structure with large numbers of output files. Rather than require Snakemake to keep track of these outputs as global outputs, thecheck_directorycommand-line utility validates output directories against a known set of files, and returns a small file containing a return code (0) if the directory of interest was successfully validated.check_directorythrows an error and does not write the return code file if it is unable to validate the contents according to the given requirements.The options and requirements are specified in the usage message and can be retrieved using the-hor--helpflags.usage: check_directory [-h] [-o OUT] FILES [FILES ...] DIR

Validates dynamic directory contents against expectations. If each of the required files is present in the directory, the
program writes '0' to the output file. otherwise, it returns an Error

positional arguments:
  FILES                 set of filepaths to check against dir contents
  DIR                   filepath of directory to verify

optional arguments:
  -h, --help            show this help message and exit
  -o OUT, --output OUT  name of return code output filePositional Options\FILES[required] a list of whitespace separated files to search for in the passed directory. these file names should be specified without their path extensions (i.e. a file whose full path is/home/user/analysis/myoutputs/output1.txtshould be passed asoutput1.txtif theDIRis indicated to be/home/user/analysis/myoutputs/)DIR[required] is the full path of the directory to verify.~/conventions are acceptable but shell variable syntax such as$WORKDIRare not supported. Relative path functionality remains in active development but is not guaranteed to work as of the current version (0.4.1)Flagged Options\--ouput -o[required] specifies the name of the file generated (containing the return code) when the passed directory is successfully validated.check_dir-orc.outoutput1.txtoutput2.txt~/myanalysis/outputs/which should return a file calledrc.outif the folder~/myanalysis/outputs/has at least two files -->output1.txtandoutput2.txt