Package: illusional.wdlgen
python-wdlgenWorkflow Description Languageis way to describe tasks and workflows in a "human readable and writable way". It was initially developed and offered byBroad Instituteto be paired with their workflow engineCromwell, however it has since been made open source with other engines such asToilandDNAnexus*.WARNINGThis module now only generates developmental WDL, this includes Directories and wrapping all inputs in an input block.
To use this generated WDL, you must use a version ofCromwell higher than 37.This module automatically includesversion developmentin the Workflow and Task outputs.
The guides below may not reflect the current version of this repository, but will be updated soon.This syntax is based on theDevelopmental Workflow Description Language specification.MotiviationI needed an easy way to generate someBASICWDL through some in memory objects, and I was using (a forkof)common-workflow-language/python-cwlgen, I figured I could open this up to see what use it has.Installationpip install illusional.wdlgenGeneral supportThis software is provided as-is, without warranty of any kind ... and so on.It's a pretty dumb wrapper that uses string interpolation to generate the structure. It wouldn't handle automatically escaping illegal characters.Generally it supports:Types - All types are represented as aWdlType, which can either be aPrimitiveType, or anArrayType(seegoal). Also supports the postfix quantifiers.Workflow creation (wdlgen.Workflow)manual imports (wdlgen.Workflow.WorkflowImport)inputs (wdlgen.Input)outputs (wdlgen.Output)calls:general call (wdlgen.WorkflowCall)scatter (wdlgen.WorkflowScatter(WorkflowCall[]))meta:wdlgen.Metaparameter_meta:wdlgen.ParameterMetaTask creation (wdlgen.Task) - This is based similar to howCWL constructs its commands.inputs:wdlgen.Inputoutputs:wdlgen.Outputruntime:wdlgen.Task.Runtimecommand:wdlgen.Task.Commandarguments:wdlgen.Task.Command.Argumentinputs:wdlgen.Task.Command.Inputmeta:wdlgen.Metaparameter_meta:wdlgen.ParameterMetaHow to useThis will give you abriefoverview on how tousepython-wdlgen. Goals are to improve the write a proper documentation spec, but if you have a moderate understanding of workflows in either CWL or WDL, this code will hopefully be fairly intuitive.Every class inherits from aWDLBasewhich means it must have aget_string()method which returns the string representation of the class, calling this on any children it may have.TypesAll types are represented as a WDLType, which has a parse method. It's a little overkill in some cases, but makes managing attributes a bit easier.parsed_string=wdlgen.WdlType.parse("String")# WdlType<PrimitiveType<String>>parsed_op_str=wdlgen.WdlType.parse("String?")# WdlType<PrimtiveType<String>>parsed_array=wdlgen.WDLType.parse("File[]")# WdlType<ArrayType<File>>parsed_ar_oq=wdlgen.WdlType(parse("Int?[]+"))# WdlType<ArrayType<Int?> (+)>You can also construct these manually:parsed_string=WdlType(PrimitiveType("String"))parsed_op_str=WdlType(PrimtiveType("String",optional=True))parsed_array=WdlType(ArrayType(WdlType(PrimitiveType("File"))))parsed_ar_q=WdlType(ArrayType(WdlType(PrimitiveType("Int"),optional=True),requires_multiple=True))Input / OutputInput:wdlgen.Input(data_type: WdlType, name: str, expression: str = None)Output:wdlgen.Output(data_type: WdlType, name: str, expression: str = None)both of which output something like:{WdlType} {name} [= {expression}]TaskA task is a collection of Inputs, Outputs and a Command that are identified by aname. Inputs and Outputs are as above. Note that you can use functions such asstdout()or other for the expression.If you don't want to play by these rules, don't include any inputs or outputs and just provide your whole string to the initializer for command.t=wdlgen.Task("task_name")t.inputs.append(wdlgen.Input(wdlgen.WdlType.parse("String"),"taskGreeting"))# command in next sectiont.outputs.append(wdlgen.Output(wdlgen.WdlType.parse("File"),"standardOut","stdout()"))CommandThe command is broken up similar to how CWL breaks its command generation up, by itself it has abase command. Each component has a corresponding input (else use thewdlgen.Task.Command.Argumentclass), optionality, position, prefix (and whether the value should be separated from prefix; think-o {val}vsoutputDir={val}) and potentially a default.Construct a command like the following:command=wdlgen.Task.Command("echo")command.inputs.append(wdlgen.Task.Command.CommandInput("taskGreeting",optional=False,position=None,prefix="-a",separate_value_from_prefix=True,default=None))command.inputs.append(wdlgen.Task.Command.CommandInput("otherInput",optional=True,position=2,prefix="optional-param=",separate_value_from_prefix=False,default=None))# t is the taskt.command=commandprint(command.get_string())This will result in the following WDL command:echo\-a${taskGreeting}\${"optional-param="+ otherInput}Task output:The combination of the task and command outputs:version development

task task_name {
  input {
    String taskGreeting
  }
  command {
    echo \
      -a ${taskGreeting} \
      ${"optional-param=" + otherInput}
  }

  output {
    File standardOut = stdout()
  }
}WorkflowYou should have moderate idea of the structure of WDL as there's no cleverness or abstraction done anywhere. Beware: there's also no checking attributes (to see if yourinputMapactually corresponds to inputs).The structure of a workflow is mw=wdlgen.Workflow("workflow_name")w.imports.append(wdlgen.Workflow.WorkflowImport("tool_file",""))w.inputs.append(wdlgen.Input(wdlgen.WdlType.parse("String"),"inputGreeting"))inputs_map={"taskGreeting":"inputGreeting"}w.calls.append(wdlgen.WorkflowCall("Q.namspaced_task_identifier","task_alias",inputs_map))w.outputs.append(wdlgen.Output(wdlgen.WdlType.parse("File"),"standardOut","task_alias.standardOut")Which outputs:version development

import "tools/tool_file.wdl"

workflow workflow_name {
  input { 
    String inputGreeting
  }
  call Q.namspaced_task_identifier as task_alias {
    input:
      taskGreeting=inputGreeting
  }
  output {
    File standardOut = task_alias.standardOut
  }
}Known limitationsI'm not a fan of the string interpolation generation of WDL that this module does. I think trying to build anAbstract syntax treeand then there should be something that convert that into the DSL that WDL uses.You could also cause syntax errors in generated WDL by providing illegal characters.GoalsImprove code-level documentation.Increase the testing coverage + quality of unit tests.Better represent the WDL spec.Find an easier distribution / release method - such as PIP.Automate testing and delivery through TravisCI / CircleCI or similar.Validate each value byWDL's language specifications.Add support for structsLong goalsWrite a documentation site.Make classes convert into AST and then into DSL.Issues and Pull RequestsFeel free to log issues and make pull requests. I make no guarantee to the existence or timeliness of replies.Links:WDL description:https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md