Package: sevenbridges-cwl-draft2-upgrader
SevenBridges CWL Draft2 Upgrader (BETA)SevenBridges CWL Draft2 Upgrader is a Python3 package that comes with three command line tools:sbg_cwl_upgradersbg_cwl_decomposersbg_validate_js_cwl_v1Command line toolsbg_cwl_upgraderallows you to upgrade CWL
tools and workflows written in sbg:draft-2 to CWL v1.0 or v1.1. The source CWL can be on your local machine or on the Seven Bridges platform and the
converted CWL can be output to your local machine or onto the Seven Bridges platform. For more details seethis section.Note that the conversion process is semi-automated, manual intervention may be required. For more details see theknown limitations section.Manual part of the conversion process may be tedious and require changes to the CWL code. Therefore, strong understanding of the CWL syntax is desirable.Command line toolsbg_cwl_decomposerinstalls individual components of a workflow and re-links them back in the workflow. For more details see theusage section.Command line toolsbg_validate_js_cwl_v1checks a CWL document for non-strict syntax in JavaScript expressions and produces warnings. For more details see theusage section.InstallationThis tool requires Python3. In general, it is helpful to install user space
Python applications invirtual environments.Install using pipThe package can be installed withpip, usingpython3 -m pip install sevenbridges-cwl-draft2-upgraderInstall from source codeInstall from source by cloning the repo, usinggit clone https://github.com/sbg/sevenbridges-cwl-draft2-upgrader.git
python3 -m pip install ./sevenbridges-cwl-draft2-upgraderUsagePackage commands use Platform authentication by reading default config from
the~/.sevenbridges/credentialsfile.
For additional ways of providing authentication, please consultsbg_cwl_upgrader --help.Save upgraded CWL to a local fileAs an example, we'll take a publicly available workflow on the Seven Bridges platform, the
GATK4 WES workflow, convert it to CWL v1.0 and save it locally:sbg_cwl_upgrader -i admin/sbg-public-data/whole-exome-sequencing-bwa-gatk-4-0 -o wes.cwlThe arguments to-iand-oare interpreted as local CWL files if they end
in.cwl,.yml,.yaml, or.jsonand Apps on the Seven Bridges platform otherwise.Save upgraded CWL to a Platform projectWe can also directly upload the converted CWL1.0 workflow to the Platform:sbg_cwl_upgrader -i admin/sbg-public-data/whole-exome-sequencing-bwa-gatk-4-0 -o username/usernames-demo-project/wesYou will additionally be asked if you want to decompose the workflow after installation.Save upgraded CWL in version v1.1The CWL version of the upgraded app can be controlled with-cparameter:sbg_cwl_upgrader -i admin/sbg-public-data/whole-exome-sequencing-bwa-gatk-4-0 -c v1.1 -o username/usernames-demo-project/wesDecompose a Platform workflowSometimes, you want all workflow components to be available in the same project as the workflow. This can be done using thesbg_cwl_decomposertool.This tool will:Find all component tools of a workflowInstall them in the project where the workflow is installedInsert them in the workflow so they are editable in that projectExample command:sbg_cwl_decomposer -a username/usernames-demo-project/wesDecompose a local workflowSometimes, you want all workflow components to be available under thestepsfolder, next to the main workflow. This can be done using thesbg_cwl_decomposertool.This tool will:Find all component tools of a workflowInstall them in thestepsfolder alongside the main workflowInsert them in the workflow as strings (relative paths to the main workflow)Example command:sbg_cwl_decomposer -a workflow.cwlRecommended draft2 to CWL1.0 upgrade flowConvert a Platform draft2 workflow to CWL1.0 and run it on the Seven Bridges PlatformPrepare a development project and copy the workflow there.Run a task in the development project with the draft2 workflow.Run the upgrader script with the same app ID for-iand-oparameters to create a new revision over the draft2 wf:sbg_cwl_upgrader -i <app_id> -o <app_id> -d -u.This will upgrade the workflow and the tools/subworkflows, and install them in the project as separate apps.Rerun the task that used the draft2 workflow with the same inputs.Debug failures. Consult theknown limitations sectionand do a manual interventionIf a tool in the workflow requires modifications, find the tool in the project, apply a fix there, and update the tool in the workflow. Good practice is to test the modified tool individually before updating in the workflow.Once the CWL1.0 task completes, check if all the expected outputs are produced properly.Convert a Platform draft2 workflow to CWL1.0 and run it using cwltoolRun the upgrader script with the-oparameter pointing to a local file:sbg_cwl_upgrader -i <app_id> -o /path/to/app.cwl -d -v.This will upgrade the workflow and the tools/subworkflows, and install them in thestepsfolder alongside the main workflow.Validate JS expressions in the workflow -check this section.Validate the workflow withcwltool --validate /path/to/app.cwland check for potential errors - see theportability notessection.Prepare workflow inputs for local execution and describe them in<name>_inputs.jsonfile (can be either JSON or YAML).Run with cwltool usingcwltool /path/to/app.cwl <name>_inputs.json.Debug failures. Consult theknown limitations sectionand do a manual intervention.If a tool in the workflow requires modifications, find the tool in thestepsfolder, apply a fix there, and rerun the workflow. Good practice is to test the modified tool individually before updating in the workflow.Once the CWL1.0 task completes, check if all the expected outputs are produced properly.Known limitationsConversion notesThe conversion process is semi-automated, meaning some differences between CWL versions can't be automatically resolved and require manual intervention.
Manual intervention may be required when:Tool parses thejob.jsonfile, which is a Seven Bridges Platform specific log file. This should be avoided in tools.File inInitialWorkdirRequirementis created in a subfolder, e.g.entryname: foo/bar. Files need to be created directly in the working directory.Glob is catching files outside the working directory, e.g.glob: $(inputs.input.path). In CWL, every job has its own working directory in order to keep things clean. Glob should only capture files inside the working directory (meaning tools should only write outputs in the working directory).JS expression is missing areturnstatement. Every expression should return something (at leastnull), and must usereturn.JS expression is editing theinputsvariable and then accessing it. The tool converts$job.inputstoinputs, so in CWL1.0 JS expressions should no longer redefine theinputsvariable.The converter is not fully tested on upgrading mixed-version workflows.Portability notesThe CWL standard specifies that glob patterns must follow POSIX(3) pattern matching and JS expressions should follow ES5.1 strict syntax. The Seven Bridges platform allows some additional features for glob and JS, which may not work on executors that strictly follow the CWL standard. Common examples include:Glob pattern with brace expand syntaxnot capturing outputs in cwltool, e.g.glob: {*.txt,*.pdf}. Using brace expand pattern (curly brackets) is a bash extension to the glob specification. This pattern will work on the Seven Bridges platform, but not with cwltool.To get this glob working with all executors, break down the single string pattern into a list of string patterns, i.e. convertglob: "{*.txt,*.pdf}"toglob: ["*.txt","*.pdf"]Some CWL executors require strict type matching between connected ports, e.g. aFileobject can't connect to aFile[]input and vice versa.These connection mismatches can be handled by adding a step inputvalueFromexpression to transform aFiletoFile[]withvalueFrom: $([self]), or transformFile[]toFilewithvalueFrom: $(self[0]).Some CWL executors require Javascript expressions to be written in ES5.1 strict mode (more infohere).Note on Javascript versions in expressionsWhile the Seven Bridges platform executor correctly evaluates bothJS ES6 and JS ES5.1code in expressions, some CWL executors only evaluate JS ES5.1 in strict mode. If a converted
workflow contains JS ES6 or JS with non-strict syntax, the workflow may not be executable on all executors.Checking Javascript version compatibility in expressionsPreliminary validation of JS expressions in an app can be done using thesbg_validate_js_cwl_v1tool. Running the command:sbg_validate_js_cwl_v1  -i app.cwlwill write the expression evaluation result to the stderr stream.To get a list of undeclared variables without duplicates, run:sbg_validate_js_cwl_v1 -i app.cwl 2>&1 >/dev/null | awk 'NR % 3 == 0' | uniqand addvarto the first instances of all undeclared variables.