The SeqWare default behaviour is manual_output=true, since the output files are provisioned to output_prefix/output_dir. SPB default behaviour places the files in an enforced unique directory, something like output_prefix/output_dir/Workflowname_version/${random}. This constitutes manual_output=false. We need both methods in our workflows.

Control over manual output is required for at least three use cases that we know of: automated output file checking, removing the 'random' directories in output provisioning, and workflows within workflows.

1) Automated output file checking: necessary for regression testing and probably other applications in the future, this allows us to know where to look for our output files to run automated checks over them.
2) Output provisioning : currently SeqWare places provisioned files into a random numbered directory. In the future, we may move to project-style directories. We need both to handle this situation.
3) Workflows of workflows: In order for workflows to call other workflows and use and provision the output, they need to know exactly where the files will be written. Hence manual output.

Since this is a core feature for us, I added it to the archetype.

--
Morgan Taschuk

Manager, Sequencing Pipeline Development and Evaluation
Ontario Institute for Cancer Research
MaRS Centre, South Tower
101 College Street, Suite 800
Toronto, Ontario, Canada M5G 0A3

Phone: 647-260-6486
Toll-free: 1-866-678-6427
www.oicr.on.ca

This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
________________________________
From: Denis Yuen [notifications@github.com]
Sent: December 4, 2013 5:16 PM
To: SeqWare/seqware
Cc: Morgan Taschuk
Subject: Re: [seqware] Feature/pde 428 update workflow archetype (#162)


In seqware-archetypes/seqware-archetype-java-workflow/src/main/resources/archetype-resources/workflow/config/__workflow-name__Workflow.ini:

> @@ -0,0 +1,19 @@
> +#set( $symbol_dollar = '$' )
> +# key=input_file:type=file:display=F:file_meta_type=text/plain
> +input_file=${workflow_bundle_dir}/Workflow_Bundle_${workflow-name}/${workflow-version}/data/input.txt
> +# key=greeting:type=text:display=T:display_name=Greeting
> +greeting=Testing
> +
> +cat=${workflow_bundle_dir}/Workflow_Bundle_${workflow-name}/${workflow-version}/bin/gnu-coreutils-5.67/cat
> +echo=${workflow_bundle_dir}/Workflow_Bundle_${workflow-name}/${workflow-version}/bin/gnu-coreutils-5.67/echo
> +
> +# the output directory is a convention used in many workflows to specify a relative output path
> +output_dir=seqware-results
> +# the output_prefix is a convention used to specify the root of the absolute output path or an S3 bucket name
> +# you should pick a path that is available on all cluster nodes and can be written by your user
> +output_prefix=./
> +# manual output determines whether or not SeqWare should enforce the uniqueness of the final directory or not.


I don't see any code in SeqWare that handles something called "manual_output." Could you comment on this?

—
Reply to this email directly or view it on GitHub<https://github.com/SeqWare/seqware/pull/162/files#r8112746>.