Initial framework for Clusterk/AWS parallelization
Initial bootstrapping script to spin up bcbio-nextgen worker node on an empty AWS instance
Provide command line runfn command called from clusterk parallel.
Enable tool-only installations without associated data files
Handle input galaxy *.loc files with multiple lines per genome. Fixes bcbio-nextgen docker test suite
Integrate S3 pack information into clusterk runs. Move to point where we need to map local files to buckets for packing and shipping
Update to match latest bcbio-nextgen version and improve createsetuser script to avoid home directory warnings
Improved support for shipping to S3; translate files into batch specific and general biodata S3 buckets
Install/update improvements thanks to feedback from @jwm: make docker image name configurable, sanity check docker image before running and remove --inplace update functionality
Correctly set default image name for docker instance on first installation. Fixes #8
Simplify docker image install using native docker import. Avoids need for download and progress bar, and avoids security issue with command line provided image
Initial set of ansible scripts for automating build of bcbio vm docker image on EC2
Additional work on ansible scripts to automate builds of bcbio-nextgen docker containers; ec2 connectivity
Additional work on bcbio docker bootstrapping on EC2: improve setting of defaults, fix issues with ssh caching and docker group, ensure termination even on failure, install gof3r and correctly set keys for S3 import
Finalize and document ansible automation for building bcbio-nextgen images. Fixes AWS connectivity and timeout issues for long running builds with async
Move to using new shared bcbio organization for conda recipes
Improve update/install process for bcbio_vm
Ship reference data and analysis inputs to S3 buckets using boto and gof3r
Remove directory only inputs to S3 separated analysis. Set up testing framework for evaluating S3 download
Generalize s3 unpacking to handle biodata and work directory retrieval