BACKGROUND
the cost of genome sequencing has been rapidly decreasing due to the introduction of a number of new affordable next-generation sequencing technologies.
coupled with the decreasing costs is an increase in the volume of data produced by sequencing machines.
as a consequence, the genomics field has been rapidly changing: larger amounts of sequence data are not only being produced at lower costs, but also more and more often by small to midsize research groups outside of the large sequencing centers all over the world  <cit> .
this is a trend, which is likely to continue, as newer generation sequencing technologies continue to drive down costs.
high-throughput sequencing technologies have decentralized sequence acquisition, increasing the number of research groups in demand of sequence analysis all over the world.
the increasing volume of data from next-generation sequencing has led to increased computational and bioinformatics needs and concern of a bioinformatics bottleneck  <cit> .
technical challenges in use of bioinformatics software  <cit>  and difficulties in utilization of available computational resources  <cit>  impede analysis, interpretation and full exploration of sequence data.
the installation, operation, and maintenance of software tools for bioinformatics analysis can be cumbersome and require significant technical expertise leading to efforts that pre-package and bundle bioinformatics tools  <cit> .
while, many bioinformatics software tools routinely used in sequence analysis are open source and freely available, the installation, operation, and maintenance can be cumbersome and require significant technical expertise  <cit> .
in addition, individual tools are often insufficient for sequence analysis and, rather, need to be integrated with others into multi-step pipelines for thorough analysis.
to aid with this, bioinformatics workflows systems and workbenches, such as galaxy  <cit> , ergatis  <cit> , genepattern  <cit> , taverna  <cit>  provide user interfaces to simplify execution of tools and pipelines on centralized servers.
prior to analysis, researchers utilizing genomics approaches are faced with a multitude of choices of analysis protocols and best practices are often poorly documented  <cit> .
complexities of analysis pipelines and lack of transparent protocols can limit reproducibility of computed results  <cit> .
use of workbenches that store pipeline metadata and track data provenance can improve reproducibility  <cit> .
bioinformatics service providers, such as rast  <cit> , mg-rast  <cit> , isga  <cit> , and the igs annotation engine  <cit> , have attempted to address challenges in microbial genome analysis by providing centralized services, where users submit sequence data to a web site for analysis using standardized pipelines.
in this model, the service provider operates the online resource, dedicating the necessary personnel and computational resources to support a community of users.
bioinformatics workflow systems  <cit>  also operate on central servers, utilizing dedicated or shared network based storage, and clusters of computers for improved processing throughput.
other efforts have bundled tools into portable software packages for installation on a local computer, including mothur  <cit>  and qiime  <cit>  for 16s ribosomal rna analysis and diya  <cit>  for bacterial genome annotation.
a virtual machine  encapsulates an operating system with pre-installed and pre-configured software in a single executable file that can be distributed and run elsewhere.
vms provide a means to eliminate complex software installations and adaptations for portable execution, directly addressing one of the challenges involved with using bioinformatics tools and pipelines.
cloud computing offers leasable computational resources on-demand over a network  <cit> .
the cloud computing model can simplify access to a variety of computing architectures, including large memory machines, while eliminating the need to build or administer a local computer network addressing challenges in access and deployment of infrastructure for bioinformatics  <cit> .
cloud computing platforms have been emerging in the commercial sector, including the amazon elastic compute cloud   <cit> , and in the public sector to support research, such as magellan  <cit>  and diag  <cit> .
in combination with virtual machines, cloud computing can help improve accessibility to complex bioinformatics workflows in a reproducible fashion on a readily accessible distributed computing platform  <cit> .
there is considerable enthusiasm in the bioinformatics community for use of cloud computing in sequence analysis  <cit> .
while cloud computing platforms that provide ready access to computing resources over the internet on-demand can improve processing throughput, utilization of bioinformatics tools and pipelines on such distributed systems requires technical expertise to achieve robust operation and intended performance gains  <cit> .
cluster management software, workflow systems, or databases may be installed, maintained, and executed across multiple machines.
also, challenges in data storage and transfer over the network add to the complexity of using cloud computing systems  <cit> .
map-reduce algorithms  <cit>  using the cloud-ready framework hadoop are available for sequence alignment and short read mapping  <cit> , snp identification  <cit> , rna expression analysis  <cit> , amongst others demonstrating the usability of cloud services to support large-scale sequence processing.
despite emergence of methods in cloud-ready frameworks, many bioinformatics tools, analysis pipelines, and standardized methods are not readily transferable to these frameworks but are trivially parallelized using batch processing systems  <cit> .
in this paper, we describe a new application, cloud virtual resource , that relies on two enabling technologies, virtual machines  and compute clouds, to provide improved access to bioinformatics workflows and distributed computing resources.
clovr provides a single vm containing pre-configured and automated pipelines, suitable for easy installation on the desktop and with cloud support for increased analysis throughput.
in building the clovr vm, we have addressed the following technical challenges in using cloud computing platforms:
i) elasticity and ease-of-use: clouds can be difficult to adopt and use requiring operating system configuration and monitoring; many existing tools and pipelines are not designed for dynamic environments and require re-implementation to utilize cloud-ready frameworks such as hadoop;
ii) limited network bandwidth: internet data transfers and relatively slow peer-to-peer networking bandwidth in some cloud configurations can incur performance and scalability problems; and
iii) portability: reliance on proprietary cloud features, including special storage systems can hinder pipeline portability; also, virtual machines, while portable and able to encapsulate complex software pipelines, are themselves difficult to build, configure, and maintain across cloud platforms.
the architecture of clovr addresses these challenges by
i) simplifying use of cloud computing platforms by automatically provisioning resources during pipeline execution;
ii) using local disk for storage and avoiding reliance on network file systems;
iii) providing a portable machine image that executes on both a personal computer and multiple cloud computing platforms;
in the presented work, we evaluate four features of the clovr architecture: portability across different local operating systems and remote cloud computing platforms, support for elastic provisioning of local and cloud resources, scalability of the architecture, use of local data storage on the cloud, and built process of the clovr vm from base image using recipes.
implementation
clovr architecture
clovr is a vm that executes on a desktop  computer, providing the ability to run analysis pipelines on local resources .
clovr is invoked using one of two supported vm players, vmware  <cit>  and virtualbox  <cit> ; at least one of which is freely available on all major desktop platforms: windows, unix/linux, and mac os.
on a local computer, clovr utilizes local disk storage and compute resources, as supported by the vm player, including multi-core cpus if available.
to access data stored on the local computer, users can copy files into a "shared folder" that is accessible on both the vm and the local desktop and uses available hard drive space on the computer.
once inside the shared folder, clovr can read this data for processing.
similarly, clovr writes output data to this shared folder, making the pipeline output available on the desktop.
this shared folder feature is supported by both vmware and virtualbox.
optionally, the clovr vm can be configured to automatically access a cloud provider for additional resources.
supported clouds include the commercial amazon elastic compute cloud  <cit>  and the academic platforms diag  <cit>  and magellan  <cit> .
in utilizing the cloud, multiple copies of the clovr vm execute concurrently and interact as a cluster for parallel processing of data .
clusters of clovr vms running different applications on the cloud are independent and not shared between users or pipelines.
<dig>  components of the clovr vm
to address technical challenges associated with software installations and pipeline configurations, the clovr vm comes bundled with all required software pre-installed and pre-configured .
the bundled software includes a base operating system , job schedulers , and a workflow system .
in addition, numerous open source bioinformatics tools are pre-installed and bundled into automated pipelines for pre-defined analysis protocols  <cit> .
<dig>  building vm images from the clovr skeleton vm
an automated build and configuration process is used to generate different vm images in formats compatible with both supported vm players and all supported cloud computing platforms.
a specially configured vm  running the hudson continuous integration server  <cit>  is used to schedule and automate the build process.
building the vm begins with a skeleton ubuntu  <dig>  disk image  <cit> .
during the build process, a series of recipes are applied to the skeleton image to install all the necessary software, resulting in three bundles, called "base", "build" and "standard", whereby the "standard" bundle represents the clovr disk image with all fully installed pipelines.
for simplicity and maximum flexibility shells scripts are being used to build the clovr bundles.
the bundles and corresponding recipes used are in version control and freely available  <cit> .
in order to allow external developers to modify, extend, or exchange pipelines or build custom appliances using the clovr framework, customized recipes can be written based on the skeleton or base image.
each disk image is converted into formats for vmware  and virtualbox .
to speed up launching clovr on the cloud, the raw disk image is also uploaded to amazon ec <dig>  and diag .
to support use of clovr on amazon ec <dig>  our group is permanently maintaining copies of the latest clovr image as well as several reference datasets on the amazon simple storage service   <cit> .
<dig>  components of a clovr automated pipeline
the clovr vm  includes four pre-packaged and automated analysis protocols , which are described in detail in the referenced sops:  a parallelized blast  <cit>  search protocol ;  a comparative 16s rrna sequence analysis pipeline ;  a comparative metagenomic sequence analysis pipeline ; and  a single microbial genome assembly and annotation pipeline .
for each protocol, a limited set of configuration options and pre-defined input files are supported, such as sff, fasta, qual and fastq; output files are generated in standardized formats, such as fasta and genbank flat files, and include summary reports, tables and graphical visualizations of the analysis results.
clovr-16s
abbreviations: nt, nucleotide; pep, peptide; gbk, genbank.
; sqn, sequin ;
key bioinformatics tools utilized in each protocol are listed.
for input, only the required inputs from the user for each analysis track are listed.
for outputs, only the data saved from each step is listed.
1- inputs may require adapter and qc trimming prior to assembly
each clovr protocol is implemented as two discrete pipelines: i) a worker pipeline and ii) a wrapper pipeline.
clovr uses the ergatis workflow engine  <cit>  to describe and execute each of these pipelines.
the worker pipeline implements and performs the particular analysis protocol, while the wrapper pipeline handles data management and automated use of the cloud from the desktop using the local vm client, if this execution mode is selected .
each wrapper pipeline is composed of seven primary phases:  pre-processing, including quality and integrity checks of input data;  starting a remote cluster for distributed processing;  data upload to the cloud;  execution of the worker pipeline locally or on the cloud;  monitoring of the worker pipeline;  data download from the cloud and  post-processing.
steps , ,  are only executed when utilizing a remote cloud platform.
to implement each of these steps in the wrapper pipeline, we built a set of utilities and a web services application programming interface  in a software package called vappio http://vappio.sf.net.
vappio is built on top of the amazon ec <dig> api  <cit> , which includes functions for managing images, instances, and authentication key pairs.
by comparison, vappio provides functions for managing clusters, datasets, and analysis protocols.
a summary of the vappio functions and web services follows:
clusters
on the cloud, clusters of clovr vm instances are configured for parallel processing.
clovr utilizes these clusters as temporary resources during pipeline processing, provisioning a new cluster for each pipeline, and terminating the cluster upon pipeline completion.
each cluster runs an instance of both grid engine  <cit>  and hadoop  <cit>  for job scheduling.
clusters are composed of a single master node and one or more worker nodes .
the client clovr vm running on the user's desktop is also considered a cluster, named 'local' that is both a master and worker type.
the first vm that is started in a cluster is designated as the master node.
subsequent vms are designated as worker nodes, automatically registered with the master node and added to the cluster upon boot of the image.
worker nodes are configured in grid engine queues in order to receive work units based on the number of cpus that are available on the instance.
communication between clusters
all communication and data transfer between a user's desktop and the cloud is managed by the client clovr vm running on a local computer.
the local client vm communicates with the master clovr vm on the cloud to transfer data, invoke worker pipelines, and monitor the pipeline state .
to provide security and help ensure data privacy, each remote cluster of clovr vms uses a unique, randomly generated authentication key.
this key is used to enable secure data transfer between instances with secure shell  both within the cloud and over the internet and between the local client vm and master cloud clovr vms.
cluster management
to manage the cluster on the cloud, vappio provides web services to dynamically start , resize  and terminate  clusters of vm instances.
these web services in turn utilize ec <dig> api calls  <cit> , including ec2-run-instances, ec2-terminate-instances, and ec2-describe-instances.
in addition to executing the ec <dig> api calls, the vappio web services manage the configuration of grid engine and hadoop on each instance, as the instance is started and added to or terminated and removed from the cluster.
user authentication on the cloud
in order to access the cloud, user account and authentication information is required and obtained from the cloud provider.
to simplify access to the cloud during pipeline execution and without jeopardizing security, vappio provides a unique identifier, called a credential name, for each cloud account.
during an initial configuration, the credential name is configured and associated with the cloud account and the authentication keys using the vappio web service, vp-add-credentials.
this credential name is then used to refer to the account in subsequent vappio web service calls during pipeline execution.
datasets
in vappio, datasets are described as lists of files or uniform resource locators  that are accessible by a cluster or the local client clovr vm.
user-provided sequence data, reference data, and output generated by the clovr analysis pipelines are all managed as datasets.
datasets are moved between a local desktop and disk storage on the remote cluster as needed for processing .
vappio provides utilities for 1) registering new datasets with a cluster , 2) transferring datasets between the cloud and local desktop , and 3) requesting information about a dataset .
protocols
pre-defined analysis protocols are invoked for data analysis using a single configuration file .
vappio provides utilities for configuration and invocation of analysis protocols with the services vp-describe-protocol and vp-run-pipeline.
an example of the configuration file for clovr-microbe run on  <dig> sequence data produced by vp-describe-protocol is shown in figure  <dig>  the specification file includes references to input data sets, configurable analysis parameters, and, optionally, references to account credentials for accessing the cloud.
protocols are executed with vp-run-pipeline, which accepts the specification file as input.
once executed, the running instance of the protocol is referred to as a pipeline.
the status of pipelines is monitored with the service vp-describe-pipeline.
transparency and reproducibility
for complex pipelines, reproducibility becomes increasingly difficult and virtualization and clouds have been recognized as ideal platforms to promote pipeline reproducibility  <cit> .
clovr realizes this potential for reproducible research by executing all steps on a portable vm that encapsulates the entire runtime environment, included versioned protocols and analysis results.
to ensure transparency of the clovr-supported analysis, each clovr protocol is described by two documents:  an abstract workflow xml file that is used by the ergatis workflow engine to execute the protocol and  a human readable standard operating procedure  document that describes the protocol in detail and has been published with open access elsewhere.
the abstract workflow xml is an exact description of the executions used to perform the analysis.
the sops describe each step of the pipeline, including software tools, software versions, and parameters used.
to ensure reproducibility of individual analysis results, clovr uses the following additional principles: 1) all pipelines are executed using the ergatis workflow system that tracks process flow and exact parameters invoked at each step in the xml file.
2) as part of the clovr software installation process, versioning is applied to each analysis protocol, reference data set, and to the clovr vm image itself.
all results generated during clovr pipeline runs have references to these versions.
<dig>  data storage and transfers
all input data and results generated during a clovr pipeline execution are stored in the "shared folder" on the local client vm and can be accessed from the local computer.
when using the cloud, input data is copied to the cloud as needed and output data is retrieved as part of the pipelines.
to improve network transfer performance, clovr uses high performance secure shell   <cit>  to transfer files.
the synchronization utility rsync  <cit>  is used to avoid redundant data transfers.
since all network transfers between the local desktop and the cloud are managed by clovr vms, data transfer is automatic, invisible to the user and does not require further software installations or configurations by the user.
upon pipeline completion, the final output is transferred from the master node on the cloud to the local vm and, subsequently, the entire cluster terminated on the cloud.
the pipelines in clovr are configured to avoid unnecessary data transfers for both local and cloud-based execution modes.
for example, publicly available reference datasets used by several of the supported protocols are either permanently hosted on the cloud or at an internet accessible url  <cit> .
when executing clovr pipelines on amazon ec <dig>  pipelines utilize reference datasets hosted on amazon s <dig> where configured.
for local execution, reference datasets are first downloaded to the local vm over the internet.
clovr ensures such local transfers happen only once, the first time the data is accessed, and the reference data is then saved locally for subsequent access.
for data storage of intermediate results or temporary files during pipeline execution on the cloud, clovr utilizes the local ephemeral disks provided to each cloud instance and does not require access to a shared file system, such as a nfs server.
under this model, worker nodes must receive copies of input data from the master node before beginning work, which is implemented using the "job prolog" feature of grid engine.
similarly, output data is copied back to the master node using the "job epilog" feature of grid engine.
to provide robustness and scalability, all data transfers to and from the master node are also scheduled as jobs in grid engine queues named staging.q and harvesting.q.
the number of slots in these queues allows for control over how many simultaneous transfers a master node will process.
hpn-ssh and rsync are used to perform the transfer between instances in the cloud.
in some cases, pipelines use reference datasets or intermediate outputs that need to be accessed on every instance in a cluster.
a single directory  is used to mirror such data to all instances in the cluster.
rather than rely exclusively on the master node to provide a mirror of the data, a custom built peer-to-peer transfer scheme is used to improve transfer throughput.
under this strategy, worker nodes share copies of the staging directory in a peer-to-peer fashion using rsync and hpn-ssh.
sun grid engine is used to schedule these transfers and limits the number of transfers per host, thereby avoiding overloading any single host.
upon receiving a complete copy of the staging directory, a worker nodes is added to a grid engine queue  indicating that they can mirror copies to peers.
<dig>  automatic resource provisioning in the cloud
during execution of clovr pipelines in the cloud, cluster sizes of clovr vm instances are configured automatically, i.e.
additional vm instances are automatically provisioned, if necessary.
pipelines that are configured to run exclusively on a local clovr vm instance skip resource allocation steps.
to determine the number of compute instances needed for the different clovr protocols, custom scripts follow a hierarchy of the following factors: 1) hard-coded assumptions about expected resource utilizations, 2) cluster size limits set by the cloud provider, and 3) runtime estimations by the user based on input data.
an example of a hard-coded resource provisioning is the short-read illumina sequence data assembly step using velvet  <cit> , which is part of the clovr-microbe pipeline and requires larger ram allocations than comparable roche/ <dig> sequence data assemblies.
when executed on the amazon ec <dig> cloud, clovr-microbe starts a single high-memory instance type  that provides  <dig>  gb of ram, which in our testing is sufficient for assembly of single bacterial genomes.
local execution is limited by available ram on the machine.
for three of the pre-packaged protocols in clovr , blast searches are the primary processing bottleneck.
in these cases, an estimation of total blast runtime can serve as a good approximation to predict the overall pipeline runtime.
based on our observations, blast runtimes can vary for a particular search database depending on the length and composition of query sequences.
for the clovr-search and clovr-metagenomics protocols, total blast runtimes are estimated based on the input data with the cunningham tool  <cit>  and used to determine how many instances to start prior to search.
cunningham, which was implemented as part of the clovr project, rapidly estimates blast runtime by comparing the kmer profiles  of a pre-calculated reference database and the input query sequence.
first, a default minimum of five c <dig> xlarge instances providing a total of  <dig> cpus is started to support blast steps in these pipelines.
second, cunningham is used to determine the expected cpu hours required for the total blast search and to allocate additional machine instances, not exceeding a user configurable upper limit.
the cloud provider may impose a limit on the maximum number of instances that can be started by a user .
for each clovr pipeline, users also have the option to set an instance limit in the configuration file, which prevents attempts to start more than the specified number of instances for a particular pipeline.
also impacting blast runtimes are the number and size of partitions that are used for parallel processing.
in clovr, blast searches are run in parallel by dividing the input query multi-fasta files into partitions and executing a search of each partition concurrently against the reference database.
over-partitioning of the data leads to very short durations of individual jobs and can result in inefficient use of resources and increased runtimes due to the overhead in the scheduling and invocation associated with each job.
provided the cunningham blast runtime estimate, the partition size p for each blast query in the clovr pipeline is obtained by
p=nqt∕r
where nq is the total number of query sequences, t is the estimated cpu runtime from cunningham, and r is a configurable parameter for the preferred execution time for a single data partition .
the support for runtime estimates is provided as a configurable module that reads the pipeline configuration and produces an estimate.
this allows for custom modules for runtime prediction in the future using some other logic.
RESULTS
<dig>  clovr runs on the desktop and dynamically utilizes cloud computing providers
to demonstrate the portability of clovr, we executed a single analysis protocol  on a local desktop computer and two cloud computing platforms .
the input data comprised of  <dig>   <dig> flx titanium  <dig> kb paired-end sequencing reads of the bacterium acinetobacter baylyi totaling ~ <dig> mbp and expected to cover the ~ <dig>  mbp genome at 25-fold coverage.
identical output, comprised of  <dig> contigs  and  <dig>  predicted coding genes was obtained on all three platforms.
for local analysis, a clovr instance with  <dig> cpus and  <dig> gb of ram was used.
when using the cloud platforms, the local client vm can be executed in as little as  <dig> gb of ram.
the diag and ec <dig> platforms allowed for the parallelization of several steps of the protocol offering 4-cpus per "medium" instance type on diag  and 8-cpus per "c <dig> xlarge" instance type on ec <dig> .
max no.
instances: 5
max no.
instances: 18
our evaluation of the clovr-microbe protocol demonstrates the ability to run the same genome assembly and annotation protocol both locally and on the cloud for increased throughput .
a single configuration setting is changed to invoke the pipeline on either the local desktop or the supported clouds.
<dig>  clovr provides automated resource provisioning in the cloud
elasticity, i.e.
dynamic provisioning of resources, is a primary feature of the cloud and allows for the addition of computational resources on-demand.
figure  <dig> shows the automatic allocation of clovr vm instances to the cluster on the cloud and the subsequent termination of idle instances upon job completion for the microbial genome assembly and annotation steps of the clovr-microbe pipeline and demonstrates dynamic capabilities provided by clovr.
figure  <dig> shows a blastx comparison using clovr-search on clusters composed of up to  <dig> c <dig> xlarge instances, comprising  <dig>  cpus.
this blastx search ran on amazon ec <dig> with a throughput of ~ <dig>  mbp per c <dig> xlarge instance, at an estimated total cost of ~$ <dig> per hour for all  <dig> instances.
resource provisioning for clovr-microbe is automatic; for clovr-search it is configured by the user but does not require the direct user interaction with the remote cluster on the cloud.
<dig>  clovr uses local disks and does not rely on network file systems during pipeline execution
bottlenecks in reading or writing data on a shared, network-based file system, such as nfs  <cit> , can cause performance problems during processing, especially when many concurrent processes are executing against the shared resource.
to avoid data transfer bottlenecks clovr uses local disks space of the instance running on the desktop or cloud, requiring input files to be transferred to each compute host within the cluster.
for data input, these file transfers between master and worker node types are made prior to computation, for data output subsequent to job completion.
in addition, reference data sets and intermediate outputs need to be accessed by all vm instances in a cluster.
to improve distribution of these data sets, a peer-to-peer data transfer scheme is used for sharing intermediate results and reference data sets.
figure  <dig> shows data transfers within a cluster of  <dig> clovr instances during a run of clovr-microbe on amazon ec <dig>
to evaluate the performance of data transfers, the throughput for providing  <dig>  gb of compressed reference data to a cluster of  <dig> c <dig> xlarge clovr vm instances was tested .
instances came online in a staggered fashion and received copies of the reference data upon boot of the instance.
the aggregate data throughput exceeded  <dig>  gb per second.
by comparison, network transfer speeds between a pair of c <dig> xlarge instance types on the amazon ec <dig> network were found to typically fall below ~ <dig> mb per second .
discussion
clovr reduces bottlenecks in sequence analysis by using two related technologies: virtual machines  and cloud computing.
clovr simplifies deployment of complex bioinformatics workflows by providing a single executable  that is portable across all major operating systems.
by default, clovr runs on a desktop but enables seamless access to large distributed computing resources including amazon ec <dig> and other cloud computing platforms, providing a potential alternative to building and maintaining in-house infrastructure for computational analysis.
while genomic workbench systems focus on providing extensive customization, i.e.
choices of multiple tools for easy integration into user-customized pipelines, many projects instead rely on static, standardized analysis pipelines.
to accommodate this scenario, clovr provides pre-defined standard pipelines that integrate tools for particular analysis objectives so that no additional configuration or expertise with individual tools by the user is required.
this level of automation is intended to enable genomics applications for users that find choice of bioinformatics tools overwhelming and instead seek recommendations for best practices.
ongoing work includes a web-based user interface , which will run locally on a user's desktop, and further simplify execution of analysis protocols.
clovr utilizes a recipe-driven process to build vms for both the desktop and cloud computing platforms, which allows for construction of customized vms by other software developers.
while clovr currently includes a set of protocols for microbial genome analysis, the clovr vm can serve as a general platform for the integration of additional tools and protocols developed by the research community.
to add new protocols to clovr, developers build recipes to install new software, deploy and test pipelines in the ergatis workflow system, and create new configuration files for the clovr api.
a first step in this direction has been made by the use of clovr to create a vm for the qiime package  <cit> .
we plan to create a wizard to simplify this process in the future and support custom repositories of build scripts.
in contrast to internet-based software-as-a-service solutions for sequence analysis, such as galaxy  <cit>  or taverna  <cit> , which provide centralized that typically execute on dedicated resources and require users to upload data clovr follows a decentralized model where each of multiple users executes a local client vm that is independent from other clovr instances in a multi-user environment.
by running on the local desktop, clovr provides the opportunity to utilize substantial computing power provided by multi-core desktop cpus, potentially avoiding the need for data transfer over the internet and for use of the cloud entirely.
the decentralized clovr architecture saves all pipeline outputs locally on the personal computer, providing users additional controls on maintaining data privacy.
although clovr transfers data to cloud servers for processing, clovr uses the cloud as a temporary resource and does not require that either inputs or results are stored permanently on the cloud.
the architecture of clovr, which utilizes grid engine  <cit>  for job scheduling and local disks for storage, allows for migration of tools and pipelines from non-cloud versions to the cloud without reimplementation.
this approach is in contrast to the use of tools developed for cloud-ready frameworks like hadoop, which are algorithms that follow mapreduce  <cit> .
the availability of these tools is, however, still relatively limited, since utilization of the hadoop framework requires new methods or reimplementation of existing tools.
as more tools that utilize mapreduce  <cit>  are becoming available, hadoop is included on the vm for their potential future integration.
clovr provides utilities for building private clusters of vm instances on-demand in the cloud, without expecting users start, manage, or resize clusters in the cloud.
a few other systems, such as nimbus one-click clusters  <cit> , galaxy cloudman  <cit>  and starcluster  <cit> , are also designed to deploy clusters of instances in the cloud.
in contrast to these systems, clovr pipelines include steps to provision these resources automatically.
this enables cost savings in the case of commercial clouds, by allocating resources only as they are needed .
to help ensure compatibility with multiple cloud providers and support emerging cloud computing platforms, clovr avoids reliance on proprietary features of individual cloud providers, instead utilizing only three ec <dig> api calls during pipeline execution .
such core functions of the ec <dig> api are becoming a standard in middleware that provides cloud services and are expected to be supported by public and private clouds.
with the increasing volume of next-generation sequencing data, data storage and transfer is increasingly important component of analysis pipelines.
compute clusters often rely on centralized, shared storage systems or file servers to simplify access to data for users and pipelines.
as part of the design to be both portable and scalable on cloud computing networks, clovr does not rely on a shared, network file system, such as nfs, for storage.
instead, clovr relies on local disk storage on either the users' desktop to store pipeline input and output, or temporary disk storage available on the cloud vm instances during pipeline execution.
by using local disk for storage rather than the network, clovr can be expected to run on commodity cloud systems with relatively slow networking and without reliance on the specialized storage features of cloud providers, such as amazon elastic block storage  <cit> .
increasing data volumes can be an impediment for utilizing the cloud, as this data needs to be transferred over the internet.
a strategy for moving analysis to data, rather than transferring data over the network, has been raised as a potential solution to dealing with data transfer bottlenecks  <cit> .
the portability of the vm provides such flexibility.
the clovr vm is  <dig>  gb compressed and can be easily transferred to computational resources that are co-located with large data sets.
similarly, reference datasets can be saved on the cloud to avoid data transfers over the internet, such as is done for uniref <dig>  <cit> , which is a  <dig>  gb compressed reference dataset hosted in the cloud to support the clovr-microbe protocol.
the clovr pipelines are composed of multiple steps, only some of which are computationally demanding or support parallelization on multiple cpus.
to match pipeline needs with available resources, each clovr pipeline includes steps to automatically provision cloud resources as needed.
one strategy for efficient allocation of resources is to estimate runtimes for steps that execute in parallel, in order to only provision resources that can be used.
the cunningham  <cit>  utility, which is implemented in clovr currently estimates blast runtimes during pipeline executions of clovr-search and clovr-metagenomics.
this strategy helps in avoiding over-partitioning of the input data, which introduces overhead that degrades overall performance, and starting too many instances for small searches or too few instances for larger searches.
the ability to predict runtimes can also be used to provide an a priori estimation to the user of how much an analysis will cost or whether a particular analysis is even feasible.
we plan to explore providing such estimates as future work and anticipate this will be of much interest to users of the software.
CONCLUSIONS
we have designed, built, and tested a portable virtual machine, named clovr, that provides automated analysis pipelines for microbial genomics.
clovr provides a stand-alone client vm for execution on a personal computer providing the ability to perform sophisticated analyses using local resources and cloud computing platforms if additional resources are needed for increased throughput.
by providing fully automated pipelines, the clovr vm allows users without extensive bioinformatics background to process sequence data, lowering the barrier of entry for microbial sequence analysis.
availability and requirements
the clovr vm is freely available for download from http://clovr.org
• project name: clovr
• project home page: http://clovr.org
• operating system: platform independent
• other requirements: vmware, virtualbox virtual machine players
• license: bsd
• any restrictions to use by non-academics: none
authors' contributions
sva and wff conceived and designed the project with the help of ow.
sva, mm, ag implemented the supporting api and custom clovr vm.
sva, kg, mv, drr, ca, jrw implemented and tested the clovr pipelines.
sva and jrw ran experiments for the paper.
sva and wff drafted the manuscript.
all authors read and approved the final manuscript.
supplementary material
additional file 1
clovr screencast.
a short screencast launching and clovr and using the web interface to launch an analysis.
click here for file
acknowledgements and funding
this research was supported by funds from the national human genome research institute, nih/nhgri, arra under grant no.
rc <dig> hg005597- <dig>  the national science foundation, nsf, under grant no.
<dig> and the amazon web services in education research grants program.
we also thank michelle giglio gwinn for help integrating the microbial annotation pipeline and anup mahurkar, dave kemeza, victor felix, and brian cotton for it assistance and help using the diag.

 the cost of genome sequencing has been rapidly decreasing due to the introduction of a number of new affordable next-generation sequencing technologies. coupled with the decreasing costs is an increase in the volume of data produced by sequencing machines. as a consequence, the genomics field has been rapidly changing: larger amounts of sequence data are not only being produced at lower costs, but also more and more often by small to midsize research groups outside of the large sequencing centers all over the world. this is a trend, which is likely to continue, as newer generation sequencing technologies continue to drive down costs. high-throughput sequencing technologies have decentralized sequence acquisition, increasing the number of research groups in demand of sequence analysis all over the world. the increasing volume of data from next-generation sequencing has led to increased computational and bioinformatics needs and concern of a bioinformatics bottleneck. technical challenges in use of bioinformatics software and difficulties in utilization of available computational resources impede analysis, interpretation and full exploration of sequence data. the installation, operation, and maintenance of software tools for bioinformatics analysis can be cumbersome and require significant technical expertise leading to efforts that pre-package and bundle bioinformatics tools. while, many bioinformatics software tools routinely used in sequence analysis are open source and freely available, the installation, operation, and maintenance can be cumbersome and require significant technical expertise. in addition, individual tools are often insufficient for sequence analysis and, rather, need to be integrated with others into multi-step pipelines for thorough analysis. to aid with this, bioinformatics workflows systems and workbenches, such as galaxy, ergatis, genepattern, taverna provide user interfaces to simplify execution of tools and pipelines on centralized servers. prior to analysis, researchers utilizing genomics approaches are faced with a multitude of choices of analysis protocols and best practices are often poorly documented. complexities of analysis pipelines and lack of transparent protocols can limit reproducibility of computed results. use of workbenches that store pipeline metadata and track data provenance can improve reproducibility. bioinformatics service providers, such as rast, mg-rast, isga, and the igs annotation engine, have attempted to address challenges in microbial genome analysis by providing centralized services, where users submit sequence data to a web site for analysis using standardized pipelines. in this model, the service provider operates the online resource, dedicating the necessary personnel and computational resources to support a community of users. bioinformatics workflow systems also operate on central servers, utilizing dedicated or shared network based storage, and clusters of computers for improved processing throughput. other efforts have bundled tools into portable software packages for installation on a local computer, including mothur and qiime for 16s ribosomal rna analysis and diya for bacterial genome annotation. a virtual machine encapsulates an operating system with pre-installed and pre-configured software in a single executable file that can be distributed and run elsewhere. vms provide a means to eliminate complex software installations and adaptations for portable execution, directly addressing one of the challenges involved with using bioinformatics tools and pipelines. cloud computing offers leasable computational resources on-demand over a network. the cloud computing model can simplify access to a variety of computing architectures, including large memory machines, while eliminating the need to build or administer a local computer network addressing challenges in access and deployment of infrastructure for bioinformatics. cloud computing platforms have been emerging in the commercial sector, including the amazon elastic compute cloud, and in the public sector to support research, such as magellan and diag. in combination with virtual machines, cloud computing can help improve accessibility to complex bioinformatics workflows in a reproducible fashion on a readily accessible distributed computing platform. there is considerable enthusiasm in the bioinformatics community for use of cloud computing in sequence analysis. while cloud computing platforms that provide ready access to computing resources over the internet on-demand can improve processing throughput, utilization of bioinformatics tools and pipelines on such distributed systems requires technical expertise to achieve robust operation and intended performance gains. cluster management software, workflow systems, or databases may be installed, maintained, and executed across multiple machines. also, challenges in data storage and transfer over the network add to the complexity of using cloud computing systems. map-reduce algorithms using the cloud-ready framework hadoop are available for sequence alignment and short read mapping, snp identification, rna expression analysis, amongst others demonstrating the usability of cloud services to support large-scale sequence processing. despite emergence of methods in cloud-ready frameworks, many bioinformatics tools, analysis pipelines, and standardized methods are not readily transferable to these frameworks but are trivially parallelized using batch processing systems. in this paper, we describe a new application, cloud virtual resource, that relies on two enabling technologies, virtual machines and compute clouds, to provide improved access to bioinformatics workflows and distributed computing resources. clovr provides a single vm containing pre-configured and automated pipelines, suitable for easy installation on the desktop and with cloud support for increased analysis throughput. in building the clovr vm, we have addressed the following technical challenges in using cloud computing platforms: i) elasticity and ease-of-use: clouds can be difficult to adopt and use requiring operating system configuration and monitoring; many existing tools and pipelines are not designed for dynamic environments and require re-implementation to utilize cloud-ready frameworks such as hadoop; ii) limited network bandwidth: internet data transfers and relatively slow peer-to-peer networking bandwidth in some cloud configurations can incur performance and scalability problems; and iii) portability: reliance on proprietary cloud features, including special storage systems can hinder pipeline portability; also, virtual machines, while portable and able to encapsulate complex software pipelines, are themselves difficult to build, configure, and maintain across cloud platforms. the architecture of clovr addresses these challenges by i) simplifying use of cloud computing platforms by automatically provisioning resources during pipeline execution; ii) using local disk for storage and avoiding reliance on network file systems; iii) providing a portable machine image that executes on both a personal computer and multiple cloud computing platforms; in the presented work, we evaluate four features of the clovr architecture: portability across different local operating systems and remote cloud computing platforms, support for elastic provisioning of local and cloud resources, scalability of the architecture, use of local data storage on the cloud, and built process of the clovr vm from base image using recipes. implementation clovr architecture clovr is a vm that executes on a desktop computer, providing the ability to run analysis pipelines on local resources. clovr is invoked using one of two supported vm players, vmware and virtualbox; at least one of which is freely available on all major desktop platforms: windows, unix/linux, and mac os. on a local computer, clovr utilizes local disk storage and compute resources, as supported by the vm player, including multi-core cpus if available. to access data stored on the local computer, users can copy files into a "shared folder" that is accessible on both the vm and the local desktop and uses available hard drive space on the computer. once inside the shared folder, clovr can read this data for processing. similarly, clovr writes output data to this shared folder, making the pipeline output available on the desktop. this shared folder feature is supported by both vmware and virtualbox. optionally, the clovr vm can be configured to automatically access a cloud provider for additional resources. supported clouds include the commercial amazon elastic compute cloud and the academic platforms diag and magellan. in utilizing the cloud, multiple copies of the clovr vm execute concurrently and interact as a cluster for parallel processing of data. clusters of clovr vms running different applications on the cloud are independent and not shared between users or pipelines. components of the clovr vm to address technical challenges associated with software installations and pipeline configurations, the clovr vm comes bundled with all required software pre-installed and pre-configured. the bundled software includes a base operating system, job schedulers, and a workflow system. in addition, numerous open source bioinformatics tools are pre-installed and bundled into automated pipelines for pre-defined analysis protocols. building vm images from the clovr skeleton vm an automated build and configuration process is used to generate different vm images in formats compatible with both supported vm players and all supported cloud computing platforms. a specially configured vm running the hudson continuous integration server is used to schedule and automate the build process. building the vm begins with a skeleton ubuntu disk image. during the build process, a series of recipes are applied to the skeleton image to install all the necessary software, resulting in three bundles, called "base", "build" and "standard", whereby the "standard" bundle represents the clovr disk image with all fully installed pipelines. for simplicity and maximum flexibility shells scripts are being used to build the clovr bundles. the bundles and corresponding recipes used are in version control and freely available. in order to allow external developers to modify, extend, or exchange pipelines or build custom appliances using the clovr framework, customized recipes can be written based on the skeleton or base image. each disk image is converted into formats for vmware and virtualbox. to speed up launching clovr on the cloud, the raw disk image is also uploaded to amazon ec and diag. to support use of clovr on amazon ec our group is permanently maintaining copies of the latest clovr image as well as several reference datasets on the amazon simple storage service. components of a clovr automated pipeline the clovr vm includes four pre-packaged and automated analysis protocols, which are described in detail in the referenced sops: a parallelized blast search protocol; a comparative 16s rrna sequence analysis pipeline; a comparative metagenomic sequence analysis pipeline; and a single microbial genome assembly and annotation pipeline. for each protocol, a limited set of configuration options and pre-defined input files are supported, such as sff, fasta, qual and fastq; output files are generated in standardized formats, such as fasta and genbank flat files, and include summary reports, tables and graphical visualizations of the analysis results. clovr-16s abbreviations: nt, nucleotide; pep, peptide; gbk, genbank.; sqn, sequin; key bioinformatics tools utilized in each protocol are listed. for input, only the required inputs from the user for each analysis track are listed. for outputs, only the data saved from each step is listed. 1- inputs may require adapter and qc trimming prior to assembly each clovr protocol is implemented as two discrete pipelines: i) a worker pipeline and ii) a wrapper pipeline. clovr uses the ergatis workflow engine to describe and execute each of these pipelines. the worker pipeline implements and performs the particular analysis protocol, while the wrapper pipeline handles data management and automated use of the cloud from the desktop using the local vm client, if this execution mode is selected. each wrapper pipeline is composed of seven primary phases: pre-processing, including quality and integrity checks of input data; starting a remote cluster for distributed processing; data upload to the cloud; execution of the worker pipeline locally or on the cloud; monitoring of the worker pipeline; data download from the cloud and post-processing. steps,, are only executed when utilizing a remote cloud platform. to implement each of these steps in the wrapper pipeline, we built a set of utilities and a web services application programming interface in a software package called vappio http://vappio.sf.net. vappio is built on top of the amazon ec api, which includes functions for managing images, instances, and authentication key pairs. by comparison, vappio provides functions for managing clusters, datasets, and analysis protocols. a summary of the vappio functions and web services follows: clusters on the cloud, clusters of clovr vm instances are configured for parallel processing. clovr utilizes these clusters as temporary resources during pipeline processing, provisioning a new cluster for each pipeline, and terminating the cluster upon pipeline completion. each cluster runs an instance of both grid engine and hadoop for job scheduling. clusters are composed of a single master node and one or more worker nodes. the client clovr vm running on the user's desktop is also considered a cluster, named'local' that is both a master and worker type. the first vm that is started in a cluster is designated as the master node. subsequent vms are designated as worker nodes, automatically registered with the master node and added to the cluster upon boot of the image. worker nodes are configured in grid engine queues in order to receive work units based on the number of cpus that are available on the instance. communication between clusters all communication and data transfer between a user's desktop and the cloud is managed by the client clovr vm running on a local computer. the local client vm communicates with the master clovr vm on the cloud to transfer data, invoke worker pipelines, and monitor the pipeline state. to provide security and help ensure data privacy, each remote cluster of clovr vms uses a unique, randomly generated authentication key. this key is used to enable secure data transfer between instances with secure shell both within the cloud and over the internet and between the local client vm and master cloud clovr vms. cluster management to manage the cluster on the cloud, vappio provides web services to dynamically start, resize and terminate clusters of vm instances. these web services in turn utilize ec api calls, including ec2-run-instances, ec2-terminate-instances, and ec2-describe-instances. in addition to executing the ec api calls, the vappio web services manage the configuration of grid engine and hadoop on each instance, as the instance is started and added to or terminated and removed from the cluster. user authentication on the cloud in order to access the cloud, user account and authentication information is required and obtained from the cloud provider. to simplify access to the cloud during pipeline execution and without jeopardizing security, vappio provides a unique identifier, called a credential name, for each cloud account. during an initial configuration, the credential name is configured and associated with the cloud account and the authentication keys using the vappio web service, vp-add-credentials. this credential name is then used to refer to the account in subsequent vappio web service calls during pipeline execution. datasets in vappio, datasets are described as lists of files or uniform resource locators that are accessible by a cluster or the local client clovr vm. user-provided sequence data, reference data, and output generated by the clovr analysis pipelines are all managed as datasets. datasets are moved between a local desktop and disk storage on the remote cluster as needed for processing. vappio provides utilities for 1) registering new datasets with a cluster, 2) transferring datasets between the cloud and local desktop, and 3) requesting information about a dataset. protocols pre-defined analysis protocols are invoked for data analysis using a single configuration file. vappio provides utilities for configuration and invocation of analysis protocols with the services vp-describe-protocol and vp-run-pipeline. an example of the configuration file for clovr-microbe run on sequence data produced by vp-describe-protocol is shown in figure the specification file includes references to input data sets, configurable analysis parameters, and, optionally, references to account credentials for accessing the cloud. protocols are executed with vp-run-pipeline, which accepts the specification file as input. once executed, the running instance of the protocol is referred to as a pipeline. the status of pipelines is monitored with the service vp-describe-pipeline. transparency and reproducibility for complex pipelines, reproducibility becomes increasingly difficult and virtualization and clouds have been recognized as ideal platforms to promote pipeline reproducibility. clovr realizes this potential for reproducible research by executing all steps on a portable vm that encapsulates the entire runtime environment, included versioned protocols and analysis results. to ensure transparency of the clovr-supported analysis, each clovr protocol is described by two documents: an abstract workflow xml file that is used by the ergatis workflow engine to execute the protocol and a human readable standard operating procedure document that describes the protocol in detail and has been published with open access elsewhere. the abstract workflow xml is an exact description of the executions used to perform the analysis. the sops describe each step of the pipeline, including software tools, software versions, and parameters used. to ensure reproducibility of individual analysis results, clovr uses the following additional principles: 1) all pipelines are executed using the ergatis workflow system that tracks process flow and exact parameters invoked at each step in the xml file. 2) as part of the clovr software installation process, versioning is applied to each analysis protocol, reference data set, and to the clovr vm image itself. all results generated during clovr pipeline runs have references to these versions. data storage and transfers all input data and results generated during a clovr pipeline execution are stored in the "shared folder" on the local client vm and can be accessed from the local computer. when using the cloud, input data is copied to the cloud as needed and output data is retrieved as part of the pipelines. to improve network transfer performance, clovr uses high performance secure shell to transfer files. the synchronization utility rsync is used to avoid redundant data transfers. since all network transfers between the local desktop and the cloud are managed by clovr vms, data transfer is automatic, invisible to the user and does not require further software installations or configurations by the user. upon pipeline completion, the final output is transferred from the master node on the cloud to the local vm and, subsequently, the entire cluster terminated on the cloud. the pipelines in clovr are configured to avoid unnecessary data transfers for both local and cloud-based execution modes. for example, publicly available reference datasets used by several of the supported protocols are either permanently hosted on the cloud or at an internet accessible url. when executing clovr pipelines on amazon ec pipelines utilize reference datasets hosted on amazon s where configured. for local execution, reference datasets are first downloaded to the local vm over the internet. clovr ensures such local transfers happen only once, the first time the data is accessed, and the reference data is then saved locally for subsequent access. for data storage of intermediate results or temporary files during pipeline execution on the cloud, clovr utilizes the local ephemeral disks provided to each cloud instance and does not require access to a shared file system, such as a nfs server. under this model, worker nodes must receive copies of input data from the master node before beginning work, which is implemented using the "job prolog" feature of grid engine. similarly, output data is copied back to the master node using the "job epilog" feature of grid engine. to provide robustness and scalability, all data transfers to and from the master node are also scheduled as jobs in grid engine queues named staging.q and harvesting.q. the number of slots in these queues allows for control over how many simultaneous transfers a master node will process. hpn-ssh and rsync are used to perform the transfer between instances in the cloud. in some cases, pipelines use reference datasets or intermediate outputs that need to be accessed on every instance in a cluster. a single directory is used to mirror such data to all instances in the cluster. rather than rely exclusively on the master node to provide a mirror of the data, a custom built peer-to-peer transfer scheme is used to improve transfer throughput. under this strategy, worker nodes share copies of the staging directory in a peer-to-peer fashion using rsync and hpn-ssh. sun grid engine is used to schedule these transfers and limits the number of transfers per host, thereby avoiding overloading any single host. upon receiving a complete copy of the staging directory, a worker nodes is added to a grid engine queue indicating that they can mirror copies to peers. automatic resource provisioning in the cloud during execution of clovr pipelines in the cloud, cluster sizes of clovr vm instances are configured automatically, i.e. additional vm instances are automatically provisioned, if necessary. pipelines that are configured to run exclusively on a local clovr vm instance skip resource allocation steps. to determine the number of compute instances needed for the different clovr protocols, custom scripts follow a hierarchy of the following factors: 1) hard-coded assumptions about expected resource utilizations, 2) cluster size limits set by the cloud provider, and 3) runtime estimations by the user based on input data. an example of a hard-coded resource provisioning is the short-read illumina sequence data assembly step using velvet, which is part of the clovr-microbe pipeline and requires larger ram allocations than comparable roche/ sequence data assemblies. when executed on the amazon ec cloud, clovr-microbe starts a single high-memory instance type that provides gb of ram, which in our testing is sufficient for assembly of single bacterial genomes. local execution is limited by available ram on the machine. for three of the pre-packaged protocols in clovr, blast searches are the primary processing bottleneck. in these cases, an estimation of total blast runtime can serve as a good approximation to predict the overall pipeline runtime. based on our observations, blast runtimes can vary for a particular search database depending on the length and composition of query sequences. for the clovr-search and clovr-metagenomics protocols, total blast runtimes are estimated based on the input data with the cunningham tool and used to determine how many instances to start prior to search. cunningham, which was implemented as part of the clovr project, rapidly estimates blast runtime by comparing the kmer profiles of a pre-calculated reference database and the input query sequence. first, a default minimum of five c xlarge instances providing a total of cpus is started to support blast steps in these pipelines. second, cunningham is used to determine the expected cpu hours required for the total blast search and to allocate additional machine instances, not exceeding a user configurable upper limit. the cloud provider may impose a limit on the maximum number of instances that can be started by a user. for each clovr pipeline, users also have the option to set an instance limit in the configuration file, which prevents attempts to start more than the specified number of instances for a particular pipeline. also impacting blast runtimes are the number and size of partitions that are used for parallel processing. in clovr, blast searches are run in parallel by dividing the input query multi-fasta files into partitions and executing a search of each partition concurrently against the reference database. over-partitioning of the data leads to very short durations of individual jobs and can result in inefficient use of resources and increased runtimes due to the overhead in the scheduling and invocation associated with each job. provided the cunningham blast runtime estimate, the partition size p for each blast query in the clovr pipeline is obtained by p=nqt∕r where nq is the total number of query sequences, t is the estimated cpu runtime from cunningham, and r is a configurable parameter for the preferred execution time for a single data partition. the support for runtime estimates is provided as a configurable module that reads the pipeline configuration and produces an estimate. this allows for custom modules for runtime prediction in the future using some other logic.  clovr runs on the desktop and dynamically utilizes cloud computing providers to demonstrate the portability of clovr, we executed a single analysis protocol on a local desktop computer and two cloud computing platforms. the input data comprised of flx titanium kb paired-end sequencing reads of the bacterium acinetobacter baylyi totaling ~ mbp and expected to cover the ~ mbp genome at 25-fold coverage. identical output, comprised of contigs and predicted coding genes was obtained on all three platforms. for local analysis, a clovr instance with cpus and gb of ram was used. when using the cloud platforms, the local client vm can be executed in as little as gb of ram. the diag and ec platforms allowed for the parallelization of several steps of the protocol offering 4-cpus per "medium" instance type on diag and 8-cpus per "c xlarge" instance type on ec. max no. instances: 5 max no. instances: 18 our evaluation of the clovr-microbe protocol demonstrates the ability to run the same genome assembly and annotation protocol both locally and on the cloud for increased throughput. a single configuration setting is changed to invoke the pipeline on either the local desktop or the supported clouds. clovr provides automated resource provisioning in the cloud elasticity, i.e. dynamic provisioning of resources, is a primary feature of the cloud and allows for the addition of computational resources on-demand. figure shows the automatic allocation of clovr vm instances to the cluster on the cloud and the subsequent termination of idle instances upon job completion for the microbial genome assembly and annotation steps of the clovr-microbe pipeline and demonstrates dynamic capabilities provided by clovr. figure shows a blastx comparison using clovr-search on clusters composed of up to c xlarge instances, comprising cpus. this blastx search ran on amazon ec with a throughput of ~ mbp per c xlarge instance, at an estimated total cost of ~$ per hour for all instances. resource provisioning for clovr-microbe is automatic; for clovr-search it is configured by the user but does not require the direct user interaction with the remote cluster on the cloud. clovr uses local disks and does not rely on network file systems during pipeline execution bottlenecks in reading or writing data on a shared, network-based file system, such as nfs, can cause performance problems during processing, especially when many concurrent processes are executing against the shared resource. to avoid data transfer bottlenecks clovr uses local disks space of the instance running on the desktop or cloud, requiring input files to be transferred to each compute host within the cluster. for data input, these file transfers between master and worker node types are made prior to computation, for data output subsequent to job completion. in addition, reference data sets and intermediate outputs need to be accessed by all vm instances in a cluster. to improve distribution of these data sets, a peer-to-peer data transfer scheme is used for sharing intermediate results and reference data sets. figure shows data transfers within a cluster of clovr instances during a run of clovr-microbe on amazon ec to evaluate the performance of data transfers, the throughput for providing gb of compressed reference data to a cluster of c xlarge clovr vm instances was tested. instances came online in a staggered fashion and received copies of the reference data upon boot of the instance. the aggregate data throughput exceeded gb per second. by comparison, network transfer speeds between a pair of c xlarge instance types on the amazon ec network were found to typically fall below ~ mb per second. discussion clovr reduces bottlenecks in sequence analysis by using two related technologies: virtual machines and cloud computing. clovr simplifies deployment of complex bioinformatics workflows by providing a single executable that is portable across all major operating systems. by default, clovr runs on a desktop but enables seamless access to large distributed computing resources including amazon ec and other cloud computing platforms, providing a potential alternative to building and maintaining in-house infrastructure for computational analysis. while genomic workbench systems focus on providing extensive customization, i.e. choices of multiple tools for easy integration into user-customized pipelines, many projects instead rely on static, standardized analysis pipelines. to accommodate this scenario, clovr provides pre-defined standard pipelines that integrate tools for particular analysis objectives so that no additional configuration or expertise with individual tools by the user is required. this level of automation is intended to enable genomics applications for users that find choice of bioinformatics tools overwhelming and instead seek recommendations for best practices. ongoing work includes a web-based user interface, which will run locally on a user's desktop, and further simplify execution of analysis protocols. clovr utilizes a recipe-driven process to build vms for both the desktop and cloud computing platforms, which allows for construction of customized vms by other software developers. while clovr currently includes a set of protocols for microbial genome analysis, the clovr vm can serve as a general platform for the integration of additional tools and protocols developed by the research community. to add new protocols to clovr, developers build recipes to install new software, deploy and test pipelines in the ergatis workflow system, and create new configuration files for the clovr api. a first step in this direction has been made by the use of clovr to create a vm for the qiime package. we plan to create a wizard to simplify this process in the future and support custom repositories of build scripts. in contrast to internet-based software-as-a-service solutions for sequence analysis, such as galaxy or taverna, which provide centralized that typically execute on dedicated resources and require users to upload data clovr follows a decentralized model where each of multiple users executes a local client vm that is independent from other clovr instances in a multi-user environment. by running on the local desktop, clovr provides the opportunity to utilize substantial computing power provided by multi-core desktop cpus, potentially avoiding the need for data transfer over the internet and for use of the cloud entirely. the decentralized clovr architecture saves all pipeline outputs locally on the personal computer, providing users additional controls on maintaining data privacy. although clovr transfers data to cloud servers for processing, clovr uses the cloud as a temporary resource and does not require that either inputs or results are stored permanently on the cloud. the architecture of clovr, which utilizes grid engine for job scheduling and local disks for storage, allows for migration of tools and pipelines from non-cloud versions to the cloud without reimplementation. this approach is in contrast to the use of tools developed for cloud-ready frameworks like hadoop, which are algorithms that follow mapreduce. the availability of these tools is, however, still relatively limited, since utilization of the hadoop framework requires new methods or reimplementation of existing tools. as more tools that utilize mapreduce are becoming available, hadoop is included on the vm for their potential future integration. clovr provides utilities for building private clusters of vm instances on-demand in the cloud, without expecting users start, manage, or resize clusters in the cloud. a few other systems, such as nimbus one-click clusters, galaxy cloudman and starcluster, are also designed to deploy clusters of instances in the cloud. in contrast to these systems, clovr pipelines include steps to provision these resources automatically. this enables cost savings in the case of commercial clouds, by allocating resources only as they are needed. to help ensure compatibility with multiple cloud providers and support emerging cloud computing platforms, clovr avoids reliance on proprietary features of individual cloud providers, instead utilizing only three ec api calls during pipeline execution. such core functions of the ec api are becoming a standard in middleware that provides cloud services and are expected to be supported by public and private clouds. with the increasing volume of next-generation sequencing data, data storage and transfer is increasingly important component of analysis pipelines. compute clusters often rely on centralized, shared storage systems or file servers to simplify access to data for users and pipelines. as part of the design to be both portable and scalable on cloud computing networks, clovr does not rely on a shared, network file system, such as nfs, for storage. instead, clovr relies on local disk storage on either the users' desktop to store pipeline input and output, or temporary disk storage available on the cloud vm instances during pipeline execution. by using local disk for storage rather than the network, clovr can be expected to run on commodity cloud systems with relatively slow networking and without reliance on the specialized storage features of cloud providers, such as amazon elastic block storage. increasing data volumes can be an impediment for utilizing the cloud, as this data needs to be transferred over the internet. a strategy for moving analysis to data, rather than transferring data over the network, has been raised as a potential solution to dealing with data transfer bottlenecks. the portability of the vm provides such flexibility. the clovr vm is gb compressed and can be easily transferred to computational resources that are co-located with large data sets. similarly, reference datasets can be saved on the cloud to avoid data transfers over the internet, such as is done for uniref, which is a gb compressed reference dataset hosted in the cloud to support the clovr-microbe protocol. the clovr pipelines are composed of multiple steps, only some of which are computationally demanding or support parallelization on multiple cpus. to match pipeline needs with available resources, each clovr pipeline includes steps to automatically provision cloud resources as needed. one strategy for efficient allocation of resources is to estimate runtimes for steps that execute in parallel, in order to only provision resources that can be used. the cunningham utility, which is implemented in clovr currently estimates blast runtimes during pipeline executions of clovr-search and clovr-metagenomics. this strategy helps in avoiding over-partitioning of the input data, which introduces overhead that degrades overall performance, and starting too many instances for small searches or too few instances for larger searches. the ability to predict runtimes can also be used to provide an a priori estimation to the user of how much an analysis will cost or whether a particular analysis is even feasible. we plan to explore providing such estimates as future work and anticipate this will be of much interest to users of the software.  we have designed, built, and tested a portable virtual machine, named clovr, that provides automated analysis pipelines for microbial genomics. clovr provides a stand-alone client vm for execution on a personal computer providing the ability to perform sophisticated analyses using local resources and cloud computing platforms if additional resources are needed for increased throughput. by providing fully automated pipelines, the clovr vm allows users without extensive bioinformatics background to process sequence data, lowering the barrier of entry for microbial sequence analysis. availability and requirements the clovr vm is freely available for download from http://clovr.org • project name: clovr • project home page: http://clovr.org • operating system: platform independent • other requirements: vmware, virtualbox virtual machine players • license: bsd • any restrictions to use by non-academics: none authors' contributions sva and wff conceived and designed the project with the help of ow. sva, mm, ag implemented the supporting api and custom clovr vm. sva, kg, mv, drr, ca, jrw implemented and tested the clovr pipelines. sva and jrw ran experiments for the paper. sva and wff drafted the manuscript. all authors read and approved the final manuscript. supplementary material additional file 1 clovr screencast. a short screencast launching and clovr and using the web interface to launch an analysis. click here for file acknowledgements and funding this research was supported by funds from the national human genome research institute, nih/nhgri, arra under grant no. rc hg005597- the national science foundation, nsf, under grant no. and the amazon web services in education research grants program. we also thank michelle giglio gwinn for help integrating the microbial annotation pipeline and anup mahurkar, dave kemeza, victor felix, and brian cotton for it assistance and help using the diag.
in addition clovr supports use of remote cloud computing resources to improve performance for large-scale sequence processing. clovr is implemented as a single portable virtual machine that provides several automated analysis pipelines for microbial genomics, including 16s, whole genome and metagenome sequence analysis. cloud computing platforms provide on-demand access to computing infrastructure over the internet and can be used in combination with custom built virtual machines to distribute pre-packaged with pre-configured software. the clovr vm runs on a personal computer, utilizes local computer resources and requires minimal installation, addressing key challenges in deploying bioinformatics workflows. we describe the cloud virtual resource, clovr, a new desktop application for push-button automated sequence analysis that can utilize cloud computing resources. the clovr vm and associated architecture lowers the barrier of entry for utilizing complex analysis protocols on both local single- and multi-core computers and cloud systems for high throughput data processing. in a case study, we demonstrate the use of clovr to automatically process next-generation sequencing data on multiple cloud computing platforms. next-generation sequencing technologies have decentralized sequence acquisition, increasing the demand for new bioinformatics tools that are easy to use, portable across multiple platforms, and scalable for high-throughput applications.
 next-generation sequencing technologies have decentralized sequence acquisition, increasing the demand for new bioinformatics tools that are easy to use, portable across multiple platforms, and scalable for high-throughput applications. cloud computing platforms provide on-demand access to computing infrastructure over the internet and can be used in combination with custom built virtual machines to distribute pre-packaged with pre-configured software.  we describe the cloud virtual resource, clovr, a new desktop application for push-button automated sequence analysis that can utilize cloud computing resources. clovr is implemented as a single portable virtual machine that provides several automated analysis pipelines for microbial genomics, including 16s, whole genome and metagenome sequence analysis. the clovr vm runs on a personal computer, utilizes local computer resources and requires minimal installation, addressing key challenges in deploying bioinformatics workflows. in addition clovr supports use of remote cloud computing resources to improve performance for large-scale sequence processing. in a case study, we demonstrate the use of clovr to automatically process next-generation sequencing data on multiple cloud computing platforms.  the clovr vm and associated architecture lowers the barrier of entry for utilizing complex analysis protocols on both local single- and multi-core computers and cloud systems for high throughput data processing. 