Trends in Use of Scientific Workflows 1 7 th International Digital Curation Conference December 2011 Trends in Use of Scientific Workflows : Insights from a Public Repository and Recommendations for Best Practices

Scientific workflows are typically used to automate the processing, analysis, and management of scientific data. Most scientific workflow programs provide a user-friendly graphical user interface that enables scientists to more easily create and visualize complex workflows that may be comprised of dozens of processing and analytical steps. Furthermore, many workflows provide mechanisms for tracing provenance and methodologies that foster reproducible science. Despite their potential for enabling science, few studies have examined how the process of creating, executing, and sharing workflows can be improved. In order to promote open discourse and access to scientific methods as well as data, we analyzed a wide variety of workflow systems and publiclyavailable workflows on the public repository myExperiment. It is hoped that understanding the usage of workflows and developing a set of recommended best practices will lead to increased contribution of workflows to the public domain. 2 Trends in Use of Scientific Workflows 7 International Digital Curation Conference December 2011 Background Scientists need to process, analyze, visualize, and manage increasing amounts of scientific data. For complex analyses, scientists often must combine multiple processing steps into larger “analysis pipelines” that can involve a number of custom algorithms, specialized tools (e.g., statistics packages, geographic information systems), local and remote databases, and web services. Such scientific workflows (Figures 1,2) are typically executed repeatedly, with different combinations of inputs and parameters. Most current research efforts lack any form of workflow automation or partially accomplish such goals via a loose collection of programming scripts that link various computational steps. In recent years, scientific workflow systems such as Kepler (Altintas et al., 2004), Taverna (Hull et al., 2006; Oinn et al., 2006), VisTrails (Callahan et al., 2006), and many others have emerged as a promising technology to further simplify the creation, execution, and sharing of computational workflows among scientists and laboratories. Scientific workflow systems often support visual workflow design, execution monitoring, fault-tolerance and recovery, and the use of distributed and parallel computing resources. Perhaps the most powerful feature of state-of-the-art workflow systems is the ability to record data lineage and other provenance information during execution, thus allowing scientists to "replay" processing steps, study data dependencies, and the datasets and parameters specified (or not used! ) during any workflow run. Thus, scientific workflow systems may also foster more transparent and reproducible science. In several cases, workflows have already been either used in diagnosis of inconsistent methodology (Coombes et al., 2007), or in eliminating years of software development for researchers (Fisher et al., 2007). [Caption] Figure 1 Taverna workflow BiomartAndEMBOSSAnalysis in myExperiment for retreiving sequences from different species, aligning them, and plotting the result. © 2009 Alan Williams. Scientific workflows are usually represented as directed graphs, whose nodes 1 Kepler Project. http://www.kepler-project.org 2 Taverna. http://www.taverna. org.uk 3 Vistrails. http://www.vistrails. org 4 BiomartAndEMBOSSAnalysis. http://www.myexperiment. org/workflows/158.html Richard Littauer et al. 3 7 International Digital Curation Conference December 2011 represent workflow steps that are linked via dataflow edges, thus prescribing serial or parallel execution of nodes. Fig.1, e.g., shows a Taverna workflow; Fig.2 depicts a Kepler metagenomics workflow (Hartman et al., 2010). [Caption] Figure 2 Kepler workflow for the Alignment, Taxonomy, and Ecology of Ribosomal Sequences (Hartman et al., 2010). Workflow Repository Study Increasingly, scientists are sharing not only their data, but also software tools and scientific workflows. myExperiment.org is a public workflow repository currently containing over 2,000 workflows with more than 5,000 registered users (Goble et al., 2010). myExperiment allows one to discover workflows of interest, which can then be used or adapted for specific requirements. myExperiment also provides social tools that foster development of virtual communities around topics of interest. A feedback and attribution mechanism allows workflow developers to gain credibility and recognition from their peers. In order to understand patterns of workflow use, we studied various attributes of workflows that were deposited in myExperiment since it became available in 2007. Specifically, we sought to identify characteristics of workflows that were related to their degree of use (e.g., number of downloads). Based on our findings, we recommend a set of best practices that may increase the usability and amount of re-use of workflows. The study focused on publicly available information gathered from myExperiment both through the SPARQL endpoint that accessed the central RDF triplestore in the myExperiment ontology and through the HTML source code on the public site. The information on the HTML pages for workflows, packs, files, groups, statistics, and users was harvested using Python; all of the code used in this research has been publicly uploaded to GitHub. This harvesting, along with calibration and development of the scripts, unfortunately resulted in some of the data being affected by the research: amount of views and downloads for each workflow were marginally affected. Noise was already 5 myExperiment. http://www.myexperiment. org 6 myExperiment ontology. http://rdf.myexperiment. org/ 7 DataONE Workflow Project GitHub. https://github.com/RichardLitt/Understanding-Workflows 4 Trends in Use of Scientific Workflows 7 International Digital Curation Conference December 2011 present to some extent in the myExperiment system; for instance, the first workflow that appears by ranking (the default setting) when accessing the workflow repository on myExperiment has been viewed around 10,000 times and downloaded 4,000 times. The disparity between this high amount of usage and the average use of other workflows on myExperiment can be accounted for by it’s prominent location on myExperiment where all beginner users would likely view it as a first step when exploring the site. myExperiment can store scientific workflows from several different workflow plartforms, including Kepler, Taverna, VisTrails, Knime, and Rapid Miner. Of the workflows on the site at the time of this analysis, Rapid Miner had 153 workflows, Taverna 1 had 479 workflows, and Taverna 2 had 684 workflows. Other workflow platforms represent a combined total of 281 workflows. As Taverna workflows utilize the XML language SCUFL (Simplified Conceptual Uniform Flow Language), myExperiment stores information about Taverna workflows internal properties, whereas the other workflows are generally stored as inscrutable objects. While Rapid Miner also has some internal property information stored on the site, this study focused on Taverna workflows when studying workflow-internal metrics. This practice is not unprecedented; Wassink et al. (2009) did a similar study on the 415 Taverna workflows then on the site. [Caption] Figure 3 Sample findings from the myExperiment analysis. Richard Littauer et al. 5 7 International Digital Curation Conference December 2011 Findings Figure 3 shows some of the findings from our initial study of the myExperiment repository, based on data publicly available from the website. The numbers of views and downloads were used as a measure of workflow popularity and as a proxy for workflow use. The distribution of workflow complexity is shown in Fig. 3A. It indicates that a large percentage of workflows consist of few components. The complexity of a workflow was determined by aggregating the number of beanshells, processors, inputs, coordinations, datalinks, and operators. The number of tasks included in Taverna workflows ranged from 1 to 250. The average workflow supports 24.3 tasks; with a standard deviation of 26.6 tasks. The total number of tasks in all Taverna workflows together is 28411 (not including 76 workflows for which this information was unavailable). This is a large increase on the Wassink et al. (2009) study, which found that: “The number of tasks per workflow ranges from 1 to 70 tasks. The average workflow size is 8.8 tasks; the standard deviation is 11.7 tasks. The total number of tasks in all workflows together is 3660. ” Many workflows have other workflows embedded within them, indicating that workflows may be shared and and reused. The total number of embedded workflows is 694; the average number of embedded workflows is 0.6; the standard deviation is 1.7 embedded workflows. In addition, many workflows access web services; the average amount of web service tasks (e.g., WSDL, Biomart, SOAP, or XML services) in a workflow is 2; the standard deviation is 3.5. The total amount is 2,368 such tasks. A large percentage of workflow components are shims (components that are used to make output from one step conform to the format expected by a subsequent step), indicating the value of workflows to “glue” pre-existing components together. Shims are hard to define clearly using the SCUFL information on myExperiment, as many different sorts of components may be used for data conversaion. Using a loose aggregate of Beanshells and Processors that do not access online databases (which may themselves perform shimming processes), the average amount of shim components per workflow is 6.3, with a standard deviation of 9.3. The total aggregate comes to 7,405 components; indicating that approximately 38% of workflow components are shims. The shimming problem has received some attention in the literature (Cui et al., 2009); this figure suggests that it remains a significant problem for workflow developers. Most workflow contributors submit a single workflow,