AE of DataFlow Lifecycles
Creators: name: Hyungro Lee
Publisher: name: Zenodo
Year: 2023

description: Artifacts of data flow lifecycles, reproducing the analysis with 1000 genome workflow application regarding to Sankey diagram, caterpillar tree and ranking table. <pre><strong># Artifact Description AD </strong> A critical performance challenge in distributed scientific workflows is coordinating tasks and dataflows on distributed resources. To guide these decisions, this paper introduces data lifecycle analysis. Workflows are commonly represented using directed acyclic graphs (DAGs). Data Flow lifecycles enrich task DAGs with data objects and properties that describe data flow and how tasks interact with that flow. Lifecycles enable analysis from several important perspectives: task, data, and data flow. This paper describes representation, measurement, analysis, visualization, and opportunity identification for dataflow lifecycles. Our measurement is both distributed and scalable, using space that is constant per data file. We use lifecycles and opportunity analysis to reason about improved task placement and reduced data movement for five scientific workflows with different characteristics. Case studies show improvements of 15x 1.9x, and 10–30x. Our work is implemented in the DataLife toolsuite. There are three steps. The first step is monitoring a workflow’s I/O during executions with representative inputs. DataLife’s monitoring tool overrides I/O calls using typical dynamic linking techniques e.g., Linux LD_PRELOAD. DataLife’s focuses on POSIX and C I/O, which includes all variants of open, close, read, write, fseek, etc. Second, DataLife constructs a dataflow lifecycle graph (DFL). A DFL is a property graph, where both vertices and edges can be annotated with property values formed from raw histogram statistics. Both tasks and data files become vertices in the graph. A read operation (by a consumer task) is denoted by a directed edge from a data source (vertex) to a task sink. A write operation (by a producer task) is denoted by a directed edge from a task source to data sink. Each directed edge in the graph is annotated with a lifecycle property value representing execution times, data volumes, footprints, flow rates, and access patterns. The third step is analysis and visualization. To visualize dataflow lifecycle graphs, DataLife generates Sankey diagrams, a representation designed for data flows. To quickly highlight interesting graph entities and relations, DataLife associates property values with them, forms a projection of the entity, and rank orders the results. Finally, DataLife finds opportunities for performance improvement based on DFL patterns. <strong># Artifact Evaluation AE </strong> <strong>## Artifact Dependencies and Requirements </strong> There are no specific hardware requirements to run AE, but the minimum resources will be: - x86 64-bit CPU - 1 GB disk space (for cloning inputs from git repo) The most modern operating systems will be sufficient to run AE, including: - Linux Debian, RHEL and CentOS For the software environment, we use: - Python 3.7 - NetworkX - Plotly - pandas - matplotlib - gcc 7.5.0+ packages for the DAG graph and Sankey visualization, respectively. For the full list of requirements is provided in the requirements.txt (https://github.com/pnnl/datalife/blob/main/src/requirements.txt) <strong>## Artifact Installation &amp; Deployment Process </strong> The requirement for evaluating Dataflow Lifecycles is to obtain all of the artifacts. First, we must ensure you have 1) input data and 2) software packages available and then repeat the process to obtain the results provided in the original study. <strong>## Input data </strong> With the central application, 1000 Genomes workflow, you will find the input tar file inside the following directory `artifacts_sc23/inputs`: therefore, the input tar file can be extracted on your machine: ``` tar xvzf inputs/1000genome_inputs.tar.gz ``` ### Docker Image We provide a Docker container image already set up with input data: ``` docker pull hrlee/dataflow:sc23 ``` Inside of the container, you will find the input file. First you obtain the bash shell prompt: ``` docker run –rm -it hrlee/dataflow:sc23 bash ``` and untar the file: ``` ls -al inputs/1000genome_inputs.tar.gz tar xvzf inputs/1000genome_inputs.tar.gz ``` <strong>## Scripts to Run </strong> We provide a list of commands to repeat analysis steps to capture the results of the original study in chronological order. 1000 genomes is a central application to artifacts for evaluation, and the following commands will include all of the key components described in the paper. First update PYTHONPATH: ``` export PYTHONPATH=$PYTHONPATH:dfl_src ``` ### Figure 2 (f). Producer-Consumer Ranking Table: `python scripts/1000genome_dfl.py --task ranking` will produce `1kgenomes-producer-consumer-ranking-table.csv` +----+-----------------------------------------------------+--------------+ | | node (edge) | value (GB) | |----+-----------------------------------------------------+--------------| | 58 | ('ALL.chr1.250000.vcf', 'individuals_ID0000001') | 22.86 | | 59 | ('ALL.chr1.250000.vcf', 'individuals_ID0000002') | 22.86 | | 60 | ('ALL.chr1.250000.vcf', 'individuals_ID0000004') | 22.86 | | 61 | ('ALL.chr1.250000.vcf', 'individuals_ID0000003') | 22.86 | | 67 | ('ALL.chr1........vcf', 'sifting_ID0000006') | 14.22 | | 66 | ('sifting_ID0000006', 'sifted.SIFT.chr1.txt') | 1.41 | |----+-----------------------------------------------------+--------------| ### Figure 4 (a). DFL caterpillars: `python scripts/1000genome_dfl.py --task ctree` will produce `1kgenomes-ctree.pdf` file. ![1000genomes ctree with sankey](1kgenomes-ctree.png "1000genomes caterpillar with Sankey"){width=40%} ### Figure 5. DFL caterpillar by data branches and task joins: `python scripts/1000genome_dfl.py --task dflg-ctree` will produce `1kgenomes-dflg-ctree.pdf` file. ![1000genomes ctree with data branches and task joins](1kgenomes-dflg-ctree.png "1000genomes caterpillar with data branches and task joins"){width=40%} ### Entrypoint script to run all at once With an interactive mode using Docker, you will find `run.sh` script which executes a series of commands to produce results in order. First you obtain a shell prompt by:: `` docker run --rm -it hrlee/dataflow:sc23 bash '' and execute the ``/ae/scripts/run.sh'' script by hand. ### Figure 6. Multi-storage tier, evaluation runs: Figure 6 in the paper shows our tasks parallelism, distribution design and data staging strategies using 1000 genomes application. We provide intructions to replicate similar computing environment using Chameleon Cloud, which has a separate page for deploying Chameleon Cloud in two options: OPT1 Running tests on general compute nodes (where HDD/SDD and /dev/shm tmpfs provided) and OPT2 Running tests on stroage hierarchy node, where NVME is mounted additionally. ### Chameleon Image ID With the 1000 Genome workflow application, ``1kg-tiered-06262023'' image is prepared in the SC23 Reproducibility project for managing different storage options. You may launch an Chameleon Cloud instance with this image to have storage tier including HDD and SSD. It is also worth noting that ChameleonCloud has limited options for storage tiers, e.g., two ``storage_hierarchy'' nodes (NVMe/SSD) out of 586 compute nodes. Therefore, we can only replicate the performance of file system resources mostly available, i.e., SATA Hard Drive Disks using general computing node instances (OPT1) to evaluate the filesystem performance. The instruction of using Chameleon with the options: https://github.com/pnnl/datalife/tree/main/artifacts_sc23/chameleon_scripts/README.md ## Other Notes After much discussion with AE organizers, it makes sense for us to include single-node execution and eliminate multi-node parallelism, as ChameleonCloud does not support the SLURM cluster platform or parallel filesystems. The benefit of our DFL-G analysis is that we can still identify the patterns of data flow lifecycles independently from the execution methods and find congested data flow and significant tasks from Sankey diagrams and caterpillar trees. We select the 1000 genomes workflow application (Section 6.2 in the paper) that can be executed on a single Chameleon instance. Finally, the DLF analysis will recover all latent parallelism even if it was executed serially, and we can still match it to the paper results conceptually. Because of that our approach, the DFL-DAG's structure is independent of execution, coordination framework, and resource assignment as long as multiple workflow executions of the same input have the same DAG. It is also worth noting that ChameleonCloud has minimal options for storage tiers, e.g., two ``storage_hierarchy'' nodes (NVMe/SSD) out of 586 compute nodes. Therefore, we can only replicate the performance of file system resources mostly available, i.e., SATA Hard Drive Disks using general computing node instances to evaluate the filesystem performance. ## Artifacts We also provide the artifacts in the DOI from Zenodo. - DOI: https://doi.org/10.5281/zenodo.7846089 </pre>