Package: vipe
AboutThis is a tool for visualizing Apache Oozie workflows as data flow
pipelines.Fig. 1: Visual summary of what the tool does.The tool is a command-line application that ingests imperative
description of a workflow in Apache Oozie XML file and converts it to a
data pipeline representation in PNG image file. Note that in order for
the application to be able to extract the pipeline representation,
content of the Oozie XML file has to follow certain conventions (e.g.,
the names of Oozie action properties that correspond to ports have to
follow a convention of being prefixed with “input” or “output” string).
See filevipe/oozie/converter/iis.pyfor a code which follows such
conventions used in workflow definitions ofOpenAIRE IIS
project.How to install and runRunpip install vipeto install the stable version of the software
from PyPI repository. After installing the software, you can run it by
executingvipe-oozie2png(runvipe-oozie2png--helpfor usage
instructions).Note that the following libraries have to be installed in the system for
the tool to work:libyaml(this is required bypyyamllibrary used by the
solution) - on Ubuntu 14.04 system, this can be installed by runningapt-getinstalllibyaml-devGraphViz- on Ubuntu 14.04 system, this can
installed by runningapt-getinstall graphviz.GoalsThere are two main goals of the solution:Show existingworkflows without distracting technical details(i.e. a high-level/business view). In order to achieve it, the
application shows only data dependencies between workflow nodes, i.e.
if one node is a producer of data consumed by other workflow node, a
link between them is shown. If two nodes are executed one after
another but their order is not really important (in such case the
order is defined just out of convenience or because both of them need
to have access to full computational resources), the information
about their order is not visible. The user of the application can
also specify a detail level of the visualization.Makedata passed between workflow nodes a first-class citizen.
The user of the visualization should focus on the most important
aspect of the defined workflows - flow of the data between the
modules.Example visualizationsThis section contains example visualization of various workflows. The
visualization were generated with the application version 0.5.Simple workflowBelow we show visualization of Oozie workflow`vipe/oozie/test/data/bypass/workflow.xml<vipe/oozie/test/data/bypass/workflow.xml>`__.
Internally, this workflow is converted toOozieGraphrepresentation
(see its YAML representation in`vipe/oozie/test/data/bypass/workflow.yaml<vipe/oozie/test/data/bypass/workflow.yaml>`__)
and then subsequently toPipelinerepresentation (see its YAML
representation in`vipe/oozie/test/data/bypass/pipeline.yaml<vipe/oozie/test/data/bypass/pipeline.yaml>`__)
and then finally to a PNG image.See Fig. 2-5 for visualizations of the workflow with different levels of
details as specified by the user.Fig. 2: Simple workflow visualized with the lowest level of details.Fig. 3: Simple workflow visualized with medium level of details.Fig. 4: Simple workflow visualized with medium level of details with
input and output ports shown.Fig. 5: Simple workflow visualized with the highest level of detail
with input and output ports shown.Workflows from OpenAIRE IIS projectIn this section, we show visualizations generated for real-life
workflows fromOpenAIRE IIS
project- see Fig. 6-8.Fig. 6: Primary-main workflow from OpenAIRE IIS project with medium
level of detail.Fig. 7: Primary-processing workflow from OpenAIRE IIS project with the
lowest level of detail.Fig. 8: Primary-processing workflow from OpenAIRE IIS project with
medium level of detail.FeaturesUser-visible featuresFeatures visible to the user of the application are listed below. Note
that we use a notion of port (see chapter 3 of Gregor Hohpe, Bobby
Woolf: “Enterprise Integration Patterns: Designing, Building, and
Deploying Messaging Solutions”, Addison-Wesley, 2003) corresponding to a
join point between node and connection in a data pipeline graph.The produced pipeline representation can be eitherPNG image or
YAML-formatted text file.Each workflow node can have itsinput and output ports shown;
ports are connected using arrows to show the producer-consumer
dependencies.There aremany detail levelson which the graph can be shown. The
amount of detail shown on each detail level depends on priority
assigned to different kind of nodes and on options given to the
application. The priority is implemented in a well-separated part of
the code responsible for interpreting custom conventions used in
Oozie XML file (filevipe/oozie/converter/iis.pyin the source
code contains such code for conventions used in workflow definitions
ofOpenAIRE IIS project) and
thus it is reasonably easy for a developer to adjust it to Oozie XML
conventions used in a different project.The produced graph can have eitherhorizontal or vertical
orientation.Developer-visible featuresIn this section, we describe internal features of the solution that are
of interest of people who want to extend its code.Extensibility areas. The application was designed and implemented
with extensibility in mind - we wanted to make iteasily extensible in
the following areas.Input descriptions of workflow, e.g. possibility to analyze
source code instead of Oozie XML file.Conventions used in the Oozie XMLfile, i.e. possibility to use
different conventions of describing workflows, other than the ones
used inOpenAIRE IIS project.
Namely, the developer should be required only to implement a newPipelineConverter-derived class.Output artefacts, e.g. producing website or interactive web
applications instead of static images.Processing stages. In order to attain mentioned extensibility goals,
the processing in the application was separated into stages shown in
Fig. 9.Fig. 9: Data processing in the application. Boxes correspond to data
structures or files while the arrows correspond to processing steps. The
area enclosed with dotted line shows discussed potential future
extensions of the application. Names highlighted in gray correspond to
names of classes in the source code.Intermediate representations. It is worth noting that there are two
intermediate representations of the workflow (as shown in Fig. 9):OozieGraphclass that corresponds directly to objects defined in
Oozie XML workflow file,Pipelineclass corresponding to data pipeline representation of
the processing. It contains information about dependencies between
workflow nodes and data passed between them. It doesn’t store
information about the order in which the workflow nodes are defined.APipelineConverter-derived class is used to translateOozieGraphintoPipeline.Code developmentPython packages that the application depends on are listed in therequirements.txtfile. Note that the project is written in Python 3,
so you need to install Python 3 version of these dependencies (on Ubuntu
14.04 system you can do it by executing, e.g.sudo pip3 install pytest).Thedocstringsin the code followGoogle style
guidewith types declared in accordance toSphinx’stype annotating
conventions.
Note that you have to use Sphinx version at least 1.3 if you want to
generate documentation with type annotations.Future workPossible future extensions of the application are listed below.Generate an interlinked website containing visualization of all
workflows and subworkflows along with some additional information,
like a list of all input and output ports with the type of data they
ingest or produce.Show types of data related to each port.Show links from the names of types of data related to each port to
their schemas (extracted from surrounding system’s source code).Show link to the Oozie XML workflow corresponding to given diagram
(it should be extracted from surrounding system’s source code).Show comments and descriptions from the original Oozie workflow
definition.Show some statistics related to the workflow (e.g., number of nodes).Check whether data passed between workflow nodes is compatible (i.e.
check that there is no such situation that a data produced by a
certain workflow node is incompatible with data consumed by its
consumer). This would be akin to static type checking for the
workflow.LicenseThe code is licensed under Apache License, Version 2.0