Package: noworkflow
Copyright (c) 2016 Universidade Federal Fluminense (UFF). Copyright (c)
2016 Polytechnic Institute of New York University. All rights reserved.The noWorkflow project aims at allowing scientists to benefit from
provenance data analysis even when they don’t use a workflow system.
Also, the goal is to allow them to avoid using naming conventions to
store files originated in previous executions. Currently, when this is
not done, the result and intermediate files are overwritten by every new
execution of the pipeline.noWorkflow was developed in Python and it currently is able to capture
provenance of Python scripts using Software Engineering techniques such
as abstract syntax tree (AST) analysis, reflection, and profiling, to
collect provenance without the need of a version control system or any
other environment.Installing and using noWorkflow is simple and easy. Please check our
installation and basic usage guidelines below, and thetutorial videos
at our Wiki
page.TeamThe main noWorkflow team is composed by researchers from Universidade
Federal Fluminense (UFF) in Brazil and New York University (NYU), in the
USA.João Felipe Pimentel (UFF) (main developer)Juliana Freire (NYU)Leonardo Murta (UFF)Vanessa Braganholo (UFF)CollaboratorsDavid Koop (University of Massachusetts Dartmouth)Fernando Chirigati (NYU)Paolo Missier (Newcastle University)PublicationsMURTA, L. G. P.; BRAGANHOLO, V.; CHIRIGATI, F. S.; KOOP, D.; FREIRE,
J.; noWorkflow: Capturing and Analyzing Provenance of Scripts. In:
International Provenance and Annotation Workshop (IPAW), 2014,
Cologne,
Germany.PIMENTEL, J. F. N.; FREIRE, J.; MURTA, L. G. P.; BRAGANHOLO, V.;
Collecting and Analyzing Provenance on Interactive Notebooks: when
IPython meets noWorkflow. In: Theory and Practice of Provenance
(TaPP), 2015, Edinburgh,
Scotland.PIMENTEL, J. F.; FREIRE, J.; BRAGANHOLO, V.; MURTA, L. G. P.;
Tracking and Analyzing the Evolution of Provenance from Scripts. In:
International Provenance and Annotation Workshop (IPAW), 2016,
McLean,
Virginia.PIMENTEL, J. F.; FREIRE, J.; MURTA, L. G. P.; BRAGANHOLO, V.;
Fine-grained Provenance Collection over Scripts Through Program
Slicing. In: International Provenance and Annotation Workshop (IPAW),
2016, McLean,
Virginia.PIMENTEL, J. F.; DEY, S.; MCPHILLIPS, T.; BELHAJJAME, K.; KOOP, D.;
MURTA, L. G. P.; BRAGANHOLO, V.; LUDÄSCHER B.; Yin & Yang:
Demonstrating Complementary Provenance from noWorkflow & YesWorkflow.
In: International Provenance and Annotation Workshop (IPAW), 2016,
McLean,
Virginia.PIMENTEL, J. F.; MURTA, L. G. P.; BRAGANHOLO, V.; FREIRE, J.;
noWorkflow: a Tool for Collecting, Analyzing, and Managing Provenance
from Python Scripts. In: International Conference on Very Large Data
Bases (VLDB), 2017, Munich,
Germany.HistoryThe project started in 2013, when Leonardo Murta and Vanessa Braganholo
were visiting professors at New York University (NYU) with Juliana
Freire. At that moment, David Koop and Fernando Chirigati also joined
the project. They published the initial paper about noWorkflow in IPAW
2014. After going back to their home university, Universidade Federal
Fluminense (UFF), Leonardo and Vanessa invited João Felipe Pimentel to
join the project in 2014 for his PhD. João, Juliana, Leonardo and
Vanessa integrated noWorkflow and IPython and published a paper about it
in TaPP 2015. They also worked on provenance versioning and fine-grained
provenance collection and published papers in IPAW 2016. During the same
time, David, João, Leonardo and Vanessa worked with the YesWorkflow team
on an integration between noWorkflow & YesWorkflow and published a demo
in IPAW 2016. The research and development on noWorkflow continues and
is currently under the responsibility of João Felipe, in the context of
his PhD thesis.Quick InstallationThis version of noWorkflow only supportsPython 2.7andPython
3.5. If you want to use Python 3.6, please try the2.0-alphaversion.To install noWorkflow, you should follow these basic instructions:If you have pip, just run:$pipinstallnoworkflow[all]This installs noWorkflow, PyPosAST, SQLAlchemy, python-future, flask,
IPython, Jupyter and PySWIP. The only requirements for running
noWorkflow are PyPosAST, SQLAlchemy and python-future. The other
libraries are only used for provenance analysis.If you only want to install noWorkflow, PyPosAST, SQLAlchemy and
python-future please do:$pipinstallnoworkflowIf you do not have pip, but already have Git (to clone our repository)
and Python:$gitclonegit@github.com:gems-uff/noworkflow.git$cdnoworkflow/capture$pythonsetup.pyinstallThis installs noWorkflow on your system. It will download the
dependencies from PyPIUpgradeTo upgrade the version of a previously installed noWorkflow using pip,
you should run the following command:$pipinstall--upgradenoworkflow[all]Basic UsagenoWorkflow is transparent in the sense that it requires neither changes
to the script, nor any laborious configuration. Run$now--helpto learn the usage options.noWorkflow comes with a demonstration project. To extract it, you should
run$nowdemo1$cddemo1To run noWorkflow with the demo script calledsimulation.pywith input
datadata1.datanddata2.dat, you should run$nowrun-vsimulation.pydata1.datdata2.datThe-voption turns the verbose mode on, so that noWorkflow gives you
feedback on the steps taken by the tool. The output, in this case, is
similar to what follows.$nowrun-vsimulation.pydata1.datdata2.dat[now]removingnoWorkflowboilerplate[now]settinguplocalprovenancestore[now]collectingdefinitionprovenance[now]registeringuser-definedfunctions[now]collectingdeploymentprovenance[now]registeringenvironmentattributes[now]searchingformoduledependencies[now]registeringprovenancefrom1369modules[now]collectingexecutionprovenance[now]executingthescript[now]theexecutionoftrial10finishedsuccessfullyEach new run produces a different trial that will be stored with a
sequential identification number in the relational database.Verifying the module dependencies is a time consuming step, and
scientists can bypass this step by using the-bflag if they know that
no library or source code has changed. The current trial then inherits
the module dependencies of the previous one.It is possible to collect more information than what is collected by
default, such as variable usages and dependencias. To perform a dynamic
program slicing and capture those information, just run$nowrun-eTracersimulation.pydata1.datdata2.datTo list all trials, just run$nowlistAssuming we run the experiment again and then runnow list, the
output would be as follows. Note that 9 trials were extracted from the
demonstration.$nowlist[now]trialsavailableintheprovenancestore:Trial1:simulation.pydata1.datdata2.datwithcodehash9f13b9b35f5215a82f9b12f9f32238dddf02646aranfrom2016-01-1319:06:53.740877to2016-01-1319:07:13.250622Trial2:simulation_complete.pywithcodehash705471548f6253da20302333f0a3f79059d79e40ranfrom2016-01-1319:07:13.583000to2016-01-1319:07:39.225553Trial3:simulation.pydata1.datdata2.datwithcodehashba58136d9eac420930d352c127a78988c226dff8ranfrom2016-01-1319:07:39.530637to2016-01-1319:07:58.513666Trial4:simulation.pydata2.datdata1.datwithcodehash9f13b9b35f5215a82f9b12f9f32238dddf02646aranfrom2016-01-1319:07:58.953236to2016-01-1319:08:20.822072Trial5:simulation.py<restore3>withcodehash16d9ba96a1dfa97d26fd5009b19f872a4fa5cb57ranfrom2016-01-1319:08:21.146970toNoneTrial6:simulation.pydata1.datdata2.datwithcodehashba58136d9eac420930d352c127a78988c226dff8ranfrom2016-01-1319:08:42.827121to2016-01-1319:09:02.137061Trial7:simulation.pydata1.datdata2.datwithcodehash16d9ba96a1dfa97d26fd5009b19f872a4fa5cb57ranfrom2016-01-1319:09:02.430346toNoneTrial8:simulation_complete.pywithcodehash705471548f6253da20302333f0a3f79059d79e40ranfrom2016-01-1319:09:22.637177to2016-01-1319:09:46.327150Trial9:simulation.pydata1.datdata2.datwithcodehash9f13b9b35f5215a82f9b12f9f32238dddf02646aranfrom2016-01-1319:09:46.711818to2016-01-1319:10:10.998172Trial10:simulation.pydata1.datdata2.datwithcodehash9f13b9b35f5215a82f9b12f9f32238dddf02646aranfrom2016-01-1319:10:21.587332to2016-01-1319:10:41.900566Trial11:simulation.pydata1.datdata2.datwithcodehash9f13b9b35f5215a82f9b12f9f32238dddf02646aranfrom2016-01-1319:11:00.033094to2016-01-1319:11:25.632197To look at details of an specific trial, use$nowshow[trial]This command has several options, such as-mto show module
dependencies;-dto show function definitions;-eto show the
environment context;-ato show function activations; and-fto show
file accesses.Running$nowshow-a1would show details of trial 1. Notice that the function name is preceded
by the line number where the call was activated.$nowshow-a1[now]trialinformation:Id:1InheritedId:NoneScript:simulation.pyCodehash:9f13b9b35f5215a82f9b12f9f32238dddf02646aStart:2016-01-1319:06:53.740877Finish:2016-01-1319:07:13.250622[now]thistrialhasthefollowingfunctionactivationgraph:54:/home/joao/demotest/demo1/simulation.py(2016-01-1319:07:12.135981-2016-01-1319:07:13.250515)Returnvalue:None38:run_simulation(2016-01-1319:07:12.136067-2016-01-1319:07:12.201430)Arguments:data_a='data1.dat',data_b='data2.dat'Returnvalue:[['0.0','0.6'],['1.0','0.0'],['1.0','0.0'],...To restore files used by trial 1, run$nowrestore1By default, the restore command will restore the trial script, imported
local modules and the first access to files. Use the option-sto
leave out the script; the option-lto leave out modules; and the
option-ato leave out file accesses. The restore command track the
evolution history. By default, subsequent trials are based on the
previous Trial (e.g. Trial 2 is based on Trial 1). When you restore a
Trial, the next Trial will be based on the restored Trial (e.g. Trial 3
based on Trial 1).The restore command also provides a-f pathoption. This option can be
used to restore a single file. With this command there are extra
options:-t path2specifies the target of restored file;-i ididentifies the file. There are 3 possibilities to identify files: by
access time, by code hash, or by number of access.$nowrestore1-fdata1.dat-i"A|2016-01-13 19:06:59"$nowrestore1-foutput.png-i90451b101-toutput_trial1.png$nowrestore1-fsimulation.py-i1The first command queries data1.dat of Trial 1 accessed at “2016-01-13
19:06:59”, and restores the resulting content after the access. The
second command restores output.png with subhash 90451b101, and save it
to output_trial1.png. The third command restores the first access to
simulation.py, which represents the trial script.The option-fdoes not affect evolution history.The remaining options of noWorkflow arediff,export,history,dataflow, andvis.Thediffoption compares two trials. It has options to compare modules
(-m), environment (-e), file accesses (-f). It has also an option
to present a brief diff, instead of a full diff (–brief)Theexportoption exports provenance data of a given trial to Prolog
facts, so inference queries can be run over the database.Thehistoryoption presents a textual history evolution graph of
trials.Thedataflowoption exports fine-grained provenance data (captured
through-e Tracer) to a graphviz dot representing the dataflow. This
command has many options to change the resulting graph. Please, run “now
dataflow -h” to get their descriptions.$nowdataflow6-l-mprospective|dot-Tpng-oprospective.pngThevisoption starts a visualization tool that allows interactive
analysis:$nowvis-bThe visualization tool shows the evolution history, the trial
information, an activation graph. It is also possible to compare
different trials in the visualization tool.The visualization tool requires Flask to be installed. To install Flask,
you can run$pipinstallflaskIPython InterfaceAnother way to run, visualize, and query trials is to use Jupyter
notebook with IPython kernel. To install Jupyter notebook and IPython
kernel, you can run$pipinstalljupyter$pipinstallipython$jupyternbextensioninstall--py--sys-prefixnoworkflow$jupyternbextensionenablenoworkflow--py--sys-prefixThen, to run Jupyter notebook, go to the project directory and execute:$jupyternotebookIt will start a local webserver where you can create notebooks and run
python code.Before loading anything related to noworkflow on a notebook, you must
initialize it:In[1]:%load_extnoworkflow...:importnoworkflow.now.ipythonasnipIt is equivalent to:In[1]:%load_extnoworkflow...:nip=%now_ipAfter that, you can either run a new trial or load an existing object
(History,Trial,Diff).There are two ways to run a new trial:1- Load an external fileIn[1]:arg1="data1.dat"arg2="data2.dat"In[2]:trial=%now_runsimulation.py{arg1}{arg2}...:trialOut[2]:<Trial12># Loads the trial object represented as a graph2- Load the code inside a cellIn[3]:arg=4In[4]:%%now_run--namenew_simularion--interactive...:l=range(arg)...:c=sum(l)...:print(c)6Out[4]:<Trial13># Loads the trial object represented as a graphIn[5]:cOut[5]:6Both modes supports all thenow runparameters.The–interactivemode allows the cell to share variables with the
notebook.Loading existing trials, histories and diffs:In[6]:trial=nip.Trial(1)# Loads trial with Id = 1...:trial# Shows trial graphOut[6]:<Trial1>In[7]:history=nip.History()# Loads history...:history# Shows history graphOut[7]:<History>In[8]:diff=nip.Diff(1,3)# Loads diff between trial 1 and 3...:diff# Shows diff graphOut[8]:<Diff13>To visualize the dataflow of a trial, it is possible to use the dot
attribute of trial objects: ```python In [9]: trial.dot Out [9]:This command requires an installation of graphviz.There are attributes on those objects to change the graph visualization,
width, height and filter values. Please, check the documentation by
running the following code on jupyter notebook:In[10]:trial?In[11]:history?It is also possible to run prolog queries on IPython notebook. To do so,
you will need to install SWI-Prolog with shared libraries and the pyswip
module.You can install pyswip module with the command:$pipinstallpyswip-altCheck how to install SWI-Prolog with shared libraries athttps://github.com/yuce/pyswip/blob/master/INSTALLTo query a specific trial, you can do:In[12]:result=trial.query("activation(_, 550, X, _, _, _)")...:next(result)# The result is a generatorOut[12]:{'X':'range'}To check the existing rules, please do:In[13]:%now_schemaprolog-tOut[13]:[...]Finally, it is possible to run the CLI commands inside ipython notebook:In[14]:!nowexport{trial.id}Out[14]:%...:%FACT:activation(trial_id,id,name,start,finish,caller_activation_id)....:%...:...Included SoftwareParts of the following software were used by noWorkflow directly or in
an adapted form:The Python DebuggerCopyright (c) 2001-2016 Python Software Foundation.All Rights Reserved.AcknowledgementsWe would like to thank CNPq, FAPERJ, and the National Science Foundation
(CNS-1229185, CNS-1153503, IIS-1142013) for partially supporting this
work.License TermsThe MIT License (MIT)Copyright (c) 2013 Universidade Federal Fluminense (UFF), Polytechnic
Institute of New York University.Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
“Software”), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.