Package: pdpp
pdppInstallation PrerequisitesInstallationExampleUsage from the Command Linepdpp initpdpp newpdpp custompdpp subpdpp rigpdpp runordoitpdpp graphpdpp extantpdpp enablepdpppdppis a command-line interface for facilitating the creation and maintainance of transparent and reproducible data workflows.pdppadheres to principles espoused by Patrick Ball in his manifesto on'Principled Data Processing'.pdppcan be used to create 'tasks', populate task directories with the requisite subdirectories, link together tasks' inputs and outputs, and executing the pipeline using thedoitsuite of automation tools.pdppis also capable of producing rich visualizaitons of the data processing workflows it creates:Every project that comforms with Patrick Ball's'Principled Data Processing'guidelines uses the 'task' as the quantum of a workflow. Each task in a workflow takes the form of a directory that houses a discrete data operation, such as extracting records from plaintext documents and storing the results as a.csvfile. Ideally, each task should be simple and conceptually unified enough that a short 3- or 4-word description is enough to convey what the task accomplishes.^[In practical terms, this implies that PDP-compliant projects tend to use many more distinct script files to perform what would normally be accomplished in the space of a single, longer script.]Each task directory contains at minimum three subdirectories:input, which contains all of the task's local data dependenciesoutput, which contains all of the task's local data outputs (also referred to as 'targets')src, which all of the task's source code^[Which, ideally, would be contained within a single script file.]Thepdpppackage adds two additional constraints to Patrick Ball's original formulation of PDP:All local data files needed by the workflow but which are not generated by any of the workflow's tasks must be included in the_import_directory, whichpdppplaces at the same directory level as the overall workflow during project initialization.All local data files produced by the workflow as project outputs must be routed into the_export_directory, whichpdppplaces at the same directory level as the overall workflow during project initialization.These additional constraints disambiguate the input and output of the overall workflow, which permitspdppworkflows to be embedded within one another.Installation PrerequisitesAside from an up-to-date installation ofpythonandpip(installation instructions for which can be foundhere), thepdpppackage depends ongraphviz, which must be installed before attempting to installpykrusch. Installation instructions forgraphvizcan be found at theGraphViz installation instructions page.InstallationTo installpykrusch, usepip:pipinstallpdppExampleThe first step when usingpdppis to initialize a new project directory, which must be empty. To initialize a new project directory, use the following command:pdppinitDoing so should produce a directory tree similar to this one:For the purposes of this example, a.csvfile containing some toy data has been added to the_import_directory.At this point, we're ready to add our first task to the project. To do this, we'll use thenewcommand:pdppnewUpon executing the command,pdppwill request a name for the new task. We'll call it 'task_1'. After supplying the name,pdppwill display an interactive menu which allows users to specify which other tasks in the project contain files that 'task_1' will depend upon.At the moment, this isn't a very decision to make, as there's only one other task in the project that can files can be imported from. Select it (using spacebar) and press the enter/return key.pdppwill then display a nested list of all the files available to be nominated as a dependency of 'task_1':Select 'example_data.csv' and press enter.pdppwill inform you that your new task has been created. At this point, the project's tree diagram should appear similar to this:The tree diagram shows that 'task_1' exists, that its input directory containsexample_data.csv(which is hard linked to theexample_data.csvfile in_import_, meaning that any changes to one will be reflected in the other). Thesrcdirectory also contains a Python script titledtask_1.py-- for the sake of convenience,pdpppopulates new tasks with a synonymous Python script.^[This file should be deleted or renamed if the task requires source code by a different name or using a different programming language.] For this example, we'll populate the Python script with a simple set of instructions for loadingexample_data.csv, adding one to each of the items in it, and then saving it as a new.csvfile in theoutputsubdirectory of thetask_1directory:importcsvnew_rows=[]withopen('../input/example_data.csv','r')asf1:r=csv.reader(f1)forrowinr:new_row=[int(row[0])+1,int(row[1])+1]new_rows.append(new_row)withopen('../output/example_data_plus_one.csv','w')asf2:w=csv.writer(f2)forrowinnew_rows:w.writerow(row)After runningtask_1.py, a new file calledexample_data_plus_one.csvshould be intask_1'soutputsubdirectory. If one assumes that producing this new.csvis one of the workflow's overall objectives, then it is important to letpdppknow as much. This can be accomplished using therigcommand, which is used to re-configure existing tasks. In this example, it will be used to configure the_export_directory (which is a special kind of task). Run the command:pdpprigSelect_export_from the list of tasks available, then selecttask_1(and not_import_); finally, selectexample_data_plus_one.csvas the only dependency for_export_.Once_export_has been rigged, this example project is a complete (if exceedingly simple) example of apdppworkflow. The workflow imports a simple.csvfile, adds one to each number in the file, and exports the resulting modified.csvfile.pdppworkflows can be visualized using the built-in visualization suite like so:pdppgraphThe above command will prompt users for two pieces of information: the output format for the visualization (defaults to.png) and the colour scheme to be used (defaults to colour, can be set to greyscale). Accept both defaults by pressing enter twice. Wheneverpdppgraphs a workflow, it produces four different visualizations thereof. For now, examine thedependencies_all.pngfile, which looks like this:Inpdppvisualizations, the box-like nodes represent tasks, the nodes with the folded-corners repesent data files, and the nodes with two tabs on the left-hand side represent source code.One may execute the entire workflow by using one of the two following commands (both are functionally identical):doitpdpprunIf everything worked correctly, running the example workflow should result in the following console message:.task_1When a workflow is run, thedoitautomation suite -- atop whichpdppis built -- lists the name of each task in the workflow. If the task needed to be executed, a period is displayed before the task. If the task didnotneed to be executed, two dashes are displayed before the task name, like so:--task_1This is becausedoitchecks the relative ages of each tasks' inputs and outputs at runtime; if any given task has any outputs^[Or 'targets,' indoitnomenclature.] that are older than one or more of the task's inputs,^[Or 'dependencies,' indoitnomenclature] that task must be re-run. If all of a task's inputs are older than its outputs, the task does not need to be run. This means that apdpp/doitpipeline can be run as often as the user desires without running the risk of needlessly wasting time or computing power: tasks will only be re-run if changes to 'upstream' files necessitate it. You can read more about this impressive feature of thedoitsuitehere.Usage from the Command LineWith the exception ofpdpp init, allpdppcommands must be run from the bottom-most project directory.pdpp initInitializes a newpdppproject in an empty directory.pdpp newAdds a new basic task to apdppproject and launches an interactive rigging session for it (seepdpp rigbelow for more information).pdpp customAdds a new custom task to apdppproject and launches an interactive rigging session for it (seepdpp rigbelow for more information). Custom tasks make no assumptions about the kind of source code or automation required by the task; it is up to the user to write a customdodo.pyfile that will correctly automate the task in the context of the larger workflow. More information on writing customdodo.pyfiles can be foundhere.pdpp subAdds a new sub-project task to apdppproject and launches an interactive rigging session for it (seepdpp rigbelow for more information). Sub-project tasks are distinctpdppprojects nested inside the main project -- structurally, they function identically to all otherpdppprojects. Their dependencies are defined as any local files contained inside their_import_directory (which functions as if it were aninputdirectory for a task) and their targets are defined as any local files contained inside their_export_directory (which functions as if if were anoutputdirectory for a task).pdpp rigLaunches an interactive rigging session for a selected task, which allows users to specify the task's dependencies (inputs), targets (outputs), and source code (src).^[In practice, explicit rigging of a task's source code shouldn't be necessary, as doing so is only required if there are more than one files inside a task'ssrcdirectory, or ifpdppdoesn't recognize the language of one or more of the source files. If this is the case, users are encouraged to consider using a custom task (pdpp custom) instead of a basic task.]pdpp runordoitRuns the project. Thepdpp runcommand provides basic functionality; users may pass arguments to thedoitcommand that provides a great deal of control and specificity. More information about thedoitcommand can be foundhere.pdpp graphProduces four visualizations of thepdppproject:dependencies_sparseonly includes task nodes.dependencies_fileincludes task nodes and data files.dependencies_sourceincludes task nodes and source files.dependencies_allincludes task nodes, source files, and data files.pdpp extantIncorporates an already-PDP compliant directory (containinginput,output, andsrcdirectories) that was not created usingpdppinto thepdppproject.pdpp enableAllows users to toggle tasks 'on' or 'off'; tasks that are 'off' will not be executed whenpdpp runordoitis used.