Package: data-depgraph
depgraphis a tiny (<500 LOC) Python library for expressing networks
of datasets and their relationships. In this way, it is superficially
similar toAirflowandLuigi, although those tools
contain significantly more functionality.Networks are declared in terms of the relationships (graph edges)
between source and target datasets (graph nodes). Target datasets can
then report sets of precursor datasets in the correct order. This makes
it simple to throw together a build script and construct dependencies,
sequentially or with parallelization.Traditionally, eachDatasetis designed to correspond to a file. ADatasetGroupclass handles cases where multiple files can be
considered a single file (e.g. a binary data file and its XML metadata).Different kinds of resources, such as database tables, can be used
as long as they can be queried to determine whether they exist (how
how old they are, in order to tak advantage of age-based incremental
building).When aDatasetrequires a different dataset to be built to satisfy
its dependencies, it provides a reason, such as:theDatasetis missing, and so must be builttheDatasetis out of datedepgraphis intended to be a reusable component for constructing
scientific dataset build tools. Important considerations for such a
build tool are that it must:permitreproducible
analysisbe documenting so thata workflow can be easily
reportedperform fast rebuilds to enable experimentationBeyond the standard library,depgraphhas no dependencies of its own,
so it is easy to include in projects running on a laptop, on a large
cluster, or in the cloud.depgraphsupports modern Python
implementations (Python 2, Python 3, PyPy), and works on Linux, OS X,
and Windows.Important partsDatasetdefines an individual data product, represented by a
filename,name. Additional keyword arguments may be provided in order
to facilitate the build process.The ancestors of a dataset can be retrieved withDataset.parents(n),
wherenis the number of generations to include.n=0means include
only the direct parents, whilen=1includes grandparents.n=-1includes every ancestor.Dataset.roots()returns the top-level
ancestors, i.e. those with no additional parents.Similarly,Dataset.children(n)yields the descendants of a dataset,
if any.Relationships are defined withDataset.dependson(obj), whereobjis anotherDatasetinstance. Relationships can be defined
programmatically to construct large dependency graphs.A user definedbuild(dataset, reason)function (name unimportant)
takes a dataset and constructs it based on its ancestors and any other
attributes of theDataset. Thereasonis aReasonobject that
specifies the motivation for a build step.Thedepgraph.buildall()function orDataset.buildnext()method
can be used to obtain ancestor datasets and reason pairs to feed to thebuild()function. Alternatively, thebuild()function can be
decorated with thebuildmanagerdecorator, which creates a function
that automatically constructs a dataset by assembling its dependencies
in order (see the examples below).Complex dependency graphs can be visualized by using thegraphviz()function, which returns aDOT
languagestring
encoding the visual graph.ExampleDeclare a set of dependencies resembling the graph below:R0      R1      R2      R3         [raw data]
  \     /       |       |
    DA0         DA1    /
        \      /  \   /
           DB0     DB1
            \     / |  \
             \   /  |   \
              DC0  DC1  DC2        [products]fromdepgraphimportDataset,buildmanager# Define Datasets# Use an optional keyword `tool` to provide a key instructing our build tool# how to assemble this product. Here we've used strings, but another pattern# would be to provide a callback functionR0=Dataset("data/raw0",tool="read_csv")R1=Dataset("data/raw1",tool="read_csv")R2=Dataset("data/raw2",tool="database_query")R3=Dataset("data/raw3",tool="read_hdf")DA0=Dataset("step1/da0",tool="merge_fish_counts")DA1=Dataset("step1/da1",tool="process_filter")DB0=Dataset("step2/db0",tool="join_counts")DB1=Dataset("step2/db1",tool="join_by_date")DC0=Dataset("results/dc0",tool="merge_model_obs")DC1=Dataset("results/dc1",tool="compute_uncertainty")DC2=Dataset("results/dc2",tool="make_plots")# Declare dependency relationships so that depgraph and determine the order of# the buildDA0.dependson(R0,R1)DA1.dependson(R2)DB0.dependson(DA0,DA1)DB1.dependson(DA1,R3)DC0.dependson(DB0,DB1)DC1.dependson(DB1)DC2.dependson(DB1)# Option 1:# Define a function that builds individual dependencies. The *buildmanager*# decorator transforms it into a loop that builds all dependencies above a# target@buildmanagerdefbatchbuilder(dependency,reason):# [....]returnexitcodebatchbuilder(DC1)# Option 2:# Implement the build loop manuallyfromdepgraphimportbuildalldefbuild(dependency,reason):# This may have the same logic as `batchbuilder` above, but we# will call it directly rather than wrapping it in @buildmanager# [....]returnexitcodeforstageinbuildall(DC1):# A build stage is a list of dependencies whose own dependencies are met and# that are independent, i.e. they can be built in parallelfordep,reasoninstage:# Each target is a dataset with a 'name' attribute and whatever# additional keyword arguments where defined with it.# The 'reason' is a depgraph.Reason object that codifies why a# particular target is necessary (e.g. it's out of date, it's missing# and required by a subsequent target, etc.)print("Building{0}with{1}because{2}".format(dep.name,dep.tool,reason))# Call a function or start a subprocess that will result in the# target being built and saved to a filereturn_val=build(dep,reason)# Perform logging, clean-up, or error handling operations# [....]Changes0.4Performance improvementsbuildallgenerator function, which is more efficient than
repeatedly callingDataset.buildnext()0.3Cyclic graph detectionGraphviz export0.2Rewrite, droppingDependencyGraphand makingDatasetthe
primary class0.1First version, copied fromdepchainmodule of asputil package