Package: prodmodel
Prodmodel is abuild systemfor data science pipelines.
Users, testers, contributors are welcome!Motivation·Concepts·Installation·Usage·Contributing·Contact·LicenceMotivationPerformance. No need to rerun things, everything is cached, switching between multiple versions is super easy. Prodmodel canfigure out if a particular partial code path has already been executed using a particular piece of dataand just use the cached output.Easy debugging. Every single dependency - code or data - is version controlled and tracked.Deploy to production. Models are more than just a file. Prodmodel makes sure that the correct version of label encoders,
feature transformation code and data and model files are all packaged together.ConceptsA build system is aDAGofrules(transformations),inputsandtargets.
In Prodmodelinputscan bedata,Python code,and configuration.Aruleis transforming any of the above to an output (which can in turn be depended on by other rules). Therefore rules need to be
re-executed (and their outputs re-created) if any of their dependencies change. Prodmodel keeps track all of these dependencies.The outputs of therulesaretargets. Everytargetcorresponds to an output (e.g. a model or a dataset). These outputs
are cached and version controlled.Prodmodel therefore ensurescorrectness, by executing every code (e.g. feature transformation, model building, tests) which can potentially be affected by a change, andperformance, by executing only the necessary code, saving time compared to rerunning the whole pipeline.RulesEvery rule is a statically typed function, where the inputs are targets, data, or configs. The execution of
a rule outputs some data (e.g. a different feature set or a model), which can be used in other rules.In order to use Prodmodel your code has to be structured as functions which the rules can call into.TargetsTargets are created by rule functions. Targets can be executed to generate output files.IterableDataTargetis a special target
which can be used as an iterable ofdictsto make iterating over datasets easier. RegularDataTargetscan represent any
Python object.InstallationProdmodel requires at least Python3.6. Usepipto install prodmodel.pipinstallprodmodel--userUsageCreate abuild.pyfile in your data science folder. The build file contains references to your inputs and the build rules you can execute.fromprodmodel.rulesimportrulescsv_data=rules.data_source(file='data.csv',type='csv',dtypes={...})my_model=rules.transform(objects={'data':csv_data},file='kmeans.py',fn='compute_kmeans')Now you can build your model by runningprodmodel my_modelfrom the directory ofbuild.py,
orprodmodel <path_to_my_directory>:my_modelfrom any directory.Prodmodel creates a.prodmodeldirectory under the home directory of the user to store log and config files.DocumentationCheck out a completeexample projectfor more examples.The complete list of build rules can be foundhere.Prodmodel searches for a config file under<user home dir>/.prodmodel/config. The config file can be created manually
based on thistemplate.Arguments--force_external: Some data sources are remote (e.g. an SQL server), therefore tracking changes is not always feasible.
This argument gives the user manual control over when to reload these data sources.--cache_data: Cache local data files if changed. This can be useful for debugging / reproducibility by making sure every
data source used for a specific build is saved.--output_format: One ofnone,str,bytesandlog. The output format of the data produced by the build target
written to stdout.List targets in build fileRunprodmodel ls <path_to_build>to list targets in a build file where<path_to_build>to the build file or its directory.Cleaning old cache filesRunprodmodel clean <target> --cutoff_date=<cutoff datetime>to delete output cache files of a target created before
the cutoff datetime, which has to be in%Y-%m-%dT%H:%M%S(YYYY-mm-ddTHH:MM:SS) format.ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.ContactFeel free to email me atgergely.svigruha@prodmodel.comif you have any question, need help or would like to contribute to the code.LicenceApache 2.0