Package: lazydata
lazydata: scalable data dependencieslazydatais a minimalist library for including data dependencies into Python projects.Problem: Keeping all data files in git (e.g. via git-lfs) results in a bloated repository copy that takes ages to pull. Keeping code and data out of sync is a disaster waiting to happen.Solution:lazydataonly stores references to data files in git, and syncs data files on-demand when they are needed.Why: The semantics of code and data are different - code needs to be versioned to merge it, and data just needs to be kept in sync.lazydataachieves exactly this in a minimal way.Benefits:Keeps your git repository clean with just code, while enabling seamless access to any number of linked data filesData consistency assured using file hashes and automatic versioningChoose your own remote storage backend: AWS S3 or (coming soon:) directory over SSHlazydatais primarily designed for machine learning and data science projects.
Seethis medium postfor more.Getting startedIn this section we'll show how to uselazydataon an example project.InstallationInstall with pip (requires Python 3.5+):$pipinstalllazydataAdd to your projectTo enablelazydata, run in project root:$lazydatainitThis will initialiselazydata.ymlwhich will hold the list of files managed by lazydata.Tracking a fileTo start tracking a file usetrack("<path_to_file>")in your code:my_script.pyfromlazydataimporttrack# store the file when loadingimportpandasaspddf=pd.read_csv(track("data/my_big_table.csv"))print("Data shape:"+df.shape)Running the script the first time will start tracking the file:$pythonmy_script.py## lazydata: Tracking a new file data/my_big_table.csv## Data shape: (10000,100)The file is now tracked and has been backed-up in your local lazydata cache in~/.lazydataand added tolazydata.yml:files:-path:data/my_big_table.csvhash:2C94697198875B6E...usage:my_script.pyIf you re-run the script without modifying the data file, lazydata will just quickly check that the data file hasn't changed and won't do anything else.If you modify the data file and re-run the script, this will add another entry to the yml file with the new hash of the data file, i.e. data files are automatically versioned. If you don't want to keep past versions, simply remove them from the yml.And you are done! This data file is now tracked and linked to your local repository.Sharing your tracked filesTo access your tracked files from multiple machines add a remote storage backend where they can be uploaded. To use S3 as a remote storage backend run:$lazydataadd-remotes3://mybucket/lazydataThis will configure the S3 backend and also add it tolazydata.ymlfor future reference.You can now git commit and push yourmy_script.pyandlazydata.ymlfiles as you normally would.To copy the stored data files to S3 use:$lazydatapushWhen your collaborator pulls the latest version of the git repository, they will get the script and thelazydata.ymlfile as usual.Data files will be downloaded when your collaborator runsmy_script.pyand thetrack("my_big_table.csv")is executed:$pythonmy_script.py## lazydata: Downloading stored file my_big_table.csv ...## Data shape: (10000,100)To get the data files without running the code, you can also use the command line utility:# download just this file$lazydatapullmy_big_table.csv# download everything used in this script$lazydatapullmy_script.py# download everything stored in the data/ directory and subdirs$lazydatapulldata/# download the latest version of all data files$lazydatapullBecauselazydata.ymlis tracked by git you can safely make and switch git branches.Data dependency scenariosYou can achieve multiple data dependency scenarios by puttinglazydata.track()into different parts of the code:Jupyter notebook data dependencies by using tracking in notebooksData pipeline output tracking by tracking saved filesClass-level data dependencies by tracking files in__init__(self)Module-level data dependencies by tracking files in__init__.pyPackage-level data dependencies by tracking files insetup.pyComing soon...Examine stored file provenance and propertiesFaceting multiple files into portable datasetsStoring data coming from databases and APIsMore remote storage optionsStay in touchThis is an early stable beta release. To find out about new releases subscribe to ournew releases mailing list.ContributingThe library is licenced under Apache-2 licence. All contributions are welcome!