Data Version control & machine learning process control using DVC & GitHub | by Nisarg Dave | Towards Data Science

Even with all the success we’ve seen today in machine learning (ML), specifically deep learning and its applications in business, the data science community still lacks good practices for organizing their projects and effectively collaborating across their varied ML projects.
Improving the productivity of machine learning workflow is necessary. As a Data Scientist at Group K Diagnostics, I deal with the hundreds of gigabytes of data. It is indeed very tough for data scientist to manage and track the data. The data tracking is necessary thing for any data science workflow. The need for data versioning is extreme, thus Data Version control (https://dvc.org) is very crucial tool for any data scientist. This is very handy and useful tool for your data science projects. The main reasons are,
DVC tracks ML models and data sets: DVC is built to make ML models shareable and reproducible.
ML project version control: Version control machine learning models, data sets and intermediate files.
ML experiment management: Harness the full power of Git branches to try different ideas instead of sloppy file suffixes and comments in code.
Deployment & Collaboration: Instead of ad-hoc scripts, use push/pull commands to move consistent bundles of ML models, data, and code into production, remote machines, or a colleague’s computer.
Major advantage here is that, we can use it with Amazon AWS S3, Microsoft Azure and other data/resource management systems. We can configure those systems to store, read, write and snap data. The setup and usage is very simple,
Install DVC using, (assuming you have python environment ready)
Install DVC using, (assuming you have python environment ready)
pip install dvc
2. Perform the DVC initialization, (assuming you are in Github repository, if not then perform)
git initdvc init
3. Now let’s assume that our project has ‘data’ folder which has various data directory (connected & downloaded through AWS S3)
├── data   │    ├── exp1│       ├── x1.png│       ├── y1.png│    ├── exp2│       ├── x2.png│       ├── y2.png│    └── exp3│       ├── x3.png│       ├── y3.png│    └── exp4│       ├── x4.png│       ├── y4.png
Now after using this data first time, we will perform the following steps.
dvc add ./data/exp1 && git add ./data/exp1.dvcdvc add ./data/exp2 && git add ./data/exp2.dvcdvc add ./data/exp3 && git add ./data/exp3.dvcdvc add ./data/exp4 && git add ./data/exp4.dvc
This will create .dvc file for each folder and that file will be committed to GitHub so we know which version of model uses which version of data.
After this we need to push the dvc changes if we are using remote data storages,
dvc push filename 
This command will push the recent changes and will update the cache, Let’s do,
dvc status
This will check for any potential change in data, if any then it will track it and we can commit it.
DVC makes collaboration very easy. Collaborators can set it up in the same way or they can pull up the data with, ‘ dvc pull filename’ command.
This procedure also works for data versions containing a lot more data than currently persistent in the data folder as dvc stores differences of arbitrary size between different versions in its cache and can therefore recreate older or newer states of the data directories by its checkout command. The checkout is of course also possible in the other direction.
For more information on commands visit their documentation here.
For using DVC with Amazon AWS S3, please visit https://dvc.org/doc/user-guide/external-dependencies
The best overall highlights and features are,
It is Git-compatible! DVC runs on top of any Git repository and is compatible with any standard Git server or provider (GitHub, GitLab, etc). Data file contents can be shared by network-accessible storage or any supported cloud solution. DVC offers all the advantages of a distributed version control system — lock-free, local branching, and versioning.It’s storage agnoistic. You can work with S3, Google cloud and Azure.It comes with Metric tracking and includes a command to list all branches, along with metric values, to track the progress or pick the best version.Language- & framework-agnosticProvides end to end support for ML pipeline framework.It can track failures!It is also compatible with HDFS, Hive & Apache Spark.
It is Git-compatible! DVC runs on top of any Git repository and is compatible with any standard Git server or provider (GitHub, GitLab, etc). Data file contents can be shared by network-accessible storage or any supported cloud solution. DVC offers all the advantages of a distributed version control system — lock-free, local branching, and versioning.
It’s storage agnoistic. You can work with S3, Google cloud and Azure.
It comes with Metric tracking and includes a command to list all branches, along with metric values, to track the progress or pick the best version.
Language- & framework-agnostic
Provides end to end support for ML pipeline framework.
It can track failures!
It is also compatible with HDFS, Hive & Apache Spark.
So give it a shot and make your life easy! Click here to get started! As a Data Scientist, I highly recommend it to each and every data scientist to use it within their ML pipeline.