Introduction to DVC: Data
Version Control Tool for Machine Learning Projects | by Khuyen Tran | Towards Data Science

Git is a powerful tool for version control. It allows you to go back and forth between different versions of your code without being afraid of losing the code you change. As a data scientist, you might not only want to control different versions of your code but also control different versions of your data for the same reason: you don’t want to lose the previous data when the data is changed.
But Git is not ideal for database version control because for two reasons:
It is messy to compare the change in different versions of data with Git. I remember that I need to manually accept about 100 lines of data that were changed between 2 commits when using git pulland it was a pain
Binary files are usually big. If you commit them to your repository, its size will become much larger. It can also take a long time to commit data to Git.
Wouldn’t it be nice if you can store your data in your favorite storage services such as Amazon S3, Google Drive, Google Cloud Storage, or your own local machine while still being able to switch back and forth between different versions of data? That is when DVC comes in handy.
DVC is a system for data version control. It is essentially like Git but is used for data. With DVC, you can keep the information about different versions of your data in Git while storing your original data somewhere else.
Better yet, DVC syntax is just like Git! If you already know Git, learning DVC is a breeze.
Let’s see how to use DVC from example. I will use my repository as an example of this article. You can follow along by cloning the repository.
Start with installing the package with pip
pip install dvc
or with conda
conda install -c conda-forge dvc
Find the instruction on more ways to install DVC here.
After DVC is installed, in a Git project, initialize it by running
dvc init
.dvc/.gitignore and .dvc/config is created. Commit this change with
git commit -m "Initialize DVC"
Here is the structure of my data directory
data├── processed│   ├── test_text.txt│   ├── train_label.txt│   ├── train_text.txt│   ├── val_label.txt│   └── val_text.txt└── raw    ├── mex20_test_full.txt    ├── mex20_train_labels.txt    ├── mex20_train.txt    ├── mex20_val_labels.txt    └── mex20_val.txt
To start tracking a file or directory, use dvc add
To track the data directory, use
dvc add data
The information of the added file or directory will be stored in .dvc file nameddata.dvc . This is a small text file that stores information on how to access the original data but not the original data itself.
Since the size of this text file is small, it can be versioned like source code with Git.
Now simply commit the dvcfile as you would with source code. Make sure to add data to .gitignore beforehand to avoid committing the data.
git add data.dvcgit commit -m "add data"
Cool! Now we created a file to store the information about the original data. The next step is to figure out where to store the data
Just like how we usegit add remote to store the URL of the remote repository, we also want to use dvc remote add to store the location of the remote storage.
DVC allows us to store our data on Google Drive, Amazon S3, Azure Blob Storage, Google Cloud Storage, Aliyun OSS, SSH, HDFS, and HTTP. Since I assume everybody has Google Drive, let’s learn how to store our data on Google Drive
Start with creating a folder on Google Drive
Once we created a folder, our link will be something like this https://drive.google.com/drive/folders/1ynNBbT-4J0ida0eKYQqZZbC93juUUbVH
Simply add that link to DVC to store the location of the remote storage
dvc remote add -d remote gdrive://1ynNBbT-4J0ida0eKYQqZZbC93juUUbVH
-d stands for default. The information about the storage will be saved under .dvc/config
[core]remote = remote['remote "remote"']url = gdrive://1ynNBbT-4J0ida0eKYQqZZbC93juUUbVH
Now simply commit the config file
git commit .dvc/config -m "Configure remote storage"
And push the data to Google Drive
dvc push
That’s it! Now all of the data is pushed to Google Drive. Checkoutdvc remote add for more ways to store your data in other storage services.
To push the change we have committed so far to our remote repository, type
git push origin <branch>
Your data is stored somewhere safe. But now you or your teammates want to access the data, what should you or your teammates do?
It’s straight forward. Just use git pull to pull the change in code like you normally do. Now you have the .dvc file in your local directory.
To retrieve the data, simply type
dvc pull
to pull the data from the remote storage. That’s it!
If you are following in tutorial and you still have the data directory in your local machine, delete it to see how the dvc pullcan retrieve your data from remote storage.
To make changes, use
dvc add datagit commit data.dvc -m 'Data updates'dvc pushgit push origin <branch>
It is straight forward, isn’t it?
The whole point of using DVC is that we can switch between different versions of our data. So how exactly can we do that? Again, just like how we switch between different versions of our code with Git, use dvc checkout to switch between different versions of our data
git checkout <...>dvc checkout
For example, if we want to switch to the previous version of the data, type
git checkout HEAD^1 data.dvcdvc checkout
Now when the data reverts to the previous version, use
git commit data.dvc -m "revert to previous version"
to save the change.
Basically, we switch to another version of our code with git chechout. dvc checkout restores the corresponding versions of the .dvc files and directories from .dvc/cacheto the workspace.
Pretty cool isn’t it?
Congratulations! You have just learned about how to control different versions of your data with DVC! If you are still hesitant to use DVC for your data science project, it is time to give it a try. The syntax is similar to Git. Use:
dvc add to track the file
dvc push to push the data to the remote storage
dvc pull to pull the data from the remote storage
dvc checkout to download the other versions of your data
You and your teammates will be happy to save the changes in your data in Git with simple command lines like above while storing your data in your favorite storage service!
Feel free to play with the code in this article here
github.com
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: