Are you still not using Version Control for Data? | by Roman Orac | Towards Data Science

Recently, in a company meeting, one of my colleagues asked: Do we use some kind of version control for data? I gave him an astonished look. Do you mean version control for code? No for data, my colleague insisted. I never heard or thought about version control for data, but it got me thinking so I did a bit of research on this topic. At the end of the post, I also share my opinion.
Here are a few links that might interest you:
- Labeling and Data Engineering for Conversational AI and Analytics- Data Science for Business Leaders [Course]- Intro to Machine Learning with PyTorch [Course]- Become a Growth Product Manager [Course]- Deep Learning (Adaptive Computation and ML series) [Ebook]- Free skill tests for Data Scientists & Machine Learning Engineers
Some of the links above are affiliate links and if you go through them to make a purchase I’ll earn a commission. Keep in mind that I link courses because of their quality and not because of the commission I receive from your purchases.
Is there a product that offers version control for data? Is there a need for it? Let’s find out.
Any game-changing product idea that I type into Google, there are already ten mature products on the market for it — what a time to be alive. There was no difference in googling for Version Control for Data. The product that caught my eye is dolt.
Dolt is the true Git for data experience in a SQL database, providing version control for schema and cell-wise for data, all optimized for collaboration. With Dolt, you can view a human-readable diff of the data you received last time versus the data you received this time. You can easily see updates you did not expect and fix the problem before you deploy the new data.
Authors say that it is like Git but for data. It is an open-source SQL database with Git-style versioning. When working on Data Science projects, we version datasets by ourselves. Many times we don’t remember what is the difference between v5 and v6. Dolt stores commit logs when we commit the code to the repository so it is easier to go back and see the changes.
Like Github is to Git, DoltHub is to Dolt. DoltHub is only free for datasets that are public on DoltHub. The price is 50$ per month, to host private repositories.
This is the million-dollar question, that I’ve been asking myself. The authors claim that the guiding use case was sharing data on the Internet. Dolt allows you to share the database instead, including schema and views, allowing you to delete all the code used to transfer data.
DoltHub allows you to “try before you buy” data. You can run SQL queries on the web to see if the data matches your needs. The data provider can even build sample queries to guide the consumer’s exploration. Via the commit log, you can see how often the data is updated. You can see who changed the data and why.
DoltHub is free for public datasets. You can sign in with Google’s Account. I created a new public repository called iris (data scientists know the dataset I am talking about).
Then you can clone the repository like wit Git — It all feels so familiar. But, before you can clone the repo, you need to install dolt CLI. Developers thought about this, so they put the install command right next to clone.
sudo curl -L https://github.com/liquidat
After you install dolt, you need to login in with the CLI tool to associate the key with your account. This will open the page in a web browser.
dolt login
Now, you can clone the repository.
dolt clone romanorac/iris
Let’s put some data in it. I took the iris dataset from sklearn, converted it into pandas DataFrame and saved it to CSV.
import numpy as npimport pandas as pdfrom sklearn import datasetsiris = datasets.load_iris()df = pd.DataFrame(data=np.c_[iris['data'], iris['target']],                  columns=iris['feature_names'] + ['target'])df.target = df.target.astype(int)df = df.reset_index()df = df.rename(    columns={        'index': 'identifier',        'sepal length (cm)': 'sepal_length',        'sepal width (cm)': 'sepal_width',        'petal length (cm)': 'petal_length',        'petal width (cm)': 'petal_width'    })df.to_csv('~/Downloads/iris/iris.csv', index=False)
To put the CSV file into Dolt, you need to create SQL table.
dolt sql -q "create table iris ( identifier int, sepal_length float, sepal_width float,petal_length float ,petal_width float, target int, primary key (identifier) )"
Then you simply import the table.
dolt table import -u -pk=identifier iris iris.csv
After the table is imported, you need to add the files, write a commit message and push it to the repository — like with Git.
git add .dolt commit -m "Initial commit"dolt remote add origin romanorac/irisdolt push origin master
Let’s check how the data looks in DoltHub. We see the familiar iris dataset.
Now, let’s change an iris sample to see the diff in action.
dolt sql --query 'UPDATE iris SET target=0 where identifier = 100'git add .dolt commit -m "Change sample 100"dolt push origin master
As a Data Scientist, I know that managing multiple versions of datasets can be messy. Especially for models that are already in production and multiple Data Scientists are working on it. Here I see a potential for Dolt. The diff functionality looks useful with smaller datasets, but I am not sure how it could be helpful with bigger datasets, where each row changes. I would also have to stress test Dolt with a bigger dataset (a few gigs of data). Does it get slower over time?
I think the real-world use case for Dolt and DoltHub is sharing data. Companies that sell the data can log each change, so customers have a transparent way of observing changes. DoltHub also showed its usability during the coronavirus epidemic, as you can manage feeds of a different quality that have the same schema using branches.
Follow me on Twitter, where I regularly tweet about Data Science and Machine Learning.