Creating a solid Data Science development environment | by Gabriel dos Santos Gonçalves | Towards Data Science

Starting a Data Science project is usually fun, at least in the beginning. You get some data, start asking questions and exploring it, make some plots, try some models and after a few minutes, you have a bunch of interesting and chaotic insights and more data wrangling to do. Then you realize you have to organize your Jupyter notebook, start commenting and versioning your code, and that you need to dedicate some time to the “not so fun” part of your analysis. And more problems ahead if you need to share your findings with others, or put the model into production, as you find out that you don’t know exactly which libraries and versions you used during your analysis.
In general, we Data Scientists tend to focus more on the results (a model, a visualization, etc) than the process itself, meaning that we don’t pay enough attention to documentation and versioning as a much as a software engineer.
That being so, there is a general need to establish good practices for the development of Data Science projects, using the proper tools available today.
OBJECTIVE: The goal of this article is to present tools and directions for Data Scientists to manage their projects in a solid and reproducible manner by using four key tools: Conda, Git, DVC, and JupyterLab. By the end of this tutorial, you are going to be able to create a repository, version your scrips, datasets and models, and replicate this same development environment on new machines.
This tutorial was performed on a Linux machine running Ubuntu 18.04, but can be easily replicated on Mac or Windows using other command-line package managers like Homebrew (Mac), or Chocolatey (Windows).
Also, we are going to use AWS S3 to store our data files with DVC. To follow the same steps on the tutorial you need an AWS account with awscli installed and configured.
The project repository created following this tutorial can be accessed on my GitHub page.
Conda is an environment and package manager that can replace pipenv and pip in Python. It is part of Anaconda, a Python (and R) distribution focused on Data Science. You can choose to install the full version (Anaconda, around 3GB) or the light version (Miniconda, around 400MB). I recommend using Miniconda as you’ll only install the libraries you need. For a broader review, please take a look on Gergely Szerovay's article on Conda.
Git is a version control system for managing software development. With Git you can keep track of all the changes made on your code stored on your repository folder. You usually use cloud services like GitHub, Bitbucket or GitLab to connect to your local repositories to manage and store your repositories. We are going to use GitHub to store our project repository, so you need an active account to follow the steps of the tutorial.
DVC (Data Version Control) is the Git equivalent for managing your datasets and machine learning models. You link your Git repository to a cloud (AWS, Azure, Google Cloud Platform, etc) or local storage through DVC to store large files, as Git is not suited for files larger than 100MB. For a complete tutorial on DVC, please take a look on Dmitry Petrov’s article.
JupyterLab is an interactive development environment for Jupyter notebooks, code, and data. It is the latest release from Project Jupyter and it offers all the functionalities found in traditional Jupyter notebooks, with a more solid interface. Notebooks have become popular on Data Science projects for doing data wrangling and visualization, as it offers a great way to dynamically explore your data.
Code Editors are a must-have tool for programmers, and there are many open-source and paid options available today. So feel free to choose the code editor that better suits your needs.
Git Clients are tools that offer a graphical user interface for versioning your code and can be an interesting addition to the set of tools to help you manage your project.
To start organizing our development environment we need first to install the tools. We’ll start by installing Git (1) and configuring it (2) using our terminal.
# 1) Install Gitsudo apt-get install git# 2) Configure your Git accountgit config --global user.name "Your Name" git config --global user.email "yourmail@mail.com"
Next, we’ll install Miniconda, by downloading its latest release (3), changing the permission for the installation file (4) and running it (5). It is also important to add the Miniconda folder to your system’s path (6) to run its command by just typing conda on the terminal.
# 3) Download Miniconda latest release for Linuxwget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh# 4) Change the permission to run the Miniconda bash filechmod +x Miniconda3-latest-Linux-x86_64.sh# 5) Run Miniconda installation file./Miniconda3-latest-Linux-x86_64.sh# 6) Export the path to Miniconda installation folderexport PATH=/home/YOURNAME/miniconda3/bin:$PATH
Now that we have the tools installed it’s time to start setting our development environment.
First, we are going to define the variables with our GitHub information (8), create a remote repository on GitHub(9), and check if the creation was successful (10). Next, we create a local folder to store our project repository (11) and README file (12). Then we initiate our local Git repository (13) and push our first commit to GitHub (14).
# 8) Define the your GitHub information as variablesGitHubName=<YourGitHubName>GitHubPassword=<YourGitHubPassword># 9) Create a new git repository on GitHub #    named "DataScience_DevEnv"curl -u $GitHubName:$GitHubPassword https://api.github.com/user/repos -d '{"name":"DataScience_DevEnv"}'# 10) Check if your new repository is available on GitHubcurl "https://api.github.com/users/$GitHubName/repos?per_page=100" | grep -w clone_url | grep -o '[^"]\+://.\+.git'# 11) Create a folder with the name of your repositorymkdir DataScience_DevEnvcd DataScience_DevEnv# 12) Create a README file for your repositoryecho "# Data Science development environment repository" >> README.md# 13) Initiate our local Git repositorygit init# 14) Add, commit and push README.md to GitHubgit add README.mdgit commit -m "first commit with README file"git remote add origin https://github.com/GabrielSGoncalves/DataScience_DevEnv.gitgit push -u origin master
We can check on our GitHub page if the repository were properly created with the first commit containing the README file.
Now that we have our Git repository all set, we are going to create our conda environment (15). We only need to define the name of our environment (-n), the python version and the libraries (eg. pandas and scikit-learn) we want to install. After the creation, we only need to type conda activate and the name of the environment (16).
# 15) Create o Conda environmentconda create -n datascience_devenv python=3.7 pandas scikit-learn# 16) Activate your environmentconda activate datascience_devenv
Now that we are working inside our conda environment, we can install JupyterLab (17) and DVC (18). Another great aspect of using conda is that it can also be used to install packages, just like we do with pip.
# 17) Install JupyterLab with# condaconda install -c conda-forge jupyterlab# or pippip install jupyterlab# 18) Install DVC with# condaconda install -c conda-forge dvc# or pippip install dvc
We can list the libraries available on our current environment by using the command list(19). We can also generate requirements files for your environments using conda or pip (20).
# 19) List your packages installed# with condaconda list# with pippip list# 20) Create requirements file# with condaconda list --export > requirements.txt# with pippip freeze > requirements.txt
To use DVC to store your large data files you need to configure a remote storage folder. We are going to use AWS S3 on our tutorial, but you have other options (local folder, Azure Blob Storage, Google Cloud Storage, Secure Shell, Hadoop Distributed File System, HTTP, and HTTPS protocols). During DVC installation you have to define the type of storage you are going to use and specify it under brackets (21). After the installation of the AWS S3 dependency for DVC, we initialize our DVC repository (22). Next, we are going to create a folder named data inside our repository to store our data files and to be versioned with DVC (23). Then we create an S3 bucket to remotely store our data files (24). It’s important to remember that we have already configured our awscli with our IAM credentials to run AWS commands using the terminal. After creating the S3 bucket, we define it as our DVC remote folder (25) and check if the last step was accepted (26). Now we can download a csv file to our data folder (27) and start versioning it with DVC (28).
# 21) Install DVC and its dependecies for connection with S3pip install dvc[s3]# 22) Initialize DVC repositorydvc init# 23) Create folder on repository to store data filesmkdir data# 24) Create S3 bucketaws s3 mb s3://dvc-datascience-devenv# 25) Define the new bucket as remote storage for DVCdvc remote add -d myremote s3://dvc-datascience-devenv# 26) List your DVC remote folderdvc remote list # 27) Download data filewget -P data/ https://dvc-repos-gsg.s3.amazonaws.com/models_pytorch_n_params.csv# 28) Add data file to DVCdvc add data/models_pytorch_n_params.csv
Whenever we add a file to DVC, it creates a .dvc file that tracks the changes made on the original file and can be versioned with Git. DVC also creates a .gitignore inside the data folder and adds the data file to it so that Git can ignore it, and we don't need to set it manually (29). Finally, we push our data file to our remote folder (the S3 bucket we created) using DVC (30).
# 29) Start tracking DVC file and .gitignore with Gitgit add data/.gitignore data/models_pytorch_n_params.csv.dvcgit commit -m "Start versioning csv file stored with DVC on S3 bucket"git push# 30) Push data file to DVC remote storage on S3 bucketdvc push
DVC can also help us built pipelines and perform experiments, making it easier to test and reproduce specific ETL steps. For more information on DVC features, please check out Gleb Ivashkevich’s article.
After installing JupyterLab we can run it by typing jupyter lab on our terminal. As a default setting, JupyterLab uses our base Python installation as the kernel, so if we try importing a library that you installed on our newly created conda environment (but not on base Python environment) we’ll get a ModuleNotFoundError . To fix this issue, we need to install the ipython kernel (31) from our environment (32). By doing this, we’ll have a kernel that corresponds to our conda environment, so every installed and newly installed library is going to be available on our JupyterLab environment. We can also check the available Jupyter kernels installed on our machine (33).
# 31) Install ipython using condaconda install ipykernel# 32) Install your kernel based on your working environmentipython kernel install --user --name=datascience_devenv# 33) List the kernels you have availablejupyter kernelspec list
As mentioned in the introduction, one important aspect of a solid development environment is the possibility to easily replicate it. One way to do it is by exporting the information about your conda environment to a YAML file (34). Remember that in order to do it you need to activate the environment first.
# 34) To export your current conda environment to YAMLconda env export > datascience_devenv.yaml# 35) Add the yaml file to our GitHub repositorygit add datascience_devenv.yamlgit commit -m 'add environment yaml to repo'git push
Until now our project repository has the following structure (36).
# 36) Project repository structuretree.├── data│   ├── models_pytorch_n_params.csv│   └── models_pytorch_n_params.csv.dvc├── datascience_devenv.yaml├── README.md└── requirements.txt
If we use the parameter -a with the command tree we can have a better notion of the configuration files that make Git and DVC (37). As mentioned before, DVC creates a .gitignore for each data file we add, so that Git avoid tracking it.
# 37) Detailed repository structuretree -a.├── data│   ├── .gitignore│   ├── models_pytorch_n_params.csv│   └── models_pytorch_n_params.csv.dvc├── datascience_devenv.yaml├── .dvc│   ├── cache│   │   └── 6f│   │       └── 387350081297a29ecde86ebfdf632c│   ├── config│   ├── .gitignore│   ├── state│   ├── tmp│   └── updater├── .git│   ├── branches│   ├── COMMIT_EDITMSG│   ├── config│   ├── description│   ├── HEAD│   ├── hooks│   │   ├── applypatch-msg.sample│   │   ├── commit-msg.sample│   │   ├── fsmonitor-watchman.sample│   │   ├── post-update.sample│   │   ├── pre-applypatch.sample│   │   ├── pre-commit.sample│   │   ├── prepare-commit-msg.sample│   │   ├── pre-push.sample│   │   ├── pre-rebase.sample│   │   ├── pre-receive.sample│   │   └── update.sample│   ├── index│   ├── info│   │   └── exclude│   ├── logs│   │   ├── HEAD│   │   └── refs│   │       ├── heads│   │       │   └── master│   │       └── remotes│   │           └── origin│   │               └── master│   ├── objects│   │   ├── 10│   │   │   └── c06accd2ad99b6cde7fc6e3f3cd36e766ce88f│   │   ├── 19│   │   │   └── 193f4a173c56c8d174ecc19700204d250e9067│   │   ├── 4e│   │   │   └── 0790499d1d09db63aaf1436ddbd91bfa043058│   │   ├── 52│   │   │   └── 4cb7d319626c1bcf24ca5184d83dc1df60c307│   │   ├── 5f│   │   │   └── 694b1bd973389b9c0cdbf6b6893bbad2c0ebc6│   │   ├── 61│   │   │   └── d5f990a1bee976a2f99b202f1dc14e33b43702│   │   ├── 67│   │   │   └── 3b06660535a92d0fdd72fe51c70c9ada47f22d│   │   ├── 70│   │   │   └── 1490f13b01089d7da8fa830bae3b6909d12875│   │   ├── 72│   │   │   └── a0ddbcc242d223cd71ee5a058fc99de2fa53cc│   │   ├── a3│   │   │   └── b5ebf7e3b752fa0da823aeb258b96e007b97ef│   │   ├── af│   │   │   └── 8017769b22fcba5945e836c3c2d454efa16bd1│   │   ├── c1│   │   │   └── 694ff5e7fe6493206eebf59ac31bf493eb7e6b│   │   ├── d7│   │   │   └── 39682b1f99f9a684cecdf976c24ddf3266b823│   │   ├── e4│   │   │   └── 5eca3c70f6f47e0a12f00b489aabc526c86e8b│   │   ├── e6│   │   │   └── 9de29bb2d1d6434b8b29ae775ad8c2e48c5391│   │   ├── ee│   │   │   └── 75f0e66a68873ac2f767c212c56411cd729eb2│   │   ├── info│   │   └── pack│   └── refs│       ├── heads│       │   └── master│       ├── remotes│       │   └── origin│       │       └── master│       └── tags├── README.md└── requirements.txt
Next, we create a .gitignore in the root of your repository for the rest of the files we don’t want to track (eg. Python compiled bytecode files, .pyc) with Git (38).
# 38) Add .gitignore for script files on our repositoryecho "*.pyc" >> .gitignoregit add .gitignoregit commit -m 'Add .gitignore for regular files'git push
Now we have our development environment configured and ready. Our JupyterLab has the kernel matching our conda environment, our data files are being versioned by DVC and our Git repository is tracking the rest of the files. So any change made on our project is going to be documented and can easily be replicated and tracked.
After setting our Git repository and configuring our DVC storage folder, we can replicate it on any new machine. Simply clone the repository (39), create a conda environment from the YAML file (40), activate it (41), create a JupyterLab kernel for our environment (42) and finally pull the data file from S3 bucket using DVC (43).
# 39) On a new machine, clone the repositorygit clone https://github.com/$GitHubName/DataScience_DevEnv.git# 40) Create conda environmentconda env create --file=datascience_devenv.yaml# 41) Activate environmentconda activate datascience_devenv# 42) Install the JupyterLab kernelipython kernel install --user --name=datascience_devenv# 43) Pull the data file from the S3 bucket using DVCdvc pull
So we can have the exactly same development environment on a new machine (including the data files and installed libraries) with just 5 commands.
In this article, we presented key tools to create a solid and reproducible development environment for Data Scientists. We do believe that Data Science is a field that can become even more mature by using best practices in project development and that Conda, Git, DVC, and JupyterLab are key components of this new approach
For a perspective on Data Science development environment more focused on practices and methods, please take a look at Will Koehrsen’s article.
You can find my other articles on my profile page 🔬
If you enjoyed it and want to become Medium a member you can use my referral link to also support me 👍