Create Reproducible Data Science and Machine Learning Environments using Conda | by Elizabeth Ter Sahakyan | Towards Data Science

Data science and machine learning projects have dependencies that we rely on to make our code run, like pandas or numpy. These dependencies get updated over time and have different versions which can cause code to break if the wrong version is used. If you work with other team members, you also need to maintain the same local development environment on different computers for the code to run successfully. To create reproducible environments for projects, it’s good practice to use an environment manager that keeps track of all the dependencies — like Conda.
Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language.
- Anaconda Documentation
This article will help you develop a better workflow for managing your environments using Conda. Feel free to skip over any of the sections and jump to the code you need!
Installing Anaconda
Creating and activating a Conda environment
Installing and uninstalling dependencies
Exporting a Conda environment
Using Conda with JupyterLab
Deactivating or deleting a Conda environment
Summary of commands
Example workflow for reproducible environments
If you don’t already have Anaconda installed, follow the instructions here to install it for your operating system. You should see Anaconda-Navigator in your applications once it’s installed.
You can either create a brand new Conda environment for when you’re starting a new project or recreate an existing environment from a yaml file.
To create a new Conda environment, open up your terminal and type the following command:
conda create -n datasci-env
You will see the following prompt asking you to proceed. Type y to continue creating the environment.
If you have an existing file containing project dependencies, you can recreate the environment by running the following:
conda env create -f environment.yaml
In this case, the first line of the yaml file will set the name of the environment. The name of an environment created with the following yaml file will be called datasci-env.
Once your environment is created, you can activate it using:
conda activate datasci-env
You can distinguish whether an environment is activated or not by looking at the left hand side of your terminal. You should see it change from base to datasci-env like below:
We can check the version of python in our newly created Conda environment with the following command:
python --versionOR python3 --version
Note: By default macOS Catalina has python as Python 2, so in order to run anything in Python 3 you must use python3
This is what you’ll see when running those commands in your terminal:
This means this conda environment we just created is running version Python 3.7.3. If this is the python version you want, you can skip over the next section and move on to installing dependencies.
If you need a different version of python for your environment you will need to specify the version at the time of installation. First, let’s deactivate the environment we just created:
conda deactivate
If we want to create a Python 3.5 environment for example, we can run the following command:
conda create -n datasci-env-py35 python=3.5
Below I’m confirming that conda installed the proper version of python by activating the environment and running python3 --version.
You can list all available environments using:
conda env list
First, make sure the correct environment is activated (it should appear in parentheses or brackets on the left side of your terminal). Installing dependencies in the wrong place can cause problems if you’re working on multiple projects, so it’s important to make sure you’re in the correct environment.
Now you should be able to install your dependencies using conda install. Let’s try it with some popular data science libraries:
conda install pandas matplotlib numpy scikit-learn
You’ll receive another prompt asking you to proceed and after typing y you should see all of the packages install successfully.
If you need to install a specific version of a package, you can just name the version when installing:
conda install pandas=0.25.1
This command above also works to downgrade libraries and packages after you’ve installed them.
If you want to uninstall a package, you can use the following:
conda uninstall PACKAGE_NAME
You can check all packages and versions in your current environment using conda list which will display the following:
Notes:- If the conda install command doesn’t work for the library you’re trying to install you can try googling how to install that specific one.- Be careful of upgrading your python version when installing dependencies. You can see above that the installation upgraded my python version to 3.8.2.
Now that we have an environment with a few dependencies installed, we can export them for versioning and reproducibility. Ideally, you’ll be exporting these files to your project directory and pushing them to github to have a proper record of the version history. You can either export a txt or yaml file depending on your needs.
First make sure you have the conda environment activated and that you’re in the correct project directory in your terminal. To export the dependencies to a txt file, run the following:
# with condaconda list --export > requirements.txt# with pippip freeze > requirements.txt
To export the dependencies to a yaml file, run the following:
conda env export > environment.yaml# OR if you run into issues, try --no-buildsconda env export --no-builds > environment.yaml
After running any of these commands, you should see a new file appear in your project’s directory containing all of the dependencies. You should push this file to github so that your teammates or yourself at a later date can access and recreate the exact project environment.
If you’re working with a team, you might want to update the environment on your end at some point if others make changes by installing new dependencies. You can update an existing conda environment from a yaml file using:
conda env update --file environment.yaml
If you want to use conda environments inside JupyterLab, you’ll need to create a specific kernel for that environment.
First install jupyterlab andipykernel inside your environment, then create a new kernel:
conda install -c conda-forge jupyterlab ipykernelipython kernel install --user --name=datasci-env
Note: the name of the kernel can be different from the environment but you should keep them the same so you can remember which kernel is linked to which environment.
Now you should be able to launch JupyterLab from your terminal using the following command:
jupyter lab
And once JupyterLab is launched you can open up a new or existing notebook, click the kernels on the top right corner, and select your new kernel from the dropdown. If you don’t see the kernel, try restarting Jupyter.
If you want to list the kernels you have available in terminal:
jupyter kernelspec list
When you’re done using a conda environment, you can deactivate it in your terminal with the following command:
conda deactivate
Your terminal should now show (base) as the environment on the left:
If you’re done with a project and no longer need an environment, you can remove it using the following command:
conda env remove -n datasci-env
# create new environmentconda create -n ENV_NAME# create new environment with different version of pythonconda create -n ENV_NAME python=VERSION# create environment from existing environment.yamlconda env create -f environment.yaml# update existing environment from environment.yamlconda env update --file environment.yaml# activate environmentconda activate ENV_NAME# deactivate environmentconda deactivate# delete/remove environmentconda env remove -n ENV_NAME# list all environmentsconda env list# export requirements.txt with condaconda list --export > requirements.txt# export requirements.txt with pippip freeze > requirements.txt# export environment.yamlconda env export > environment.yaml# export environment.yaml with no buildsconda env export --no-builds > environment.yaml# install dependenciesconda install PACKAGE_NAMEORconda install -c conda-forge PACKAGE_NAME# install dependencies with a specific versionconda install PACKAGE_NAME=VERSION# uninstall dependenciesconda uninstall PACKAGE_NAME# list all dependenciesconda list# create ipython kernel (for jupyter and jupyterlab)conda install -c conda-forge jupyterlab ipykernelipython kernel install --user --name=KERNEL_NAME# list all ipython kernelsjupyter kernelspec list# uninstall/remove existing ipython kerneljupyter kernelspec uninstall KERNEL_NAME
Create new Github repo for projectClone repo locally using git clonecd into project directoryCreate a new Conda envActivate envInstall dependenciesWork on project in Jupyter or other IDE(optional) if working in Jupyter create kernel from envExport conda env to yaml or txt fileSave codeCreate .gitignore to avoid files you don’t want to add to gitAdd files to git using git add .Push environment file and code to Github
Create new Github repo for project
Clone repo locally using git clone
cd into project directory
Create a new Conda env
Activate env
Install dependencies
Work on project in Jupyter or other IDE
(optional) if working in Jupyter create kernel from env
Export conda env to yaml or txt file
Save code
Create .gitignore to avoid files you don’t want to add to git
Add files to git using git add .
Push environment file and code to Github
To reproduce environment:
Clone repo on a different computerCreate environment from yaml or txt file (don’t have to use Conda)Run code
Clone repo on a different computer
Create environment from yaml or txt file (don’t have to use Conda)
Run code
Note: If you plan on deploying your project you will probably want to maintain a separate deployment environment that contains only the dependencies needed for the deployed aspects.
You should now be able to create, activate, and install dependencies into conda environments to create reproducible data science and machine learning environments.
Feel free to connect with me on twitter @elizabethets or LinkedIn!