A Quick and Easy Guide to Managing Conda Environments
By: Renu Khandelwal
Tags: Python, Anaconda, Machine Learning, Conda, Deep Learning

A Quick and Easy Guide to Managing Conda Environments

A simple guide to managing Conda environments

Photo by Grant Ritchie on Unsplash

In this article, you will learn

Need for Conda environments

Difference between Conda environments and Virtual environments

Commands to manage Conda environments

Preserving Conda environments from existing Conda environments

You are working on multiple projects like data analysis and visualization where you need matplotlib and seaborn libraries, multi-class classification using TensorFlow 1.14, and Keras and another project where you want to use BERT model using TensorFlow 2.0 for the Sentiment Analysis.

Consider another situation where you have completed the development of your project and want to deploy the deep learning model on a different computer. You want to ensure that everything works smoothly.

How can you handle these scenarios with ease?

Option 1: Install required libraries on different computers to isolate the environment and manage dependencies or

Option 2: Create separate environments isolating the library conflicts and managing dependencies on the same computer. Also, easily extract all the dependent libraries in a file to be restored on another system.

Option 2 can be achieved by creating Conda environments or Virtual environments in Python.

What is the Conda environment?

Conda environment is a dedicated directory where we can install specific libraries and their dependencies.

Conda environments are language agnostic. You can manage Python as well as R libraries and packages.

You can manage Python as well as R libraries and packages. Conda allows us to create, export, list, remove and update environments with different versions of libraries or packages installed in them by isolating them in different environments. Thus Conda is an Environment as well as a Package manager.

Conda uses a satisfiability (SAT) solver for dependency relationships within an environment . SAT solver verifies if all of the required dependent libraries are installed in the environment, this check takes extra time but prevents the creation of broken environments.

. SAT solver verifies if all of the required dependent libraries are installed in the environment, this check takes extra time but prevents the creation of broken environments. Conda libraries are stored as binary files.

Conda uses different channels, including remote channels, for downloading packages. Channels are locations where packages are stored.

Can we not achieve the same with Virtual environments?

A Virtual environment is also used to manage python packages for different projects; however, there are few differences between Conda environments and Virtual environments.

Virtual environments are specifically for Python and hence are not language agnostic .

. Virtual environments depend on the base system install of Python, whereas Conda environments are independent of the base install.

whereas Conda environments are independent of the base install. Conda packages are binary, whereas libraries in Virtual environments are packaged as wheels or source distributions.

How do we create and manage Conda environments?

Listing all Conda environments

conda env list

Listing all Conda environments

The active environment is shown with a *

Creating a new Conda environment

conda create — name env_data_viz python=3.7

Creates a new environment with the name “env_data_viz” with the base as python 3.7

creating a new Conda environment with Python 3.7 as the base

Activating Conda environment

conda activate env_data_viz

Once the Conda environment is activated, the prompt changes to the newly set Conda environment

Activation performs two primary functions:

Adds entries to PATH for the environment and

Runs any activation scripts that the environment may contain

Install packages in Conda environment

conda install -n env_data_viz matplotlib seaborn

Installing matplotlib and seaborn to the env_data_viz Conda environment

List all packages in the Conda environment

conda list

Lists all packages and version of libraries in the current Conda environment

To list the history of each change to the current environment, you can use the revision option

conda list — revision

Revision history of installed packages

You can also revert to a previous version.

conda install — revision 0

You are reverting to revision 0 that does not contain matplotlib and seaborn after you click on “y” to confirm the deletion of the packages.

Viewing the revision again, you will see the base revision without matplotlib and seaborn.

Deactivating the current Conda environment

conda deactivate

Deactivates the current Conda environment and returns to the base

Created a new Conda environment for multi-class classification env_multi_class_tf15 with the base as python 3.7 and then installed TensorFlow 1.14 and deactivated the condo environment

conda create — name env_multi_class_tf15 python=3.7

conda install tensorflow==1.14 keras

conda deactivate

Updating env_multi_class_tf15 with TensorFlow 1.15 from base

conda install -n env_multi_class_tf15 tensorflow==1.15

Adding Channels

Conda channels are the locations where packages are stored. When we install a package, then the Conda command searches the default set of channels to download packages.

When some packages are not available in default packages, we can add the channels to Condo environments and download the packages.

Here we install PyTorch from channel “conda-forge” which is not the default channel.

conda install — channel conda-forge pytorch

conda list

Listing all the packages in the Conda environment, you can now see that some of the packages were installed from channel “conda-forge”

packages installed from default channel and conda-forge channel

Whenever we install a package in the Conda environment, it searches all the channels listed at locations in the .condarc file.

You can add the channel “conda-forge” to the .condarc file using the following command

conda config — add channels conda-forge

Restoring or Moving Conda environments

We can restore or move Conda environments using multiple methods.

Clone

Spec list using condo list

Environment.yml using the export option

Clone

Makes an exact copy of the specified Conda environments

conda create — name test_clone — clone env_multi_class_tf15

test_clone is the new cloned environment from the source Conda environment env_multi_class_tf15

Spec List

Reproduces Conda environment on the same operating system

The Spec list can be created with or without the explicit option, as shown below.

conda list >spec_list.txt #or conda list — explicit >spec_list_exp.txt

An explicit spec file shows the platform where the environment was created. This platform is the one where this spec file is known to work.

To recreate the Conda environment from the explicit spec file.

conda create — name test_spec — file spec_list_exp.txt

Export Conda environments using environment.yml file

When we want to share our project environment with other team members, we can use an export option with the .yml file to extract all the information required to create the Conda environment. The environment.yml file is not operating system specific and formatted using YAML.

environment.yml lists all the package names, and it is left to Conda to build the environment based on the package name. The export command also includes packages installed using pip.

Creating a yaml file from an existing Conda environment

conda env export --file environment.yml

The output will be environment.yml file.

Sample yml file

Recreating the Conda environment from environment.yml file. If the environment already exists, then you will get an error.

CondaValueError: prefix already exists

conda env create -f environment.yml

Removing a Conda environment

You can remove the Conda environment using the below command.

conda env remove — name test_clone — all

Listing the Conda environments, and you can see the test clone environment no longer exists.

test_clone COnda environment is removed

Conclusion:

You can create, export, list, activate, deactivate, remove, install packages, and update environments with different versions of Python and/or packages installed in them. You can export all the dependent libraries to a file that can be used to recreate the Conda environments.

References:

https://docs.conda.io/projects/conda/en/4.6.0/_downloads/52a95608c49671267e40c689e0bc00ca/conda-cheatsheet.pdf

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

https://www.anaconda.com/blog/moving-conda-environments