Virtual Environments — Setup and Importance in Python | by Robbie Prior | Towards Data Science

Virtual environments tend to be one of those Data Science tasks that I find myself repeatedly returning to Google for, to check my syntax and such in my day-to-day usage. I hope to walk through that process and all its noteworthy elements, as a reminder to myself and hopefully as a bookmark-worthy resource for handy reference down the road for others.
I will be using virtualenv for Python on my Macbook Pro. I am mostly using JupyterLab, which will show up in some of my references later on. In this article, I cover:
Importance of Virtual EnvironmentsCreating Virtual Environments (virtualenv)Finalizing the Setup — Activating & Deactivating Virtual EnvironmentsConnecting Virtual Environment to PythonManaging Requirements/Dependencies With Pipfile (pipenv)
Importance of Virtual Environments
Creating Virtual Environments (virtualenv)
Finalizing the Setup — Activating & Deactivating Virtual Environments
Connecting Virtual Environment to Python
Managing Requirements/Dependencies With Pipfile (pipenv)
As your project portfolio grows, managing package dependencies becomes more important. Jumping from one project to another means different libraries used for various purposes and the need for version adjustment to ensure compatability. All packages are not maintained at the same cadence, so since the latest pandas release, it may no longer be compatible with that one-person maintained package you found during some late night googling.
Virtual Environments solve this issue for us. Instead of creating one work environment that we try to constantly adapt to the various projects we are working on, an enourmous hassle for the multi-taskers in-particular, we create a separate and isolated work environment for each project. Sort of like a series of containers to house and keep our project workspaces separate and independent. This allows us to enter a work environment without worrying about any other projects affecting the stability of the code we are currently working on. Thus, no more updating that pandas version each time we switch between a project to get our code to run. Instead, we simply create an environment that has certain project dependencies and versioning in place. We can switch to that environment when picking up the respective project where we left off or when needing a test bed to ensure we have the correct environment setup before shipping our work off for release.
Let’s get started with creating the first virtual environment. In your terminal you need to ensure you have virtualenv installed, you can use pip to install it. Virtualenv can be installed as shown below if needed.
pip install virtualenv
Now that the virtualenv is installed, it can be used to setup our first virtual environment. Still within the terminal, navigate to the location where you want the virtual environment folder to be created using the “ls” and “cd” commands within the terminal. Any errors encountered in the terminal about the referenced file not being found generally require a check of the current directory to ensure the correct location is being used. The following code will create our first virtual environment within the current working directory.
The {...} used, including the brackets themselves, represent the area of the code that I have used as a stand-in for whatever name the coder chooses to be used for the respective project
virtualenv {name_for_your_virtualenv}
Example: virtualenv finance_web_app
A folder should now be present in the working directory with the name of your virtual environment. It should contain a couple other nested folders such as a python installation and at a later date the programs for any future libraries installed within the environment.
Now before we start working on the project we just created the environment for, we need to activate that environment to install any needed libraries from the terminal. We should be using the same working directory used to create the environment or we will need to specify the additional path details for the virtual environment to be located . Note, “bin” is a folder inside your {name_for_your_virtualenv} folder we just created in step 2.
source {name_for_your_virtualenv}/bin/activate
Example: source finance_web_app/bin/activate
OPTIONAL: A validation that the virtual environment has been activated is performed by checking that the following code “which python” run in the terminal returns a path with the virtual environment name (ex: “finance_web_app”) in it.
which python
Once the necessary libraries for the project have been installed (covered in Section 5) we shut down the virtual environment with the following code below. After deactivation, a different virtual environment can now be activated for maintenance or updating.
deactivate
Now Given that the virtual environment is setup, we still need to connect it to the python to ensure it is recognized and can be used for our project within the shell program as a kernel.
Run the following code to “connect” the virtual environment to Python as a kernel option for each virtual environment created. Multiple virtual environments can be connected at a time and toggled between within the Python shell program.
ipython kernel install --user --name={name_for_your_virtualenv}
Example: ipython kernel install — user — name=finance_web_app
Now for running the project with the desired virtual environment (kernel). In Jupyterlab when I start a new notebook, I am shown the option below to select the environment I wish to run my project in. Python 3 is the default kernel where all libraries are installed without virtual environments. Note that the finance_web_app virtual environment we created previously shows up in the list now.
These kernel options can also be brought up by clicking in an already open notebook on the bottom left of the page, the location where it currently says “Python 3,” the designation of the current kernel with which the project is running.
In Jupiter Notebook, the kernel selection locations are slightly different from above but resemble a similar experience when creating a new project or using the kernel drop down menu in a current project.
SUCCESS! At this point though, the virtual environment created is ready to go. So watch out world, as this virtual environment is ready to give a big “hello!”
Granted, depending on the libraries needed for the project, additional installations may be needed. Installing packages is covered in the next and last section.
Removing old virtual environments later is fairly easy, the file itself can be deleted and the following code can be run to keep it from showing as an available environment in the Python shell. Just make sure the current directory matches up with the location of the file.
jupyter kernelspec uninstall {name_for_your_virtualenv}
Example: jupyter kernelspec uninstall finance_web_app
There are a couple different options for installing packages within your environment.
First, the respective virtual environment needs to be activated. Then a pip install of the package can be performed and it should be an importable library in your shell at this point (a refresh may be needed). An example of the install is seen below for the most recent version of numpy and a specific version of pandas.
pip install numpypip install pandas==0.21
If that library cannot be found afterwards when importing it in the shell, it could be that your terminal is installing to your default python location instead of your virtual environment.
If using the dependency tracking capabilities of pipenv with Pipfile, ensure the intended target virtual environment is the only one in its folder. This is because the created Pipfiles will track installed packages for any virtual environments in the same parent folder. This may require using the “cd” command to navigate the working directory to this new folder/path.
An alternative approach that should alleviate the encountering of these types of issues is using pipenv, a wrapper for the pip command. Pipenv will ensure that all packages installed are done so in a virtual environment. It can be installed with the following code without the virtual environment active.
pip install pipenv
Pipenv is optionally setup for additional functionality beyond just installing packages (more details below) by activiting the virtual environment and then running the following code. This should create Pipfile and Pipfile.lock files in the same parent folder as the virtual environment.
pipenv install
Now that we are setup, the code below shows an example of program installation using pipenv for the most recent version of numpy and a specific version of pandas.
pipenv install numpypipenv install pandas==1.2.1
In addition to this pipenv approach ensuring these packages are installed to your virtual environment, it has built in functionality to help with documenting project dependencies.
The “requirements.txt” file is thrown around a lot as a standard approach for documenting the required packages for a project. The challenge here is that is often maintained manually, which brings up potential for errors or incompleteness. The benefit of pipenv is that once the “pipenv install” command is run, two Pipfiles, or pseudo“requirements.txt” files, are created that update automatically as packages are installed (and uninstalled). The Pipfile contains a hash so that if a python file and the Pipfile are shared across to another user or downloaded from Github, they just need to run the following code below (“pipenv install”) and the Pipfiles will automatically be located using the hashed location and used to setup the environment with the necessary packages.
pipenv install
If a “requirements.txt” was previously used, those dependencies listed within can also easily be installed with pipenv as seen below.
pipenv install -r requirements.txt
This article by Murtaza Gulamali does a good job of walking through this usage of pipenv in more detail as an additional reference.
This concludes my walkthrough of virtual environment setup and day-to-day usage. Hopefully it can help others in addition to myself as a easy reference in those moments of cloudiness regarding syntax that are an everyday occurence for a beginner and even still an oft occuring brain fog moment for the experienced.
Feel free to reach out with any comments or suggestions, I would enjoy the opportunity to connect.