Common Pipenv Errors. A collection of Pipenv errors and their... | by Edward Krueger | Towards Data Science

By: Edward Krueger Data Scientist and Instructor and Douglas Franklin Teaching Assistant and Developer Advocate.
In this article, we’ll talk about Pipenv virtual environments, where to use them on your machine, how to utilize the Pipfile and Pipfile.lock feature, and some issues that can arise when these are done incorrectly. We begin with why environments are essential for data science and software development.
For more information on Environments or getting started with Pipenv, check out this article
Data Scientists are often interdisciplinary and haven’t developed the coding skills to work collaboratively with others and push projects into production. Hence, environment and dependency management skills are often lacking. This causes difficulties with code reproducibility and collaboration on projects. Data science projects that others can easily recreate, modify, and run are reproducible data science projects
Properly performed environment management practices reduce dependency version conflicts between your projects and keep the development environment from becoming bloated and unmanageable, helping users to create reproducible projects.
Pipenv combines package management and virtual environment control into one tool for installing, removing, tracking, and documenting your dependencies; and to create, use, and manage your virtual environments. Pipenv combines pip and virtualenv into a single product. For more information on Pipenv or virtual environments, check out this article!
Pipenv’s combination of package management and virtual environment control into one tool makes it a fantastic tool for data scientists and developers.
Pipenv manages dependencies on a per-project basis, so it is best to use Pipenv within your project directory. To install packages, change into your project’s directory and run:
$ cd myproject$ pipenv install requests
When you begin a project with Pipenv, the tool automatically creates a virtual environment, a Pipfile, and a Pipfile.lock. The virtual environment is named after the project directory where it is initialized. The Pipfile stores the packages or dependencies required to run the project’s code. For those who have developed with pip, the Pipfile is very similar to a requirements.txt file. The Pipfile.lock stores specific dependency versions that enable the deterministic building of an environment. When you use pipenv install to get a new dependency, the Pipfile and Pipfile.lock are automatically updated with that new dependency. In the above example, your Pipfile will contain the package 'requests,' and your Pipfile.lock will include the version installed.
With Pipenv, you’ll want to make a new directory for each project. This allows you to isolate dependencies and their associated files. This is a good practice generally and especially when creating isolated development environments.
> mkdir new_project> cd new_project> pipenv install pandas matplotlib
After the above commands, you will have the infrastructure set up for a project that needs Python, Pandas, and Matplotlib. This infrastructure consists of the directory new_project, a Pipfile containing the installed packages, and a Pipfile.lock containing their specific versions.
Creating a new directory and Pipenv environment for each project will keep your machine organized and allow for seamless transitioning from one project to another. To contribute to a project, change into a new directory and activate that directory’s environment by running:
> pipenv install> pipenv shell
Let’s say for some reason you have a Pipfile in a weird place, like at the root directory. When you use
> pipenv install
at a lower directory, ~/Desktop/new_project, for example, Pipenv will find and use the Pipfile at the root. This effectively blocks you from using the package to manage dependencies. In this example, there is one Pipfile at the root sourcing all the Pipenv virtual environments you create in lower directories. If this happens to you, you must delete the stray Pipfile that is causing issues.
Now we will walk through recreating this error. Below is a less extreme example than having the Pipfile at the root but will illustrate the same concept, having a Pipfile at one level of a path, then attempt creating another Pipfile at a lower level in that path.
Here we have created the directory pipenv_nesting_test and run pipenv install pandas. Notice this creates both the Pipfile and Pipfile.lock in this directory.
Notice that the Pipfile is at the same level as nested_directory. Now we’ll navigate into this directory and runpipenv install matplotlib.
Notice after running ls there is no Pipfile or Pipfile.lock generated. That is because Pipenv has found the Pipfile higher up in the path and modified the data in those files to include matplotlib. Below we use cat ../Pipfile to visualize that matplotlib has indeed been added to the Pipfile higher in the path.
Keeping Pipfiles in their project directories is the best way to avoid any issues on your machine with ambiguous or stray Pipftiles. Additionally, when you have a Pipfile in a project repository online, Github for example, other users can easily clone your repo, recreate your environment from the Pipfile, and run your code!
When you use the command pipenv install , a Pipfile.lock file is created. Additionally, the commandpipenv lock can be used to create a Pipfile.lock, that declares all dependencies (and sub-dependencies) of your project in their latest available versions, and stores the current hashes for the downloaded files. That means that all specific versions of dependencies and sub-dependencies are stored in the Pipfile.lock. This ensures repeatable, and most importantly, deterministic builds. However, many poor practices can cause locking to fail. Below we'll cover some methods to avoid and solve this issue.
The command pipenv install first attempts to build a virtual environment from the Pipfile.lock. If the Pipfile.lock is unavailable, Pipenv will make the virtual environment from the Pipfile and then create a Pipfile.lock.
The Pipfile.lock will not lock when it tries to install packages that don’t exist, i.e., typos.
pipenv install panadas
Pipenv looks for the package you specify, whether it exists or not. You see the following errors, and many more, in the terminal when the package does not exist or can’t be found for another reason. For now, we will focus on these two.
Locking Failed! Notice that ‘panadas’ is added to the Pipfile’s packages. This typo will be an issue because now our Pipfile contains an error, and anyone who uses this Pipfile will encounter locking and dependency errors. Be aware that the ‘Installation Succeeded’ alert is untrue; there is no Panadas to install. All that s occurred is the addition of ‘panadas’ to the Pipfile. Typos in the Pipfile are the most common reason that locking fails.
As long as this typo remains in the Pipfile, installing a package, existent or not, causes the locking to fail. You’ll need to fix the Pipfile in a text editor. I’m using nano. You can use whatever editor you are comfortable with; Sublime, VS Code, Atom, etc. Simply correct Panadas to Pandas then pipenv install.
Now let’s look at some other error messages that populated the console when locking failed.
Notice the suggestionpipenv install --skip-lock. This ought to be only used in extreme or causal programming circumstances as it bypasses the Pipfile .lock's primary purpose; locking package versions and creating the formula for installing the environment dependencies.
When cloning repository made on another machine, you can run into Pipfile.lock incompatibilities. This is because the Pipfile.lock is system-specific. If collaborating developers are using different systems, i.e., macOS and Windows, it is best to delete the Pipfile.lock and let each system solve its dependencies. In this case, make sure everyone is using the same versions of packages by pinning them manually in the Pipfile.
Another common reason locking fails is if a package in the Pipfile only exists as a pre-release version. To allow prereleases of a package, you much add the — pre flag to your install command, otherwise locking will fail.
pipenv install --pre black
You can resolve pre-release version conflicts by adding the pre flag to the installation command pipenv install --pre. After this command, your Pipfile has a new variable; 'allow_prereleases = true.' Note that this will allow the installation of any pre-release package into your environment. Now is when pinning specific package versions is useful. Pinning versions is an excellent habit to possess and can prevent dependency issues in the future.
Pipfile Management: specifying Python versions
Notice in the Pipfile image above under the requires section; you see python_version = 3.7. Pipenv can be used to specify Python versions and package versions.
To build a Pipfile or environment with a specific Python version, use the following syntax.
pipenv --python <path>#orpipenv --python python3
Note: package versions can be specified when needed.
SHOW WHAT THE PIPFILE LOOKS LIKE and mention that you can also modify this directly in the Pipfile
pipenv install flask1.1.1
Notice that this version is pinned in the Pipfile below.
You might also want to edit your Pipfile directly to change versions.
Many data scientists use Jupyter for exploratory data analysis and model selection. Jupyter can be difficult to use with a package manager. However, we can easily create an environment with Pipenv and connect it to Jupyter with a kernel.
In Jupyter, you have access to Jupyter kernels, which point to virtual environments. Virtual environments are those made with Pipenv.
Pipenv automatically maps projects to their specific virtual environments. The commandpipenv shellpoints to a virtual environment on your computer. The virtual environment is stored globally with the name of the project's root directory plus the hash of the full path to the project's root (i.e. my_project-a3de50). Because of this, if you change your project's path, you break such a default mapping, and Pipenv will no longer be able to find and to use the project's virtual environment.
Before using the below command, you will need to know your Python version and have Jupyter installed. A guide for setting this up can be found in another article, Virtual Environments for Data Science, under the ‘Pipenv Workflow Tutorial: Jupyter’ header.
To connect a Pipenv virtual env to a Jupyter kernel. First, run pipenv shell.Then inside the shell, create the Jupyter kernel so that it points to the virtual environment.
python3.7 -m ipykernel install --user --display-name matpltlib --name matpltlib
When making the kernel specify everything, to be safe. Some computers have the kernel always named Python 3, overwriting the other virtual environments when --display-name or --name are not passed. We recommend using the same name for--name and --display-name to avoid any confusion.
if you only have one version of Python, you should be able to use python rather than python3.7 above. If you are using a Mac, you will have a Python version by default. If you are unsure which Python version you have, check with:
python --version#and/orpython3 --version
After completing the above, you should now be able to run:
jupyter notebook
Or if you prefer the Lab user interface and added functionality.
jupyter lab
In Jupyter Notebook, select ‘new’ to see a dropdown to see a list of available kernels. Note that ‘example_env’ created in step 3 above is the name of the kernel.
In Jupyter Lab, you will have to click the kernel box in the upper right to access a kernel list. Or you can click the kernel tab and select ‘change kernel’ to access the same dropdown.
In a new .ipynb file, attempt to run the below code in a cell.
import pandas
If successful, you are on your way to improving your development workflow!
If any issues arose, be sure you have the correct kernel selected, that you have run pipenv shellbefore connecting the kernel and before running jupyter labor jupyter notebook.
Pipenv is an excellent tool for developers and data scientists. However, when learning these tools and methodologies, we encounter some common errors. With Pipenv, the Pipfile.lock pins specific versions by default; however, due to locking issues mentioned and other complications, the lock might fail when shared between developers. Thus, It is a best practice to pin your dependencies in your Pipfile for production and to be cautious when building environments from a Pipfile.lock. Additionally, you’ll want to check your Pipfile for typos before sharing it or using it to build an environment!
Jupyter notebooks are a useful tool for learning code or for making reports that use code to create visualizations. Using Jupyter in a Pipenv virtual environment can make dependency management easy by allowing you to import all your environment dependencies into your notebook. Connecting the virtual environment to a Jupyter Kernel can be tricky, but if you follow the above steps, you should have no difficulty!
Proper development, package, and virtual environment management practice are all critical skills to learn as a Data Scientist and Developer. Using Pipenv to manage virtual environments accelerates production and allows you to create deterministic builds for reproducing code and for pushing projects into production.
Keeping Pipfiles in their project directories is the way to avoid any issues on your machine with ambiguous or stray Pipftiles. Additionally, when you have a Pipfile in an online repository, Github for example, other users can easily clone your repo, recreate your environment, and play with your code. With Pipenv and the Pipfile, you present to others a standardized way to install project dependencies and testing and development requirements. Anyone who has a copy of your Pipfile can recreate your environment with the following.
pipenv install
This command creates an environment specified by the Pipfile. Just make sure it is typo-free and specified proper package and Python versions before sharing it!
We hope this has been informative and helpful. Good luck, and keep coding!