Environments, Conda, Pip, aaaaah! | by Dennis Bakhuis | Towards Data Science

There are many ways to install Python and unfortunately, it is very easy to create a mess. In this blog post I describe a method that works pretty well. I am aware of other great solutions, but this is generally the way I recommend, especially to beginners.
👉 This is part of my Python-10-minutes-a-day course!
While there is not much to code along, this document is still available as a Jupyter Notebook on my Github. You can also find my previous tutorials there.
A short overview of the topics we will be discussing:
Python is great, but...A way that worksA typical workflowRound up
Python is great, but...
A way that works
A typical workflow
Round up
As many others, I ❤️ Python! You can very rapidly translate your ideas into readable code solutions. One big reason Python is such a success is the very active community in which amazing people share their awesome solutions. This is why you do not have to write Data structures from scratch, but you simply import Pandas. Write the data to an hdf5 file format? Import h5py! Plot some figure, xkcd style? Import matplotlib! Even better, there are multiple flavors so if you prefer to plot your data in a different way, import one of the various other plotting system, e.g. Plotly, Bokeh, ggplot, to name a few. All this sharing goodness makes that Python is quite popular in many rapid evolving fields, such as Machine Learning.
Unfortunately, all effort from the large community comes at a price. The packages you used get updated, restructured, improved, or just rewritten, just because the authors came up with a better way to solve their problem. These changes can be breaking changes for the code you have written. Popular packages, such as Numpy or Matplotlib are very reliable, and chances that you get breaking changes are slim. However, using packages that are not as popular, breaking changing can happen, especially when upgrading the package or Python itself.
A way, the Python community solved this problem is with the use of virtual environments. These create isolated Python installations with their own set of packages. It is good practice to have a unique environment for each project or task. This ensures that dependencies of one project will not create breaking changes for another. This solution works great, but also creates some bookkeeping as you have different Python installations, for which each might come with their own package manager Pip. All of these references pointing Python and/or Pip can create a mess quite quickly and we end up in the famous diagram van XKCD below:
While all XKCDs are funny as they generally contain some truth and indeed, if you do not have some sort of system to do the bookkeeping, your Python installations can become a mess. As with all things in Python, there are many different ways to organize this, including some great tools such as Poetry, Pipenv, and many more. In the next section I will describe the system I am using. This works for me and this might also work for you. These are just my two cents, and you should of course choose whatever suits you best.
While there are many ways to organize your Python versions, virtual environments, and packages I do the following:
conda for Python and virtual environmentspip for package management inside the virtual environments
conda for Python and virtual environments
pip for package management inside the virtual environments
There are probably very nice and flexible graphical user interfaces (GUIs) out there, however I prefer the command line interace (CLI) over them. They give me the impression that I am more in control. I am not sure if that is true, but at least I get confirmation over each step that I am doing. There are many different shells to choose from and it really does not matter that much. Use the one you have available is generally the easiest. When you spend more time in the CLI, you might get pickier one day and choose a different flavor.
Windows has two different CLIs installed by default, the Command Line Prompt (CMD) and Windows Powershell. Both are fine, but the power shell gives more of an shell feeling. If you have never used a CLI, it can be useful to watch a tutorial of the power shell on Youtube. While I have not used Windows for a while, there are other options including running the famous Linux Bash shell on windows.
MacOs has by default Bash (MacOs Catalina has Zsh) which is great. You can access the shell using the Terminal application which is a way to interact with bash or zsh. I personally prefer Zsh as a shell and iTerm2 to interact with it. Both can be installed using home-brew. If you are not familiar with Bash, I highly advice to watch a tutorial on it, as it is incredibly useful.
Linux users are probably already familiar with a shell. Which shell and terminal application is installed, depends on the distribution you have installed. Generally all are fine, use what you have available.
While your shell will most likely look quite different, here is an image of my shell:
Now that we are familiar with a shell, lets get the next requirement: a conda distribution. Conda is a popular package manager for Python (and many other languages) and gives you access practically all Python versions and packages. It includes a easy system to manage virtual environments. While conda can be used to install packages, I only use it for virtual environments and Python versions. Conda has a feature called dependency checking, which works quite well, but sometimes can be a bit slow. Also, some packages are more up-to-date using Pip and therefore I choose to only use pip. Mixing both will probably work but it is probably better to use a single tool.
Often, conda is installed using Anaconda, which is a full-fledged distribution, including many packages, tools and a GUI. It installs many packages you will probably never use and I find the GUI slow to work with. The other option to install conda is to use Miniconda, another distribution which is much smaller as the name suggests. Miniconda is a base installation with a Python system, Pip, Conda, and some other tools. While it is straight forward to install, here are some simple guide lines:
Install in your home directory if and only if you do not have spaces in the full path. For example, if you have a username using a space, e.g. "dennis bakhuis", your home directory path will also contain a space (/home/dennis bakhuis/). This can cause problems with some packages as not all imports use quotes around paths, which is required for spaces folder names. If you happen to have a space in the folder, install miniconda in a different location. For example in Windows, just use the root directory: 'C:/miniconda3'
After installing miniconda, you should have the conda command available in your shell. To test this, open your shell and type "conda --version". If the command is not found, the path to miniconda has to be added to the global path variable.
For powershell users, there is an additional step which should be entered in the powershell: "conda init powershell"
This is all there is in installing the tools required to work with Python en virtual environments. In the next section I will explain a typical workflow.
Miniconda comes without any GUIs and therefore, your starting point is always the shell.
If you now open a shell, you have your new Python, Conda, and Pip ready for your bidding. You could install packages straight into your ‘base’ environment, however I highly advice against this. If you, for what ever reason make a mess of you ‘base’ environment, there is not way to delete it. Options are a reinstall (which actually is not all that bad) or removing packages by hand. Maybe there are tricks, but much easier is to just create virtual environments.
I would create an environment for each project or task, just to keep things separate. As mentioned before, I create an environment using conda:
conda create --name tutorial python=3.7
This will create a new environment with the name ‘tutorial’ and Python version 3.7.x. Because we used a single ‘=’, we tell conda to use the latest version in the Python 3.7 tree. At the moment this is version 3.7.7. If we would have used two equal signs ‘==’ we would tell conda to give exactly version 3.7, so there is a subtle difference.
After the environment is created, we have to switch to the newly created environment. For this conda has the activate command:
conda activate tutorial
Now you are in the isolated python environment called ‘tutorial’, which has its own version of Python, Conda, and pip. It is possible that you are not completely sure how a previously created environment was called. To check the available environments, you can always use:
conda env list
The previous figure shows all my available environments. These are just subdirectories in your miniconda folder. The currently active environment is shown with an asterix. Let’s now install some packages in our fleshly created environment in the next section.
In your activated environment, it is dead easy to install packages using the command ‘pip install’. For this example we will install the packages numpy, pandas, jupyterlab, matplotlib. While the dependency checking of pip is not as sophisticated as conda, it does however know that Pandas depends on numpy, and will install the dependecy if it is missing. To install the packages type:
 pip install pandas matplotlib jupyterlab
After the install, the packages are ready to go. For example to start Jupyter Lab type:
jupyter lab
Sometimes it happens when you work in a notebook, bot forgot to install that one package. For example, want to use tqdm to have some progress bars. To install that package, open another shell next to the one you are already using, activate the environment, and use pip to install tqdm. The package is immediately ready for use in the notebook you are working in. Or install it directly in you notebook using:
The ‘!’ is used to execute shell commands. For example to do an *list directory* (ls) we do:
So using ‘!pip install tqdm’ we can directly install the package in the current environment. The shell method is in my opinion a bit more explicit as you are absolutely sure in which environment you are installing the package, but both are great!.
Another great way of installing packages is using the requirements.txt file. This is a list generated using the ‘pip freeze’ command and gives exact versions of packages used and is a great way to reproduce environments from previous projects. To create a requirements.txt yourself, type the following in a shell:
pip freeze > requirements.txt
Of course you can also invoke this command from jupyter:
If you open the file, or run it without the ‘> requirements.txt’ you will see a list of all packages in your environment and the exact version behind the double equal sign. If you provide this file in the root folder of your project or git-repo, others can install all required packages with a single command:
pip install -r requirements.txt
As you can see, requirements.txt is a great way to share and reproduce environments.
After a while, you will be collecting quite some environments, which can introduce some clutter to your system. To remove environments that are not needed anymore, we can simply delete them. If you would require it again, using the requirements.txt it is dead easy to recreate the environment. Before we can remove an environment, we have to deactivate it. To do that type:
conda deactivate
Now we can delete an environment by typing:
conda env remove --name tutorial
To verify, the environment is indeed gone:
conda env list
Some commands that might be useful:Clone an existing environment:
conda create --clone tutorial --name tutorial2
Search for available packages:
pip search tensorflow
This was it for managing Pythons, ENVs, PIPs, and all the snakes. As I mentioned before, this is one way to do it and there are many others. The other ways might work better, but this works for me. Feel free to comment on how this process can be done better or what works for you.
Please let me know if you have any questions or suggestions.