Data Science Best Practices: Python Environments | by Jerry Yu | Towards Data Science

Virtual environments are isolated coding spaces where Python packages can be installed, upgraded, and used, instead of packages being installed globally in your working environment. Virtual environments can be installed with a .txt or .yml file, that lists the package names and their versions, through the terminal/command prompt/anaconda prompt. When you install or upgrade a package, the old versions stay installed in your directory, so different virtual environments can utilize different package versions.
Picture this, you spend months cleaning, manipulating, creating features, training your model, improving the model, and evaluating your model performance; you operationalize your model and put it into production. At this point in time, you haven’t heard about virtual environments, so you move onto the next project. You find a Python package you need for the next project, but you first have to update say, Pandas, so you decided to update all of your packages all in one go using:
conda update --all
You go about your day, but get a warning from your last project, which is currently in production, telling you something is broken. If you’re lucky and the bug isn’t obscure, you are able to fix it. However, if you’re unlucky like my coworkers, and happen to run your production models on a server, AND you break the whole Python path environment, well, bad luck. Time to wipe it clean and host a new server.
You would also want to use project-specific virtual environments to ensure code reproducibility by indicating all packages with their specified versions in the environment. You can also bypass admin rights that might be needed when installing a package, by using a virtual environment. Lastly, using environments is another step you can take at better organizing your projects, or avoid installing packages to your default path when you only need it for one project.
Here is one of the way to create an environment using conda
conda create --name myenv    ## Replace myenv with environment name
The above code will create the environment in the current path using the same version of Python that you are currently using since you did not specify a version. You can activate this environment by:
conda activate myenv  ## Replace myenv with environment name
Once the environment is activated, you can install packages normally and it will only be added to that environment. If you want to specify the packages before creating and activating the environment, you can create a file and change the file-extension to .yml with the format below. The name section will be the environment name used when you activate the environment.
After you have your .yml file ready, do this to create the environment from the file:
conda env create -f environment.yml
environment.yml is the name of the .yml file.
Now, this is a very important step, for the individuals that like working with Jupyter Notebooks.
python -m ipykernel install --user --name myenv
If this line is not executed, you will not be able to use the environment you just created in your Jupyter Notebook.
If you need to update your environment due to:
a new version of one of your core dependencies was released
need to install additional packages
or you no longer need a package
Edit your .yml file, edit the package names, then save it. Open terminal/command prompt and navigate to the directory of your .yml file and run the following code:
conda env update --file environment.yml --prune
You can make an exact copy of an environment by using the following code in the terminal/command prompt:
conda create --name clonename --clone myenv
clonename will be the name you want to name your cloned environment, and myenv is the name of the environment that you want to clone.
If you found this content helpful and would like to read my other content, feel free to follow me at:TwitterInstagramFacebookI post data science tips and tricks that you may find useful!