Create Your Custom, private Python Package That You Can PIP Install From Your Git Repository | by Mike Huls | Towards Data Science

You’ve created some convenient script that you would like your coworkers or others to use. In many companies, code like this is copied and emailed to one another. Although email is a very accessible tool to share code, we’re not living in the 90s anymore so let’s distribute your code in a smart way.
This article tackles the problem described above in two steps: packaging and distribution. First, we’ll focus on transforming your code into a python package so that people can easily install it. Then we’ll put the package into a repository (like on Github or Bitbucket) so people have access to it. At the end of this article you:
understand the requirements of a Python package
are able to build a Python package or transform an existing project into a package
are able to pip install the self-built package from the repository
are able to update your package
Let’s code!
We work in a company that performs analytics for restaurants. The owner of a restaurant collects data about guests, meals, and prices. They send us a data file with a specific question like, “What type of guest eats the canard a l’orange?”, “Is our pumpkin soup overprices?”, and “Do we see an increase in customers since we’ve lowered prices on deserts?”
In my years of working here I’ve noticed I use a lot of the same code, just copying it over from previous assignments. Our goal is to create a “Toolbox” package that contains some very generic pieces of code that my coworkers and I can easily install with pip. Then, every time one of us thinks of another handy function, we can add it and update our package.
In order to achieve this, we’re going to first package our existing code in a Python package. Then we’ll focus on distributing the package so it’s available to my coworkers.
First, we’ll take our functions and create a Python package out of it. This is necessary if you want to use pip for the installation. You can check out all of my folders and files in this repository. We’ll walk through creating one in these steps:
1. Create a venv.Create a virtual environment and add a gitignore, otherwise, we’ll create an unnecessarily large package. Check this article on how to do this.
2. Create package folderCreate a folder with the name of your package. In my case, this is “toolbox”.This will be the package we’re installing. We’ll create some files in the package folder:
toolbox/functions.pyThis file will hold some of the functions that we want to share. I’ve included 3 functions: listChunker, weirdCase and report.toolbox/__init__.pyThis will tell Python that the toolbox folder is a python package. This file can also be used to import functions so that we can import listChunker from toolbox in addition to from toolbox.functions import listChunker. Creating this file is required but the content is optional
toolbox/functions.pyThis file will hold some of the functions that we want to share. I’ve included 3 functions: listChunker, weirdCase and report.
toolbox/__init__.pyThis will tell Python that the toolbox folder is a python package. This file can also be used to import functions so that we can import listChunker from toolbox in addition to from toolbox.functions import listChunker. Creating this file is required but the content is optional
3. Create setup.pyThis file is required for telling pip what your package needs for it to be installed. Let’s take a look at the setup.py that I’ve used.
Below we walk through the setup file lines that need a little more explanation.
Line 3: load README.md in a variable called long_description. This is optional.
Line 7: Give our package a name. Must match your package folder name
Line 8. Which is the version of our package. Pip uses this version to see if packages need updating so make sure to increment it if you want users to be able to update
Line 12 and 13: loads the README.md from line 3; line 13 indicates the format of the readme.
Line 14: URL to your repo
Line 15: optionally list some handy URLs
Line 18: how can users use your package? Check choosealicense.com
Line 19: list of all packages that need to be built: make sure this matches your package folder name
Line 20: list of packages that your package relies upon. Even though none of my functions use requests I’ve decided to include it for demonstration purposes. Including a package here makes sure that when pip installing the toolbox package, requests gets installed first so that toolbox can use it.
4. Other optional filesI’ve decided to include a README.md and a LICENSE file. These are simple text files that are not really required but a very nice addition.
Our repository is complete! Let’s find out how to distribute it!
Now that our package is created we can use a repository for distribution. First, we’ll create the repository and use it to pip install our package. Finally, we’ll update our package after we’ve changed the source code.
First, create a repository. You can do this on any platform that uses Git (GitHub, BitBucket, etc). Then add all your files, make sure to gitignore unnecessary files, and push to the repo.
Copy the URL of your repository. You can pip install your package with this URL like this:
pip install git+https://github.com/mike-huls/toolbox.git
That’s it! Easy isn’t it? Also note that you can install from both a public (like toolbox) and a private repository!
Imagine my coworker has come up with a new function and decided to commit it to the repository. I can use pip to update my package. Every time I call pip install git+https://github.com/mike-huls/toolbox.git pip checks the version number in the setup.py file. If my coworker remembered to increment the version then my package gets updated. Easy!
There is a very easy trick to use your package in Docker. Check out this article for a detailed description.
GitHub offers a place to document issues, have a nice readme on the ‘front page’, and even offers a wiki for when your package requires even more explanation.
As we’ve seen in this article, combining the power of Python packaging and Git offers a lot of advantages:
Easy distribution, installation, and updates from one central source (one source of the truth)Version control on our package and the ability to collaborateAbility to update the package once it’s modifiedAbility to pip install and update packages from a private repository
Easy distribution, installation, and updates from one central source (one source of the truth)
Version control on our package and the ability to collaborate
Ability to update the package once it’s modified
Ability to pip install and update packages from a private repository
I hope I’ve clarified a lot of the process of creating and distributing Python packages. If you have suggestions/clarifications please comment so I can improve this article. In the meantime, check out my other articles on all kinds of programming-related topics like these:
Multi-tasking in Python: speed up your program 10x by executing things simultaneously
Create a fast auto-documented, maintainable and easy-to-use Python API in 5 lines of code with FastAPI
Python to SQL — UPSERT Safely, Easily and Fast
Create and publish your own Python package
Create Your Custom, private Python Package That You Can PIP Install From Your Git Repository
Virtual environments for absolute beginners — what is it and how to create one (+ examples)
Dramatically improve your database insert speed with a simple upgrade
Happy coding!
— Mike
P.S: like what I’m doing? Follow me!