How to Build Your First Python Package | by Shaked Zychlinski | Towards Data Science

This blogpost is now available in Polish too, read it on BulldogJob.pl
About two years ago I published my very first data-science related blogpost. It was about Categorical Correlations, and I honestly thought no-one will find it useful. It was just experimental, and for myself. 1.7K claps later, I’ve learned that I cannot determine what other people will find useful, and I’m quite happy I can assist others on the web like others on the web assist me.
I was also quite new to Python and Github at that time, so I also experimented with writing the code to these categorical-correlations that I wrote about, and publishing it on Github. I gave that piece of code a name: Dython, as in Data tools for pYTHON.
But this isn’t what this blogpost is about — it’s about the last experimental thing I did after everything I just mentioned: I created a Python library and uploaded it to PyPI — which you might know as pip install. Again, this all began just as a learning exercise — but with around 1.5K installs per month, I realized again that I underestimated my code ability to be useful to others.
But why am I telling you all this? Because almost everyone has some cool code lying somewhere on their laptop or on their Github account. This cool code might be very helpful to others — though most us think that’s probably not true, and “what value can others find in my tiny piece of code”. I thought so too, and I found out that I was wrong. You might be wrong too, and I’m here to persuade you — and assist you — to make your nifty piece of code into a full grown Python package in just a few steps — so you can save others hours of programming with a single pip install. Be the person that assists other programmers — and let me show you how to accomplish this in just six steps. Let’s do this.
I believe it quite goes without saying, but still — I’d like to make sure we’re on the same page. Programmers are divided into two types: the ones which write readable code, and the ones which use variables named x, x1, x2, yx, etc. If you belong to the first type, you can skip ahead. If you’re the second type — it’s time to switch to the other side. Remember that other people will read your code, so make sure it’s readable. Give things meaningful names, break long functions to short-coded one-purpose methods, etc. You know, like they teach in all those coding courses.
Another thing I assume is that you’re using some source-control, probably GitHub. While this isn’t really required, it is more than recommended. If you’ve never used GitHub before, this is a good point to take a few minutes and learn how — check out the official tutorial, or this blog-post, and get started.
The first real step into turning your code into a package, is deciding how users should use it — and make it importable. We’ll split this into two actions:
Your code is probably made of several functions and methods, but not all are meant to be used by the users — some are internal functions, supposed to be used only by your code. While Python doesn’t support private methods, the convention is to mark private methods with an underscore prefix: def _private_function(). Rename all your internal functions like so, so users will easily understand what is supposed to be exposed to them and what isn’t.
Pro tip: You can use the special variable __all__ to define exactly which functions to expose and which not when the users use from your_package import *. Check out this thread on StackOverflow for more information.
If your code is made out of several modules (files), looking like this:
your_package  |-- module1.py  |-- module2.py
You’ll need to add another file, __init__.py to your package in order to make Python interpret the your_package directory as a Python package. This means now it should look like this:
your_package  |-- __init__.py  |-- module1.py  |-- module2.py
You can leave the file empty as a starting point, but it has to be there in order to allow from your_package import module1.
Pro tip: The code in __init__.py is executed once your package is being imported, and this allows you to do all sorts of cool stuff. For example, you can use __init__.py to shorten your API. Let’s say that the most important method of your code, imp_func, is found in module1 . You can allow the users to import it as from your_package import imp_func instead of from your_package.module1 import imp_func, by simply adding:
from .module1 import imp_func
to your __init__.py file.
Documentation is like lining up in a queue — we wish everyone would simply do it, but we tend to cut ourselves some slack whenever we can. Step 2 here relates to the previous Before We Begin section, as it meant to allow other people to understand what does your code do. There are two types of documentations you’ll need to add:
Each function in your library should have a doc-string, which summarizes what the function is all about. It should contain:
Summary: Explain in a simple language what the function suppose to do
Parameters: Explain what are the parameters and parameter-types which the function expects
Return value: Explain what the function returns
Example: While not a must, it is always useful to add a usage example
Here’s an example:
You can learn more about doc-string formats in this super cool article on DataCamp.
A Read-Me file is a summary of what your package is all about. When placed in the top directory of your package, GitHub uses it as your package “landing-page”, being displayed to visitors as the first thing they see.
A good Read-Me file will have an overview of the package, installation information (such as requirements) and some Quick-Start examples, describing basic usage of your package.
Read-Me files are usually written in a format known as MarkDown, which you can learn more about here. This is why they’re usually named README.md.
As your code is about to get public, you should attach a license to it, explaining others how they should use your code. You’ll probably want to use a common one like the MIT License or Apache 2.0 License. GitHub will assist you choosing one by clicking the License option, and it will add a new file named LICENSE to your package. If you insist on not using GitHub, you can add the file manually.
At this point, your package is supposed to look like this:
your_package    |-- README.md   |-- LICENSE  |-- __init__.py  |-- module1.py  |-- module2.py
Now we’ll add the file that builds an installable Python package out of your code. For this, you’ll have to add a new file named setup.py, and rearrange your files in the following way:
your_package   |-- setup.py   |-- README.md   |-- LICENSE  |-- your_package        |-- __init__.py        |-- module1.py        |-- module2.py
The setup file dictates everything Python installer needs to know when installing your package. A very basic one which you can simply copy-paste, looks like this:
Except for editing your personal info, there are two things you’ll have to keep track of:
Version: Each time you’ll release a new build to PyPI (which we’ll discuss in a second), you’ll have to bump your version number
Install requires: This is a list of all the external dependencies of your package. Be sure to list everything here, so Python will install them along with your package
We’re almost there! Next up: go to pypi.org and create an account. You’ll need it to upload your new library.
We are pretty much done. All that’s left is running some commands which will build an installable Python package out of your code — and deploy it to PyPI.
Before we begin, we’ll need to install twine, which will allow us to deploy to PyPI. It is as simple as:
pip install twine
Next, we’ll create an installable package. Go to the directory where the setup.py file is, and run:
python setup.py sdist bdist_wheel
This will create several new directories, such as dist, build and your_package.egg-info. It’s dist which we care about now, as it contains the installation files we want to deploy to PyPI. You’ll find two files in there: a compressed tar file and a wheel file. And in a second, they’ll be available to the whole world.
Pro tip: Add these directories to your .gitignore file, to prevent pushing installation files to your repo. (Never heard of .gitignore? Read this)
Next up, verify the distribution files you just created are okay by running:
twine check dist/*
Time to upload your package to PyPI. I recommend deploying first to the PyPI test domain, so you can verify everything looks as you intended. Do this using:
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Go to test.pypi.org and check your new library. Satisfied? Great, let’s push it to the real PyPI:
twine upload dist/*
That’s it. From now on everyone can install your package using pip install. Great job!
Pro tip: You can make your life easier with a Bash script, which will build, check and deploy your package with a single command. Create a new file named build_deploy.sh in the same directory as setup.py. Copy-paste the following code into this file:
Now all you need to do is run:
./build_deploy.sh
From the directory where the file is at, and that’s it. You can also run:
./build_deploy.sh --test
to upload to the test domain. (Note you’ll have to make the script executable before running it the first time. Simply run chmod +x build_deploy.sh in the directory where the file is located.)
Your awesome code is now available for everyone to use — but people are still unaware of your awesome code which is now available to them. How can you let them know? Write a blog-post, and tell why you build this code in the first place, and what are its use-cases. Share your story, so others will know how much effort you just saved them with your awesome code.
Taking your code into the next level — publishing it to the world — might seem frightening when doing it for the first time, both for personal reasons (“who will want to use it?”) and for technical reasons (“how can I do it?”). I hope that in this post, I assisted you to overcome both these setbacks, and now you too can assist thousands of other programmers worldwide. Well done, and welcome to the open-source community!