Making Python Packages Part 2: How to Publish & Test Your Package on PyPI with Poetry | by Skylar Kerzner | Towards Data Science

In the last edition of this guide, we published a python package to PyPI using a traditional setup.py.
It required a strange Manifest.in file, didn’t come with any testing, and we didn’t even discuss dependency management with a requirements.txt. Why? Because that is not the way in 2021, my friends.
Dependency management is about the Python libraries used by your package. If you import numpy and use its classes/functions, your package will depend on the version of numpy.
If a new version of numpy uses different syntax or gets rid of a function, then your package won’t work on that version of numpy. We need to keep track of what version you were using.
Or what if you use numpy and pandas, and they both depend on different versions of some other library. Who will keep track of these things?!
Poetry will.
Poetry is a Python library that we can use to create a package, publish it to PyPI more easily, AND it will deal with dependency management for us. Yay!
To get started, we just install poetry
pip install poetry
As always, try pip3 (the version for python3)if you don’t have pip.
Poetry new
Our first command is to create the directory. No more mkdir’s at the beginning of our project when we have poetry.
Really, don’t make a directory for your project before you do this, because poetry is going to create a directory with your project name and put everything in there including a subdirectory also named for your project, where the source code will live.
So if you make a directory for your project and do this in there, you’re gonna end up with a russian doll situation: my_package/my_package/my_package/yikes.
Instead, just go to the directory where all your packages will live (so when you run ls, you see your other projects), and run:
poetry new
Now you can see your poetry directory with all its parts, inclduing the all-important pyproject.toml.
And now we can start building a package with Poetry!
Virtual environments
But first, a quick detour on how poetry works as a virtual environment.
A virtual environment is an isolated digital environment where libraries, codes and interpreters are completely separated from the others on your computer.
If you go in a digital environment and try to import numpy, it won’t exist, because it isn’t installed in that digital environment.
Conversely, if you install a library in a digital environment, it wont be installed on the rest of your computer.
Essentially, the libraries available in an environment are all stored in some directory in that environment. A virtual environment uses its own directory of available libraries, so it has no relationship with your computer’s directory of available libraries.
When we ran poetry new, we created a virtual environment! When we run code from the command line while in a poetry directory, we are not in the virtual environment. However if we start a command with
poetry run the-rest-of-the-command
then we can run that command in the poetry environment.
Or, if we run
poetry shell
then we will enter our poetry folder’s virtual environment, where we can use the libraries that we will install through poetry.
Exit a virtual shell with ctrl+d.
Poetry add
Indeed let’s look at installing libraries with poetry, which will manage our dependencies nicely.
Let’s go into that pyproject.toml we created. It’s going to contain two major things:
The documentation for our project (well, including referencing the README and LICENSE). We’ll get back to documentation after we finish this libraries business.The range of library versions our package needs
The documentation for our project (well, including referencing the README and LICENSE). We’ll get back to documentation after we finish this libraries business.
The range of library versions our package needs
When we make our package, we’ll need to tell poetry what versions of libraries we are using. That’s easy with the command:
poetry add library-name
where library-name is the name of whatever library we are using, like numpy.
That command will add the library version to our pyproject.toml. It’s really a range of versions following semantic versioning. You can see the syntax here: https://classic.yarnpkg.com/en/docs/dependency-versions/.
It will also install the library in our Poetry virtual environment.
poetry.lock
If you poetry add a library, which will also install that library, you’ll not only see your pyproject.toml automatically updated — you’ll also gain a new file: poetry.lock.
That pyproject.toml notes the range of library versions that are acceptable for your package, and only lists the libraries that you’ve directly added with poetry add.
On the other hand, the poetry.lock notes the exact library versions that are installed in your package. And it includes every library that your dependencies need.
So if you poetry add seaborn,
then while your pyproject.toml will just gain seaborn,
your poetry.lock will gain:
atomicwrites, attrs, colorama, cycler, fonttools, kiwisolver, matplotlib, more-itertools, numpy, packaing, pandas, pillow, pluggy, py, pyparsing, pytest, pytz, scipy, setuptools-scm, six, tomli, wcwidth, and seaborn
because that’s every library that seaborn depends on, plus every library that each of those depend on, recursively until every library that seaborn needs has the libraries it needs.
Side note on others using your source code
Now, if someone clones your package from github, they will have your pyproject.toml, but didn’t install your libraries yet. They’ll need to run
poetry install
If they also have your poetry.lock, they will install the exact versions stated in your poetry.lock.
If they don’t have your poetry.lock, then poetry will do what it did for you — figure out the write versions to install, and write it in a poetry.lock.
In summary, the poetry.lock lists all exact library versions installed when a package runs, so that users of your package can all use the same versions of everything.
Project Description with pyproject.toml
Alrighty so I’ll skip toward the PyPI publishing ASAP, we just need to be a bit decent with our package documentation here.
That section on the top of your pyproject.toml will contain everything that setup.py contains in the traditional packaging method without Poetry.
The top of your pyproject.toml should look like:
pyproject.toml
[tool.poetry]name = “” #your package nameversion = “0.0.1”description = “” #a short description of your package that will be used in PyPI searchauthors = [“your-name <your-email>”]license = “MIT”readme = “README.md”homepage = “” #can be the reporepository = “”keywords = [“test”, “dependencies”, “documentation”]classifiers = [“Development Status :: 5 — Production/Stable”,“Intended Audience :: Education”,“Operating System :: MacOS”,“License :: OSI Approved :: MIT License”,“Programming Language :: Python :: 3”,]include = [“LICENSE”,]
Go ahead and copy that into your toml and update the fields: fill in your name and email, project name and version, short description, repo/homepage, and fill out those classifiers using the options here: https://pypi.org/pypi?%3Aaction=list_classifiers
As you can see, we are referencing a README.md, so change your README.rst to an md (markdown) file if you like. You’ll put the long project description including the change log in that README.md, so it can look something like:
README.md
My long description blah blah blah blah.Change Log================0.0.1 (Dec 1, 2021) — — — — — — — — — — — — — — — -- First Release0.0.2 (Dec 2, 2021) — — — — — — — — — — — — — — — -- Did some more stuff
Publishing to PyPI
Okay here we go. I’ll skip to the juicy part, and then I’ll go back and offer some tips.
You can publish to PyPI in one line. Afffter you setup your credentials one time with:
poetry config http-basic.pypi username password
Those are your credentials from pypi.org, which you can get from making an account on their webpage.
After you set up those credentials, then here’s that one line publish:
poetry publish --build my_package
And you’re on PyPI just like that!
There is a separate poetry build command but you can just do the build at the same time as the publish with that build flag ( — build).
Testing on Test PyPI
But maybe before we go throwing our half-baked package on PyPI, we can test it on a test version of PyPI.
This test PyPI will allow us to mimic updating to PyPI, then pip installing our own package to see if it works as expected.
The first step to do this is to make a new account on test.pypi.org (yes, you need a separate account from your pypi account, but it can have the same username).
You can then add the repository to poetry:
poetry config repositories.testpypi https://test.pypi.org/legacy/
publish to that repository:
poetry publish –-build -r testpypi
and pip install your package from that repository:
pip install --index-url https://test.pypi.org/simple/ my_package
Sidebar: Troubleshooting that pip install
If you’re having trouble, this longer command line function solves some of the common issues:
python3 -m pip install -i https://test.pypi.org/simple/ — extra-index-url https://pypi.org/simple my_package==0.0.1
The python -m before pip is to make sure you’re using the right pip version. You can also try looking into your pip version and python version with
which pipwhich pythonpip --versionpython --version 
or updating pip with
 python -m pip install --upgrade pip
As always you might need to switch python with python3 or pip with pip3.
The extra index url points to the normal PyPI repo (rather than the test), which your dependencies need to be downloaded from.
Finally, I’ve also added the version, which you should replace with your package version number, so that it gets the right one.
Installing and importing in a new virtual environment
When you’re trying that pip install, I would go into a new virtual environment, just so that your package module name isn’t confused with the name of the directory we were in, which was also named your package name.
I like to just go to my top level project directory, create a new directory with poetry new, enter its virtual environment with
poetry shell
and pip install it from the test PyPI.
Then I run the python interpreter in the virtual environment with
poetry run python
Now, import your package and functions in Python! You can first try:
import my_package
Next, try importing your functions. If they’re in the __init__.py you can import them directly, as in
from my_package import my_function
Or if they’re in a file at the same level as the __init__.py, you’ll specify the file name without the .py to get its module name. e.g. utils.py is just called utils.
from my_package.my_module import my_function
If there’s a folder at that level, with .py files in it, turn the folder into a package by adding an __init__.py to that folder too. The init files can be empty.
So if
my_package is the whole library,inside the my_package folder is my_subpackage folder, which contains .py files and an init file,inside my_subpackage folder is my_module.pyinside my_module.py lives my_function,
my_package is the whole library,
inside the my_package folder is my_subpackage folder, which contains .py files and an init file,
inside my_subpackage folder is my_module.py
inside my_module.py lives my_function,
it would be:
from my_package.my_subpackage.my_module import my_function
You can also check what functions are in an imported module with:
dir(my_package.my_module)
Bonus: Updating all dependencies with poetry update
Why is there always a bonus section about updating things?
Anyway, in order to update all the libraries (dependencies) and write them to your poetry.lock, you run:
poetry update
Don’t confuse that with updating poetry, which can be done via
pip install --upgrade poetry
or
poetry self update
Thanks for reading!
Please follow me to learn more about Python and Data Science!
Check out Part 1 of this guide to learn how to publish to PyPI the old fashioned way with a setup.py, some others tips, plus some good humor. I was on a roll that day: