Package: lancstro
Lancstro: an example of creating a Python packageThis repository and the following text is intended as a basic tutorial on creating and publishing a
Python package. It was created for a seminar given to the Lancaster UniversityObservational
Astrophysics
group, but
may be more widely applicable.What is a Python packageIn general, when talking about aPython
packageit means an set of Python modules
and/or scripts and/or data, that are installable under a common namespace (the package's name). A
package might also be referred to as a library. This is different from a collection of individual
Python files that you have in a folder, which will not be under a common namespace and are only
accessible if their path is in yourPYTHONPATHor you use them from the directory in which they
live.A couple of examples of common Python packages used in research in the physical sciences are:NumPySciPyNote: "namespace" basically refers to the name of the package as you would import it, e.g., if you
import numpy withimport numpy, then you will access all NumPy's functions/classes/modules via
thenumpynamespace:numpy.sin(2.3)A package can contain everything within a single namespace, or contain various submodules, e.g.,
parts that contain common functionality that naturally fits together in it's own namespace. For
example, in NumPy, therandomsubmodule contains functions and classes for generating random numbers:importnumpynumpy.random.randn()# generate a normally distributed random numberWhy package my code?So, why should you package (and publish) your Python code rather than just having local scripts?
Well, there are several reasons:It creates an installable package that can be imported without having to have the Python
script/file in your path.It creates a “versioned” package that can have specified features/dependencies. This is very
important for reproducibility of results, where a specific code version used for an analysis
can be pointed to.You can share you package with others (you can make itpip installableviaPyPI, orconda installableviaconda-forge),
which can be important when working with collaborators.You will gain developer kudos! Software development is a major skill you learn during your
research, so show off what you’ve done and add it to your CV.Project structureTo create a Python package you should structure the directory containing you code in the following
way (the directory name containing this information does not have to match the package name, but
often they will):repo/
├── LICENSE
├── pyproject.toml
├── README.md
├── setup.cfg
├── setup.py
├── pkgname/
│   ├── __init__.py
│   └── example.py
└── bin/
    └── executable_script.pyThere are other slight variations on this, for example, using asrcdirectory in which your
package directories live, as described in theofficial
guidelines).In this project the structure is:lancstro/
├── LICENSE
├── pyproject.toml
├── README.md
├── setup.cfg
├── setup.py
├── lancstro/
│   ├── __init__.py
│   ├── base.py
|   ├── members/
|   |   ├── __init__.py
|   |   └── staff.py
|   └── data/
|       └── office_numbers.txt
└── bin/
    └── favourite_object.pyHere, there is a "submodule" calledmemberswithin the mainlancstropackage.Using GithubYour packageshouldbe in aversion controlsystem and ideally hosted somewhere that provides a backup. It is now very common to usegitfor version control and it is sensible to host the project onGithub/Gitlab/bitbucketor similar. On Github you can have public or private repositories.If using Github, it is best to start the project by creating new repository there first, then
cloning that repository to you machine before then adding in your code. When creating a Github
repository (I might use "repo" for short later) you can initialise it with alicense
fileand aREADME file.Note: this is not a tutorial on using git, so you'll have to find thatelsewhere.The LICENSE fileYou should give your code a license describing the terms of use and copyright. Often you'll want
your code to be open source, so a good choice is theMIT
license, which is very permissive in terms of reuse of the
code. Avariety of other open source licensesare
available, although these often differ slighty on the permissiveness, i.e., whether others can use
your code in commercial and non-open source projects or not.TheLICENSEfile will contain a plain ascii text copy of your license.The pyproject.toml fileThis file tells thepiptool used for installing
packages how it should build the package. In this repo we have used thefile
contentssuggestedhere, which
means that thesetuptoolspackage is used for
the build.The README.md fileThis is the file that you are currently reading! It should provide a basic description of your
package, maybe including information about how to install it. Ideally it should be brief and not be
seen as a replacement for having properdocumentationfor you code available
elsewhere.In this case the suggested format for the file isMarkdown(the.mdextension), but it could be a
plain ascii text file orreStructedText. Markdown and
reStructuredText will be automatically rendered if you host your package on, e.g.,Github.The setup.cfg and setup.py filesIn many packages you might just see asetup.pyfile, which is the build script used by setuptools.
However, it is now good practice to put"static"
metadataabout
your package in thesetup.cfgconfiguration
file. By
"static" I mean any package information that does not have to be dynamically defined during the
build process (such as defining and building Cythonextensions). In many cases, like
this repository, this can mean thesetup.pyfile can be very simple and just contain:fromsetuptoolsimportsetupsetup()The layout of the configuration file is describedhere. I'll reproduce theone from this projectbelow with additional inline comments:[metadata]# the name of the packagename=lancstro# the package author information (multiple authors can just be separated by commas)author=MatthewPitkinauthor_email=m.pitkin@lancaster.ac.uk# a brief description of the packagedescription=PackagedefiningtheLancasterObservationalAstronomygroup# the license type and license filelicense=MITlicense_files=LICENSE# a more in-depth description of the project that will appear on it's PyPI page,# in this case read in from the README.md filelong_description=file:README.mdlong_description_content_type=text/markdown# the projects URL (often the Github repo URL)url=https://github.com/mattpitkin/lancstro# standard classifiers giving some information about the projectclassifiers=IntendedAudience::Science/ResearchLicense::OSIApproved::MITLicenseNaturalLanguage::EnglishProgrammingLanguage::PythonProgrammingLanguage::Python::3ProgrammingLanguage::Python::3.6ProgrammingLanguage::Python::3.7ProgrammingLanguage::Python::3.8ProgrammingLanguage::Python::3.9Topic::Scientific/EngineeringTopic::Scientific/Engineering::AstronomyTopic::Scientific/Engineering::Physics# the package's current version (this isn't actually in the file in this repo, see later!)version=0.0.1[options]# state the Python versions that the package requires/supportspython_requires=>=3.6# state packages and versions (of necessary) required for running the setupsetup_requires=setuptools>=43wheel# state packages and versions (if necessary) required for installing and using the packageinstall_requires=astropyastroquery>=0.4.3# automatically find all modules within this packagepackages=find:# include data in the package defined belowinclude_package_data=True# any executable scripts to include in the packagescripts=bin/favourite_object.py[options.package_data]# any data files to include in the package (lancsrto shows they are in the# lancstro package and then the paths are given)lancstro=data/office_numbers.txtFor a list of the standard "classifiers" that you can add seehere.In this project, we have added a "data" file that come bundled with the package. It is not required
to include data in your package.Adding a package versionIn the above case the package version is set manually in thesetup.cfgfile. It is up to you how
you define the version string, but it is often good to useSemantic
Versioning. In this format the version consists of three full-stop separated
numbers: MAJOR.MINOR.PATCH.The Semantic Versioning site gives the following definitions of when to change the numbers:MAJOR version when you make incompatible API changes,MINOR version when you add functionality in a backwards compatible manner, andPATCH version when you make backwards compatible bug fixes.Additional labels for pre-release and build metadata are available as extensions to the
MAJOR.MINOR.PATCH format.To update the version you can just edit the value in thesetup.cfgfile. When youinstallthis will be the package's version.This allows the package manager (e.g., pip) to know what version of the package is installed.
However, it is often useful to provide the version number as a variable within the package itself,
so that the user can check it if necessary. Most often you will find this as a variable called__version__, e.g.,:importnumpyprint(numpy.__version__)1.21.2There are several ways to set this, but it is best to make sure that there's only one place that you
have to edit the version number rather than multiple places. One method (used in this package) is to
include the version number in your package's main__init__.pyfile by adding
the line:__version__="0.0.1"Then, withinsetup.cfg, theversionline can be:version = attr: lancstro.__version__Among the other options, a good one to use is through setting the version with a tools such assetuptools-scm, which gathers the version information
fromgit tagsin your repo.The MANIFEST.in fileYou can specify which additional files that you want to be bundled with the package's source
distribution using aMANIFEST.in. With
modern versions of setuptools (e.g., greater than 43) most of the standard files such as the README
file and setup files, and any license file given insetup.cfg, are automatically included in the
source distribution by default. Hence, not include aMANIFEST.infile in this repository.However, you may want to include other files. If you had, say, atestdirectory with multiple
Python test scripts that you wanted in the package, you could add andMANIFEST.infile containing:recursive-include test/ *.pywhich will include all.pyfiles withintest.The package source directoryIn this project the directory containing the package source code, i.e., the Python files, is calledlancstro/. In this case has two files in it (although it can contain any number of Python files, each of
which will be a module that is available in the package):__init__.pybase.pyThebase.pyfile contains some Python code, in this case a class calledGroupMembers, which is part of our package.The__init__.pyfile is very important. It is what tells Python that this
directory is apackage. The__init__.pyfile can be completely empty, but itdoesneed to be present. It can contain any
Python code (you could define your whole package in the__init__.pyfile if you wanted), but often
it is used to import things from submodules/subpackages into the package's namespace. In this case
the__init__.pyfile contains the following code:from.baseimportGroupMemberfrom.importmembers__version__="0.0.2"# the version number of the codeThe first line imports theGroupMemberclass from thebase.pyfile, so that
theGroupMemberclass can be used from thelancstronamespace rather than thelancstro.basenamespace. E.g., this means that when using the package we could do:from lancstro import GroupMemberrather thanfrom lancstro.base import GroupMemberalthough both will work. You may want to do this for commonly used function or classes, but it is
not necessary.Thelancstro/directory also contains the directorymembers/, which is a subpackage of the
package (any subpackage must also contain their own__init__.pyfile). The second line of the__init__.pyfile imports thememberssubmodule into thelancstronamespace. E.g., if I just do:importlancstrothen I can access things from thememberssubpackage usinglancstro.members.staffrather than doing:from lancstro.members import staffalthough (again) both will work.The final line in the__init__.pyfile sets theversion numberof the
package.The data directoryYou might want to include some data files in your package, e.g., a look-up table for a calculation,
a catalogue, etc. In this case I've added a JSON file,office_numbers.txt, in a directory calleddata/(any name
can be used, butdataseems quite sensible!). This directory does not need an__init__.pyas it
is not a package. To include this file in the package you need to have the line:include_package_data = Truein yoursetup.cfgfile and also list it in the[options.package_data]section, e.g.,:[options.package_data]
lancstro = 
    data/office_numbers.txtIntra-package referencesIn your package you canimportthings from the
various submodules/subpackages using the.notation.For example, to import things between Python files in the same part of the package (e.g., at thelancstro/level), you can do:from.baseimportGroupMemberwhich imports from thebase.pyfile.If a file in a subpackage wants to import from the level below, e.g., a Python file inlancstro/memberswants to import from a file inlancstro/, the you could use:from..baseimportGroupMemberI.e, use two dots..to specify going down one package level.The bin directoryYou may want to include executable scripts in your package. It is good to place them in a directory
called, for example,bin/in the root directory of your repository. To make these part of the
package you need to list these in thesetup.cfgfile in ascriptssection, e.g.,scripts =
    bin/favourite_object.pyOnce the packages are installed these scripts should be in you path and usable with, e.g.,:$favourite_object.py-hInstalling the packageIt is best practice to install Python packages usingpip(the
"package installer for Python"), so you should have that installed. Once you have the above
structure you can install the package (from it's root directory) using:pipinstall.where the.just refers to the current directory. The standard install locations are describedhere, but I would recommend
usingvirtual environments, such as
provided viaconda, in which case the package will be installed
only in the environment.That's it! Open up a Python terminal (from any location except in the package directory, otherwise
it'll get confused!) and you should be able to do:importlancstroprint(lancstro.__version__)0.0.1or run thefavourite_object.pyscript from the command line:$favourite_object.py-husage: favourite_object.py [-h] name nameGet a staff member's favourite objectpositional arguments:name        The staff member's full nameoptional arguments:-h, --help  show this help message and exitYou can then tell other people to clone your Github repo and install things in the same way, or evenpip installdirectly from the repo with, e.g.:$pipinstallgit+git://github.com/mattpitkin/lancstro.git#egg=lancstroThese methods will install the very latest code from the repo, so not necessarily a specific version
(although that can be done if you've tagged a version or work from a particular the git hash).Publishing the package on PyPIRather than getting people to install code directly from your Github repo, it is often better to
publish versioned releases of your code. You can publish Python packages on thePyPI(Python Package Index) repository from which they will then bepip installableby anyone!Firstly, you'll need toregister an accounton PyPI. Anyone is
able to do this. Secondly, you'll need to install thetwinepackage, which is used for uploading packages to
PyPI.Within your repo's root directory (containingsetup.py) you can now build aPython
wheel(a zipped binary format of the package designed for speedier
installation) containing your package with:pythonsetup.pybdist_wheelsdistNote: if your code is pure Python, creating a wheel should work straightforwardly, but if not the
wheel generation may not work. In these cases you can just build a tarball containing the package
using:pythonsetup.pysdistThis should create adist/directory containing a file with the extension.whl(built by
including thebdist_wheelargument). This is the Python wheel. It should also contain a tarball of
the package (built by including thesdistargument).It is often best to first upload these products toPyPI's testing
repository(you'll need toregister a
separate accountfor this), which can be done usingtwinewith:twineupload-rtestpypidist/*Note: make sure thedist/directory is empty before generating the new package version withpython setup.py bdist_wheel sdistotherwise you might end up uploading multiple versions.You should be prompted for your username and password, although there are ways to set these asenvironment variablesorusingkeyring, so that you don't have to enter them each time. If
the upload is successful you should be able to see the project on the Test PyPI site, e.g., athttps://test.pypi.org/project/lancstro/0.0.2/.You can test that the package installs correctly from the Test PyPI repository by running
(potentially in a new virtual environment):pipinstall-ihttps://test.pypi.org/simple/lancstroIf you're happy with the package you can proceed to upload it to the main PyPI repository using:twineuploaddist/*Et voilà! Now you just need to tell people to run:pipinstalllancstroto install yourpackage. If they want to install a
particular version they can use, e.g.,:pipinstalllancstro==0.0.2Or, if there's a lower or upper version that must be used the inequality operators can be used
instead, e.g.,:pipinstalllancstro<=0.0.2Publishing the package on conda-forgeYou may (andshould!) install Python packages in a virtual environment that is relevant for the
particular project that you are working on. A popular virtual environment/package manager tool isconda, which is installed as part ofAnaconda. Conda is a package manager for a variety of software, not
just Python packages, so if creating a conda package for your Python project you can make it
dependent on specific versions of non-Python libraries (maybe you want to use a specific version ofGSL!).You canbuild a conda
packageand host it in your own account on Anaconda.org. However, a popular repository for hosting projects
isconda-forge. An advantage of hosting your package on conda-forge is
that it will have been automatically verified by a test suite and reviewed by an actual person, so
hopefully will be more robust for other users.Getting a package on conda-forge is quite a bit more involved than uploading to PyPI, although if
you already have your package on PyPI that is an advantage (and is what I'll assume in the example
below). The basic steps are givenhere, but you will need a
Github account. I'll detail these a bit more below.Note: you will need to have uploaded the package source tarball to PyPI for these instructions to
work.Go tohttps://github.com/conda-forge/staged-recipesand fork the repository to your own account.In your fork of the repository create a new branch. If you've cloned your fork of the repository
you might do:gitcheckout-badd_lancstro_to_conda_forgeIn therecipes/directory create a new directory with the name of your package and copy themeta.yamlfile from theexample/directory into it:cdrecipes
mkdirlancstro
cpexample/meta.yamllancstroOpen up the copiedmeta.yamlfile in a text editor and change it to look something like below
(I've removed a lot of the comments):{%set name = "lancstro" %}{%set version = "0.0.1" %}package:name:{{name|lower}}version:{{version}}source:url:https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz# get the SHA256 check sum of the file (on the PyPI page for the package# click on "Download files" and then "View" under the "Hashes" heading)sha256:2873bb17f5e8cc84ac19e22307cc8567273fcdc57e5dd1f57fe52b2b1a6b1da3build:noarch:pythonnumber:0script:"{{PYTHON}}-mpipinstall.-vv"requirements:host:# packages required to build and install the package-python-pip-setuptoolsrun:# packges required to run the package-astropy-astroquery >= 0.4.3-pythontest:# make sure the package can at least be imported (other tests can be added)imports:-lancstroabout:home:https://github.com/mattpitkin/lancstrolicense:MITlicense_family:MITsummary:'Mygreatpackage'description:|An example package for showing how to package a package.doc_url:https://lancstro.readthedocs.io/dev_url:https://github.com/mattpitkin/lancstroextra:recipe-maintainers:# github ids for maintainers-mattpitkinCommit the changes and push them to your fork of thestaged-recipesrepository.Open up a pull request (PR) between your branch and conda-forge'sstaged-recipesrepo. Call the
PR something like "Add lancstro". Create the pull request.After a while check that the test builds in the PR have completed successfully. If not try and
fix the issue by editing the (forked)meta.yamlfile.Answer and respond to any questions/comments from the assigned reviewer (you shouldn't have to
assigned a reviewer, but sometimes you need to prod the appropriate channel).Wait for a reviewer to sign-off and merge the PR.At this point your package should be installable from conda-forge using, e.g.,:condainstall-cconda-forgelancstroDocumentationYou should try not to just write code for yourself. Academic results should be transparent and
reproducible, so the code you write and use should be usable by others, thereforeWrite The Docs!Creating documentation for your code doesn't just mean that your code should contain comments (which
it definitely should!), but there should also be documentation (on, e.g., a website) on how to
install and use your code. This should include information on the code'sAPI(just a fancy way of saying show how to use the functions
and classes in your package). It is also important to have examples of use cases as it's often good
to "show not tell". You can store the documentation source files in the same repository as you
package (e.g., adocs/folder).I'm not going to describe in detail how to add documentation to a package (I haven't added it into
this package yet, but I may add this in the future!), but will just point towards some resources.
Two packages that you may want to look into for building documentation are:SphinxmkdocsBoth of these allow you to write documentation in Markdown or reStructuredText and automatically
include (via various extensions/plugins) code docstrings. They can also include Jupyter notebooks.For repositories hosted on Github, you can easily and freely set up building and hosting of the
documentation onRead the Docs. You can also publish yourdocumentation
directlyon
Github usingGithub Pages.There is an example of using Sphinx for documenting a packagehere.ContributionsYour code may be the product of many developer's work. If it's open source you may also be open to
having other developers contributing to it. You should therefore have instructions on how people
should contribute and guidelines on the expected behaviour of contributors.Often you will see aCONTRIBUTING.mdMarkdown file in package repositories that describes how to
contribute. If a contributor wants to add/request a new feature, or fix a bug, then they may want to
open a Github issue (or post on an appropriate forum) to see if the feature is useful/bug is known.
If they have coded up a bug fix/feature then adding that into the repository often involves a
"fork-and-pull request" workflow process
(this is the process for many projects, e.g.,NumPy,astropy):fork the repository to your own Github accountcreate a new branch on your fork for developmentadd and commit your changes making sure that they work and don't break the packagepush your commits to your forkcreate a pull request with the upstream (i.e., original) repositoryrespond to any comments on the changemerge the request into the original repositoryCode of conductYou should also consider adding a code of conduct to your project outlining expected behaviours
during interactions between developers/contributors. There are many examples of code's of conduct
that you can often use verbatim (many are licensed usingCreative
Commonslicenses) or adapt to your needs:astropy code of conductNumFOCUS code of conductPython Community code of conductCode styleYou may want to enforce a particular style for your code. Many projects follow thePEP8style guide. There are packages that you can run
on your code to automatically make them conform to this style, e.g.,blackorflake8,
so you should tell contributors to run these on any code they submit (and make sure you run them
yourself!). You can also add thepep8speaksapp on Github that will check
that any pull request conforms to PEP8 and inform the committer of any violations of the style.You can force checks to happen automatically by using thepre-commitpackage to add"pre-commit" hooksto
git, so that it automatically runs, e.g., black, on any committed code.Making code citableYour code is a very large part of your academic output, so it's good to make your package citable.
This way you can receive appropriate acknowledgement when people use it and show evidence of your
output. There are a variety of ways of doing this (skewed towards Astro/Physics):For packages on Github, link your repository toZenodowhich will provide
a citableDOIfor you project.Get it linked onto theAstrophysics Source Code Library(ASCL). This is
indexed on NASA ADS, but does not give a DOI.Write a paper for theJournal of Open Source Software(JOSS). This is a
very light touch, but peer reviewed publication that also provides a DOI and is indexed on NASA
ADS. It does require you to have proper documentation for your package as an acceptable level of
documentation is part of the review.Write a paper for a standard journal. Many journals (MNRAS,ApJ,PASP, etc) do now accept
papers on software, although it's likely that they should also include a description of a
practical use case for the software.Not covered here!There are many additional useful things that I've not covered here. These include:using entry point console scripts rather than, or as well as, including executable scriptsincluding C/C++/FORTRAN code, orCython-ized code, in your packagecreating atest suitefor your package (and checking itscoverage)setting upcontinuous integrationfor
building and testing (and automatically publishing) your code (e.g., withGithub
Actions, TravisCI, ...)I may add these at a later date.Other resourcesFor other descriptions of creating your Python code see:Packaging Python Projectssetuptoolsdocumentation"How to package your Python code"