Package: publication
Setting expectations around what APIs you can rely on in a Python
library is very difficult.Publication makes it easy.The ProblemAsHyrum’s Lawsomewhat grimly states,With a sufficient number of users of an API,it does not matter what you promise in the contract:all observable behaviors of your systemwill be depended on by somebody.In general, Python famously has a somewhat different philosophical view of this
reality.  We assume each other to beresponsible usersof the libraries
we consume.  Mucking with implementation details might break every time you
upgrade, but it’s sufficiently useful for testing, debugging, and
experimentation that retaining that ability is worth paying the cost.But, critical to this assumption is that everybodyknowswhen they’re
breaking into the “private” area of the library’s interface.  Here, there’s a
mismatch of expectations:library authorswrite documentation, and then think that users sit down and
read the documentation, front to back, and learn about what the “public”
interface is by doing so.  they then assume that users will know that they’ve
used private implementation details if they ever deviate from these
documented features.library userspip installa thing, open up a REPL, import the module,
and discover the library by doingdir()on the module and its contents,
assuming that their program is not using any private implementation details
as long as they never had to typelibrary._something_private()while
doing so.  If they ever encounter a traceback they may consult the
documentation, briefly, until it is resolved.Publication makes it possible to align the wildly divergent expectations of
these groups, so that users can still get the benefits of being able to use
internal details if they want, but they’ll know that they’re doing so.  It
makes the runtime namespace of your module look like the public documentation
of your library.How does this look in practice?You, a prospective library author, want to write a library that makes it easy to zorf a sprocket.
Great! You do, and it looks like this:# sprocket_zorfer.pyfromsprocketimportsprocket_with_namefromzorfimportzorfable_thingdefzorf_sprocket_internal(sprocket,zorfulations):...defcompute_zorfulations():...defzorf_sprocket_named(sprocket_name,how_much):sprocket=sprocket_with_name(sprocket_name)zorfulations=compute_zorfulations(how_much)returnzorf_sprocket_internal(sprocket,zorfulations)__all__=['zorf_sprocket_named']Your intent here, of course, is that you have exposed a module with a
single function:zorf_sprocket_named, and everything else is an
implementation detail. You even said so, explicitly, with__all__.
Your API documentation says the same.However, reading reference documentation and cleanly respecting
conventions is not how working programmers really figure out how to use
stuff. Your users all do stuff like:Load up an interactivepythoninterpreter and calldir()on
your moduleInstall Jupyter and tab-complete their way around your module to find
what they wantuse the auto-import function in PyCharm to grab some private
implementation detailand, before you know it, you have thousands of users of your library
with code likefromsprocket_zorferimportcompute_zorfulations,zorf_sprocket_internal,sprocket_with_namesprocket=sprocket_with_name(name)zorf_sprocket_internal(sprocket,compute_zorfulations(7)*2)Now you can never changeanyof your implementation details! Worse
yet,sprocket_with_nameisn’t even your own code; that’s something
you got from a library! But when someone doesimport sprocket_zorfer;sprocket_zorfer.<tab>in an interactive
shell, none of that information comes through.Underscore ParanoiaThe convention in Python is that we use_to indicate private names.
So when we library authors notice this problem starting to happen, a
common reaction is to start putting_in front ofeverything–
class names, function names, module names – and only explicitly export
those things that should be public by “moving” them via an import and an
entry in a public module’s__all__.However, this has a bunch of disadvantages:Most code inspection tooling and IDEs won’t see that the public name
is “moved”, so code exploration just makes it seem likeeverythingis an implementation detail now, rather than making it seem like
nothing is.All your__repr__s now have ugly and inaccurate function and
class names in them, at least from the perspective of your users; how
are they supposed to knowzorf_sprocket_namedis actually defined
inzorf_sprocket._impl_details.funcs._zorf_sprocket_publicnow?
How are they supposed to find the good, public name once they’re
looking at the goofy internal one?You constantly need to remember to putallof your code in these
ugly_-prefixed modules, and educate new contributors as to the
risks of creating new modules in your package that are not carefully
hidden away from public users.A Better WorldWhat if you could write all your codeas ifit were just regular
public code, and have all your implementation details and imports
automatically squirreled away in an underscore namespace so that curious
coders won’t accidentally find every module you ever imported and every
temporary helper function you ever defined and think they’re part of the
permanent public face of your library?Enterpublication.publicationuses the existing convention of__all__and a little
runtime hackery to hide everything that you have not marked as
explicitly public, like so:# sprocket_zorfer.pyfrompublicationimportpublishfromsprocketimportsprocket_with_namefromzorfimportzorfable_thingdefzorf_sprocket_internal(sprocket,zorfulations):...defcompute_zorfulations():...defzorf_sprocket_named(sprocket_name,how_much):sprocket=sprocket_with_name(sprocket_name)zorfulations=compute_zorfulations(how_much)returnzorf_sprocket_internal(sprocket,zorfulations)__all__=['zorf_sprocket_named']publish()That’s it! Now,from sprocket_zorfer import zorf_sprocket_namedworks as intended, butfrom sprocket_zorfer import compute_zorfulationsis anImportError.But what about…Other modules in my package, like tests, that need to peek at implementation details?Don’t worry, your code didn’t go anywhere. The original module is still
available as a special pseudo-module called<your_module>._private.
In the example above,sprocket_zorfer.py’s tests can still do:fromsprocket_zorfer._privateimportcompute_zorfulationsdeftest_compute_zorfulations():assertcompute_zorfulations(0)>7Mypy?Your types shouldprobablyjust be part of your published API, if
you’re expecting that users will need to know about them. But, if there
are cases which need to be type-checked internally in your library, as
far as Mypy is concerned, all your private classes are still there. So,
in the simple case you can just do this:fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:fromsomethingimportTdefreturns_a()->"T":...and in the hopefully very unusual case you need to mix runtime and
type-checking access to a different module’s private details,fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:fromsomethingimportTelse:fromsomething._privateimportTdefreturns_a()->A:...