Package: transmute
Automatically update Python Eggs on application startup – or, you know,
whenever, really.OverviewThe idea behind this module is to support self-updating Python
applications, namely command-line tools. Transmute probes remote
repositories for updated components, fetches updates and adds them tosys.path, making them available for use in the application.Components are assumed to be standard Python Eggs. Repositories are then
simple containers for these eggs. Currently PyPI and S3 “folders” are
supported as repositories. (Mostly for testing purposes, local
directories are also supported as repositories).Under the hood,pkg_resources(fromsetuptools) is used to parse
and fulfill requirements, based on listings of available eggs obtained
from each repository. Once updated packages are made available, their
modules can be imported or the application can be re-launched with an
adjusted environment to pick up updated modules.The application writer controls the packages to update, repositories
each package will be grabbed from, and when an update is actioned.The philosophy has been that an absent or flaky network should not
prevent (or significantly delay) an application from running on top of
outdated packages, if they’ve been cached locally. That said, there are
currently no provisions for testing and verifying a successful update or
rolling back a failed update.ExampleThis is a script that requests the ‘hello’ package to be updated from
thedistfolder in the current working directory:importtransmutetransmute.require(['hello'],sources=['dist'])transmute.update()importhellohello.greet('world')Bootstrapping an application withbootstrap.pyThe submodule in`transmute/bootstrap.py<https://github.com/comoyo/python-transmute/blob/master/transmute/bootstrap.py>`__
can be used on its own to bootstrap other Python modules and
applications. It is capable of downloading packages from PyPI. In this
way,transmuteitself can be loaded and further used to download
additional packages.The script provides a bunch of hooks where users can place their code.
In particular,main()can be filled in to fetch application specific
packages and actually launch the application. At the point it is calledtransmutehas been added tosys.path(after downloading from
PyPI, if needed).defmain():importtransmuteimporttransmute.s3transmute.require(['foobar'],sources=['s3://foobar-repository/eggs/foobar'])transmute.update()importfoobar.clireturnfoobar.cli.run()It can also be used as a placeholder for a Python module. If the module
itself is available from PyPI, the corresponding package name would be
added to therequirementsvariable. For other use cases, I sense a
pull request coming :-)At the time of this writing, the only non-standard dependency of the
script is thepkg_resourcesmodule from thesetuptoolspackage.
The assumption here is that the module is more or less available
everywhere. If this turns out to be a problem in practice, I suppose the
script could be simplified to not require it.Supported package formatsCurrently, only standard Python eggs are supported. I don’t mind adding
support for other formats, formats supported natively by Python are
preferred.In that regard, source tarballs look particularly interesting for pure
Python packages, and seem to be more generally available from PyPI.
Unpacking and importing the packages locally could be a way forward.Python wheels also look interesting and gaining some traction.Supported repositoriesLocal repositoriesJust a directory with eggs in it. This is mostly useful for testing.transmute.require(['foobar'],sources=['/opt/basket'])PyPITransmute supports PyPI’sPyPIJSONinterface.transmute.require(['foobar'],sources=[transmute.PYPI_SOURCE])Amazon Simple Storage Service (S3)Packages can be uploaded to a directory in S3.While technically Amazon’s S3 doesn’t have the concept of a folder, the
slash ('/') in S3 key names is abused to sustain the illusion of
directories.Credentials can be provided as environment variables.transmuterecognizes a few fairly standard variables:- AWS_CREDENTIAL_FILE
- AWS_ACCESS_KEY, AWS_SECRET_KEY, and (optionally) AWS_SECURITY_TOKEN
- AWS_DEFAULT_REGION and EC2_REGIONWhen running in an EC2 instance, transmute may also pick credentials
from the IAM role associated with it.importtransmute.s3transmute.require(['foobar'],sources=['s3://bucket/key-prefix'])Missing a repository format?I’m missing a pull request. :-)Open issuesLogging is sorely missing. This can be helpful in debugging, but also
to keep track of updates and possibly tie in to enabling rollbacks.Rolling back a b0rked update.Provide hooks for verifying an update before activating it.We shouldn’t use the network on every run of a given command. Keeping
track of metadata about repository queries would allow us to limit
updates to daily or weekly schedules.Currently MD5 hashes are used to verify integrity of downloaded
packages, as advertised by repositories. It would be nice to be able
to verify package signatures.Your pet peeve?