astropy/astropy — pull request #1563 by embray (2013-12-16 23:36:36+00:00)
Title: Create a separate ``astropy-distutils`` package to contain common infrastructure code for the core and affiliated packages

(This conversation started in https://github.com/astropy/package-template/pull/32, but I will attempt to summarize here)

At the moment, affiliated packages need to import Astropy in order to have access to infrastructure code such as code to help populate the ``setup`` metadata. This is a bad idea, because packages shouldn't have to import a large package such as Astropy to be able to run the ``setup.py`` script.

@embray summarized it better than I could:

> The astropy package contains a number of useful utilities and hacks on distutils for building and installing, and distributing a package that depends on Numpy and Cython and that developed on git (as it includes utils for including the git changeset in the version info). The problem is that, as all those utilities reside in the astropy package, it has to be installed before they can be used in another package's setup.py. The ``setup_requires`` feature allows a package to be made available when running a project's setup.py script. Unfortunately, the astropy package itself being fairly sizeable and slow to build makes running any setup.py requiring astropy very slow. Furthermore, the way the feature is implemented requires running the setup() function, which supply's all the project's metadata, even though some of that metadata might require some of astropy's setup utilities to be generated (such as the full version string).``

The solution we have converged to in https://github.com/astropy/package-template/pull/32 is to create a separate repository containing the common infrastructure code. Again, to quote @embray:

> The first, and my personal preference especially from the view of affiliated packages (though also potentially beneficial for the main Astropy package) is to create a separate subproject, say, "astropy-distutils" that contains all the utilities used in setting up Astropy or affiliated packages using our framework. This could potentially include all the test stuff too. This would be a much smaller and simpler package than Astropy itself, and so can easily be used with the setup_requires feature of setuptools (where a package listed in setup_requires is downloaded and added to sys.path if it's not already installed).
>
> I have some experience with this: STScI has a package I wrote called "stsci-distutils" that serves the same purpose: a common collection of build and installation tools shared by all our packages. This works rather well. A particular advantage of this is that in many cases, if a tweak is needed to the build infrastructure (say, for example, Apple changes their default C compiler again) support for that can be added simply by releasing a new "astropy-distutils", rather than having to release a whole new astropy to fix builds on the platform in question. Of course, this won't always be possible, but it would work in many cases.

There are some minor questions, such as:

* whether this package should be a namespace package for ``astropy`` (i.e. it would be separate, but share the namespace with ``astropy``), or whether it should be its own independent package/namespace.

* whether it could be included via sub-modules (which doesn't seem necessary)

Also, we need to think which of the following it makes sense to move to this separate package - i.e. where do we stop? Do we include only what is strictly necessary for ``setup.py``? For example:

* ``setup_helpers.py``
* ``version_helpers.py``
* ``utils/release.py``

What about the testing framework? Or the configuration framework? (once it's refactored)

Of course, we don't want to do something so drastic before 0.3, but we should do it as soon as possible after 0.3.0 is out so that we have many months to iron out any bugs.