astropy/astropy — pull request #3181 by embray (2014-12-05 23:48:07+00:00)
Title: liberfa Python further simplification

This builds on top of the PR #3176 by @jwoillez by removing the need to build Python functions from a text template entirely. Instead it dumps the information gathered by the `cython_generator.py` script into a JSON file (that should obviously be distributed/installed with the source code). The metadata in this file is then used to generated the Python functions in the `erfa` module.

If the JSON file seems like too much indirection another alternative would be to just dump this dict directly into a Python file, but I think this is pretty nice since it could in principle be used in other contexts (documentation comes to mind...).

A further enhancement that I *want* to make to this, if it seems worthwhile, is to make the `erfa` module into a special module subtype that would be loaded via an import hook, so that erfa functions can be generated on demand when accessed, rather than all at once when the module is imported.

One downside to using JSON is that the JSON file, while smaller than the generated Python module, is still fairly large. And generating a function from it still means either:

1) Reading/parsing the entire JSON file every time a function is used for the first time.
2) Keeping the data structure read from JSON in memory.

Another possibility would be to use a different storage format that is better for on-demand key lookups, but the options for that tend to be problematic, and there other advantages to having a plain-text format. At the end of the day we're still talking half a megabyte so it's not a huge deal.

The current PR still saves memory at the end of the day, because once the JSON file has been read and the functions generated, the JSON data can be freed from memory, so all that is kept in memory by the Python interpreter is the code objects themselves (and all the docstrings, which take up the majority of space). This is in contrast to having the code objects *and* the source code in memory, which the Python interpreter does by default.

Another possible enhancement would be to make the docstring for these functions into a property that reads the function's docstring in from the JSON database on demand, but doesn't otherwise load the docstrings.