ga4gh/server — pull request #39 by jeromekelleher (2014-11-17 16:12:00+00:00)
Title: Generate schemas from Avro definitions

This is an early PR for discussion; it will be updated and cleaned up before proposed merging.

We want to generate our protocol element definitions from the Avro IDL in as automated manner as possible --- see issue #5 for further discussion. This PR is an initial pass at generating Python code from the Avro schemas. Before putting any more work in on this, it would be good to get some feedback on the basic design, and make sure we are all happy with it.

The basic idea is that we pull down the schema definitions in Avro IDL format from Github and then transform them into the standard JSON representation using the Java tools. We then read in the JSON definitions, generate Python code for each of the classes, and write this out to the file `ga4gh/_protocol_definitions.py`. This code can then be checked in to Github, where it will mostly go unmodified. Each time we tag a release, we can rerun the script and regenerate the protocol definitions. The protocol definitions are imported into the existing `ga4gh.protocol` module, and as far as client code is concerned, nothing has changed. The current code works, and passes the unit tests.

Is this model of generating code from a tagged version in Github a good idea?

TODO:

1. We currently don't use the Python Avro package to parse the schemas, which is pretty dumb. Expect much of the current parsing code to disappear.
2. Combine the functionality of the `preprocess_schemas.sh` script and the `generate_schemas.py` script and add a CLI so that we can choose the tagged version (or Git revision?) to download and process.
3. Add type and extra information to the Python classes so that we can validate server input and also improve our unit testing (see also issue #22).