bigdatagenomics/adam — pull request #224 by laserson (2014-04-23 03:46:39+00:00)
Title: New proposal for ADAMFeature

This PR serves as a sketch for another way of doing ADAMFeatures. Please comment!

The considerations/principles were as follows:

* Embrace the Hadoop philosophy of not being over-opinionated about how things should be done. Let the user work with features however she wishes.

* Encourage use of a type system to allow for compile-time checks that the user's code behaves as expected.

Some alternatives have problems:

* Having an Avro type for each feature type (e.g., `ADAMGFFFeature`, `ADAMBEDFeature`) is too messy and not as forward looking as I would like. One way to go, however, is to define a very slimmed down `ADAMFeature` type that we would encourage people to use going forward.

* Trying to take the intersection of all different feature formats and model a common format for which they all work. I think there is just too much diversity between the feature types. You'd probably end up with the current schema anyway. This is the same hurdle that Hadoop tries to eliminate with respect to people having to shape their data in a very particular way to match some SQL schema.

So my favorite sol'n at the moment is the following:

* Have an `ADAMFeature` Avro type that includes all possible fields under the sun. Each popular feature format (e.g., BED, GFF) should be able to represent all fields in this schema.

* Have a type hierarchy rooted at `BaseFeature` with subclasses like `BEDFeature` or `GFFFeature` that provide getters to access an underlying `ADAMFeature` object. This way, a user can work with `RDD[BEDFeature]` and only access the fields that it should logically provide. The `adamLoad` function will then produce the proper RDD type that, when serialized to disk, will write out `ADAMFeature`s.

This approach depends on the good behavior of the user, as they could simply circumvent the system and work with `ADAMFeature` objects if they wish, which I think is both a positive and negative.

To add additional feature types, you'd simply have to subclass `BaseFeature`, implement the `FeatureParser` interface, and register the extension so it maps to the correct machinery.

Thoughts?