Changing the type of a schema field will definitely break backward compatibility.

I gave @tdanford misinformation about schema compatibility before. To keep the schema compatible by making sure the same field is not defined with different types or incompatible repetition (meaning the field needs to be at the same nesting level).

Since ADAM is still such a new project, we should feel free to break things to make them better for the long-term. Once we mature and our used more broadly in production, we won't have the luxury.

I agree that we're starting to see a theme emerging here about how to best (de-)normalize the data. On the ADAM mailing list, I sent the following:

> We could address this limitation by creating a separate sequence dictionary file during the conversion from
> BAM to ADAM. I think that Timothy and Carl at GenomeBridge would have a more informed idea of how to
> manage sequence dictionaries than I do. As far as I see it, we can: (1) keep records fully de-normalized, (2)
> add a sequence dictionary file separate from the de-normalized records with its own schema or (3) switch to
> normalized records with a separate sequence dictionary or (4) have a separate dictionary and allow users
> decide whether to normalize records.
> Which approach do you guys like best?

But it was buried deep inside a thread.

My preference is for (1) right now but I'm always happy to be told I'm wrong. Even if we keep everything de-normalized, we'll still be the same size as BAM. I don't think that compression is as important as correctness, performance and ease-of-use. If compression becomes an issue, it would be easy to add an ADAM "archive" format later that normalizes all data and uses maximum compression for the most compact size.

Re: performance. Having a unique "RecordGroupId" will make the Mark Duplicates code much faster (not that it's very slow now). Instead of doing a groupBy on Strings, we can do a groupBy on Ints. Any other transforms that group by record group will also be much faster (e.g. BQSR).