kite-sdk/kite — pull request #199 by rdblue (2014-09-07 23:17:25+00:00)
Title: CDK-634: Add Dataset creation to MR and Crunch.

This adds an option to set a descriptor for the output URI, which is
used to create a Dataset if one doesn't already exist. For Datasets
that use the MergeOutputCommitter, the final Dataset is created in the
commitJob method and the NullOutputCommitter creates the dataset in the
setupJob method because tasks write directly to it.

This adds methods to store and retrieve a descriptor in a Configuration
to DescriptorUtil. These methods take a context String that is used to
build property names, so multiple descriptors can be stored in the same
Configuration.

Because the Descriptor can be stored in the Job Configuration, this no
longer relies on the job Dataset to serialize the descriptor to task
attempts. Task attemps no longer load the job Dataset to create the task
attempt Datasets.