kite-sdk/kite — pull request #108 by rdblue (2014-06-19 23:43:04+00:00)
Title: CDK-353: Add TemporaryDatasetRepository

This introduces `TemporaryDatasetRepository` for kite-data-mapreduce, which manages the temporary datasets written by each output task using a named temporary repository that is deleted after its contents are committed. Hive uses a regular FS dataset for its temporary space, so the Hive MetaStore is not contacted from MR tasks.

CDK-353 suggested a temporary dataset rather than a temporary repository, but this is simpler to implement because it allows descriptors to be written to disk rather than supplied when creating the dataset. It also deletes all of the temporary data at once, which is simpler to manage.