I was testing this branch out on a use case I am playing with where I run a Crunch pipeline that adds a partition to a dataset.  What I found was that the pipeline completed successfully but the data never showed up in the partition.

Debugging through I saw the cause was this rename was returning false during the merge of the task attempt dataset into the job dataset.  The task attempt dataset is partitioned but the partition is not created in the job dataset before this merge is attempted.  I believe the rename fails because the partition directory doesn't exist.

I'm not sure if it was intentional that you omitted creation of the partition on the job dataset rather than doing logic similar to what is in DatasetKeyOutputFormat.getRecordWriter()?  I played with a change where I invoked that logic from DatasetKeyOutputFormat.createJobDataset() and then my use case started working but I'm not sure what solution you might have in mind.