kite-sdk/kite — pull request #115 by rdblue (2014-06-24 01:38:04+00:00)
Title: CDK-488: Add TaskUtil.partition.

This utility method regroups a PCollection using a specified number of
reducers. This can be used to control the number of files created by a
copy job. The number of reducers is approximately the number of files
produced, but at least one file per partition will be created.

This is implemented in two ways. If the dataset is partitioned, the
entities are grouped by partition key so that each partition is handled
by one reducer and the number of files created is minimized. If the
dataset is not partitioned, this will perform the regroup operation
using the entities themselves as keys to produce the right number of
files, one per reduce task.