Load partitions from the metastore as batches of configurable size at
configurable periodic intervals