changed dataprovider pre-processing to remove the need to have batches written out on disk