Package: batchup
BatchUpPython library for extracting mini-batches of data from a data source for the purpose of training neural networks.Quick example:frombatchupimportdata_source# Construct an array data sourceds=data_source.ArrayDataSource([train_X,train_y])# Iterate over samples, drawing batches of 64 elements in random orderfor(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=True):# Processes batches here...Documentation available athttps://batchup.readthedocs.ioTable of ContentsInstallationBatch iterationProcessing data in mini-batches:quick batch iteration; a basic exampleiterating over subsets identified by indicesdata augmentationincluding sample indices in the mini-batchesinfinite batch iteration; an iterator that generates batches endlesslysample weighting to alter likelihood of samples (e.g. to compensate for class imbalance)iterating over two data sets simultaneously where their sizes differ (e.g. for semi-supervised learning)iterating over data sets that are NOT stored as NumPy arrays (e.g. on disk or generated on the fly)parallel processing to speed up iteration where loading/preparing samples could be slowGathering results and loss valuesremoving the for-loop; predict values for samples in one linecomputing mean loss/error valuesStandard datasetsBatchUp supports some standard machine learning datasets. They will be automatically downloaded if necessary.MNISTSVHNCIFAR-10CIFAR-100STLUSPSConfiguring BatchUpData paths, etc.More details further down, but briefly, use either the~/.batchup.cfgconfiguration file or theBATCHUP_HOMEenvironment varible.InstallationYou can install BatchUp with:> pip install batchupBatch iterationQuick batch iterationAssume we have a training set loaded in the variablestrain_Xandtrain_y:frombatchupimportdata_source# Construct an array data sourceds=data_source.ArrayDataSource([train_X,train_y])# Iterate over samples, drawing batches of 64 elements in random orderfor(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Processes batches here...Some notes:the last batch will be short (have less samples than the requested batch size) if there isn't enough data to fill it.usingshuffle=Truewill use NumPy's default random number generatornot specifying shuffle will process the samples in-orderIterating over subsets identified by indicesWe can specify the indices of a subset of the samples in a dataset and draw mini-batches from only those samples:importnumpyasnp# Randomly choose a subset of 20,000 samples, by indicessubset_a=np.random.permutation(train_X.shape[0])[:20000]# Construct an array data source that will only draw samples whose indices are in `subset_a`ds=data_source.ArrayDataSource([train_X,train_y],indices=subset_a)# Drawing batches of 64 elements in random orderfor(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Processes batches here...Data augmentationWe can define a function that applies data augmentation on the fly. Let's assume thattrain_Xcontains image data,
has the shape(sample, channel, height, width)and that we wish to horizontally flip some of the images:importnumpyasnp# Define our batch augmentation function.defaugment_batch(batch_X,batch_y):# Create an array that selects samples with 50% probability and convert to `bool` dtypeflip_flags=np.random.binomial(1,0.5,size=(len(batch_X),))!=0# Flip the width dimension in selected samplesbatch_X[flip_flags,...]=flip_flags[flip_flags,:,:,::-1]# Return the batch as a tuplereturnbatch_X,batch_y# Construct an array data source that will only draw samples whose indices are in `subset_a`ds=data_source.ArrayDataSource([train_X,train_y])# Apply augmentationds=ds.map(augment_batch)# Drawing batches of 64 elements in random orderfor(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Processes batches here...More complex augmentation may incurr significant runtime cost. This can be alleviated by preparing batches
in background threads. See theparallel processingsection below.Including sample indices in the mini-batchesWe can ask to be provided with the indices of the samples that were drawn to form the mini-batch:# Construct an array data source that will provide sample indicesds=data_source.ArrayDataSource([train_X,train_y],include_indices=True)# Drawing batches of 64 elements in random orderfor(batch_ndx,batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Processes batches here...Infinite batch iterationLets say you need an iterator that extracts samples from your dataset and starts from the beginning when it reaches the end:ds=data_source.ArrayDataSource([train_X,train_y],repeats=-1)Now use thebatch_iteratormethod as before.Therepeatsparameter accepts either-1for infininte, or any positive integer>= 1for a specified number of repetitions.This will also work if the dataset has less samples than the batch size; not a common use case but can happen in certain situations involving semi-supervised learning for instance.Sample weighting to alter likelihood of samplesIf you want some samples to be drawn more frequently than others, construct asampling.WeightedSamplerand pass
it as thesamplerargument to theArrayDataSourceconstructor. In the example the per-sample weights are stored
intrain_w.frombatchupimportsamplingsampler=sampling.WeightedSampler(weights=train_w)ds=data_source.ArrayDataSource([train_X,train_y],sampler=sampler)# Drawing batches of 64 elements in random orderfor(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Processes batches here...Notethat in-order is NOT supported when usingsampling.WeightedSampler, soshufflecannotbeFalseorNone.To draw from a subset of the dataset, usesampling.WeightedSubsetSampler:frombatchupimportsampling# NOTE that that parameter is called `sub_weights` (rather than `weights`) and that it must have the# same length as `indices`.sampler=sampling.WeightedSubsetSampler(sub_weights=train_w[subset_a],indices=subset_a)ds=data_source.ArrayDataSource([train_X,train_y],sampler=sampler)# Drawing batches of 64 elements in random orderfor(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Processes batches here...Class balancing helperAn alternate constructorsampling.WeightedSampler.class_balancing_sampleris available to construct a weighted sampler to compensate for class imbalance:# Construct the sampler; NOTE that the `n_classes` argument is *optional*sampler=sampling.WeightedSampler.class_balancing_sampler(y=train_y,n_classes=train_y.max()+1)ds=data_source.ArrayDataSource([train_X,train_y],sampler=sampler)# Drawing batches of 64 elements in random orderfor(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Processes batches here...Thesampling.WeightedSampler.class_balancing_sample_weightshelper method constructs an array of sample weights,
in case you wish to modify the weights first:weights=sampling.WeightedSampler.class_balancing_sample_weights(y=train_y,n_classes=train_y.max()+1)# Assume `modify_weights` is defined aboveweights=modify_weights(weights)# Construct the sampler and the data sourcesampler=sampling.WeightedSampler(weights=weights)ds=data_source.ArrayDataSource([train_X,train_y],sampler=sampler)# Drawing batches of 64 elements in random orderfor(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Processes batches here...Iterating over two data sources of wildly different sizes for semi-supervised learningIn semi-supervised learning we have a small dataset of labeled sampleslab_Xwith ground truthslab_yand a larger set of unlabeled samplesunlab_X. Lets say we want a single epoch to consist of the entire unlabeled dataset while looping over the labeled dataset repeatly. TheCompositeDataSourceclass can help us here.Without usingCompositeDataSource:rng=np.random.RandomState(12345)# Construct the data sources; the labeled data source will repeat infinitelyds_lab=data_source.ArrayDataSource([lab_X,lab_y],repeats=-1)ds_unlab=data_source.ArrayDataSource([unlab_X])# Construct an iterator to get samples from our labeled data source:lab_iter=ds_lab.batch_iterator(batch_size=64,shuffle=rng)# Iterate over the unlabled data set in the for-loopfor(batch_unlab_X,)inds_unlab.batch_iterator(batch_size=64,shuffle=rng):# Extract batches from the labeled iterator ourselvesbatch_lab_X,batch_lab_y=next(lab_iter)# Process batches here...Now usingCompositeDataSource:# Construct the data sources; the labeled data source will repeat infinitelyds_lab=data_source.ArrayDataSource([lab_X,lab_y],repeats=-1)ds_unlab=data_source.ArrayDataSource([unlab_X])ds=data_source.CompositeDataSource([ds_lab,ds_unlab])# Iterate over both the labeled and unlabeled samples:for(batch_lab_X,batch_lab_y,batch_unlab_X)inds.batch_iterator(batch_size=64,shuffle=rng):# Process batches here...The two component data sources (ds_labandds_unlab) will be shuffled independently.You can also haveCompositeDataSourcegenerate structured mini-batches that reflect the structure of the data source:# Flatten this time round:ds_struct=data_source.CompositeDataSource([ds_lab,ds_unlab],flatten=False)# Iterate over both the labeled and unlabeled samples:for((batch_lab_X,batch_lab_y),(batch_unlab_X,))inds_struct.batch_iterator(batch_size=64,shuffle=rng):# Process batches here...CompositeDataSourceinstances can be arbitrarily nested.Using data that is NOT stored as NumPy arraysThe arrays passed toArrayDataSourcedo not have to be NumPy arrays, they just have to be array-like. An array-like object should implement the__len__method that returns the number of samples and the__getitem__method that returns the samples themselves. Note that__getitem__should accept integer indices, slices, or NumPy integer arrays that give the indices of the samples to retrieve.Lets day we want to implement a data source that loads images from disk on the fly. Lets also assume that the prefix of the filename, either'cat'or'dog'gives the ground truth:importglobimportosfromscipy.miscimportimreadfrombatchupimportdata_sourceclassLoadImagesFromDisk(object):def__init__(self,paths):# Paths is a list of file pathsself.paths=paths# We have to imlement the `__len__` method:def__len__(self):returnlen(self.paths)# We have to implement the `__getitem__` method that `ArrayDataSource` will use to get samplesdef__getitem__(self,index):ifisinstance(index,(int,long)):# A single integer index; return that samplereturnimread(self.paths[index])elifisinstance(index,slice):# A sliceimages=[imread(p)forpinself.paths[index]]returnnp.concatenate([img[None,...]forimginimages],axis=0)elifisinstance(index,np.ndarray):ifindex.ndim!=1:raiseValueError('index array should only have 1 dimension, not{}'.format(index.ndim))images=[imread(self.paths[i])foriinindex]returnnp.concatenate([img[None,...]forimginimages],axis=0)else:raiseTypeError('index should be an integer, a slice or a NumPy array, ''not a{}'.format(type(index))# Get our image pathsimage_paths=glob.glob('/path/to/my/images/*.jpg')# Build our array-like data sourcetrain_X=LoadImagesFromDisk(image_paths)# Construct our ground truths as a NumPy arraytrain_y=[(1ifos.path.basename(p).startswith('dog')else0)forpinimage_paths)]train_y=np.array(train_y,dtype=np.int32)# Mixing custom array types with NumPy arrays is fineds=data_source.ArrayDataSource([train_X,train_y])for(batch_X,batch_y)inds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345)):# Process batches here...Using parallel processing to speed things upThe above example has a potential performance problem as loading the images from disk would introduce latency. We can use thework_poolmodule to prepare the mini-batches in separate threads or processes to hide this latency.Using threadsThe modifications to the previous example to use parallel processing are quite simple (lets assume that theLoadImagesFromDiskclass is defined and thattrain_X,train_yandds(anArrayDataSourceinstance) have already been built:frombatchupimportwork_pool# Build a pool of 4 worker threads:th_pool=work_pool.WorkerThreadPool(processes=4)# Construct a data source that prepares mini-batches in the background# It wraps the existing data source `ds` and will try to keep a buffer of 32# mini-batches full to eliminate latency:par_ds=th_pool.parallel_data_source(ds,batch_buffer_size=32)# As soon as we create an iterator, it will start filling its buffer; lets create an# iterator right now to get it going in the background:par_iter=par_ds.batch_iterator(batch_size=64,shuffle=np.random.RandomState(12345))# Do some other initialisation stuff that may take a while...# By now, with any luck, some batches will be ready to retrievefor(batch_X,batch_y)inpar_iter:# Process batches here...Using processesIn some cases the data source that you wish to parallelize may include some cacheing logic that is not thread safe. In such cases you can use process based pools that use separate processes rather than threads.
There are one or two gotchas, namely that using process-based pools entails a higher overhead and that the data source class and its dependent types must be declared in the top level of a module so thatpicklecan find them.# Build a pool of 4 worker processes:proc_pool=work_pool.WorkerProcessPool(processes=4)# Construct a data source that prepares mini-batches in the background# It wraps the existing non-thread-safe data source `ds` and# will try to keep a buffer of 32 mini-batches full to eliminate latency:par_ds=proc_pool.parallel_data_source(ds,batch_buffer_size=32)# ... use `par_ds` the same way as before ...Gathering results and loss valuesWe can further simplify training and evaluation procedures using thebatch_map_concatandbatch_map_meanmethods.Removing the for-loop; predict values for samples in one lineLets assume we have a prediction functionf_predof the formf_pred(batch_X) -> batch_pred_y.
If we want to predict results for our test set intest_X, we can do this in one line, without the for loop:test_ds=data_source.ArrayDataSource([test_X])(pred_y,)=test_ds.batch_map_concat(f_pred,batch_size=256)Thebatch_map_concatmethod will process all the samples intest_Xand gather the results in a tuple of arrays, hence
the(pred_y,) = .... If you wanttqdm(PyPi,GitHub) to give you a progress bar:(pred_y,)=test_ds.batch_map_concat(f_pred,batch_size=256,progress_iter_func=tqdm.tqdm)Computing mean loss/error valuesLets assume we have a evaluation functionf_evalof the formf_eval(batch_X, batch_y) -> [log_loss_sum, err_count].
Assuming that we are doing classification,f_evalreturns the sum of the per-sample log-losses and the number of errors.
Thebatch_map_meanmethod will process all of the data in the data source, gather loss and error counts and return the mean:val_ds=data_source.ArrayDataSource([val_X,val_y])mean_log_loss,mean_err_rate=val_ds.batch_map_mean(f_eval,batch_size=256)Note that as above, theprogress_iter_funcparameter can be passedtqdm.tqdmto give you a progress bar.Standard datasetsBatchUp provides support for using some standard datasets.MNIST datasetLoad the MNIST dataset:frombatchup.datasetsimportmnist# Load MNIST dataset (downloading it if necessary) and retain the last 10000# training samples for validationds=mnist.MNIST(n_val=10000)ds.train_Xis a(n, 1, 28, 28)float32array that contains the
training images.ds.train_yis a(n,)int32array that contains the ground truths.ds.val_Xandds.val_ycontain the validation samplesds.test_Xandds.test_ycontain the test samplesSVHN datasetLoad the SVHN dataset:frombatchup.datasetsimportsvhn# Load SVHN dataset (downloading it if necessary) and retain the last 10000# training samples for validationds=svhn.SVHN(n_val=10000)ds.train_Xis a(n, 3, 32, 32)float32array that contains the
training images.ds.train_yis a(n,)int32array that contains the ground truths.ds.val_Xandds.val_ycontain the validation samplesds.test_Xandds.test_ycontain the test samplesCIFAR-10 datasetLoad the CIFAR-10 dataset:frombatchup.datasetsimportcifar10# Load CIFAR-10 dataset (downloading it if necessary) and retain the last 5000# training samples for validationds=cifar10.CIFAR10(n_val=5000)ds.train_Xis a(n, 3, 32, 32)float32array that contains the
training images.ds.train_yis a(n,)int32array that contains the ground truths.ds.val_Xandds.val_ycontain the validation samplesds.test_Xandds.test_ycontain the test samplesds.class_nameslists the class names of the corresponding ground truth
indicesCIFAR-100 datasetLoad the CIFAR-100 dataset:frombatchup.datasetsimportcifar100# Load CIFAR-100 dataset (downloading it if necessary) and retain the last 5000# training samples for validationds=cifar100.CIFAR100(n_val=5000)ds.train_Xis a(n, 3, 32, 32)float32array that contains the
training images.ds.train_yis a(n,)int32array that contains the fine ground
truth classes.ds.train_y_coarseis a(n,)int32array that contains the coarse
ground truth classes.ds.val_X,ds.val_yandds.val_y_coarsecontain the validation samplesds.test_X,ds.test_yandds.test_y_coarsecontain the test samplesds.class_nameslists the class names of the corresponding fine ground
truth indicesds.class_names_coarselists the class names of the corresponding coarse
ground truth indicesSTL datasetLoad the STL dataset:frombatchup.datasetsimportstl# Load STL dataset (downloading it if necessary) and retain 1 fold of# training samples for validationds=stl.STL(n_val_folds=1)ds.train_X_u8is a(n, 3, 96, 96)uint8array that contains the
training images.ds.train_yis a(n,)int32array that contains the ground truths.ds.val_X_u8andds.val_ycontain the validation samplesds.test_X_u8andds.test_ycontain the test samplesds.class_nameslists the class names of the corresponding ground truth
indicesWe keep the image data inuint8form to save memory,USPS datasetLoad the USPS dataset (similar to MNIST hand-written digits but smaller):frombatchup.datasetsimportusps# Load USPS dataset (downloading it if necessary) and retain 729# training samples for validationds=usps.USPS(n_val=729)ds.train_Xis a(n, 1, 16, 16)float32array that contains the
training images.ds.train_yis a(n,)int32array that contains the ground truths.ds.val_Xandds.val_ycontain the validation samplesds.test_Xandds.test_ycontain the test samplesConfiguring BatchUp (paths etc).The configuration for BatchUp lives in.batchup.cfgin your home directory.By default BatchUp will store its data (e.g. downloaded datasets) in a directory called.batchupthat resides in your home directory. If you wish it to locate this data somewhere else (some of the datasets an take a few gigabytes), create the configuration file mentioned above:[paths]data_dir=/some/path/batchup_dataAlternatively you can set theBATCHUP_HOMEenvironment variable top the BatchUp data directory.