Package: dapipe
DPIPEWith dpipe you can create ready to use datasets from paths or list of files. You should
specify the type and the location of the input and target. The labels are assumed to be the name of the folder containing the file,
if you need a dataset for classification.The inputs and targets can be a list of paths, a path to be explored containing images or videos. For example:./dataset||--cat/img1.png||--cat/img2.png|--dog/img1.png|--dog/img2.pngThe functionmake_datasetoutputs adpipe.dataset_builderobject that has the method to predefined multiprocessing setups based on the recomendation of tensorflow.methodactiondataset_builder.prefetch()Preloads samples on memorydataset_builder.batch()Creates a batch datasetdataset_builder.enumerate()Creates a appends an index to the outputdataset_builder.filter()Applies a filter concurrentlydataset_builder.map()Applies a function to each element concurrentlydataset_builder.repeat()Creates a repeated datasetdataset_builder.shuffle()Shuffles the dataset after a complete runThe dataset can be specified as:fromdpipeimportmake_datasetdataset=make_dataset('image','label',x_path='./dataset',x_size=(128,128)).build()Creating dataset (more options)Additionally, we defined the dataset from functions or objects. Two use cases are presented here. A dataset can be created from a function and a list of element to parse, for example a list of files and a reading function.
For example, if we need are training a denoising autoencoder, we need image noisy and clean image pairs; this can be handled with the functiondpipe.from_function:importglob# to find the filesimportmatplotlib.imageasmpimg# to read the images (you need to install it.)importnumpyasnpfromdpipeimportfrom_functionfilelist=glob.glob('./dataset','*.png')defread_file(filename):target=mpimg.imread(filename)# read the imagenoisy_image=np.random.randn(target.shape)returnnoisy_image,target# undetermined shape is used to define dimentions that vary across shamples, in this case the height and the width of the imagesdataset=from_function(read_file,filelist,undetermined_shape=((1,2),(1,2))).build()If you are accessing your data in an object oriented way, you can usedpipe.from_object. In the next example lets consider you want use consume a list of files with records on it via generator function, this can also be handled withdpipe.from_functionthough. The code should look like thisimportosimportpandasaspdfromdpipeimportfrom_objectclassReader():def__init__(self,datapath='./dataset'):self.filelist=os.listdir(datapath)def__len__(self):returnlen(self.filelist)defmy_reading_function(self,filename):df=pd.read_csv(filename)forv,tinzip(df.values,df.targets):yieldv,treader=Reader()dataset=from_object(reader,'my_reading_function','filelist').build()Thebuild()function that creates a dataset with arguments ready to use with thefit()method of andtf.modelobject. This is used like this:training_ds=from_object(reader_training,'my_reading_function').shuffle(len(reader_training),reshuffle_each_iteration=True).batch(32).repeat().build()validation_ds=from_object(reader_validation,'my_reading_function',training=False).batch(32).build()model.fit(x=training_ds,validation_data=validation_ds,epochs=10,**training_ds.built_args,**validation_ds_ds.built_args)InstallationpipinstalldapipeIt requires to install FFMPEG (here) to work with video formats.