Package: numpy2tfrecord
numpy2tfrecordSimple helper library to convert numpy data to tfrecord and build a tensorflow dataset.Installation$gitclonegit@github.com:yonetaniryo/numpy2tfrecord.git
$cdnumpy2tfrecord
$pipinstall.or simply using pip:$pipinstallnumpy2tfrecordHow to useConvert a collection of numpy data to tfrecordYou can convert samples represented in the form of adicttotf.train.Exampleand save them as a tfrecord.importnumpyasnpfromnumpy2tfrecordimportNumpy2TFRecordConverterwithNumpy2TFRecordConverter("test.tfrecord")asconverter:x=np.arange(100).reshape(10,10).astype(np.float32)# float arrayy=np.arange(100).reshape(10,10).astype(np.int64)# int arraya=5# intb=0.3# floatsample={"x":x,"y":y,"a":a,"b":b}converter.convert_sample(sample)# convert data sampleYou can also convert alistof samples at once usingconvert_list.withNumpy2TFRecordConverter("test.tfrecord")asconverter:samples=[{"x":np.random.rand(64).astype(np.float32),"y":np.random.randint(0,10),}for_inrange(32)]# list of 32 samplesconverter.convert_list(samples)Or a batch of samples at once usingconvert_batch.withNumpy2TFRecordConverter("test.tfrecord")asconverter:samples={"x":np.random.rand(32,64).astype(np.float32),"y":np.random.randint(0,10,size=32).astype(np.int64),}# batch of 32 samplesconverter.convert_batch(samples)So what are the advantages ofNumpy2TFRecordConvertercompared totf.data.datset.from_tensor_slices?
Simply put, when usingtf.data.dataset.from_tensor_slices, all the samples that will be converted to a dataset must be in memory.
On the other hand, you can useNumpy2TFRecordConverterto sequentially add samples to the tfrecord without having to read all of them into memory beforehand..Build a tensorflow dataset from tfrecordSamples once stored in the tfrecord can be streamed usingtf.data.TFRecordDataset.fromnumpy2tfrecordimportbuild_dataset_from_tfrecorddataset=build_dataset_from_tfrecord("test.tfrecord")The dataset can then be used directly in the for-loop of machine learning.forbatchindataset.as_numpy_iterator():x,y=batch.values()...Speeding up PyTorch data loading withnumpy2tfrecord!https://gist.github.com/yonetaniryo/c1780e58b841f30150c45233d3fe6d01importosimporttimeimportnumpyasnpfromnumpy2tfrecordimportNumpy2TfrecordConverter,build_dataset_from_tfrecordimporttorchfromtorchvisionimportdatasets,transformsdataset=datasets.MNIST(".",download=True,transform=transforms.ToTensor())# convert to tfrecordwithNumpy2TfrecordConverter("mnist.tfrecord")asconverter:converter.convert_batch({"x":dataset.data.numpy().astype(np.int64),"y":dataset.targets.numpy().astype(np.int64)})torch_loader=torch.utils.data.DataLoader(dataset,batch_size=32,pin_memory=True,num_workers=os.cpu_count())tic=time.time()foreinrange(5):forbatchintorch_loader:x,y=batchelapsed=time.time()-ticprint(f"elapsed time with pytorch dataloader:{elapsed:0.2f}sec for 5 epochs")tf_loader=build_dataset_from_tfrecord("mnist.tfrecord").batch(32).prefetch(1)tic=time.time()foreinrange(5):forbatchintf_loader.as_numpy_iterator():x,y=batch.values()elapsed=time.time()-ticprint(f"elapsed time with tf dataloader:{elapsed:0.2f}sec for 5 epochs")⬇️elapsed time with pytorch dataloader: 41.10 sec for 5 epochs
elapsed time with tf dataloader: 17.34 sec for 5 epochs