Package: hpu-media-loader
Habana Media Python packagehpu_media_loaderis a package designed for easy integration of media processing on Gaudi2.
Main entry point (Python import) ishabana_frameworks.mediapipemodule that contains all the necessary functions to work with Gaudi2.StructureProperly built wheel contains:habana_frameworkspython namespace (with all the folder structure inside).mediapipefolder catering media execution on device andmedialoaderfolder catering pre-built mediapipe for tensorflow & pytorch frameworks.proper licensing.Media package (habana_frameworks.mediapipe and habana_frameworks.medialoaders)First part of media package contains media pipe which is responsible for media processing on device.Following are the steps to create mediapipeCreate a class derived fromhabana_frameworks.mediapipesuper class.In the class constructor initialize super class.Create nodes required for execution along with it's parameters.Define a methoddefinegraph()which defines the data flow between nodes created in constructor.Following are the steps to execute a standalone mediapipeInstantiate an object of defined mediapipe class.Build the mediapipe by executingbuild()method of mediapipe object.Initialize the iterator by callingiter_init()method of mediapipe object.To produce one batch of dataset, executerun()method of mediapipe object. Eachrun()method call executes and produces one batch of device tensors.To view or manipulate tensors on hostas_cpu()method of device tensor object can be called, which yields host tensor object.For numpy manipulationas_nparray()method of host tensor object can be called to get a numpy host array.Example:fromhabana_frameworks.mediapipeimportfnfromhabana_frameworks.mediapipe.mediapipeimportMediaPipefromhabana_frameworks.mediapipe.media_typesimportimgtypeasitfromhabana_frameworks.mediapipe.media_typesimportdtypeasdtfromhabana_frameworks.mediapipe.media_typesimportlayoutasltimporttimeclassmyMediaPipe(MediaPipe):def__init__(self,device,queue_depth,batch_size,channel,height,width):super(myMediaPipe,self).__init__(device,queue_depth,batch_size,channel,height,width,self.__class__.__name__,layout=lt.NHWC)mediapipe_seed=int(time.time_ns()%(2**31-1))# create reader node and setting it's paramsself.input=fn.ReadImageDatasetFromDir(dir="/path/to/jpeg/dir/",format="JPEG",shuffle=True,seed=mediapipe_seed)# create decoder node and set it's paramsself.decode=fn.ImageDecoder(output_format=it.RGB_P,resize=[224,224])# create transpose node and set it's paramsself.transpose=fn.Transpose(permutation=[2,0,1,3],tensorDim=4)defdefinegraph(self):# define actual data flow of nodesjpegs,data=self.input()images=self.decode(jpegs)images=self.transpose(images)# return output nodes of the graphreturnimages,data# test specific paramsbatch_size=4img_width=224img_height=224channels=3queue_depth=3iterations=5# instantiating defined classpipe=myMediaPipe("hpu",queue_depth,batch_size,channels,img_height,img_width)# build the pipepipe.build()# initialize iteratorpipe.iter_init()batch_count=0while(batch_count<iterations):try:# exectute and produce one batch of dataset.images,labels=pipe.run()# images and labels are device tensors.exceptStopIteration:print("stop iteration")break# as cpu will bring the device data to host and produce host tensors# as_nparray will convert host tensors to numpy array.images=images.as_cpu().as_nparray()labels=labels.as_cpu().as_nparray()batch_count=batch_count+1Second part of media package contains pre built media pipe for tensorflow and pytorch.tensorflow folder contains media_resnet_pipe containing resnet pipe for tensflow graphFollowing are the steps to use pre built mediapipe for tensorflowImportResnetPipefromhabana_frameworks.medialoaders.tensorflow.media_resnet_pipeInstantiate an object ofResnetPipewith following parametersdevice name: hpuqueue_depth: queue depth for media processing.batch_size: mediapipe output batch size.height: mediapipe output image height.width: mediapipe output image width.is_training: if is training pipe or validation pipedata_dir: jpeg data directory.out_dtype: output datatype of image.num_slices: number of slices to be done of dataset.slice_index: slice index to be used for this instance of execution.Instantiate an object ofHabanaDatasetwhich is derived from tensorflow dataset with following parametersoutput_shapes: list of output shapes of the  dataset.output_types: list of output datatype of the dataset.pipeline: media pipeline object.Above dataset can be used for dataset iterations.Example:fromhabana_frameworks.medialoaders.tensorflow.media_resnet_pipeimportResnetPipefromhabana_frameworks.tensorflow.media.habana_datasetimportHabanaDataset# network specific parametersbatch_size=256num_channels=3img_size=224is_training=Truedir_path='/jpeg/path/'media_dtype='bfloat16'num_slices=1slice_index=0queue_depth=3tf_media_dtype=tf.bfloat16tf_meta_dtype=tf.float32#pre defined mediapipe from medialoaderspipe=ResnetPipe("hpu",queue_depth,batch_size,num_channels,img_size,img_size,is_training,dir_path,media_dtype,num_slices,slice_index)# tensorflow predefine habanadataset classdataset=HabanaDataset(output_shapes=[(batch_size,img_size,img_size,num_channels),(batch_size,)],output_types=[tf_media_dtype,tf_meta_dtype],pipeline=pipe)# above dataset object is tf dataset object which is iteratable and can be fed to training node.torch folder contains media_dataloader_mediapipe containingHPUMediaPipewhich can be used to create resnet and SSD media pipe for pytorchFollowing are the steps to useHPUMediaPipefor pytorchImportHPUMediaPipefromhabana_frameworks.medialoaders.torch.media_dataloader_mediapipeInstantiate an object ofHPUMediaPipewith following parameters:a_torch_transforms: transforms to be applied on mediapipe.a_root: directory path from which to load the images.a_annotation_file: path from which to load annotation file for SSD.a_batch_size: mediapipe output batch size.a_shuffle: whether images have to be shuffled. <True/False>a_drop_last: whether to drop the last incomplete batch or round up.<True/False>a_prefetch_count: queue depth for media processing.a_num_instances:  number of devices.a_instance_id: instance id of current device.a_model_ssd: whether mediapipe is to be created for SSD. <True/False>a_device: media device to run mediapipe on.SeparateHPUMediaPipeobjects can be created for training and validation.Instantiate an object ofHPUResnetPytorchIterator(for resnet) orHPUSsdPytorchIterator(for SSD) with following parametersmediapipe: media pipe object.Example for resnet media pipe:fromhabana_frameworks.medialoaders.torch.media_dataloader_mediapipeimportHPUMediaPipefromhabana_frameworks.mediapipe.plugins.iterator_pytorchimportHPUResnetPytorchIteratornormalize=transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225])torch_transforms=transforms.Compose([transforms.RandomResizedCrop(224),transforms.RandomHorizontalFlip(),transforms.ToTensor(),normalize,])root="/JPEG/path"batch_size=256shuffle=Truedrop_last=Falseprefetch_factor=3num_instances=1instance_id=0pipeline=HPUMediaPipe(a_torch_transforms=torch_transforms,a_root=root,a_batch_size=batch_size,a_shuffle=shuffle,a_drop_last=drop_last,a_prefetch_count=prefetch_factor,a_num_instances=num_instances,a_instance_id=instance_id,a_device="hpu")iterator=HPUResnetPytorchIterator(mediapipe=pipeline)