Package: deeplite-profiler
Deeplite ProfilerTo be able to use a deep learning model in research and production, it is essential to understand different performance metrics of the model beyond just the model's accuracy.deeplite-profilerhelps to easily and effectively measure the different performance metrics of a deep learning model. In addition to the existing metrics in thedeeplite-profiler, users could seamlessly contribute any custom metric to measure using the profiler.deeplite-profilercould also be used to compare the performance between two different deep learning models, for example, a teacher and a student model.deeplite-profilercurrently supports PyTorch and TensorFlow Keras (v1) as two different backend frameworks.Deeplite ProfilerInstallationInstall using pipInstall from sourceInstall in Dev modeHow to UseFor a PyTorch ModelFor a TensorFlow ModelOutput DisplayExamplesContribute a Custom MetricInstallationInstall using pipUse following command to install the package from our internal PyPI repository.$pipinstall--upgradepip$pipinstalldeeplite-profiler[`backend`]Install from source$gitclonehttps://github.com/Deeplite/deeplite-profiler.git$pipinstall.[`backend`]One can install specificbackendmodules, depending on the required framework and compute support.backendcould be one of the following valuestorch: to install atorchspecific profilertf: to install aTensorFlowspecific profiler (this supports only CPU compute)tf-gpu: to install aTensorFlow-gpuspecific profiler (this supports both CPU and GPU compute)all: to install bothtorchandTensorFlowspecific profiler (this supports only CPU compute for TensorFlow)all-gpu: to install bothtorchandTensorFlow-gpuspecific profiler (for GPU environment) (this supports both CPU and GPU compute for TensorFlow)Install in Dev mode$gitclonehttps://github.com/Deeplite/deeplite-profiler.git$pipinstall-e.[`backend`]$pipinstall-rrequirements-test.txtTo test the installation, one can run the basic tests usingpytestcommand in the root folder.NOTE:Currently, we support Tensorflow 1.14 and 1.15 versions, for Python 3.6 and 3.7. Wedo notsupport Python 3.8+.How to UseFor a PyTorch Model# Step 1: Define native pytorch dataloaders and model# 1a. data_splits = {"train": train_dataloder, "test": test_dataloader}data_splits=/*...loaditerabledataloaders...*/model=/*...loadnativedeeplearningmodel...*/# Step 2: Create Profiler class and register the profiling functionsdata_loader=TorchProfiler.enable_forward_pass_data_splits(data_splits)profiler=TorchProfiler(model,data_splits,name="Original Model")profiler.register_profiler_function(ComputeComplexity())profiler.register_profiler_function(ComputeExecutionTime())profiler.register_profiler_function(ComputeEvalMetric(get_accuracy,'accuracy',unit_name='%'))# Step 3: Compute the registered profiler metrics for the PyTorch Modelprofiler.compute_network_status(batch_size=1,device=Device.CPU,short_print=False,include_weights=True,print_mode='debug')# Step 4: Compare two different models or profilers.profiler2=profiler.clone(model=deepcopy(model))# Creating a dummy clone of the current profilerprofiler2.name="Clone Model"profiler.compare(profiler2,short_print=False,batch_size=1,device=Device.CPU,print_mode='debug')For a TensorFlow Model# Step 1: Define native tensorflow dataloaders and model# 1a. data_splits = {"train": train_dataloder, "test": test_dataloader}data_splits=/*...loaditerabledataloaders...*/model=/*...loadnativedeeplearningmodel...*/# Step 2: Create Profiler class and register the profiling functionsdata_loader=TFProfiler.enable_forward_pass_data_splits(data_splits)profiler=TFProfiler(model,data_splits,name="Original Model")profiler.register_profiler_function(ComputeFlops())profiler.register_profiler_function(ComputeSize())profiler.register_profiler_function(ComputeParams())profiler.register_profiler_function(ComputeLayerwiseSummary())profiler.register_profiler_function(ComputeExecutionTime())profiler.register_profiler_function(ComputeEvalMetric(get_accuracy,'accuracy',unit_name='%'))# Step 3: Compute the registered profiler metrics for the Tensorflow Keras Modelprofiler.compute_network_status(batch_size=1,device=Device.CPU,short_print=False,include_weights=True,print_mode='debug')# Step 4: Compare two different models or profilers.profiler2=profiler.clone(model=model)# Creating a dummy clone of the current profilerprofiler2.name="Clone Model"profiler.compare(profiler2,short_print=False,batch_size=1,device=Device.CPU,print_mode='debug')Output DisplayAn example output of thedeeplite-profilerforresnet18model using the standardCIFAR100dataset usingPyTorchbackend looks as follows+---------------------------------------------------------------+|                    deeplite Model Profiler                    |+-----------------------------------------+---------------------+|             Param Name (Original Model) |                Value||                   Backend: TorchBackend |                     |+-----------------------------------------+---------------------+|                   Evaluation Metric (%) |              76.8295||                         Model Size (MB) |              42.8014||     Computational Complexity (GigaMACs) |               0.5567||             Total Parameters (Millions) |              11.2201||                   Memory Footprint (MB) |              48.4389||                     Execution Time (ms) |               2.6537|+-----------------------------------------+---------------------+Evaluation Metric:Computed performance of the model on the given dataModel Size:Memory consumed by the parameters (weights and biases) of the modelComputational Complexity:Summation of Multiply-Add Cumulations (MACs) per single image (batch_size=1)#Total Parameters:Total number of parameters (trainable and non-trainable) in the modelMemory Footprint:Total memory consumed by parameters and activations per single image (batch_size=1)Execution Time:OnNVIDIA TITAN V <https://www.nvidia.com/en-us/titan/titan-v/>_ GPU, time required for the forward pass per single image (batch_size=1)ExamplesA list of different examples to usedeeplite-profilerto profiler different PyTorch and TensorFlow models can be foundhereContribute a Custom MetricNOTE:If you looking for an SDK documentation, please head overhere.We always welcome community contributions to expand the scope ofdeeplite-profilerand also to have additional new metrics. Please refer to thedocumentationfor the detailed steps on how to design a new metrics. In general, we follow thefork-and-pullGit workflow.Forkthe repo on GitHubClonethe project to your own machineCommitchanges to your own branchPushyour work back up to your forkSubmit aPull requestso that we can review your changesNOTE: Be sure to merge the latest from "upstream" before making a pull request!