Package: lpot
Introduction to Intel® LPOTThe Intel® Low Precision Optimization Tool (Intel® LPOT) is an open-source Python library that delivers a unified low-precision inference interface across multiple Intel-optimized Deep Learning (DL) frameworks on both CPUs and GPUs. It supports automatic accuracy-driven tuning strategies, along with additional objectives such as optimizing for performance, model size, and memory footprint. It also provides easy extension capability for new backends, tuning strategies, metrics, and objectives.NoteGPU support is under development.Visit the Intel® LPOT online document website at:https://intel.github.io/lpot.ArchitectureIntel® LPOT features an infrastructure and workflow that aids in increasing performance and faster deployments across architectures.InfrastructureClick the image to enlarge it.WorkflowClick the image to enlarge it.Supported FrameworksSupported Intel-optimized DL frameworks are:TensorFlow*, including1.15.0 UP3,1.15.0 UP2,1.15.0 UP1,2.1.0,2.2.0,2.3.0,2.4.0,2.5.0,Official TensorFlow 2.6.0Note: Intel Optimized TensorFlow 2.5.0 requires to set environment variable TF_ENABLE_MKL_NATIVE_FORMAT=0 before running LPOT quantization or deploying the quantized model.Note: From Official TensorFlow 2.6.0, oneDNN support has been upstreamed. User just need download official TensorFlow binary for CPU device and set environment variable TF_ENABLE_ONEDNN_OPTS=1 before running LPOT quantization or deploying the quantized model.PyTorch*, including1.5.0+cpu,1.6.0+cpu,1.8.0+cpuApache* MXNet, including1.6.0,1.7.0,1.8.0ONNX* Runtime, including1.6.0,1.7.0,1.8.0InstallationSelect the installation based on your operating system.Linux InstallationYou can install LPOT using one of three options: Install just the LPOT library
from binary or source, or get the Intel-optimized framework together with the
LPOT library by installing theIntel® oneAPI AI Analytics Toolkit.Option 1 Install from binary# install stable version from pippipinstalllpot# install nightly version from pippipinstall-ihttps://test.pypi.org/simple/lpot# install stable version from from condacondainstalllpot-cconda-forge-cintelOption 2 Install from sourcegitclonehttps://github.com/intel/lpot.gitcdlpot
pipinstall-rrequirements.txt
pythonsetup.pyinstallOption 3 Install from AI KitThe Intel® LPOT library is released as part of theIntel® oneAPI AI Analytics Toolkit(AI Kit).
The AI Kit provides a consolidated package of Intel's latest deep learning and
machine optimizations all in one place for ease of development. Along with
LPOT, the AI Kit includes Intel-optimized versions of deep learning frameworks
(such as TensorFlow and PyTorch) and high-performing Python libraries to
streamline end-to-end data science and AI workflows on Intel architectures.The AI Kit is distributed through many common channels,
including from Intel's website, YUM, APT, Anaconda, and more.
Select anddownloadthe AI Kit distribution package that's best suited for you and follow theGet Started Guidefor post-installation instructions.Download AI KitAI Kit Get Started GuideWindows InstallationPrerequisitesThe following prerequisites and requirements must be satisfied for a successful installation:Python version: 3.6 or 3.7 or 3.8 or 3.9Download and installanaconda.Create a virtual environment named lpot in anaconda:# Here we install python 3.7 for instance. You can also choose python 3.6, 3.8, or 3.9.condacreate-nlpotpython=3.7
condaactivatelpotInstallation optionsOption 1 Install from binary# install stable version from pippipinstalllpot# install nightly version from pippipinstall-ihttps://test.pypi.org/simple/lpot# install from condacondainstalllpot-cconda-forge-cintelOption 2 Install from sourcegitclonehttps://github.com/intel/lpot.gitcdlpot
pipinstall-rrequirements.txt
pythonsetup.pyinstallDocumentationGet StartedAPIsexplains Intel® Low Precision Optimization Tool's API.Transformintroduces how to utilize LPOT's built-in data processing and how to develop a custom data processing method.Datasetintroduces how to utilize LPOT's built-in dataset and how to develop a custom dataset.Metricintroduces how to utilize LPOT's built-in metrics and how to develop a custom metric.Tutorialprovides comprehensive instructions on how to utilize LPOT's features with examples.Examplesare provided to demonstrate the usage of LPOT in different frameworks: TensorFlow, PyTorch, MXNet, and ONNX Runtime.UXis a web-based system used to simplify LPOT usage.Intel oneAPI AI Analytics Toolkit Get Started Guideexplains the AI Kit components, installation and configuration guides, and instructions for building and running sample apps.AI and Analytics Samplesincludes code samples for Intel oneAPI libraries.Deep DiveQuantizationare processes that enable inference and training by performing computations at low-precision data types, such as fixed-point integers. LPOT supports Post-Training Quantization (PTQ) withdifferent quantization capabilitiesand Quantization-Aware Training (QAT). Note that (Dynamic Quantization) currently has limited support.Pruningprovides a common method for introducing sparsity in weights and activations.Benchmarkingintroduces how to utilize the benchmark interface of LPOT.Mixed precisionintroduces how to enable mixed precision, including BFP16 and int8 and FP32, on Intel platforms during tuning.Graph Optimizationintroduces how to enable graph optimization for FP32 and auto-mixed precision.Model Conversionintroduces how to convert TensorFlow QAT model to quantized model running on Intel platforms.TensorBoardprovides tensor histograms and execution graphs for tuning debugging purposes.Advanced TopicsAdaptoris the interface between LPOT and framework. The method to develop adaptor extension is introduced with ONNX Runtime as example.Strategycan automatically optimized low-precision recipes for deep learning models to achieve optimal product objectives like inference performance and memory usage with expected accuracy criteria. The method to develop a new strategy is introduced.PublicationsMLPerf™ Performance Gains Abound with latest 3rd Generation Intel® Xeon® Scalable Processors(Apr 2021)3D Digital Face Reconstruction Solution enabled by 3rd Gen Intel® Xeon® Scalable Processors(Apr 2021)Accelerating Alibaba Transformer model performance with 3rd Gen Intel® Xeon® Scalable Processors (Ice Lake) and Intel® Deep Learning Boost(Apr 2021)Using Low-Precision Optimizations for High-Performance DL Inference Applications(Apr 2021)DL Boost Quantization with CERN's 3D-GANs model(Feb 2021)Full publication list please refers tohereSystem RequirementsIntel® Low Precision Optimization Tool supports systems based onIntel 64 architecture or compatible processors, specially optimized for the following CPUs:Intel Xeon Scalable processor (formerly Skylake, Cascade Lake, Cooper Lake, and Icelake)future Intel Xeon Scalable processor (code name Sapphire Rapids)Intel® Low Precision Optimization Tool requires installing the pertinent Intel-optimized framework version for TensorFlow, PyTorch, MXNet, and ONNX runtime.Validated Hardware/Software EnvironmentPlatformOSPythonFrameworkVersionCascade LakeCooper LakeSkylakeIce LakeCentOS 8.3Ubuntu 18.043.63.73.83.9TensorFlow2.6.02.5.02.4.02.3.02.2.02.1.01.15.0 UP11.15.0 UP21.15.0 UP31.15.2PyTorch1.5.0+cpu1.6.0+cpu1.8.0+cpuIPEXMXNet1.8.01.7.01.6.0ONNX Runtime1.6.01.7.01.8.0Validated ModelsIntel® Low Precision Optimization Tool provides numerous examples to show promising accuracy loss with the best performance gain. A full quantized model list on various frameworks is available in theModel List.Validated MLPerf ModelsModelFrameworkSupportExampleResNet50 v1.5TensorFlowYesLinkPyTorchYesLinkDLRMPyTorchYesLinkBERT-largeTensorFlowYesLinkPyTorchYesLinkSSD-ResNet34TensorFlowWIPPyTorchYesLinkRNN-TPyTorchWIP3D-UNetTensorFlowWIPPyTorchYesLinkValidated Quantized ModelsFrameworkVersionModelAccuracyPerformance speed upINT8 Tuning AccuracyFP32 Accuracy BaselineAcc Ratio [(INT8-FP32)/FP32]Realtime Latency Ratio[FP32/INT8]tensorflow2.5.0resnet50v1.074.24%74.27%-0.04%2.67xtensorflow2.5.0resnet50v1.576.94%76.46%0.63%2.54xtensorflow2.5.0resnet10177.21%76.45%0.99%2.46xtensorflow2.5.0inception_v170.30%69.74%0.80%1.63xtensorflow2.5.0inception_v274.27%73.97%0.41%1.74xtensorflow2.5.0inception_v377.29%76.75%0.70%2.11xtensorflow2.5.0inception_v480.36%80.27%0.11%2.59xtensorflow2.5.0inception_resnet_v280.42%80.40%0.02%1.86xtensorflow2.5.0mobilenetv173.93%70.96%4.19%2.27xtensorflow2.5.0mobilenetv271.96%71.76%0.28%1.78xtensorflow2.5.0vgg1672.13%70.89%1.75%3.86xtensorflow2.5.0vgg1972.35%71.01%1.89%3.90xFrameworkVersionModelAccuracyPerformance speed upINT8 Tuning AccuracyFP32 Accuracy BaselineAcc Ratio [(INT8-FP32)/FP32]Realtime Latency Ratio[FP32/INT8]pytorch1.8.0+cpuresnet1869.58%69.76%-0.26%2.13xpytorch1.8.0+cpuresnet5075.87%76.13%-0.34%3.11xpytorch1.8.0+cpuresnext101_32x8d79.09%79.31%-0.28%4.99xpytorch1.8.0+cpubert_base_mrpc87.92%88.73%-0.91%1.79xpytorch1.8.0+cpubert_base_cola58.33%58.84%-0.87%2.01xpytorch1.8.0+cpubert_base_sts-b88.46%89.27%-0.91%2.01xpytorch1.8.0+cpubert_base_sst-291.97%91.86%0.12%2.00xpytorch1.8.0+cpubert_base_rte69.68%69.68%0.00%1.92xpytorch1.8.0+cpubert_large_mrpc87.60%88.33%-0.83%2.47xpytorch1.8.0+cpubert_large_squad92.9993.05-0.06%1.85xpytorch1.8.0+cpubert_large_qnli91.12%91.82%-0.76%2.47xpytorch1.8.0+cpubert_large_rte71.84%72.56%-0.99%1.44xpytorch1.8.0+cpubert_large_cola61.97%62.57%-0.97%2.48xValidated Pruning ModelsTasksFWKModelfp32 baselinegradient sensitivity with 20% sparsity+onnx dynamic quantization on pruned modelaccuracy%drop%perf gain (sample/s)accuracy%drop%perf gain (sample/s)SST-2pytorchbert-baseaccuracy = 92.32accuracy = 91.97-0.381.30xaccuracy = 92.20-0.131.86xQQPpytorchbert-base[accuracy, f1] = [91.10, 88.05][accuracy, f1] = [89.97, 86.54][-1.24, -1.71]1.32x[accuracy, f1] = [89.75, 86.60][-1.48, -1.65]1.81xTasksFWKModelfp32 baselinePattern Lock on 70% Unstructured SparsityPattern Lock on 50% 1:2 Structured Sparsityaccuracy%drop%accuracy%drop%MNLIpytorchbert-base[m, mm] = [84.57, 84.79][m, mm] = [82.45, 83.27][-2.51, -1.80][m, mm] = [83.20, 84.11][-1.62, -0.80]SST-2pytorchbert-baseaccuracy = 92.32accuracy = 91.51-0.88accuracy = 92.20-0.13QQPpytorchbert-base[accuracy, f1] = [91.10, 88.05][accuracy, f1] = [90.48, 87.06][-0.68, -1.12][accuracy, f1] = [90.92, 87.78][-0.20, -0.31]QNLIpytorchbert-baseaccuracy = 91.54accuracy = 90.39-1.26accuracy = 90.87-0.73QnApytorchbert-base[em, f1] = [79.34, 87.10][em, f1] = [77.27, 85.75][-2.61, -1.54][em, f1] = [78.03, 86.50][-1.65, -0.69]FrameworkModelfp32 baselineCompressiondatasetacc(drop)%Pytorchresnet1869.7630% sparsity on magnitudeImageNet69.47(-0.42)Pytorchresnet1869.7630% sparsity on gradient sensitivityImageNet68.85(-1.30)Pytorchresnet5076.1330% sparsity on magnitudeImageNet76.11(-0.03)Pytorchresnet5076.1330% sparsity on magnitude and post training quantizationImageNet76.01(-0.16)Pytorchresnet5076.1330% sparsity on magnitude and quantization aware trainingImageNet75.90(-0.30)Additional ContentRelease InformationContribution GuidelinesLegalSecurity PolicyIntel® LPOT Website