Package: nncf
Neural Network Compression Framework (NNCF)Key Features•Installation•Documentation•Usage•Tutorials and Samples•Third-party integration•Model ZooNeural Network Compression Framework (NNCF) provides a suite of post-training and training-time algorithms for neural networks inference optimization inOpenVINO™with minimal accuracy drop.NNCF is designed to work with models fromPyTorch,TensorFlow,ONNXandOpenVINO™.NNCF providessamplesthat demonstrate the usage of compression algorithms for different use cases and models. See compression results achievable with the NNCF-powered samples atModel Zoo page.The framework is organized as a Python* package that can be built and used in a standalone mode. The framework
architecture is unified to make it easy to add different compression algorithms for both PyTorch and TensorFlow deep
learning frameworks.Key FeaturesPost-Training Compression AlgorithmsCompression algorithmOpenVINOPyTorchTensorFlowONNXPost-Training QuantizationSupportedSupportedSupportedSupportedWeights CompressionSupportedSupportedNot supportedNot supportedTraining-Time Compression AlgorithmsCompression algorithmPyTorchTensorFlowQuantization Aware TrainingSupportedSupportedMixed-Precision QuantizationSupportedNot supportedBinarizationSupportedNot supportedSparsitySupportedSupportedFilter pruningSupportedSupportedMovement pruningExperimentalNot supportedAutomatic, configurable model graph transformation to obtain the compressed model.NOTE: Limited support for TensorFlow models. The models created using Sequential or Keras Functional API are only supported.Common interface for compression methods.GPU-accelerated layers for faster compressed model fine-tuning.Distributed training support.Git patch for prominent third-party repository (huggingface-transformers) demonstrating the process of integrating NNCF into custom training pipelinesSeamless combination of pruning, sparsity and quantization algorithms. Please refer tooptimum-intelfor examples of
joint (movement) pruning, quantization and distillation (JPQD), end-to-end from NNCF optimization to compressed OpenVINO IR.Exporting PyTorch compressed models to ONNX* checkpoints and TensorFlow compressed models to SavedModel or Frozen Graph format, ready to use withOpenVINO™ toolkit.Support forAccuracy-Aware model trainingpipelines via theAdaptive Compression Level TrainingandEarly Exit Training.DocumentationThis documentation covers detailed information about NNCF algorithms and functions needed for the contribution to NNCF.The latest user documentation for NNCF is availablehere.NNCF API documentation can be foundhere.UsagePost-Training QuantizationThe NNCF PTQ is the simplest way to apply 8-bit quantization. To run the algorithm you only need your model and a small (~300 samples) calibration dataset.OpenVINOis the preferred backend to run PTQ with, and PyTorch, TensorFlow and ONNX are also supported.OpenVINOimportnncfimportopenvino.runtimeasovimporttorchfromtorchvisionimportdatasets,transforms# Instantiate your uncompressed modelmodel=ov.Core().read_model("/model_path")# Provide validation part of the dataset to collect statistics needed for the compression algorithmval_dataset=datasets.ImageFolder("/path",transform=transforms.Compose([transforms.ToTensor()]))dataset_loader=torch.utils.data.DataLoader(val_dataset,batch_size=1)# Step 1: Initialize transformation functiondeftransform_fn(data_item):images,_=data_itemreturnimages# Step 2: Initialize NNCF Datasetcalibration_dataset=nncf.Dataset(dataset_loader,transform_fn)# Step 3: Run the quantization pipelinequantized_model=nncf.quantize(model,calibration_dataset)PyTorchimportnncfimporttorchfromtorchvisionimportdatasets,models# Instantiate your uncompressed modelmodel=models.mobilenet_v2()# Provide validation part of the dataset to collect statistics needed for the compression algorithmval_dataset=datasets.ImageFolder("/path",transform=transforms.Compose([transforms.ToTensor()]))dataset_loader=torch.utils.data.DataLoader(val_dataset)# Step 1: Initialize the transformation functiondeftransform_fn(data_item):images,_=data_itemreturnimages# Step 2: Initialize NNCF Datasetcalibration_dataset=nncf.Dataset(dataset_loader,transform_fn)# Step 3: Run the quantization pipelinequantized_model=nncf.quantize(model,calibration_dataset)TensorFlowimportnncfimporttensorflowastfimporttensorflow_datasetsastfds# Instantiate your uncompressed modelmodel=tf.keras.applications.MobileNetV2()# Provide validation part of the dataset to collect statistics needed for the compression algorithmval_dataset=tfds.load("/path",split="validation",shuffle_files=False,as_supervised=True)# Step 1: Initialize transformation functiondeftransform_fn(data_item):images,_=data_itemreturnimages# Step 2: Initialize NNCF Datasetcalibration_dataset=nncf.Dataset(val_dataset,transform_fn)# Step 3: Run the quantization pipelinequantized_model=nncf.quantize(model,calibration_dataset)ONNXimportonnximportnncfimporttorchfromtorchvisionimportdatasets# Instantiate your uncompressed modelonnx_model=onnx.load_model("/model_path")# Provide validation part of the dataset to collect statistics needed for the compression algorithmval_dataset=datasets.ImageFolder("/path",transform=transforms.Compose([transforms.ToTensor()]))dataset_loader=torch.utils.data.DataLoader(val_dataset,batch_size=1)# Step 1: Initialize transformation functioninput_name=onnx_model.graph.input[0].namedeftransform_fn(data_item):images,_=data_itemreturn{input_name:images.numpy()}# Step 2: Initialize NNCF Datasetcalibration_dataset=nncf.Dataset(dataset_loader,transform_fn)# Step 3: Run the quantization pipelinequantized_model=nncf.quantize(onnx_model,calibration_dataset)Training-Time CompressionBelow is an example of Accuracy Aware Quantization pipeline where model weights and compression parameters may be fine-tuned to achieve a higher accuracy.PyTorchimporttorchimportnncf.torch# Important - must be imported before any other external package that depends on torchfromnncfimportNNCFConfigfromnncf.torchimportcreate_compressed_model,register_default_init_args# Instantiate your uncompressed modelfromtorchvision.models.resnetimportresnet50model=resnet50()# Load a configuration file to specify compressionnncf_config=NNCFConfig.from_json("resnet50_int8.json")# Provide data loaders for compression algorithm initialization, if necessaryimporttorchvision.datasetsasdatasetsrepresentative_dataset=datasets.ImageFolder("/path",transform=transforms.Compose([transforms.ToTensor()]))init_loader=torch.utils.data.DataLoader(representative_dataset)nncf_config=register_default_init_args(nncf_config,init_loader)# Apply the specified compression algorithms to the modelcompression_ctrl,compressed_model=create_compressed_model(model,nncf_config)# Now use compressed_model as a usual torch.nn.Module# to fine-tune compression parameters along with the model weights# ... the rest of the usual PyTorch-powered training pipeline# Export to ONNX or .pth when done fine-tuningcompression_ctrl.export_model("compressed_model.onnx")torch.save(compressed_model.state_dict(),"compressed_model.pth")NOTE (PyTorch): Due to the way NNCF works within the PyTorch backend,import nncfmust be done before any other import oftorchin your packageorin third-party packages that your code utilizes, otherwise the compression may be applied incompletely.TensorflowimporttensorflowastffromnncfimportNNCFConfigfromnncf.tensorflowimportcreate_compressed_model,register_default_init_args# Instantiate your uncompressed modelfromtensorflow.keras.applicationsimportResNet50model=ResNet50()# Load a configuration file to specify compressionnncf_config=NNCFConfig.from_json("resnet50_int8.json")# Provide dataset for compression algorithm initializationrepresentative_dataset=tf.data.Dataset.list_files("/path/*.jpeg")nncf_config=register_default_init_args(nncf_config,representative_dataset,batch_size=1)# Apply the specified compression algorithms to the modelcompression_ctrl,compressed_model=create_compressed_model(model,nncf_config)# Now use compressed_model as a usual Keras model# to fine-tune compression parameters along with the model weights# ... the rest of the usual TensorFlow-powered training pipeline# Export to Frozen Graph, TensorFlow SavedModel or .h5  when done fine-tuningcompression_ctrl.export_model("compressed_model.pb",save_format="frozen_graph")For a more detailed description of NNCF usage in your training code, seethis tutorial.Demos, Tutorials and SamplesFor a quicker start with NNCF-powered compression, try sample notebooks and scripts presented below.Jupyter* Notebook Tutorials and DemosA collection of ready-to-run Jupyter* notebooks tutorials and demos are available to explain and display NNCF compression algorithms for optimizing models for inference with the OpenVINO Toolkit.Notebook Tutorial NameCompression AlgorithmBackendDomainBERT QuantizationPost-Training QuantizationOpenVINONLPMONAI Segmentation Model QuantizationPost-Training QuantizationOpenVINOSegmentationPyTorch Model QuantizationPost-Training QuantizationPyTorchImage ClassificationTensorFlow Model QuantizationPost-Training QuantizationTensorflowImage ClassificationMigrating from POT to NNCFPost-Training QuantizationOpenVINOObject detectionQuantization with Accuracy ControlPost-Training Quantization with Accuracy ControlOpenVINOSpeech-to-Text,Object DetectionTensorFlow Training-Time CompressionTraining-Time CompressionTensorflowImage ClassificationJoint Pruning, Quantization and Distillation for BERTJoint Pruning, Quantization and DistillationOpenVINONLPBelow is a list of notebooks demonstrating OpenVINO conversion and inference together with NNCF compression for models from various domains.Demo ModelCompression AlgorithmBackendDomainYOLOv8Post-Training QuantizationOpenVINOObject Detection,KeyPoint Detection,Instance SegmentationYOLOv7Post-Training QuantizationOpenVINOObject DetectionSegment Anything ModelPost-Training QuantizationOpenVINOPanoptic SegmentationOneFormerPost-Training QuantizationOpenVINOPanoptic SegmentationInstructPix2PixPost-Training QuantizationOpenVINOImage-to-ImageCLIPPost-Training QuantizationOpenVINOImage-to-TextBLIPPost-Training QuantizationOpenVINOImage-to-TextLatent Consistency ModelPost-Training QuantizationOpenVINOText-to-ImageWürstchenPost-Training QuantizationOpenVINOText-to-ImageControlNet QR Code MonsterPost-Training QuantizationOpenVINOText-to-ImageSDXL-turboPost-Training QuantizationOpenVINOText-to-Image,Image-to-ImageDeepFloyd IFPost-Training Quantization,Weight CompressionOpenVINOText-to-Image,Image-to-ImageImageBindPost-Training QuantizationOpenVINOMulti-Modal RetrievalDistil-WhisperPost-Training QuantizationOpenVINOSpeech-to-TextWhisperPost-Training QuantizationOpenVINOSpeech-to-TextMMS Speech RecognitionPost-Training QuantizationOpenVINOSpeech-to-TextGrammar Error CorrectionPost-Training QuantizationOpenVINONLP, Grammar CorrectionLLM Instruction FollowingWeight CompressionOpenVINONLP, Instruction FollowingDolly 2.0Weight CompressionOpenVINONLP, Instruction FollowingLLM Chat BotsWeight CompressionOpenVINONLP, Chat BotPost-Training Quantization ExamplesCompact scripts demonstrating quantization and corresponding inference speed boost:Example NameCompression AlgorithmBackendDomainOpenVINO MobileNetV2Post-Training QuantizationOpenVINOImage ClassificationOpenVINO YOLOv8Post-Training QuantizationOpenVINOObject DetectionOpenVINO YOLOv8 QwAСPost-Training Quantization with Accuracy ControlOpenVINOObject DetectionOpenVINO Anomaly ClassificationPost-Training Quantization with Accuracy ControlOpenVINOAnomaly ClassificationPyTorch MobileNetV2Post-Training QuantizationPyTorchImage ClassificationPyTorch SSDPost-Training QuantizationPyTorchObject DetectionTensorFlow MobileNetV2Post-Training QuantizationTensorFlowImage ClassificationONNX MobileNetV2Post-Training QuantizationONNXImage ClassificationTraining-Time Compression ExamplesThese examples provide full pipelines including compression, training and inference for classification, detection and segmentation tasks.Example NameCompression AlgorithmBackendDomainPyTorch Image ClassificationTraining-Time CompressionPyTorchImage ClassificationPyTorch Object DetectionTraining-Time CompressionPyTorchObject DetectionPyTorch Semantic SegmentationTraining-Time CompressionPyTorchSemantic SegmentationTensorFlow Image ClassificationTraining-Time CompressionTensorFlowImage ClassificationTensorFlow Object DetectionTraining-Time CompressionTensorFlowObject DetectionTensorFlow Instance SegmentationTraining-Time CompressionTensorFlowInstance SegmentationThird-party repository integrationNNCF may be straightforwardly integrated into training/evaluation pipelines of third-party repositories.Used byOpenVINO Training ExtensionsNNCF is integrated into OpenVINO Training Extensions as model optimization backend. So you can train, optimize and export new models based on the available model templates as well as run exported models with OpenVINO.HuggingFace Optimum IntelNNCF is used as a compression backend within the renownedtransformersrepository in HuggingFace Optimum Intel.Git patches for third-party repositorySeethird_party_integrationfor examples of code modifications (Git patches and base commit IDs are provided) that are necessary to integrate NNCF into the following repositories:huggingface-transformersNOTE: this patch is deprecated and will be removed from NNCF repository in future releases.Installation GuideFor detailed installation instructions please refer to theInstallationpage.NNCF can be installed as a regular PyPI package via pip:pipinstallnncfIf you want to install both NNCF and the supported PyTorch version in one line, you can do this by simply running:pipinstallnncf[torch]Other viable options besides[torch]are[tf],[onnx]and[openvino].NNCF is also available viaconda:condainstall-cconda-forgenncfYou may also use one of the Dockerfiles in thedockerdirectory to build an image with an environment already set up and ready for running NNCFsample scripts.System requirementsUbuntu* 18.04 or later (64-bit)Python* 3.7 or laterSupported frameworks:PyTorch* >=2.0, <2.2TensorFlow* >=2.8.4, <=2.12.1ONNX* ~=1.13.1OpenVINO* >=2022.3.0This repository is tested on Python* 3.8.10, PyTorch* 2.1.2 (NVidia CUDA* Toolkit 11.8) and TensorFlow* 2.12.1 (NVidia CUDA* Toolkit 11.8).NNCF Compressed Model ZooList of models and compression results for them can be found at ourModel Zoo page.Citing@article{kozlov2020neural,
    title =   {Neural network compression framework for fast model inference},
    author =  {Kozlov, Alexander and Lazarevich, Ivan and Shamporov, Vasily and Lyalyushkin, Nikolay and Gorbachev, Yury},
    journal = {arXiv preprint arXiv:2002.08679},
    year =    {2020}
}Contributing GuideRefer to theCONTRIBUTING.mdfile for guidelines on contributions to the NNCF repository.Useful linksDocumentationExample scripts (model objects available through links in respective README.md files):PyTorchTensorFlowFAQNotebooksHuggingFace Optimum IntelOpenVINO Model Optimization GuideTelemetryNNCF as part of the OpenVINO™ toolkit collects anonymous usage data for the purpose of improving OpenVINO™ tools.
You can opt-out at any time by running the following command in the Python environment where you have NNCF installed:opt_in_out --opt_outMore information is available athttps://docs.openvino.ai/latest/openvino_docs_telemetry_information.html.