Package: neural-compressor
Intel® Neural CompressorAn open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, ONNX Runtime, and MXNet)Architecture|Workflow|LLMs Recipes|Results|DocumentationsIntel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks such asTensorFlow,PyTorch,ONNX Runtime, andMXNet,
as well as Intel extensions such asIntel Extension for TensorFlowandIntel Extension for PyTorch.
In particular, the tool provides the key features, typical examples, and open collaborations as below:Support a wide range of Intel hardware such asIntel Xeon Scalable Processors,Intel Xeon CPU Max Series,Intel Data Center GPU Flex Series, andIntel Data Center GPU Max Serieswith extensive testing; support AMD CPU, ARM CPU, and NVidia GPU through ONNX Runtime with limited testingValidate popular LLMs such asLLama2,Falcon,GPT-J,Bloom,OPT, and more than 10,000 broad models such asStable Diffusion,BERT-Large, andResNet50from popular model hubs such asHugging Face,Torch Vision, andONNX Model Zoo, by leveraging zero-code optimization solutionNeural Coderand automaticaccuracy-drivenquantization strategiesCollaborate with cloud marketplaces such asGoogle Cloud Platform,Amazon Web Services, andAzure, software platforms such asAlibaba Cloud,Tencent TACOandMicrosoft Olive, and open AI ecosystem such asHugging Face,PyTorch,ONNX,ONNX Runtime, andLightning AIInstallationInstall from pypipipinstallneural-compressorNote:
More installation methods can be found atInstallation Guide. Please check out ourFAQfor more details.Getting StartedQuantization with Python API# Install Intel Neural Compressor and TensorFlowpipinstallneural-compressor
pipinstalltensorflow# Prepare fp32 modelwgethttps://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/mobilenet_v1_1.0_224_frozen.pbfromneural_compressor.dataimportDataLoader,Datasetsfromneural_compressor.configimportPostTrainingQuantConfigdataset=Datasets("tensorflow")["dummy"](shape=(1,224,224,3))dataloader=DataLoader(framework="tensorflow",dataset=dataset)fromneural_compressor.quantizationimportfitq_model=fit(model="./mobilenet_v1_1.0_224_frozen.pb",conf=PostTrainingQuantConfig(),calib_dataloader=dataloader,)DocumentationOverviewArchitectureWorkflowAPIsLLMs RecipesExamplesPython-based APIsQuantizationAdvanced Mixed PrecisionPruning (Sparsity)DistillationOrchestrationBenchmarkingDistributed CompressionModel ExportNeural Coder (Zero-code Optimization)LauncherJupyterLab ExtensionVisual Studio Code ExtensionSupported MatrixAdvanced TopicsAdaptorStrategyDistillation for QuantizationSmoothQuantWeight-Only Quantization (INT8/INT4/FP4/NF4)FP8 QuantizationLayer-Wise QuantizationInnovations for ProductivityNeural InsightsNeural SolutionNote:
More documentations can be found atUser Guide.Selected Publications/EventsBlog by Intel:Effective Weight-Only Quantization for Large Language Models with Intel® Neural Compressor(Oct 2023)EMNLP'2023 (Under Review):TEQ: Trainable Equivalent Transformation for Quantization of LLMs(Sep 2023)arXiv:Efficient Post-training Quantization with FP8 Formats(Sep 2023)arXiv:Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs(Sep 2023)NeurIPS'2022:Fast Distilbert on CPUs(Oct 2022)NeurIPS'2022:QuaLA-MiniLM: a Quantized Length Adaptive MiniLM(Oct 2022)Note:
ViewFull Publication List.Additional ContentRelease InformationContribution GuidelinesLegal InformationSecurity PolicyCommunicationGitHub Issues: mainly for bug reports, new feature requests, question asking, etc.Email: welcome to raise any interesting research ideas on model compression techniques by email for collaborations.Discord Channel: join the discord channel for more flexible technical discussion.WeChat group: scan the QA code to join the technical discussion.