Package: optimum-intel
Optimum Intel🤗 Optimum Intel is the interface between the 🤗 Transformers and Diffusers libraries and the different tools and libraries provided by Intel to accelerate end-to-end pipelines on Intel architectures.Intel Extension for PyTorchis an open-source library which provides optimizations for both eager mode and graph mode, however, compared to eager mode, graph mode in PyTorch* normally yields better performance from optimization techniques, such as operation fusion.IntelNeural Compressoris an open-source library enabling the usage of the most popular compression techniques such as quantization, pruning and knowledge distillation. It supports automatic accuracy-driven tuning strategies in order for users to easily generate quantized model. The users can easily apply static, dynamic and aware-training quantization approaches while giving an expected accuracy criteria. It also supports different weight pruning techniques enabling the creation of pruned model giving a predefined sparsity target.OpenVINOis an open-source toolkit that enables high performance inference capabilities for Intel CPUs, GPUs, and special DL inference accelerators (seethe full list of supported devices). It is supplied with a set of tools to optimize your models with compression techniques such as quantization, pruning and knowledge distillation. Optimum Intel provides a simple interface to optimize your Transformers and Diffusers models, convert them to the OpenVINO Intermediate Representation (IR) format and run inference using OpenVINO Runtime.InstallationTo install the latest release of 🤗 Optimum Intel with the corresponding required dependencies, you can usepipas follows:AcceleratorInstallationIntel Neural Compressorpip install --upgrade-strategy eager "optimum[neural-compressor]"OpenVINOpip install --upgrade-strategy eager "optimum[openvino,nncf]"Intel Extension for PyTorchpip install --upgrade-strategy eager "optimum[ipex]"The--upgrade-strategy eageroption is needed to ensureoptimum-intelis upgraded to the latest version.We recommend creating avirtual environmentand upgrading
pip withpython -m pip install --upgrade pip.Optimum Intel is a fast-moving project, and you may want to install from source with the following command:python-mpipinstallgit+https://github.com/huggingface/optimum-intel.gitor to install from source including dependencies:python-mpipinstall"optimum-intel[extras]"@git+https://github.com/huggingface/optimum-intel.gitwhereextrascan be one or more ofipex,neural-compressor,openvino,nncf.Quick tourNeural CompressorDynamic quantization can be used through the Optimum command-line interface:optimum-cliincquantize--modeldistilbert-base-cased-distilled-squad--output./quantized_distilbertNote that quantization is currently only supported for CPUs (only CPU backends are available), so we will not be utilizing GPUs / CUDA in this example.To load a quantized model hosted locally or on the 🤗 hub, you can do as follows :fromoptimum.intelimportINCModelForSequenceClassificationmodel_id="Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-dynamic"model=INCModelForSequenceClassification.from_pretrained(model_id)You can load many more quantized models hosted on the hub under the Intel organizationhere.For more details on the supported compression techniques, please refer to thedocumentation.OpenVINOBelow are the examples of how to use OpenVINO and itsNNCFframework to accelerate inference.Export:It is possible to export your model to theOpenVINOIR format with the CLI :optimum-cli export openvino --model gpt2 ov_modelYou can also apply 8-bit weight-only quantization when exporting your model : the model linear and embedding weights will be quantized to INT8, the activations will be kept in floating point precision.optimum-cli export openvino --model gpt2 --weight-format int8 ov_modelTo apply quantization on both weights and activations, you can find more information in thedocumentation.Inference:To load a model and run inference with OpenVINO Runtime, you can just replace yourAutoModelForXxxclass with the correspondingOVModelForXxxclass.- from transformers import AutoModelForSeq2SeqLM+ from optimum.intel import OVModelForSeq2SeqLMfrom transformers import AutoTokenizer, pipelinemodel_id = "echarlaix/t5-small-openvino"- model = AutoModelForSeq2SeqLM.from_pretrained(model_id)+ model = OVModelForSeq2SeqLM.from_pretrained(model_id)tokenizer = AutoTokenizer.from_pretrained(model_id)pipe = pipeline("translation_en_to_fr", model=model, tokenizer=tokenizer)results = pipe("He never went out without a book under his arm, and he often came back with two.")[{'translation_text': "Il n'est jamais sorti sans un livre sous son bras, et il est souvent revenu avec deux."}]If you want to load a PyTorch checkpoint, setexport=Trueto convert your model to the OpenVINO IR.fromoptimum.intelimportOVModelForCausalLMmodel=OVModelForCausalLM.from_pretrained("gpt2",export=True)model.save_pretrained("./ov_model")Post-training static quantization:Post-training static quantization introduces an additional calibration step where data is fed through the network in order to compute the activations quantization parameters. Here is an example on how to apply static quantization on a fine-tuned DistilBERT.fromfunctoolsimportpartialfromoptimum.intelimportOVQuantizer,OVModelForSequenceClassificationfromtransformersimportAutoTokenizer,AutoModelForSequenceClassificationmodel_id="distilbert-base-uncased-finetuned-sst-2-english"model=AutoModelForSequenceClassification.from_pretrained(model_id)tokenizer=AutoTokenizer.from_pretrained(model_id)defpreprocess_fn(examples,tokenizer):returntokenizer(examples["sentence"],padding=True,truncation=True,max_length=128)quantizer=OVQuantizer.from_pretrained(model)calibration_dataset=quantizer.get_calibration_dataset("glue",dataset_config_name="sst2",preprocess_function=partial(preprocess_fn,tokenizer=tokenizer),num_samples=100,dataset_split="train",preprocess_batch=True,)# The directory where the quantized model will be savedsave_dir="nncf_results"# Apply static quantization and save the resulting model in the OpenVINO IR formatquantizer.quantize(calibration_dataset=calibration_dataset,save_directory=save_dir)# Load the quantized modeloptimized_model=OVModelForSequenceClassification.from_pretrained(save_dir)Quantization-aware training:Quantization aware training (QAT) is applied in order to simulate the effects of quantization during training, to alleviate its effects on the model’s accuracy. Here is an example on how to fine-tune a DistilBERT model on the sst-2 task while applying quantization aware training (QAT).import evaluateimport numpy as npfrom datasets import load_datasetfrom transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, default_data_collator- from transformers import Trainer+ from optimum.intel import OVConfig, OVModelForSequenceClassification, OVTrainermodel_id = "distilbert-base-uncased-finetuned-sst-2-english"model = AutoModelForSequenceClassification.from_pretrained(model_id)tokenizer = AutoTokenizer.from_pretrained(model_id)dataset = load_dataset("glue", "sst2")dataset = dataset.map(lambda examples: tokenizer(examples["sentence"], padding=True, truncation=True, max_length=128), batched=True)metric = evaluate.load("glue", "sst2")compute_metrics = lambda p: metric.compute(predictions=np.argmax(p.predictions, axis=1), references=p.label_ids)# The directory where the quantized model will be savedsave_dir = "nncf_results"# Load the default quantization configuration detailing the quantization we wish to apply+ ov_config = OVConfig()- trainer = Trainer(+ trainer = OVTrainer(model=model,args=TrainingArguments(save_dir, num_train_epochs=1.0, do_train=True, do_eval=True),train_dataset=dataset["train"].select(range(300)),eval_dataset=dataset["validation"],compute_metrics=compute_metrics,tokenizer=tokenizer,data_collator=default_data_collator,+     ov_config=ov_config,+     task="text-classification",)train_result = trainer.train()metrics = trainer.evaluate()trainer.save_model()+ optimized_model = OVModelForSequenceClassification.from_pretrained(save_dir)You can find more examples in thedocumentation.Running the examplesCheck out theexamplesdirectory to see how 🤗 Optimum Intel can be used to optimize models and accelerate inference.Do not forget to install requirements for every example:cd <example-folder>
pip install -r requirements.txt