Package: optimum
Hugging Face Optimum🤗 Optimum is an extension of 🤗 Transformers and Diffusers, providing a set of optimization tools enabling maximum efficiency to train and run models on targeted hardware, while keeping things easy to use.Installation🤗 Optimum can be installed usingpipas follows:python-mpipinstalloptimumIf you'd like to use the accelerator-specific features of 🤗 Optimum, you can install the required dependencies according to the table below:AcceleratorInstallationONNX Runtimepip install --upgrade-strategy eager optimum[onnxruntime]Intel Neural Compressorpip install --upgrade-strategy eager optimum[neural-compressor]OpenVINOpip install --upgrade-strategy eager optimum[openvino,nncf]AMD Instinct GPUs and Ryzen AI NPUpip install --upgrade-strategy eager optimum[amd]Habana Gaudi Processor (HPU)pip install --upgrade-strategy eager optimum[habana]FuriosaAIpip install --upgrade-strategy eager optimum[furiosa]The--upgrade-strategy eageroption is needed to ensure the different packages are upgraded to the latest possible version.To install from source:python-mpipinstallgit+https://github.com/huggingface/optimum.gitFor the accelerator-specific features, appendoptimum[accelerator_type]to the above command:python-mpipinstalloptimum[onnxruntime]@git+https://github.com/huggingface/optimum.gitAccelerated Inference🤗 Optimum provides multiple tools to export and run optimized models on various ecosystems:ONNX/ONNX RuntimeTensorFlow LiteOpenVINOHabana first-gen Gaudi / Gaudi2, more detailshereTheexportand optimizations can be done both programmatically and with a command line.Features summaryFeaturesONNX RuntimeNeural CompressorOpenVINOTensorFlow LiteGraph optimization:heavy_check_mark:N/A:heavy_check_mark:N/APost-training dynamic quantization:heavy_check_mark::heavy_check_mark:N/A:heavy_check_mark:Post-training static quantization:heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:Quantization Aware Training (QAT)N/A:heavy_check_mark::heavy_check_mark:N/AFP16 (half precision):heavy_check_mark:N/A:heavy_check_mark::heavy_check_mark:PruningN/A:heavy_check_mark::heavy_check_mark:N/AKnowledge DistillationN/A:heavy_check_mark::heavy_check_mark:N/AOpenVINOBefore you begin, make sure you have all the necessary libraries installed :pipinstall--upgrade-strategyeageroptimum[openvino,nncf]It is possible to export 🤗 Transformers and Diffusers models to the OpenVINO format easily:optimum-cliexportopenvino--modeldistilbert-base-uncased-finetuned-sst-2-englishdistilbert_sst2_ovIf you add--int8, the weights will be quantized to INT8. Static quantization can also be applied on the activations usingNNCF, more information can be found in thedocumentation.To load a model and run inference with OpenVINO Runtime, you can just replace yourAutoModelForXxxclass with the correspondingOVModelForXxxclass. To load a PyTorch checkpoint and convert it to the OpenVINO format on-the-fly, you can setexport=Truewhen loading your model.- from transformers import AutoModelForSequenceClassification+ from optimum.intel import OVModelForSequenceClassificationfrom transformers import AutoTokenizer, pipelinemodel_id = "distilbert-base-uncased-finetuned-sst-2-english"tokenizer = AutoTokenizer.from_pretrained(model_id)- model = AutoModelForSequenceClassification.from_pretrained(model_id)+ model = OVModelForSequenceClassification.from_pretrained("distilbert_sst2_ov")classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)results = classifier("He's a dreadful magician.")You can find more examples in thedocumentationand in theexamples.Neural CompressorBefore you begin, make sure you have all the necessary libraries installed :pipinstall--upgrade-strategyeageroptimum[neural-compressor]Dynamic quantization can be applied on your model:optimum-cliincquantize--modeldistilbert-base-cased-distilled-squad--output./quantized_distilbertTo load a model quantized with Intel Neural Compressor, hosted locally or on the 🤗 hub, you can do as follows :fromoptimum.intelimportINCModelForSequenceClassificationmodel_id="Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-dynamic"model=INCModelForSequenceClassification.from_pretrained(model_id)You can find more examples in thedocumentationand in theexamples.ONNX + ONNX RuntimeBefore you begin, make sure you have all the necessary libraries installed :pipinstalloptimum[exporters,onnxruntime]It is possible to export 🤗 Transformers and Diffusers models to theONNXformat and perform graph optimization as well as quantization easily:optimum-cli export onnx -m deepset/roberta-base-squad2 --optimize O2 roberta_base_qa_onnxThe model can then be quantized usingonnxruntime:optimum-clionnxruntimequantize\--avx512\--onnx_modelroberta_base_qa_onnx\-oquantized_roberta_base_qa_onnxThese commands will exportdeepset/roberta-base-squad2and performO2 graph optimizationon the exported model, and finally quantize it with theavx512 configuration.For more information on the ONNX export, please check thedocumentation.Run the exported model using ONNX RuntimeOnce the model is exported to the ONNX format, we provide Python classes enabling you to run the exported ONNX model in a seemless manner usingONNX Runtimein the backend:- from transformers import AutoModelForQuestionAnswering+ from optimum.onnxruntime import ORTModelForQuestionAnsweringfrom transformers import AutoTokenizer, pipelinemodel_id = "deepset/roberta-base-squad2"tokenizer = AutoTokenizer.from_pretrained(model_id)- model = AutoModelForQuestionAnswering.from_pretrained(model_id)+ model = ORTModelForQuestionAnswering.from_pretrained("roberta_base_qa_onnx")qa_pipe = pipeline("question-answering", model=model, tokenizer=tokenizer)question = "What's Optimum?"context = "Optimum is an awesome library everyone should use!"results = qa_pipe(question=question, context=context)More details on how to run ONNX models withORTModelForXXXclasseshere.TensorFlow LiteBefore you begin, make sure you have all the necessary libraries installed :pipinstalloptimum[exporters-tf]Just as for ONNX, it is possible to export models toTensorFlow Liteand quantize them:optimum-cli export tflite \
  -m deepset/roberta-base-squad2 \
  --sequence_length 384  \
  --quantize int8-dynamic roberta_tflite_modelAccelerated training🤗 Optimum provides wrappers around the original 🤗 TransformersTrainerto enable training on powerful hardware easily.
We support many providers:Habana's Gaudi processorsONNX Runtime (optimized for GPUs)HabanaBefore you begin, make sure you have all the necessary libraries installed :pipinstall--upgrade-strategyeageroptimum[habana]- from transformers import Trainer, TrainingArguments+ from optimum.habana import GaudiTrainer, GaudiTrainingArguments# Download a pretrained model from the Hubmodel = AutoModelForXxx.from_pretrained("bert-base-uncased")# Define the training arguments- training_args = TrainingArguments(+ training_args = GaudiTrainingArguments(output_dir="path/to/save/folder/",+     use_habana=True,+     use_lazy_mode=True,+     gaudi_config_name="Habana/bert-base-uncased",...)# Initialize the trainer- trainer = Trainer(+ trainer = GaudiTrainer(model=model,args=training_args,train_dataset=train_dataset,...)# Use Habana Gaudi processor for training!trainer.train()You can find more examples in thedocumentationand in theexamples.ONNX Runtime- from transformers import Trainer, TrainingArguments+ from optimum.onnxruntime import ORTTrainer, ORTTrainingArguments# Download a pretrained model from the Hubmodel = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")# Define the training arguments- training_args = TrainingArguments(+ training_args = ORTTrainingArguments(output_dir="path/to/save/folder/",optim="adamw_ort_fused",...)# Create a ONNX Runtime Trainer- trainer = Trainer(+ trainer = ORTTrainer(model=model,args=training_args,train_dataset=train_dataset,...)# Use ONNX Runtime for training!trainer.train()You can find more examples in thedocumentationand in theexamples.