Package: optimum-habana
Optimum Habana🤗 Optimum Habana is the interface between the 🤗 Transformers and Diffusers libraries andHabana's Gaudi processor (HPU).
It provides a set of tools enabling easy model loading, training and inference on single- and multi-HPU settings for different downstream tasks.
The list of officially validated models and tasks is availablehere. Users can try other models and tasks with only few changes.What is a Habana Processing Unit (HPU)?HPUs offer fast model training and inference as well as a great price-performance ratio.
Check outthis blog post about BERT pre-trainingandthis article benchmarking Habana Gaudi2 versus Nvidia A100 GPUsfor concrete examples.
If you are not familiar with HPUs and would like to know more about them, we recommend you take a look atour conceptual guide.Install the library and get example scriptsOption 1: Use the latest stable releaseTo install the latest stable release of this packagepipinstall--upgrade-strategyeageroptimum[habana]The--upgrade-strategy eageroption is needed to ensureoptimum-habanais upgraded to the latest stable release.To use the example associated with the latest stable release, run:git clone https://github.com/huggingface/optimum-habana
cd optimum-habana && git checkout v1.10.4withv1.10.4the version number of this release.Option 2: Use the latest main branch under developmentOptimum Habana is a fast-moving project, and you may want to install it from source and get the latest scripts :pipinstallgit+https://github.com/huggingface/optimum-habana.git
gitclonehttps://github.com/huggingface/optimum-habanaInstall dependenciesTo use DeepSpeed on HPUs, you also need to run the following command:pipinstallgit+https://github.com/HabanaAI/DeepSpeed.git@1.14.0To install the requirements for every example:cd<example-folder>
pipinstall-rrequirements.txtHow to use it?Quick Start🤗 Optimum Habana was designed with one goal in mind:to make training and inference straightforward for any 🤗 Transformers and 🤗 Diffusers user while leveraging the complete power of Gaudi processors.Transformers InterfaceThere are two main classes one needs to know:GaudiTrainer: the trainer class that takes care of compiling and distributing the model to run on HPUs, and performing training and evaluation.GaudiConfig: the class that enables to configure Habana Mixed Precision and to decide whether optimized operators and optimizers should be used or not.TheGaudiTraineris very similar to the🤗 Transformers Trainer, and adapting a script using the Trainer to make it work with Gaudi will mostly consist in simply swapping theTrainerclass for theGaudiTrainerone.
That's how most of theexample scriptswere adapted from theiroriginal counterparts.Here is an example:- from transformers import Trainer, TrainingArguments+ from optimum.habana import GaudiConfig, GaudiTrainer, GaudiTrainingArguments- training_args = TrainingArguments(+ training_args = GaudiTrainingArguments(# training arguments...+ use_habana=True,+ use_lazy_mode=True,  # whether to use lazy or eager mode+ gaudi_config_name=path_to_gaudi_config,)

# A lot of code here

# Initialize our Trainer- trainer = Trainer(+ trainer = GaudiTrainer(model=model,args=training_args,  # Original training arguments.train_dataset=train_dataset if training_args.do_train else None,eval_dataset=eval_dataset if training_args.do_eval else None,compute_metrics=compute_metrics,tokenizer=tokenizer,data_collator=data_collator,
)wheregaudi_config_nameis the name of a model from theHub(Gaudi configurations are stored in model repositories) or a path to a local Gaudi configuration file (you can seeherehow to write your own).Diffusers InterfaceYou can generate images from prompts using Stable Diffusion on Gaudi using theGaudiStableDiffusionPipelineclass and the [GaudiDDIMScheduler] which have been both optimized for HPUs. Here is how to use them and the differences with the 🤗 Diffusers library:- from diffusers import DDIMScheduler, StableDiffusionPipeline+ from optimum.habana.diffusers import GaudiDDIMScheduler, GaudiStableDiffusionPipelinemodel_name = "runwayml/stable-diffusion-v1-5"- scheduler = DDIMScheduler.from_pretrained(model_name, subfolder="scheduler")+ scheduler = GaudiDDIMScheduler.from_pretrained(model_name, subfolder="scheduler")- pipeline = StableDiffusionPipeline.from_pretrained(+ pipeline = GaudiStableDiffusionPipeline.from_pretrained(model_name,scheduler=scheduler,+   use_habana=True,+   use_hpu_graphs=True,+   gaudi_config="Habana/stable-diffusion",)

outputs = generator(["An image of a squirrel in Picasso style"],num_images_per_prompt=16,+   batch_size=4,)DocumentationCheck outthe documentation of Optimum Habanafor more advanced usage.Validated ModelsThe following model architectures, tasks and device distributions have been validated for 🤗 Optimum Habana:In the tables below, :heavy_check_mark: means single-card, multi-card and DeepSpeed have all been validated.Transformers:ArchitectureTrainingInferenceTasksBERT:heavy_check_mark::heavy_check_mark:text classificationquestion answeringlanguage modelingRoBERTa:heavy_check_mark::heavy_check_mark:question answeringlanguage modelingALBERT:heavy_check_mark::heavy_check_mark:question answeringlanguage modelingDistilBERT:heavy_check_mark::heavy_check_mark:question answeringlanguage modelingGPT2:heavy_check_mark::heavy_check_mark:language modelingtext generationBLOOM(Z)DeepSpeedtext generationStarCoderSingle cardtext generationGPT-JDeepSpeedSingle cardDeepSpeedlanguage modelingtext generationGPT-NeoXDeepSpeedDeepSpeedlanguage modelingtext generationOPTDeepSpeedtext generationLlama 2 / CodeLlamaDeepSpeedLoRA:heavy_check_mark:language modelingtext generationStableLMSingle cardtext generationFalconLoRA:heavy_check_mark:text generationCodeGenSingle cardtext generationMPTSingle cardtext generationMistralSingle cardtext generationT5 / Flan T5:heavy_check_mark::heavy_check_mark:summarizationtranslationquestion answeringBARTSingle cardsummarizationtranslationquestion answeringViT:heavy_check_mark::heavy_check_mark:image classificationSwin:heavy_check_mark::heavy_check_mark:image classificationWav2Vec2:heavy_check_mark::heavy_check_mark:audio classificationspeech recognitionCLIP:heavy_check_mark::heavy_check_mark:contrastive image-text trainingBridgeTower:heavy_check_mark::heavy_check_mark:contrastive image-text trainingESMFoldSingle cardprotein foldingDiffusers:ArchitectureTrainingInferenceTasksStable DiffusionSingle cardtext-to-image generationLDM3DSingle cardtext-to-image generationTRL:ArchitectureTrainingInferenceTasksLlama 2Multi cardDPO PipelineOther models and tasks supported by the 🤗 Transformers and 🤗 Diffusers library may also work. You can refer to thissectionfor using them with 🤗 Optimum Habana. Besides,this pageexplains how to modify anyexamplefrom the 🤗 Transformers library to make it work with 🤗 Optimum Habana.If you find any issues while using those, please open an issue or a pull request.Gaudi SetupPlease refer to Habana Gaudi's officialinstallation guide.Tests should be run in a Docker container based on Habana Docker images.The current version has been validated for SynapseAI 1.14.DevelopmentCheck thecontributor guidefor instructions.