Package: optimum-amd
Optimum-AMD🤗 Optimum-AMD is an extension to Hugging Face libraries enabling performance optimizations forROCm for AMD GPUsandRyzen AI for AMDNPU accelerator.InstallOptimum-AMD library can be installed through pip:pipinstall--upgrade-strategyeageroptimum[amd]Installation is possible from source as well:gitclonehttps://github.com/huggingface/optimum-amd.gitcdoptimum-amd
pipinstall-e.ROCm support for AMD GPUsHugging Face libraries natively support AMD GPUs throughPyTorch for ROCmwith zero code change.🤗 Transformers natively supportsFlash Attention 2,GPTQ quantizationwith ROCm.🤗 Text Generation Inferencelibrary for LLM deployment has native ROCm support, with Flash Attention 2,Paged Attention, fused positional encoding & layer norm kernels support.Find out more about these integrations in the documentation!In the future, Optimum-AMD may host more ROCm-specific optimizations.How to use it: Text Generation InferenceText Generation Inferencelibrary for LLM deployment supports AMD Instinct MI210/MI250 GPUs. Deployment can be done as follow:InstallROCm5.7to the host machineExample LLM server setup: launch a Falcon-7b model server on the ROCm-enabled docker.model=tiiuae/falcon-7b-instructvolume=$PWD/data# share a volume with the Docker container to avoid downloading weights every rundockerrun--cap-add=SYS_PTRACE--security-optseccomp=unconfined--device=/dev/kfd--device=/dev/dri--group-addvideo--ipc=host--shm-size1g-p8080:80-v$volume:/dataghcr.io/huggingface/text-generation-inference:1.2-rocm--model-id$modelClient setup: Open another shell and run:curl127.0.0.1:8080/generate\-XPOST\-d'{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}'\-H'Content-Type: application/json'How to use it: ONNX Runtime with ROCmOptimum ONNX Runtime integrationsupports ROCm for AMD GPUs. Usage is as follow:InstallROCm 5.7on the host machine.Use the exampleDockerfileor installonnxruntime-rocmpackage locally from source. Pip wheels are not available at the time.Run a BERT text classification ONNX model by usingROCMExecutionProvider:fromoptimum.onnxruntimeimportORTModelForSequenceClassificationfromoptimum.pipelinesimportpipelinefromtransformersimportAutoTokenizerort_model=ORTModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english",export=True,provider="ROCMExecutionProvider",)tokenizer=AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")pipe=pipeline(task="text-classification",model=ort_model,tokenizer=tokenizer,device="cuda:0")result=pipe("Both the music and visual were astounding, not to mention the actors performance.")print(result)# [{'label': 'POSITIVE', 'score': 0.9997727274894714}]Ryzen AIAMD'sRyzen™ AIfamily of laptop processors provide users with an integrated Neural Processing Unit (NPU) which offloads the host CPU and GPU from AI processing tasks. Ryzen™ AI software consists of the Vitis™ AI execution provider (EP) for ONNX Runtime combined with quantization tools and a pre-optimized model zoo. All of this is made possible based on Ryzen™ AI technology built on AMD XDNA™ architecture, purpose-built to run AI workloads efficiently and locally, offering a host of benefits for the developer innovating the next groundbreaking AI app.Optimum-AMD provides easy interface for loading and inference of Hugging Face models on Ryzen AI accelerator.Ryzen AI Environment setupA Ryzen AI environment needs to be enabled to use this library. Please refer to Ryzen AI'sInstallationandRuntime Setup.How to use it?Quantize the ONNX model with Optimum or using the RyzenAI quantization toolsFor more information on quantization refer toModel Quantizationguide.Load model with Ryzen AI classTo load a model and run inference with RyzenAI, you can just replace yourAutoModelForXxxclass with the correspondingRyzenAIModelForXxxclass.import requests
from PIL import Image- from transformers import AutoModelForImageClassification+ from optimum.amd.ryzenai import RyzenAIModelForImageClassificationfrom transformers import AutoFeatureExtractor, pipeline

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

model_id = <path of the model>- model = AutoModelForImageClassification.from_pretrained(model_id)+ model = RyzenAIModelForImageClassification.from_pretrained(model_id, vaip_config=<path to config file>)feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
cls_pipe = pipeline("image-classification", model=model, feature_extractor=feature_extractor)
outputs = cls_pipe(image)If you find any issue while using those, please open an issue or a pull request.