Package: intel-extension-for-transformers
Intel® Extension for TransformersAn Innovative Transformer-based Toolkit to Accelerate GenAI/LLM Everywhere🏭Architecture|💬NeuralChat|😃Inference on CPU|😃Inference  on GPU|💻Examples|📖Documentations🚀Latest News[2024/01] SupportedINT4 inference on Intel GPUsincluding Intel Data Center GPU Max Series (e.g., PVC) and Intel Arc A-Series (e.g., ARC). Check out theexamplesandscripts.[2024/01] DemonstratedIntel Hybrid CopilotinCES 2024 Great MindsSession "Bringing the Limitless Potential of AI Everywhere".[2023/12] SupportedQLoRA on CPUsto make fine-tuning on client CPU possible. Check out theblogandreadmefor more details.[2023/11] Releasedtop-1 7B-sized LLMNeuralChat-v3-1andDPO dataset. Check out thenice videopublished byWorldofAI.[2023/11] Published a4-bit chatbot demo(based on NeuralChat) available onIntel Hugging Face Space. Welcome to have a try! To setup the demo locally, please follow theinstructions.🏃InstallationQuick Install from Pypipipinstallintel-extension-for-transformersFor system requirements and other installation tips, please refer toInstallation Guide🌟IntroductionIntel® Extension for Transformers is an innovative toolkit designed to accelerate GenAI/LLM everywhere with the optimal performance of Transformer-based models on various Intel platforms, including Intel Gaudi2, Intel CPU, and Intel GPU. The toolkit provides the below key features and examples:Seamless user experience of model compressions on Transformer-based models by extendingHugging Face transformersAPIs and leveragingIntel® Neural CompressorAdvanced software optimizations and unique compression-aware runtime (released with NeurIPS 2022's paperFast Distilbert on CPUsandQuaLA-MiniLM: a Quantized Length Adaptive MiniLM, and NeurIPS 2021's paperPrune Once for All: Sparse Pre-Trained Language Models)Optimized Transformer-based model packages such asStable Diffusion,GPT-J-6B,GPT-NEOX,BLOOM-176B,T5,Flan-T5, and end-to-end workflows such asSetFit-based text classificationanddocument level sentiment analysis (DLSA)NeuralChat, a customizable chatbot framework to create your own chatbot within minutes by leveraging a rich set ofpluginssuch asKnowledge Retrieval,Speech Interaction,Query Caching, andSecurity Guardrail. This framework supports Intel Gaudi2/CPU/GPU.Inferenceof Large Language Model (LLM) in pure C/C++ with weight-only quantization kernels for Intel CPU and Intel GPU (TBD), supportingGPT-NEOX,LLAMA,MPT,FALCON,BLOOM-7B,OPT,ChatGLM2-6B,GPT-J-6B, andDolly-v2-3B. Support AMX, VNNI, AVX512F and AVX2 instruction set. We've boosted the performance of Intel CPUs, with a particular focus on the 4th generation Intel Xeon Scalable processor, codenamedSapphire Rapids.🔓Validated HardwareHardwareFine-TuningInferenceFullPEFT8-bit4-bitIntel Gaudi2✔✔WIP (FP8)-Intel Xeon Scalable Processors✔✔✔ (INT8, FP8)✔ (INT4, FP4, NF4)Intel Xeon CPU Max Series✔✔✔ (INT8, FP8)✔ (INT4, FP4, NF4)Intel Data Center GPU Max SeriesWIPWIPWIP (INT8)✔ (INT4)Intel Arc A-Series--WIP (INT8)✔ (INT4)Intel Core Processors-✔✔ (INT8, FP8)✔ (INT4, FP4, NF4)In the table above, "-" means not applicable or not started yet.🔓Validated SoftwareSoftwareFine-TuningInferenceFullPEFT8-bit4-bitPyTorch2.0.1+cpu,2.0.1a0 (gpu)2.0.1+cpu,2.0.1a0 (gpu)2.1.0+cpu,2.0.1a0 (gpu)2.1.0+cpu,2.0.1a0 (gpu)Intel® Extension for PyTorch2.1.0+cpu,2.0.110+xpu2.1.0+cpu,2.0.110+xpu2.1.0+cpu,2.0.110+xpu2.1.0+cpu,2.0.110+xpuTransformers4.35.2(CPU),4.31.0 (Intel GPU)4.35.2(CPU),4.31.0 (Intel GPU)4.35.2(CPU),4.31.0 (Intel GPU)4.35.2(CPU),4.31.0 (Intel GPU)Synapse AI1.13.01.13.01.13.01.13.0Gaudi2 driver1.13.0-ee32e421.13.0-ee32e421.13.0-ee32e421.13.0-ee32e42intel-level-zero-gpu1.3.26918.50-736~22.041.3.26918.50-736~22.041.3.26918.50-736~22.041.3.26918.50-736~22.04Please refer to the detailed requirements inCPU,Gaudi2,Intel GPU.🔓Validated OSUbuntu 20.04/22.04, Centos 8.🌱Getting StartedChatbotBelow is the sample code to create your chatbot. See moreexamples.Serving (OpenAI-compatible RESTful APIs)NeuralChat provides OpenAI-compatible RESTful APIs for chat, so you can use NeuralChat as a drop-in replacement for OpenAI APIs.
You can start NeuralChat server either using the Shell command or Python code.# Shell Commandneuralchat_serverstart--config_file./server/config/neuralchat.yaml# Python Codefromintel_extension_for_transformers.neural_chatimportNeuralChatServerExecutorserver_executor=NeuralChatServerExecutor()server_executor(config_file="./server/config/neuralchat.yaml",log_file="./neuralchat.log")NeuralChat service can be accessible throughOpenAI client library,curlcommands, andrequestslibrary. See more inNeuralChat.Offlinefromintel_extension_for_transformers.neural_chatimportbuild_chatbotchatbot=build_chatbot()response=chatbot.predict("Tell me about Intel Xeon Scalable Processors.")Transformers-based extension APIsBelow is the sample code to use the extended Transformers APIs. See moreexamples.INT4 Inference (CPU)fromtransformersimportAutoTokenizerfromintel_extension_for_transformers.transformersimportAutoModelForCausalLMmodel_name="Intel/neural-chat-7b-v3-1"prompt="Once upon a time, there existed a little girl,"tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)inputs=tokenizer(prompt,return_tensors="pt").input_idsmodel=AutoModelForCausalLM.from_pretrained(model_name,load_in_4bit=True)outputs=model.generate(inputs)You can also load the low-bit model quantized by GPTQ/AWQ/RTN/AutoRound algorithm.fromtransformersimportAutoTokenizerfromintel_extension_for_transformers.transformersimportAutoModelForCausalLM,WeightOnlyQuantConfig# Download Hugging Face GPTQ/AWQ model or use local quantize modelmodel_name="PATH_TO_MODEL"# local path to modelwoq_config=WeightOnlyQuantConfig(use_gptq=True)# use_awq=True for AWQ; use_autoround=True for AutoRoundprompt="Once upon a time, a little girl"tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)inputs=tokenizer(prompt,return_tensors="pt").input_idsmodel=AutoModelForCausalLM.from_pretrained(model_name,quantization_config=woq_config,trust_remote_code=True)outputs=model.generate(inputs)INT4 Inference (GPU)importintel_extension_for_pytorchasipexfromintel_extension_for_transformers.transformers.modelingimportAutoModelForCausalLMfromtransformersimportAutoTokenizerdevice_map="xpu"model_name="Qwen/Qwen-7B"tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)prompt="Once upon a time, there existed a little girl,"inputs=tokenizer(prompt,return_tensors="pt").input_ids.to(device_map)model=AutoModelForCausalLM.from_pretrained(model_name,trust_remote_code=True,device_map=device_map,load_in_4bit=True)model=ipex.optimize_transformers(model,inplace=True,dtype=torch.float16,woq=True,device=device_map)output=model.generate(inputs)Note: Please refer to theexampleandscriptfor more details.Langchain-based extension APIsBelow is the sample code to use the extended Langchain APIs. See moreexamples.fromlangchain_community.llms.huggingface_pipelineimportHuggingFacePipelinefromlangchain.chainsimportRetrievalQAfromlangchain_core.vectorstoresimportVectorStoreRetrieverfromintel_extension_for_transformers.langchain.vectorstoresimportChromaretriever=VectorStoreRetriever(vectorstore=Chroma(...))retrievalQA=RetrievalQA.from_llm(llm=HuggingFacePipeline(...),retriever=retriever)🎯Validated  ModelsYou can access the validated models, accuracy and performance fromRelease dataorMedium blog.📖DocumentationOVERVIEWNeuralChatNeural SpeedNEURALCHATChatbot on Intel CPUChatbot on Intel GPUChatbot on GaudiChatbot on ClientMore NotebooksNEURAL SPEEDNeural SpeedStreaming LLMLow Precision KernelsTensor ParallelismLLM COMPRESSIONSmoothQuant (INT8)Weight-only Quantization (INT4/FP4/NF4/INT8)QLoRA on CPUGENERAL COMPRESSIONQuantizationPruningDistillationOrchestrationNeural Architecture SearchExportMetricsObjectivesPipelineLength AdaptiveEarly ExitData AugmentationTUTORIALS & RESULTSTutorialsLLM ListGeneral Model ListModel Performance🙌DemoLLM Infinite Inference (up to 4M tokens)https://github.com/intel/intel-extension-for-transformers/assets/109187816/1698dcda-c9ec-4f44-b159-f4e9d67ab15bLLM QLoRA on Client CPUhttps://github.com/intel/intel-extension-for-transformers/assets/88082706/9d9bdb7e-65db-47bb-bbed-d23b151e8b31📃Selected Publications/EventsCES 2024:CES 2024 Great Minds Keynote: Bringing the Limitless Potential of AI Everywhere: Intel Hybrid Copilot demo(Jan 2024)Blog published on Medium:Connect an AI agent with your API: Intel Neural-Chat 7b LLM can replace Open AI Function Calling(Dec 2023)NeurIPS'2023 on Efficient Natural Language and Speech Processing:Efficient LLM Inference on CPUs(Nov 2023)Blog published on Hugging Face:Intel Neural-Chat 7b: Fine-Tuning on Gaudi2 for Top LLM Performance(Nov 2023)Blog published on VMware:AI without GPUs: A Technical Brief for VMware Private AI with Intel(Nov 2023)ViewFull Publication ListAdditional ContentRelease InformationContribution GuidelinesLegal InformationSecurity PolicyApache LicenseAcknowledgementsExcellent open-source projects:bitsandbytes,FastChat,fastRAG,ggml,gptq,llama.cpp,lm-evauation-harness,peft,trl,streamingllmand many others.Thanks to all thecontributors.💁CollaborationsWelcome to raise any interesting ideas on model compression techniques and LLM-based chatbot development! Feel free to reachus, and we look forward to our collaborations on Intel Extension for Transformers!