Package: neural-speed
Neural SpeedNeural Speed is an innovation library designed to provide the efficient inference of large language models (LLMs) on Intel platforms through the state-of-the-art (SOTA) low-bit quantization and sparsity powered byIntel Neural Compressorandllama.cpp. Highlights of this project:Support LLAMA, LLAMA2, NeuralChat series, GPT-J, GPT-NEOX, Dolly-v2, MPT, Falcon, BLOOM, OPT, ChatGLM, ChatGLM2, Baichuan, Baichuan2, Qwen, Mistral, Whisper, CodeLlama, MagicCoder and StarCoderHighly optimized low precision kernels, utilize AMX, VNNI, AVX512F, AVX_VNNI and AVX2 instruction setUp to 40x compared withllama.cpp, performance details:blogNeurIPS' 2023:Efficient LLM Inference on CPUsSupport 4bits and 8bits quantizationTensor Parallelism across sockets/nodes:tensor_parallelism.mdNeural Speed is under active development so APIs are subject to change.InstallationBuild Python package (Recommended way)pipinstall-rrequirements.txt
pipinstall.Note: Please make sure GCC version is higher than GCC 10.Quick StartThere are two approaches for utilizing the Neural Speed: 1. Transformer-like usage, you need to installITREX(intel extension for transformers)2. llama.cpp-like usage1. Transformer-like usagePytorch format HF modelfromtransformersimportAutoTokenizer,TextStreamerfromintel_extension_for_transformers.transformersimportAutoModelForCausalLMmodel_name="Intel/neural-chat-7b-v3-1"# Hugging Face model_id or local modelprompt="Once upon a time, there existed a little girl,"tokenizer=AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)inputs=tokenizer(prompt,return_tensors="pt").input_idsstreamer=TextStreamer(tokenizer)model=AutoModelForCausalLM.from_pretrained(model_name,load_in_4bit=True)outputs=model.generate(inputs,streamer=streamer,max_new_tokens=300)GGUF format HF modelfromtransformersimportAutoTokenizer,TextStreamerfromintel_extension_for_transformers.transformersimportAutoModelForCausalLM# Specify the GGUF repo on the Hugginfacemodel_name="TheBloke/Llama-2-7B-Chat-GGUF"# Download the the specific gguf model file from the above repomodel_file="llama-2-7b-chat.Q4_0.gguf"# make sure you are granted to access this model on the Huggingface.tokenizer_name="meta-llama/Llama-2-7b-chat-hf"prompt="Once upon a time"tokenizer=AutoTokenizer.from_pretrained(tokenizer_name,trust_remote_code=True)inputs=tokenizer(prompt,return_tensors="pt").input_idsstreamer=TextStreamer(tokenizer)model=AutoModelForCausalLM.from_pretrained(model_name,model_file=model_file)outputs=model.generate(inputs,streamer=streamer,max_new_tokens=300)Please referthis linkto check supported models.If you want to useTransformer-based APIinITREX(Intel extension for transformers). Please refer toITREX Installation Page.2. llama.cpp-like usage:One-click Python scriptsRun LLM with one-click python script including conversion, quantization and inference.python scripts/run.py model-path --weight_dtype int4 -p "She opened the door and see"Quantize and Inference Step By StepNeural Speed supports 1. GGUF models generated byllama.cpp2. GGUF models from HuggingFace 3. PyTorch model from HuggingFace, but quantized by Neural Speed
Neural Speed offers the scripts: 1) convert and quantize, and 2) inference for converting the model by yourself.
If the GGUF model is from HuggingFace or generated by llama.cpp, you can inference it directly.1. Convert and Quantize LLMconverting the model by following the below steps:# convert the model directly use model id in Hugging Face. (recommended)pythonscripts/convert.py--outtypef32--outfilene-f32.binEleutherAI/gpt-j-6b2. InferenceLinux and WSLOMP_NUM_THREADS=<physic_cores>numactl-m0-C0-<physic_cores-1>pythonscripts/inference.py--model_namellama-mne-q4_j.bin-c512-b1024-n256-t<physic_cores>--color-p"She opened the door and see"Windowspythonscripts/inference.py--model_namellama-mne-q4_j.bin-c512-b1024-n256-t<physic_cores|P-cores>--color-p"She opened the door and see"For details please refer toAdvanced Usage.Supported HardwareHardwareOptimizationIntel Xeon Scalable Processors✔Intel Xeon CPU Max Series✔Intel Core Processors✔Supported ModelsLLAMA, LLAMA2, NeuralChat series, GPT-J, GPT-NEOX, Dolly-v2, MPT, Falcon, BLOOM, OPT, ChatGLM, ChatGLM2, Baichuan, Baichuan2, Qwen, Mistral, Whisper, CodeLlama, MagicCoder and StarCoder.Neural Speed also supports GGUF models generated byllama.cpp, you need to download the model and use llama.cpp to create it. Validated models:llama2-7b-chat-hf,falcon-7b,falcon-40b,mpt-7b,mpt-40bandbloom-7b1.Please check more validated GGUF models from HuggingFace inlist.Advanced Usage1. Quantization and infereneceMore parameters in llama.cpp-like usage:Advanced Usage.2. Tensor Parallelism cross nodes/socketsWe support tensor parallelism strategy for distributed inference/training on multi-node and multi-socket. You can refer totensor_parallelism.mdto enable this feature.3. Custom Stopping CriteriaYou can customize the stopping criteria according to your own needs by processing the input_ids to determine if text generation needs to be stopped.
Here is the document of Custom Stopping Criteria:simple example with minimum generation length of 80 tokens4. Verbose ModeEnable verbose mode and control tracing information using theNEURAL_SPEED_VERBOSEenvironment variable.Available modes:0: Print all tracing information. Comprehensive output, including: evaluation time and operator profiling. (need to setNS_PROFILINGto ON and recompile)1: Print evaluation time. Time taken for each evaluation.2: Profile individual operator. Identify performance bottleneck within the model. (need to setNS_PROFILINGto ON and recompile)Enable New ModelYou can consider adding your own models, please follow the document:graph developer document.