explain this page like i am a layperson 

: ""Skip to content
intel
/
neural-speed

Type / to search

Code
Issues
8
Pull requests
11
Actions
Projects
Security
Insights
Owner avatar
neural-speed
Public
intel/neural-speed
Go to file
t
Add file
Folders and files
Name		
Latest commit
Zhenzhong1
Zhenzhong1
Update convert_chatglm.py (#185)
1051182
 · 
4 days ago
History
.github
[BesTLA] New thread pool and hybrid dispatcher (#118)
3 weeks ago
bestla
mha enhance (#180)
last week
docker
add docker file and readme (#14)
3 months ago
docs
Gemma-7b&&Gemma-2b (#171)
4 days ago
neural_speed
Update convert_chatglm.py (#185)
4 days ago
scripts
[Model Enabling] Support ChatGLM3 (#182)
5 days ago
tests
Gemma-7b&&Gemma-2b (#171)
4 days ago
third_party
reorg directory
3 months ago
.clang-format
update readme path and copy hidden files (#185)
3 months ago
.clang-tidy
[Neural Speed] Cont Batching in Offline and Server (GPT-J & Beam Sear…
2 months ago
.editorconfig
update readme path and copy hidden files (#185)
3 months ago
.gitignore
fix clang-tidy issues (#123)
last month
.gitmodules
reorg directory
3 months ago
.pre-commit-config.yaml
enable pre-commit CI (#113)
last month
CMakeLists.txt
[BesTLA] New thread pool and hybrid dispatcher (#118)
3 weeks ago
CMakePresets.json
[BesTLA] New thread pool and hybrid dispatcher (#118)
3 weeks ago
CODE_OF_CONDUCT.md
add code_of_conduct, contributing agreement, and security.md file
4 months ago
CONTRIBUTING.md
add code_of_conduct, contributing agreement, and security.md file
4 months ago
LICENSE
add license file (#34)
2 months ago
README.md
Improve readme (#166)
2 weeks ago
clang-format.py
Init ns doc (#9)
3 months ago
developer_document.md
[Runtime Enhence] Extend long input tokens length (#157)
2 weeks ago
requirements.txt
Improvements Targeting Windows (#136)
last month
security.md
add code_of_conduct, contributing agreement, and security.md file
4 months ago
setup.py
Improvements Targeting Windows (#136)
last month
Repository files navigation
README
Code of conduct
Apache-2.0 license
Security
Neural Speed
Neural Speed is an innovative library designed to support the efficient inference of large language models (LLMs) on Intel platforms through the state-of-the-art (SOTA) low-bit quantization powered by Intel Neural Compressor. The work is inspired by llama.cpp and further optimized for Intel platforms with our innovations in NeurIPS' 2023

Key Features
Highly optimized low-precision kernels on CPUs with ISAs (AMX, VNNI, AVX512F, AVX_VNNI and AVX2). See details
Up to 40x performance speedup on popular LLMs compared with llama.cpp. See details
Tensor parallelism across sockets/nodes on CPUs. See details
Neural Speed is under active development so APIs are subject to change.

Supported Hardware
Hardware	Supported
Intel Xeon Scalable Processors	✔
Intel Xeon CPU Max Series	✔
Intel Core Processors	✔
Supported Models
Support almost all the LLMs in PyTorch format from Hugging Face such as Llama2, ChatGLM2, Baichuan2, Qwen, Mistral, Whisper, etc. File an issue if your favorite LLM does not work.

Support typical LLMs in GGUF format such as Llama2, Falcon, MPT, Bloom etc. More are coming. Check out the details.

Installation
Install from binary
pip install neural-speed
Build from Source
pip install -r requirements.txt
pip install .
Note: GCC requires version 10+

Quick Start (Transformer-like usage)
Install Intel Extension for Transformers to use Transformer-like APIs.

PyTorch Model from Hugging Face
from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM
model_name = "Intel/neural-chat-7b-v3-1"     # Hugging Face model_id or local model
prompt = "Once upon a time, there existed a little girl,"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
streamer = TextStreamer(tokenizer)

model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True)
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)
GGUF Model from Hugging Face
from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM

# Specify the GGUF repo on the Hugginface
model_name = "TheBloke/Llama-2-7B-Chat-GGUF"
# Download the the specific gguf model file from the above repo
model_file = "llama-2-7b-chat.Q4_0.gguf"
# make sure you are granted to access this model on the Huggingface.
tokenizer_name = "meta-llama/Llama-2-7b-chat-hf"

prompt = "Once upon a time"
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
streamer = TextStreamer(tokenizer)
model = AutoModelForCausalLM.from_pretrained(model_name, model_file = model_file)
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)
PyTorch Model from Modelscope
import sys
from modelscope import AutoTokenizer
from transformers import TextStreamer
from neural_speed import Model

model_name = "qwen/Qwen1.5-7B-Chat"     # modelscope model_id or local model
prompt = "Once upon a time, there existed a little girl,"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
streamer = TextStreamer(tokenizer)
model = Model()
model.init(model_name, weight_dtype="int4", compute_dtype="int8", model_hub="modelscope")
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)
Quick Start (llama.cpp-like usage)
Single (One-click) Step
python scripts/run.py model-path --weight_dtype int4 -p "She opened the door and see"
Multiple Steps
Convert and Quantize
# skip the step if GGUF model is from Hugging Face or generated by llama.cpp
python scripts/convert.py --outtype f32 --outfile ne-f32.bin EleutherAI/gpt-j-6b
Inference
# Linux and WSL
OMP_NUM_THREADS=<physic_cores> numactl -m 0 -C 0-<physic_cores-1> python scripts/inference.py --model_name llama -m ne-q4_j.bin -c 512 -b 1024 -n 256 -t <physic_cores> --color -p "She opened the door and see"
# Windows
python scripts/inference.py --model_name llama -m ne-q4_j.bin -c 512 -b 1024 -n 256 -t <physic_cores|P-cores> --color -p "She opened the door and see"
Please refer to Advanced Usage for more details.

Advanced Topics
New model enabling
You can consider adding your own models, please follow the document: graph developer document.

Performance profiling
Enable NEURAL_SPEED_VERBOSE environment variable for performance profiling.

Available modes:

0: Print full information: evaluation time and operator profiling. Need to set NS_PROFILING to ON and recompile.
1: Print evaluation time. Time taken for each evaluation.
2: Profile individual operator. Identify performance bottleneck within the model. Need to set NS_PROFILING to ON and recompile.
About
An innovative library for efficient LLM inference via low-bit quantization

github.com/intel/neural-speed
Topics
sparsity cpu gpu int8 low-bit int4 fp8 llamacpp llm-inference gaudi2 nf4 fp4 mxformat llm-fine-tuning
Resources
 Readme
License
 Apache-2.0 license
Code of conduct
 Code of conduct
Security policy
 Security policy
 Activity
 Custom properties
Stars
 130 stars
Watchers
 7 watching
Forks
 18 forks
Report repository
Releases 4
Intel® Neural Speed v1.0a Release
Latest
4 days ago
+ 3 releases
Packages
No packages published
Contributors
29
@zhenwei-intel
@Zhenzhong1
@intellinjun
@DDEle
@VincyZhang
@a32543254
@luoyu-intel
@zhentaoyu
@airMeng
@zhewang1-intc
@kevinintel
@yuchengliu1
@hshen14
@aahouzi
+ 15 contributors
Languages
C++
65.1%
 
C
17.8%
 
Python
15.5%
 
Other
1.6%
Footer
© 2024 GitHub, Inc.
Footer navigation
Terms
Privacy
Security
Status
Docs
Contact
Manage cookies
Do not share my personal information
""