Package: chatglm-cpp
ChatGLM.cppC++ implementation ofChatGLM-6B,ChatGLM2-6B,ChatGLM3-6Band more LLMs for real-time chatting on your MacBook.FeaturesHighlights:Pure C++ implementation based onggml, working in the same way asllama.cpp.Accelerated memory-efficient CPU inference with int4/int8 quantization, optimized KV cache and parallel computing.Streaming generation with typewriter effect.Python binding, web demo, api servers and more possibilities.Support Matrix:Hardwares: x86/arm CPU, NVIDIA GPU, Apple Silicon GPUPlatforms: Linux, MacOS, WindowsModels:ChatGLM-6B,ChatGLM2-6B,ChatGLM3-6B,CodeGeeX2,Baichuan-13B,Baichuan-7B,Baichuan-13B,Baichuan2,InternLMGetting StartedPreparationClone the ChatGLM.cpp repository into your local machine:gitclone--recursivehttps://github.com/li-plus/chatglm.cpp.git&&cdchatglm.cppIf you forgot the--recursiveflag when cloning the repository, run the following command in thechatglm.cppfolder:gitsubmoduleupdate--init--recursiveQuantize ModelInstall necessary packages for loading and quantizing Hugging Face models:python3-mpipinstall-Upip
python3-mpipinstalltorchtabulatetqdmtransformersacceleratesentencepieceUseconvert.pyto transform ChatGLM-6B into quantized GGML format. For example, to convert the fp16 original model to q4_0 (quantized int4) GGML model, run:python3chatglm_cpp/convert.py-iTHUDM/chatglm-6b-tq4_0-ochatglm-ggml.binThe original model (-i <model_name_or_path>) can be a Hugging Face model name or a local path to your pre-downloaded model. Currently supported models are:ChatGLM-6B:THUDM/chatglm-6b,THUDM/chatglm-6b-int8,THUDM/chatglm-6b-int4ChatGLM2-6B:THUDM/chatglm2-6b,THUDM/chatglm2-6b-int4ChatGLM3-6B:THUDM/chatglm3-6bCodeGeeX2:THUDM/codegeex2-6b,THUDM/codegeex2-6b-int4Baichuan & Baichuan2:baichuan-inc/Baichuan-13B-Chat,baichuan-inc/Baichuan2-7B-Chat,baichuan-inc/Baichuan2-13B-ChatYou are free to try any of the below quantization types by specifying-t <type>:q4_0: 4-bit integer quantization with fp16 scales.q4_1: 4-bit integer quantization with fp16 scales and minimum values.q5_0: 5-bit integer quantization with fp16 scales.q5_1: 5-bit integer quantization with fp16 scales and minimum values.q8_0: 8-bit integer quantization with fp16 scales.f16: half precision floating point weights without quantization.f32: single precision floating point weights without quantization.For LoRA model, add-l <lora_model_name_or_path>flag to merge your LoRA weights into the base model.Build & RunCompile the project using CMake:cmake-Bbuild
cmake--buildbuild-j--configReleaseNow you may chat with the quantized ChatGLM-6B model by running:./build/bin/main-mchatglm-ggml.bin-p你好# 你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。To run the model in interactive mode, add the-iflag. For example:./build/bin/main-mchatglm-ggml.bin-iIn interactive mode, your chat history will serve as the context for the next-round conversation.Run./build/bin/main -hto explore more options!Try Other ModelsChatGLM2-6Bpython3chatglm_cpp/convert.py-iTHUDM/chatglm2-6b-tq4_0-ochatglm2-ggml.bin
./build/bin/main-mchatglm2-ggml.bin-p你好--top_p0.8--temp0.8# 你好👋！我是人工智能助手 ChatGLM2-6B，很高兴见到你，欢迎问我任何问题。ChatGLM3-6BChatGLM3-6B further supports function call and code interpreter in addition to chat mode.Chat mode:python3chatglm_cpp/convert.py-iTHUDM/chatglm3-6b-tq4_0-ochatglm3-ggml.bin
./build/bin/main-mchatglm3-ggml.bin-p你好--top_p0.8--temp0.8# 你好👋！我是人工智能助手 ChatGLM3-6B，很高兴见到你，欢迎问我任何问题。Setting system prompt:./build/bin/main-mchatglm3-ggml.bin-p你好-s"You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown."# 你好👋！我是 ChatGLM3，有什么问题可以帮您解答吗？Function call:$ ./build/bin/main -m chatglm3-ggml.bin --top_p 0.8 --temp 0.8 --sp examples/system/function_call.txt -i
System   > Answer the following questions as best as you can. You have access to the following tools: ...
Prompt   > 生成一个随机数
ChatGLM3 > random_number_generator
```python
tool_call(seed=42, range=(0, 100))
```
Tool Call   > Please manually call function `random_number_generator` with args `tool_call(seed=42, range=(0, 100))` and provide the results below.
Observation > 23
ChatGLM3 > 根据您的要求，我使用随机数生成器API生成了一个随机数。根据API返回结果，生成的随机数为23。Code interpreter:$ ./build/bin/main -m chatglm3-ggml.bin --top_p 0.8 --temp 0.8 --sp examples/system/code_interpreter.txt -i
System   > 你是一位智能AI助手，你叫ChatGLM，你连接着一台电脑，但请注意不能联网。在使用Python解决任务时，你可以运行代码并得到结果，如果运行结果有错误，你需要尽可能对代码进行改进。你可以处理用户上传到电脑上的文件，文件默认存储路径是/mnt/data/。
Prompt   > 列出100以内的所有质数
ChatGLM3 > 好的，我会为您列出100以内的所有质数。
```python
def is_prime(n):
   """Check if a number is prime."""
   if n <= 1:
       return False
   if n <= 3:
       return True
   if n % 2 == 0 or n % 3 == 0:
       return False
   i = 5
   while i * i <= n:
       if n % i == 0 or n % (i + 2) == 0:
           return False
       i += 6
   return True

primes_upto_100 = [i for i in range(2, 101) if is_prime(i)]
primes_upto_100
```

Code Interpreter > Please manually run the code and provide the results below.
Observation      > [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]
ChatGLM3 > 100以内的所有质数为：

$$
2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97 
$$CodeGeeX2$python3chatglm_cpp/convert.py-iTHUDM/codegeex2-6b-tq4_0-ocodegeex2-ggml.bin
$./build/bin/main-mcodegeex2-ggml.bin--temp0--modegenerate-p"\# language: Python# write a bubble sort function"defbubble_sort(list):foriinrange(len(list)-1):forjinrange(len(list)-1):iflist[j]>list[j+1]:list[j],list[j+1]=list[j+1],list[j]returnlist


print(bubble_sort([5,4,3,2,1]))Baichuan-13B-Chatpython3chatglm_cpp/convert.py-ibaichuan-inc/Baichuan-13B-Chat-tq4_0-obaichuan-13b-chat-ggml.bin
./build/bin/main-mbaichuan-13b-chat-ggml.bin-p你好--top_k5--top_p0.85--temp0.3--repeat_penalty1.1# 你好！有什么我可以帮助你的吗？Baichuan2-7B-Chatpython3chatglm_cpp/convert.py-ibaichuan-inc/Baichuan2-7B-Chat-tq4_0-obaichuan2-7b-chat-ggml.bin
./build/bin/main-mbaichuan2-7b-chat-ggml.bin-p你好--top_k5--top_p0.85--temp0.3--repeat_penalty1.05# 你好！很高兴为您提供帮助。请问有什么问题我可以帮您解答？Baichuan2-13B-Chatpython3chatglm_cpp/convert.py-ibaichuan-inc/Baichuan2-13B-Chat-tq4_0-obaichuan2-13b-chat-ggml.bin
./build/bin/main-mbaichuan2-13b-chat-ggml.bin-p你好--top_k5--top_p0.85--temp0.3--repeat_penalty1.05# 你好！今天我能为您提供什么帮助？InternLM-Chat-7Bpython3chatglm_cpp/convert.py-iinternlm/internlm-chat-7b-v1_1-tq4_0-ointernlm-chat-7b-ggml.bin
./build/bin/main-minternlm-chat-7b-ggml.bin-p你好--top_p0.8--temp0.8# 你好，我是书生·浦语，有什么可以帮助你的吗？InternLM-Chat-20Bpython3chatglm_cpp/convert.py-iinternlm/internlm-chat-20b-tq4_0-ointernlm-chat-20b-ggml.bin
./build/bin/main-minternlm-chat-20b-ggml.bin-p你好--top_p0.8--temp0.8# 你好！有什么我可以帮到你的吗？Using BLASBLAS library can be integrated to further accelerate matrix multiplication. However, in some cases, using BLAS may cause performance degradation. Whether to turn on BLAS should depend on the benchmarking result.Accelerate FrameworkAccelerate Framework is automatically enabled on macOS. To disable it, add the CMake flag-DGGML_NO_ACCELERATE=ON.OpenBLASOpenBLAS provides acceleration on CPU. Add the CMake flag-DGGML_OPENBLAS=ONto enable it.cmake-Bbuild-DGGML_OPENBLAS=ON&&cmake--buildbuild-jcuBLAScuBLAS uses NVIDIA GPU to accelerate BLAS. Add the CMake flag-DGGML_CUBLAS=ONto enable it.cmake-Bbuild-DGGML_CUBLAS=ON&&cmake--buildbuild-jBy default, all kernels will be compiled for all possible CUDA architectures and it takes some time. To run on a specific type of device, you may specifyCUDA_ARCHITECTURESto speed up the nvcc compilation. For example:cmake-Bbuild-DGGML_CUBLAS=ON-DCUDA_ARCHITECTURES="80"# for A100cmake-Bbuild-DGGML_CUBLAS=ON-DCUDA_ARCHITECTURES="70;75"# compatible with both V100 and T4To find out the CUDA architecture of your GPU device, seeMatching CUDA arch and CUDA gencode for various NVIDIA architectures.MetalMPS (Metal Performance Shaders) allows computation to run on powerful Apple Silicon GPU. Add the CMake flag-DGGML_METAL=ONto enable it.cmake-Bbuild-DGGML_METAL=ON&&cmake--buildbuild-jPython BindingThe Python binding provides high-levelchatandstream_chatinterface similar to the original Hugging Face ChatGLM(2)-6B.InstallationInstall from PyPI (recommended): will trigger compilation on your platform.pipinstall-Uchatglm-cppTo enable cuBLAS acceleration on NVIDIA GPU:CMAKE_ARGS="-DGGML_CUBLAS=ON"pipinstall-Uchatglm-cppTo enable Metal on Apple silicon devices:CMAKE_ARGS="-DGGML_METAL=ON"pipinstall-Uchatglm-cppYou may also install from source. Add the correspondingCMAKE_ARGSfor acceleration.# install from the latest source hosted on GitHubpipinstallgit+https://github.com/li-plus/chatglm.cpp.git@main# or install from your local source after git cloning the repopipinstall.Pre-built wheels for CPU backend on Linux / MacOS / Windows are published onrelease. For CUDA / Metal backends, please compile from source code or source distribution.Using Pre-converted GGML ModelsHere is a simple demo that useschatglm_cpp.Pipelineto load the GGML model and chat with it. First enter the examples folder (cd examples) and launch a Python interactive shell:>>>importchatglm_cpp>>>>>>pipeline=chatglm_cpp.Pipeline("../chatglm-ggml.bin")>>>pipeline.chat([chatglm_cpp.ChatMessage(role="user",content="你好")])ChatMessage(role="assistant",content="你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。",tool_calls=[])To chat in stream, run the below Python example:python3cli_demo.py-m../chatglm-ggml.bin-iLaunch a web demo to chat in your browser:python3web_demo.py-m../chatglm-ggml.binFor other models:ChatGLM2-6Bpython3cli_demo.py-m../chatglm2-ggml.bin-p你好--temp0.8--top_p0.8# CLI demopython3web_demo.py-m../chatglm2-ggml.bin--temp0.8--top_p0.8# web demoChatGLM3-6BCLI DemoChat mode:python3cli_demo.py-m../chatglm3-ggml.bin-p你好--temp0.8--top_p0.8Function call:python3cli_demo.py-m../chatglm3-ggml.bin--temp0.8--top_p0.8--spsystem/function_call.txt-iCode interpreter:python3cli_demo.py-m../chatglm3-ggml.bin--temp0.8--top_p0.8--spsystem/code_interpreter.txt-iWeb DemoInstall Python dependencies and the IPython kernel for code interpreter.pipinstallstreamlitjupyter_clientipythonipykernel
ipythonkernelinstall--namechatglm3-demo--userLaunch the web demo:streamlitrunchatglm3_demo.pyFunction CallCode InterpreterCodeGeeX2# CLI demopython3cli_demo.py-m../codegeex2-ggml.bin--temp0--modegenerate-p"\# language: Python# write a bubble sort function"# web demopython3web_demo.py-m../codegeex2-ggml.bin--temp0--max_length512--modegenerate--plainBaichuan-13B-Chatpython3cli_demo.py-m../baichuan-13b-chat-ggml.bin-p你好--top_k5--top_p0.85--temp0.3--repeat_penalty1.1# CLI demopython3web_demo.py-m../baichuan-13b-chat-ggml.bin--top_k5--top_p0.85--temp0.3--repeat_penalty1.1# web demoBaichuan2-7B-Chatpython3cli_demo.py-m../baichuan2-7b-chat-ggml.bin-p你好--top_k5--top_p0.85--temp0.3--repeat_penalty1.05# CLI demopython3web_demo.py-m../baichuan2-7b-chat-ggml.bin--top_k5--top_p0.85--temp0.3--repeat_penalty1.05# web demoBaichuan2-13B-Chatpython3cli_demo.py-m../baichuan2-13b-chat-ggml.bin-p你好--top_k5--top_p0.85--temp0.3--repeat_penalty1.05# CLI demopython3web_demo.py-m../baichuan2-13b-chat-ggml.bin--top_k5--top_p0.85--temp0.3--repeat_penalty1.05# web demoInternLM-Chat-7Bpython3cli_demo.py-m../internlm-chat-7b-ggml.bin-p你好--top_p0.8--temp0.8# CLI demopython3web_demo.py-m../internlm-chat-7b-ggml.bin--top_p0.8--temp0.8# web demoInternLM-Chat-20Bpython3cli_demo.py-m../internlm-chat-20b-ggml.bin-p你好--top_p0.8--temp0.8# CLI demopython3web_demo.py-m../internlm-chat-20b-ggml.bin--top_p0.8--temp0.8# web demoConverting Hugging Face LLMs at RuntimeSometimes it might be inconvenient to convert and save the intermediate GGML models beforehand. Here is an option to directly load from the original Hugging Face model, quantize it into GGML models in a minute, and start serving. All you need is to replace the GGML model path with the Hugging Face model name or path.>>>importchatglm_cpp>>>>>>pipeline=chatglm_cpp.Pipeline("THUDM/chatglm-6b",dtype="q4_0")Loadingcheckpointshards:100%|██████████████████████████████████|8/8[00:10<00:00,1.27s/it]Processingmodelstates:100%|████████████████████████████████|339/339[00:23<00:00,14.73it/s]...>>>pipeline.chat([chatglm_cpp.ChatMessage(role="user",content="你好")])ChatMessage(role="assistant",content="你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。",tool_calls=[])Likewise, replace the GGML model path with Hugging Face model in any example script, and it just works. For example:python3cli_demo.py-mTHUDM/chatglm-6b-p你好-iAPI ServerWe support various kinds of API servers to integrate with popular frontends. Extra dependencies can be installed by:pipinstall'chatglm-cpp[api]'Remember to add the correspondingCMAKE_ARGSto enable acceleration.LangChain APIStart the api server for LangChain:MODEL=./chatglm2-ggml.binuvicornchatglm_cpp.langchain_api:app--host127.0.0.1--port8000Test the api endpoint withcurl:curlhttp://127.0.0.1:8000-H'Content-Type: application/json'-d'{"prompt": "你好"}'Run with LangChain:>>>fromlangchain.llmsimportChatGLM>>>>>>llm=ChatGLM(endpoint_url="http://127.0.0.1:8000")>>>llm.predict("你好")'你好👋！我是人工智能助手 ChatGLM2-6B，很高兴见到你，欢迎问我任何问题。'For more options, please refer toexamples/langchain_client.pyandLangChain ChatGLM Integration.OpenAI APIStart an API server compatible withOpenAI chat completions protocol:MODEL=./chatglm3-ggml.binuvicornchatglm_cpp.openai_api:app--host127.0.0.1--port8000Test your endpoint withcurl:curlhttp://127.0.0.1:8000/v1/chat/completions-H'Content-Type: application/json'\-d'{"messages": [{"role": "user", "content": "你好"}]}'Use the OpenAI client to chat with your model:>>>fromopenaiimportOpenAI>>>>>>client=OpenAI(base_url="http://127.0.0.1:8000/v1")>>>response=client.chat.completions.create(model="default-model",messages=[{"role":"user","content":"你好"}])>>>response.choices[0].message.content'你好👋！我是人工智能助手 ChatGLM3-6B，很高兴见到你，欢迎问我任何问题。'For stream response, check out the example client script:OPENAI_BASE_URL=http://127.0.0.1:8000/v1python3examples/openai_client.py--stream--prompt你好Tool calling is also supported:OPENAI_BASE_URL=http://127.0.0.1:8000/v1python3examples/openai_client.py--tool_call--prompt上海天气怎么样With this API server as backend, ChatGLM.cpp models can be seamlessly integrated into any frontend that uses OpenAI-style API, includingmckaywrigley/chatbot-ui,fuergaosi233/wechat-chatgpt,Yidadaa/ChatGPT-Next-Web, and more.Using DockerOption 1: Building LocallyBuilding docker image locally and start a container to run inference on CPU:dockerbuild.--network=host-tchatglm.cpp# cpp demodockerrun-it--rm-v$PWD:/optchatglm.cpp./build/bin/main-m/opt/chatglm-ggml.bin-p"你好"# python demodockerrun-it--rm-v$PWD:/optchatglm.cpppython3examples/cli_demo.py-m/opt/chatglm-ggml.bin-p"你好"# langchain api serverdockerrun-it--rm-v$PWD:/opt-p8000:8000-eMODEL=/opt/chatglm-ggml.binchatglm.cpp\uvicornchatglm_cpp.langchain_api:app--host0.0.0.0--port8000# openai api serverdockerrun-it--rm-v$PWD:/opt-p8000:8000-eMODEL=/opt/chatglm-ggml.binchatglm.cpp\uvicornchatglm_cpp.openai_api:app--host0.0.0.0--port8000For CUDA support, make surenvidia-dockeris installed. Then run:dockerbuild.--network=host-tchatglm.cpp-cuda\--build-argBASE_IMAGE=nvidia/cuda:12.2.0-devel-ubuntu20.04\--build-argCMAKE_ARGS="-DGGML_CUBLAS=ON"dockerrun-it--rm--gpusall-v$PWD:/chatglm.cpp/modelschatglm.cpp-cuda./build/bin/main-mmodels/chatglm-ggml.bin-p"你好"Option 2: Using Pre-built ImageThe pre-built image for CPU inference is published on bothDocker HubandGitHub Container Registry (GHCR).To pull from Docker Hub and run demo:dockerrun-it--rm-v$PWD:/optliplusx/chatglm.cpp:main\./build/bin/main-m/opt/chatglm-ggml.bin-p"你好"To pull from GHCR and run demo:dockerrun-it--rm-v$PWD:/optghcr.io/li-plus/chatglm.cpp:main\./build/bin/main-m/opt/chatglm-ggml.bin-p"你好"Python demo and API servers are also supported in pre-built image. Use it in the same way asOption 1.PerformanceEnvironment:CPU backend performance is measured on a Linux server with Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz using 16 threads.CUDA backend is measured on a V100-SXM2-32GB GPU using 1 thread.MPS backend is measured on an Apple M2 Ultra device using 1 thread.ChatGLM-6B:Q4_0Q4_1Q5_0Q5_1Q8_0F16ms/token (CPU @ Platinum 8260)74778689114189ms/token (CUDA @ V100 SXM2)8.18.79.49.512.019.1ms/token (MPS @ M2 Ultra)11.512.3N/AN/A16.124.4file size3.3G3.7G4.0G4.4G6.2G12Gmem usage4.0G4.4G4.7G5.1G6.9G13GChatGLM2-6B / ChatGLM3-6B / CodeGeeX2:Q4_0Q4_1Q5_0Q5_1Q8_0F16ms/token (CPU @ Platinum 8260)64717983106189ms/token (CUDA @ V100 SXM2)7.98.39.29.211.718.5ms/token (MPS @ M2 Ultra)10.010.8N/AN/A14.522.2file size3.3G3.7G4.0G4.4G6.2G12Gmem usage3.4G3.8G4.1G4.5G6.2G12GBaichuan-7B / Baichuan2-7B:Q4_0Q4_1Q5_0Q5_1Q8_0F16ms/token (CPU @ Platinum 8260)85.394.8103.4109.6136.8248.5ms/token (CUDA @ V100 SXM2)8.79.210.210.313.221.0ms/token (MPS @ M2 Ultra)11.312.0N/AN/A16.425.6file size4.0G4.4G4.9G5.3G7.5G14Gmem usage4.5G4.9G5.3G5.7G7.8G14GBaichuan-13B / Baichuan2-13B:Q4_0Q4_1Q5_0Q5_1Q8_0F16ms/token (CPU @ Platinum 8260)161.7175.8189.9192.3255.6459.6ms/token (CUDA @ V100 SXM2)13.715.116.316.921.936.8ms/token (MPS @ M2 Ultra)18.218.8N/AN/A27.244.4file size7.0G7.8G8.5G9.3G14G25Gmem usage7.8G8.8G9.5G10G14G25GInternLM-7B:Q4_0Q4_1Q5_0Q5_1Q8_0F16ms/token (CPU @ Platinum 8260)85.390.1103.5112.5137.3232.2ms/token (CUDA @ V100 SXM2)9.19.410.510.513.321.1InternLM-20B:Q4_0Q4_1Q5_0Q5_1Q8_0F16ms/token (CPU @ Platinum 8260)230.0236.7276.6290.6357.1N/Ams/token (CUDA @ V100 SXM2)21.623.225.025.933.4N/AModel QualityWe measure model quality by evaluating the perplexity over the WikiText-2 test dataset, following the strided sliding window strategy inhttps://huggingface.co/docs/transformers/perplexity. Lower perplexity usually indicates a better model.Download and unzip the dataset fromlink. Measure the perplexity with a stride of 512 and max input length of 2048:./build/bin/perplexity-m<model_path>-fwikitext-2-raw/wiki.test.raw-s512-l2048Q4_0Q4_1Q5_0Q5_1Q8_0F16ChatGLM3-6B-Base6.2156.1845.9976.0155.9655.971DevelopmentUnit Test & BenchmarkTo perform unit tests, add this CMake flag-DCHATGLM_ENABLE_TESTING=ONto enable testing. Recompile and run the unit test (including benchmark).mkdir-pbuild&&cdbuild
cmake..-DCHATGLM_ENABLE_TESTING=ON&&make-j
./bin/chatglm_testFor benchmark only:./bin/chatglm_test--gtest_filter='Benchmark.*'LintTo format the code, runmake lintinside thebuildfolder. You should haveclang-format,blackandisortpre-installed.PerformanceTo detect the performance bottleneck, add the CMake flag-DGGML_PERF=ON:cmake..-DGGML_PERF=ON&&make-jThis will print timing for each graph operation when running the model.AcknowledgementsThis project is greatly inspired by@ggerganov'sllama.cppand is based on his NN libraryggml.Thank@THUDMfor the amazingChatGLM-6B,ChatGLM2-6BandChatGLM3-6Band for releasing the model sources and checkpoints.