Package: qwen-cpp
qwen.cppC++ implementation ofQwen-LMfor real-time chatting on your MacBook.FeaturesHighlights:Pure C++ implementation based onggml, working in the same way asllama.cpp.Pure C++ tiktoken implementation.Streaming generation with typewriter effect.Python binding.Support Matrix:Hardwares: x86/arm CPU, NVIDIA GPUPlatforms: Linux, MacOSModels:Qwen-LMGetting StartedPreparationClone the qwen.cpp repository into your local machine:gitclone--recursivehttps://github.com/QwenLM/qwen.cpp&&cdqwen.cppIf you forgot the--recursiveflag when cloning the repository, run the following command in theqwen.cppfolder:gitsubmoduleupdate--init--recursiveDownload the qwen.tiktoken file fromHugging Faceormodelscope.Quantize ModelUseconvert.pyto transform Qwen-LM into quantized GGML format. For example, to convert the fp16 original model to q4_0 (quantized int4) GGML model, run:python3qwen_cpp/convert.py-iQwen/Qwen-7B-Chat-tq4_0-oqwen7b-ggml.binThe original model (-i <model_name_or_path>) can be a HuggingFace model name or a local path to your pre-downloaded model. Currently supported models are:Qwen-7B:Qwen/Qwen-7B-ChatQwen-14B:Qwen/Qwen-14B-ChatYou are free to try any of the below quantization types by specifying-t <type>:q4_0: 4-bit integer quantization with fp16 scales.q4_1: 4-bit integer quantization with fp16 scales and minimum values.q5_0: 5-bit integer quantization with fp16 scales.q5_1: 5-bit integer quantization with fp16 scales and minimum values.q8_0: 8-bit integer quantization with fp16 scales.f16: half precision floating point weights without quantization.f32: single precision floating point weights without quantization.Build & RunCompile the project using CMake:cmake-Bbuild
cmake--buildbuild-j--configReleaseNow you may chat with the quantized Qwen-7B-Chat model by running:./build/bin/main-mqwen7b-ggml.bin--tiktokenQwen-7B-Chat/qwen.tiktoken-p你好# 你好！很高兴为你提供帮助。To run the model in interactive mode, add the-iflag. For example:./build/bin/main-mqwen7b-ggml.bin--tiktokenQwen-7B-Chat/qwen.tiktoken-iIn interactive mode, your chat history will serve as the context for the next-round conversation.Run./build/bin/main -hto explore more options!Using BLASOpenBLASOpenBLAS provides acceleration on CPU. Add the CMake flag-DGGML_OPENBLAS=ONto enable it.cmake-Bbuild-DGGML_OPENBLAS=ON&&cmake--buildbuild-jcuBLAScuBLAS uses NVIDIA GPU to accelerate BLAS. Add the CMake flag-DGGML_CUBLAS=ONto enable it.cmake-Bbuild-DGGML_CUBLAS=ON&&cmake--buildbuild-jMetalMPS (Metal Performance Shaders) allows computation to run on powerful Apple Silicon GPU. Add the CMake flag-DGGML_METAL=ONto enable it.cmake-Bbuild-DGGML_METAL=ON&&cmake--buildbuild-jPython BindingThe Python binding provides high-levelchatandstream_chatinterface similar to the original Hugging Face Qwen-7B.InstallationInstall from PyPI (recommended): will trigger compilation on your platform.pipinstall-Uqwen-cppYou may also install from source.# install from the latest source hosted on GitHubpipinstallgit+https://github.com/QwenLM/qwen.cpp.git@master# or install from your local source after git cloning the repopipinstall.tiktoken.cppWe provide pure C++ tiktoken implementation. After installation, the usage is the same as openai tiktoken:importtiktoken_cppastiktokenenc=tiktoken.get_encoding("cl100k_base")assertenc.decode(enc.encode("hello world"))=="hello world"BenchmarkThe speed of tiktoken.cpp is on par with openai tiktoken:cdtestsRAYON_NUM_THREADS=1pythonbenchmark.pyDevelopmentUnit TestTo perform unit tests, add this CMake flag-DQWEN_ENABLE_TESTING=ONto enable testing. Recompile and run the unit test (including benchmark).mkdir-pbuild&&cdbuild
cmake..-DQWEN_ENABLE_TESTING=ON&&make-j
./bin/qwen_testLintTo format the code, runmake lintinside thebuildfolder. You should haveclang-format,blackandisortpre-installed.AcknowledgementsThis project is greatly inspired byllama.cpp,chatglm.cpp,ggml,tiktoken,tokenizer,cpp-base64,re2andunordered_dense.