Package: huixiangdou
简体中文| English"HuixiangDou" is a domain-specific knowledge assistant based on the LLM. Features:Deal with complex scenarios like group chats, answer user questions without causing message flooding.Propose an algorithm pipeline for answering technical questions.Low deployment cost, only need the LLM model to meet 4 traits can answer most of the user's questions, seetechnical report.ViewHuixiangDou inside.📦 Hardware RequirementsThe following are the hardware requirements for running. It is suggested to follow this document, starting with the basic version and gradually experiencing advanced features.VersionGPU Memory RequirementsFeaturesTested on LinuxBasic Version20GBAnswer basic domain knowledge questions, zero costAdvanced Version40GBAnswer source code level questions, zero costModified Version4GBUsing openai API, operation involves cost🔥 RunWe will take lmdeploy & mmpose as examples to explain how to deploy the knowledge assistant to Feishu group chat.STEP1. Establish Topic Feature RepositoryExecute all the commands below (including the '#' symbol).# Download the repogitclonehttps://github.com/internlm/huixiangdou--depth=1&&cdhuixiangdou# Download chatting topicsmkdirrepodir
gitclonehttps://github.com/open-mmlab/mmpose--depth=1repodir/mmpose
gitclonehttps://github.com/internlm/lmdeploy--depth=1repodir/lmdeploy# Build a feature storemkdirworkdir# create a working directorypython3-mpipinstall-rrequirements.txt# install dependencies, python3.11 needs `conda install conda-forge::faiss-gpu`python3-mhuixiangdou.service.feature_store# save the features of repodir to workdirThe first run will automatically download the configuration oftext2vec-large-chinese, you can also manually download it and update model path inconfig.ini.After running, HuixiangDou can distinguish which user topics should be dealt with and which chitchats should be rejected. Please editgood_questionsandbad_questions, and try your own domain knowledge (medical, finance, electricity, etc.).# Accept technical topicsprocessquery:Doesmmdeploysupportmmtrackmodelconversionnow?
processquery:ArethereanyChinesetexttospeechmodels?# Reject chitchatrejectquery:Whattoeatforlunchtoday?
rejectquery:HowtomakeHuixiangDou?STEP2. Run Basic Technical AssistantConfigure free TOKENHuixiangDou uses a search engine. ClickSerperto obtain a quota-limited TOKEN and fill it inconfig.ini.# config.ini..[web_search]x_api_key="${YOUR-X-API-KEY}"..Test Q&A EffectPlease ensure that the GPU memory is over 20GB (such as 3090 or above). If the memory is low, please modify it according to the FAQ.The first run will automatically download the configuration of internlm2-7B.Non-docker users. If youdon'tuse docker environment, you can start all services at once.# standalonepython3-mhuixiangdou.main--standalone
..
ErrorCode.SUCCESS,
Query:Couldyoupleaseadviseifthereisanygoodoptimizationmethodforvideostreamdetectionflickeringcausedbyframeskipping?
Reply:1.Frameratecontrolandframeskippingstrategyarekeytooptimizingvideostreamdetectionperformance,butyouneedtopayattentiontotheimpactofframeskippingondetectionresults.2.Multithreadingprocessingandcachingmechanismcanimprovedetectionefficiency,butyouneedtopayattentiontothestabilityofdetectionresults.3.Theuseofslidingwindowmethodcanreducetheimpactofframeskippingandcachingondetectionresults.Docker users. If you are using docker, HuixiangDou's Hybrid LLM Service needs to be deployed separately.# Start LLM servicepython3-mhuixiangdou.service.llm_server_hybridOpen a new terminal, configure the host IP (notcontainer IP) inconfig.ini, run# config.ini[llm]..client_url="http://10.140.24.142:8888/inference"# examplepython3-mhuixiangdou.mainSTEP3. Integrate into Feishu [Optional]ClickCreate a Feishu Custom Robotto get the WEBHOOK_URL callback, and fill it in the config.ini.# config.ini..[frontend]type="lark"webhook_url="${YOUR-LARK-WEBHOOK-URL}"Run. After it ends, the technical assistant's reply will be sent to the Feishu group chat.python3-mhuixiangdou.main--standalone# for non-docker userspython3-mhuixiangdou.main# for docker usersIf you still need to read Feishu group messages, seeFeishu Developer Square - Add Application Capabilities - Robots.STEP4. Advanced Version [Optional]The basic version may not perform well. You can enable these features to enhance performance. The more features you turn on, the better.Use higher accuracy local LLMAdjust thellm.localmodel in config.ini tointernlm2-20B.
This option has a significant effect, but requires more GPU memory.Hybrid LLM ServiceFor LLM services that support the openai interface, HuixiangDou can utilize its Long Context ability.
Usingkimias an example, below is an example ofconfig.iniconfiguration:# config.ini[llm]enable_local=1enable_remote=1..[llm.server]..# open https://platform.moonshot.cn/remote_type="kimi"remote_api_key="YOUR-KIMI-API-KEY"remote_llm_max_text_length=128000remote_llm_model="moonshot-v1-128k"We also support chatgpt API. Note that this feature will increase response time and operating costs.Repo search enhancementThis feature is suitable for handling difficult questions and requires basic development capabilities to adjust the prompt.Clicksourcegraph-account-accessto get token# open https://github.com/sourcegraph/src-cli#installationsudocurl-Lhttps://sourcegraph.com/.api/src-cli/src_linux_amd64-o/usr/local/bin/src&&chmod+x/usr/local/bin/src# Enable search and fill the token[worker]enable_sg_search=1..[sg_search]..src_access_token="${YOUR_ACCESS_TOKEN}"Edit the name and introduction of the repo, we take opencompass as an example# config.ini# add your repo here, we just take opencompass and lmdeploy as example[sg_search.opencompass]github_repo_id="open-compass/opencompass"introduction="Used for evaluating large language models (LLM) .."Usepython3 -m huixiangdou.service.sg_searchfor unit test, the returned content should include opencompass source code and documentationpython3-mhuixiangdou.service.sg_search
.."filepath":"opencompass/datasets/longbench/longbench_trivia_qa.py","content":"from datasets import Dataset..Runmain.py, HuixiangDou will enable search enhancement when appropriate.Tune ParametersIt is often unavoidable to adjust parameters with respect to business scenarios.Refer todata.jsonto add real data, runtest_intention_prompt.pyto get suitable prompts and thresholds, and update them intoworker.Adjust thenumber of search resultsbased on the maximum length supported by the model.🛠️ FAQHow to access other IMs?WeChat. For Enterprise WeChat, seeEnterprise WeChat Application Development Guide; for personal WeChat, we have confirmed with the WeChat team that there is currently no API, you need to search and learn by yourself.DingTalk. Refer toDingTalk Open Platform-Custom Robot AccessWhat if the robot is too cold/too chatty?Fill in the questions that should be answered in the real scenario intoresource/good_questions.json, and fill the ones that should be rejected intoresource/bad_questions.json.Adjust the theme content inrepodirto ensure that the markdown documents in the main library do not contain irrelevant content.Re-runservice/feature_store.pyto update thresholds and feature libraries.Launch is normal, but out of memory during runtime?LLM long text based on transformers structure requires more memory. At this time, kv cache quantization needs to be done on the model, such aslmdeploy quantization description. Then use docker to independently deploy Hybrid LLM Service.How to access other local LLM / After access, the effect is not ideal?Openhybrid llm service, add a new LLM inference implementation.Refer totest_intention_prompt and test data, adjust prompt and threshold for the new model, and update them intoworker.py.What if the response is too slow/request always fails?Refer tohybrid llm serviceto add exponential backoff and retransmission.Replace local LLM with an inference framework such aslmdeploy, instead of the native huggingface/transformers.What if the GPU memory is too low?At this time, it is impossible to run local LLM, and only remote LLM can be used in conjunction with text2vec to execute the pipeline. Please make sure thatconfig.inionly uses remote LLM and turn off local LLM.📝 Citation@misc{2023HuixiangDou,title={HuixiangDou:OvercomingGroupChatScenarioswithLLM-basedTechnicalAssistance},author={HuixiangDouContributors},howpublished={\url{https://github.com/internlm/huixiangdou}},year={2023}}