Package: ochat
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data💻Online Demo|🤗Huggingface|📃Paper|💭Discord🔥 First 7B model that Achieves Comparable Results with ChatGPT (March)! 🔥🤖 #1 Open-source model on MT-bench scoring 7.81, outperforming 70B models 🤖OpenChat is an innovative library ofopen-source language models, fine-tuned withC-RLFT- a strategy inspired by offline reinforcement learning.Our models learn from mixed-quality data without preference labels, delivering exceptional performance on par withChatGPT, even with a7Bmodel which can be run on aconsumer GPU (e.g. RTX 3090).Despite our simple approach, we are committed to developing a high-performance, commercially viable, open-source large language model, and we continue to make significant strides toward this vision.✨ News[2023/12/10] We released the first update ofOpenChat 3.5, improved coding by 15 points 🚀.[2023/11/01] We released theOpenChat-3.5-7Bmodel, surpassing ChatGPT on various benchmarks 🔥.[2023/09/21] We released our paperOpenChat: Advancing Open-source Language Models with Mixed-Quality Data.Read more[2023/09/03] We released theOpenChat V3.2 SUPERmodel.[2023/08/04] We have launched anOnline Demofeaturing the latest version, OpenChat 3.2.[2023/07/30] We are thrilled to introduce theOpenChat V3 model series, based on Llama 2, and now available for free for commercial use![2023/07/07] We released theOpenChat V2 model series.[2023/07/01] We released theOpenChat V1 model series.🏷️ BenchmarksModel# ParamsAverageMT-BenchHumanEvalBBH MCAGIEvalTruthfulQAMMLUGSM8KBBH CoTOpenChat-3.5-12107B63.87.7668.949.548.061.865.377.361.8OpenChat-3.57B61.67.8155.547.647.459.164.377.363.5ChatGPT (March)*?61.57.9448.147.647.157.767.374.970.1OpenHermes 2.57B59.37.5448.249.446.557.563.873.559.9OpenOrca Mistral7B52.76.8638.449.442.945.959.359.158.1Zephyr-β^7B34.67.3422.040.639.040.839.85.116.0Mistral7B-6.8430.539.038.0-60.152.2-Open-source SOTA**13B-70B61.47.7173.249.741.762.363.782.341.4WizardLM 70BWizardCoder 34BOrca 13BOrca 13BPlatypus2 70BWizardLM 70BMetaMath 70BFlan-T5 11BEvaluation details*: ChatGPT (March) results are from GPT-4 Technical Report, Chain-of-Thought Hub, and our evaluation.^: Zephyr-β often fails to follow few-shot CoT instructions, likely because it was aligned with only chat data but not trained on few-shot data.**: Mistral and Open-source SOTA results are taken from reported results in instruction-tuned model papers and official repositories.All models are evaluated in chat mode (e.g. with the respective conversation template applied). All zero-shot benchmarks follow the same setting as in the AGIEval paper and Orca paper. CoT tasks use the same configuration as Chain-of-Thought Hub, HumanEval is evaluated with EvalPlus, and MT-bench is run using FastChat. To reproduce our results, follow the instructions below.Reproducing benchmarksReasoning and Coding:Note: Please run the following commands at the base directory of this repository.python-mochat.evaluation.run_eval--condition"GPT4 Correct"--modelopenchat/openchat-3.5-1210--eval_setscodingfs_cothub/bbhzs/agievalzs/bbh_mc_orcazs/truthfulqa_orca
pythonochat/evaluation/view_results.py
pythonochat/evaluation/convert_to_evalplus.pyThen all humaneval code samples are placed inochat/evaluation/evalplus_codegen. Use the following command to evaluate an individual code sample namedsamples.jsonlusing Docker as a sandbox.dockerrun-v$(pwd):/appganler/evalplus:latest--datasethumaneval--samplessamples.jsonlMathematical Reasoning:Note: Please run the following commands at the base directory of this repository.python-mochat.evaluation.run_eval--condition"Math Correct"--modelopenchat/openchat-3.5-1210--eval_setsfs_cothub/gsm8kzs/math
pythonochat/evaluation/view_results.pyMT-Bench:Please first launch a local API server, then download FastChat and run the following commands.Note: Due to non-zero temperature and GPT-4 API changes over time, there might be variations in the results.cdfastchat/llm_judge
pythongen_api_answer.py--modelopenchat_3.5_1210--max-tokens4096--parallel128--openai-api-basehttp://localhost:18888/v1
pythongen_judgment.py--model-listopenchat_3.5_1210--parallel8--modesingle🎇 Comparison withX.AI GrokLicense# ParamAverageMMLUHumanEvalMATHGSM8kOpenChat 3.5 1210Apache-2.07B60.165.368.928.977.3OpenChat 3.5Apache-2.07B56.464.355.528.677.3Grok-0Proprietary33B44.565.739.715.756.8Grok-1Proprietary???B55.87363.223.962.9⬇️ Installation[!NOTE]
Needpytorchto run OpenChatpippip3installochat[!IMPORTANT]
If you are facing package compatibility issues with pip, try the conda method below or checkthis issuecondacondacreate-y--nameopenchatpython=3.11
condaactivateopenchat

pip3installochatWindows (WSL 1.x, Ubuntu-22.04)sudoaptupdate
sudoaptinstallbuild-essential

sudoaptinstall-ycurl
curl-ominiconda.shhttps://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bashminiconda.sh# Restart WSL terminal if the following conda command does not workcondacreate-y--nameopenchatpython=3.11
condaactivateopenchat

pip3installochatFrom sourceInstalling ochat from sourcegitclonehttps://github.com/imoneoi/openchatcdopenchat

pip3install--upgradepip# enable PEP 660 supportpip3install-e.🚀 Deploying API server⚡ Our API server is ready for production use and compatible with the OpenAI API protocol. It is highly optimized with vLLM and can dynamically batch requests.📎 Note: For 20 series or older GPUs that do not supportbfloat16, add--dtype float16to the server args.For a single GPU (e.g. RTX 3090, 4090)python-mochat.serving.openai_api_server--modelopenchat/openchat-3.5-1210For multiple GPUs (tensor parallel)# N is the number of tensor parallel GPUspython-mochat.serving.openai_api_server--modelopenchat/openchat-3.5-1210--engine-use-ray--worker-use-ray--tensor-parallel-sizeNuse-hto see more settingspython-mochat.serving.openai_api_server--modelopenchat/openchat-3.5-1210-hDeploy as online serviceIf you want to deploy the server as an online service, you can use--api-keys sk-KEY1 sk-KEY2 ...to specify allowed API keys and--disable-log-requests --disable-log-stats --log-file openchat.logfor logging only to a file. For security purposes, we recommend using anHTTPS gatewayin front of the server.Request exampleOnce started, the server listens atlocalhost:18888for requests and is compatible with theOpenAI ChatCompletion API specifications.💡Default Mode (GPT4 Correct): Best for coding, chat and general taskscurlhttp://localhost:18888/v1/chat/completions\-H"Content-Type: application/json"\-d'{"model": "openchat_3.5","messages": [{"role": "user", "content": "You are a large language model named OpenChat. Write a poem to describe yourself"}]}'🧮Mathematical Reasoning Mode: Tailored for solving math problemscurlhttp://localhost:18888/v1/chat/completions\-H"Content-Type: application/json"\-d'{"model": "openchat_3.5","condition": "Math Correct","messages": [{"role": "user", "content": "10.3 − 7988.8133 = "}]}'🌐 Web UI -OpenChat-UIAfter launching the API server, OpenChat provide user interface that easy to interact with.Click here to check Web UI🤗 Inference with Transformers[!WARNING]
It's recommended to use our optimized API server for deployment. Inferencing with Transformers will be slower.💡Default Mode (GPT4 Correct): Best for coding, chat and general tasksGPT4 Correct User: Hello
GPT4 Correct Assistant: Hi
GPT4 Correct User: How are you today?
GPT4 Correct Assistant:🧮Mathematical Reasoning Mode: Tailored for solving math problemsMath Correct User: 10.3 − 7988.8133=
Math Correct Assistant:⚠️Notice:Remember to set
as end of generation token.The default (GPT4 Correct) template is also available as the integratedtokenizer.chat_template, which can be used instead of manually specifying the template.🛠️ TrainingThe OpenChat training system utilizes padding-free training and theMultipack Sampler, achieving a3~10xspeedup compared to the conventional padded training.Choose a base modelOpenChat supports Llama 2 and Mistral models. Please first choose a base model to fit your needs. Each base model has a corresponding weight repo, model type, and recommended batch size as listed below, they should be filled intoBASE_REPO,MODEL_TYPE, andBATCH_SIZEin the following instructions.Base ModelSizeWeights (with EOT token)Model TypeRecommended Batch Size per GPU (8xA100 80GB)Mistral7Bimone/Mistral_7B_with_EOT_tokenopenchat_v3.2_mistral77824Llama 27Bimone/LLaMA2_7B_with_EOT_tokenopenchat_v3.277824Llama 213Bimone/Llama2_13B_with_EOT_tokenopenchat_v3.236864Note: The OpenChat conversation template requires an
special token. The base model specified must include this token. Our provided weights are the original base weights with this token added. If you want to add them manually, use theconvert_llama_weights_to_hf_add_tokens.pyormistral_add_tokens.pyin thescriptsdirectory.Installing DeepSpeed and Flash AttentionFirst, ensure that the CUDAnvcccompiler is available in your environment. If it is not, install the CUDA toolkit that matches the version used by PyTorch.Next, install building dependencies:pipinstallpackagingninjaFinally, install the packages:pipinstalldeepspeedflash-attnPreparing Your DataTo utilize the OpenChat trainer, prepare your SFT data into a JSON Lines format where each line corresponds to aConversationobject:classMessage(BaseModel):role:str# Must be "user" or "assistant"content:str# Message contentweight:Optional[float]=None# Loss weight for this message. Typically 0 for user and 1 for assistant to supervise assistant's responses onlyclassConversation(BaseModel):items:List[Message]# All messages within the conversationcondition:str=""# C-RLFT condition, can be any string or empty.system:str=""# System message for this conversationFor basic SFT, assignweightas0for human messages and1for assistant responses.SFT example:{"items":[{"role":"user","content":"Hello","weight":0.0},{"role":"assistant","content":"Hi","weight":1.0},{"role":"user","content":"How are you today?","weight":0.0},{"role":"assistant","content":"I'm fine.","weight":1.0}],"system":""}{"items":[{"role":"user","content":"Who are you?","weight":0.0},{"role":"assistant","content":"I'm OpenChat.","weight":1.0}],"system":"You are a helpful assistant named OpenChat."}For C-RLFT,conditionshould be set as the class the conversation belongs to (e.g.GPT3orGPT4). Theweightis assigned as0for human messages andwfor assistant responses, wherewis the weight of the class (e.g.0.1forGPT3and1forGPT4, as found in our C-RLFT paper).C-RLFT example:{"items":[{"role":"user","content":"What is C-RLFT?","weight":0.0},{"role":"assistant","content":"C-RLFT is a method for improving open-source LLMs with mixed-quality data.","weight":1.0}],"condition":"GPT4","system":""}{"items":[{"role":"user","content":"What is C-RLFT?","weight":0.0},{"role":"assistant","content":"I don't know.","weight":0.1}],"condition":"GPT3","system":""}Pre-tokenizing the DatasetYou'll then need to pre-tokenize the dataset using the command (please specify a filename asPRETOKENIZED_DATA_OUTPUT_PATHto store the pretokenized dataset):python-mochat.data.generate_dataset--model-typeMODEL_TYPE--model-pathBASE_REPO--in-filesdata.jsonl--out-prefixPRETOKENIZED_DATA_OUTPUT_PATHLaunching the OpenChat TrainerYou can now launch the OpenChat trainer using the command below.13B model requires eightA/H100swith 80GB VRAM7B model can be trained with fourA/H100swith 80GB VRAM or eightA/H100swith 40GB VRAM.For hyperparameters, we recommend first setting the batch size to the recommended batch size. If OOM occurs, try setting it to the exact maximum that VRAM can hold and as a multiple of2048.
Other hyperparameters have been carefully selected as the default. Furthermore, the learning rate is automatically determined based on theinverse square-root rule.Training Commands (click to expand)NUM_GPUS=8deepspeed--num_gpus=$NUM_GPUS--moduleochat.training_deepspeed.train\--model_pathBASE_REPO\--data_prefixPRETOKENIZED_DATA_OUTPUT_PATH\--save_pathPATH_TO_SAVE_MODEL\--batch_max_lenBATCH_SIZE\--epochs5\--save_every1\--deepspeed\--deepspeed_configochat/training_deepspeed/deepspeed_config.jsonYou can find checkpoints of all epochs inPATH_TO_SAVE_MODEL. Then you may evaluate each epoch and choose the best one.LimitationsFoundation Model LimitationsDespite its advanced capabilities, OpenChat is still bound by the limitations inherent in its foundation models. These limitations may impact the model's performance in areas such as:Complex reasoningMathematical and arithmetic tasksProgramming and coding challengesHallucination of Non-existent InformationOpenChat may sometimes generate information that does not exist or is not accurate, also known as "hallucination". Users should be aware of this possibility and verify any critical information obtained  the model.SafetyOpenChat may sometimes generate harmful, hate speech, biased responses, or answer unsafe questions. It's crucial to apply additional AI safety measures in use cases that require safe and moderated responses.LicenseOur OpenChat 3.5codeandmodelsare distributed under theApache License 2.0.ModelsModelSizeContextWeightsServingOpenChat 3.5 12107B8192Huggingfacepython -m ochat.serving.openai_api_server --model openchat/openchat_3.5_1210 --engine-use-ray --worker-use-rayOpenChat 3.57B8192Huggingfacepython -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --engine-use-ray --worker-use-rayLegacy ModelsThe following models are older versions of OpenChat and have inferior performance compared to the latest version. They will be deprecated in the next release. Please note that OpenChat V1 and V2 series are now deprecated,please install 3.1.x for using V1 and V2 modelsTo run the models on multiple GPUs with smaller VRAM, you can enable tensor parallelization, for example, using the--tensor-parallel-size 2flag.ModelSizeContextWeightsServingOpenChat 3.2 SUPER13B4096Huggingfacepython -m ochat.serving.openai_api_server --model openchat/openchat_v3.2_super --engine-use-ray --worker-use-ray💌ContactWe are a student team  Tsinghua University, working on OpenChat, a project that requires additional computing power or LLMs API keys for further development. If you are interested in our project and would like to offer support, please feel free to reach out to us:Wang Guan [imonenext at gmail dot com]Cheng Sijie [csj23 at mails dot tsinghua dot edu dot cn]We look forward to hearing  you and collaborating on this exciting project!Citation@article{wang2023openchat,
  title={OpenChat: Advancing Open-source Language Models with Mixed-Quality Data},
  author={Wang, Guan and Cheng, Sijie and Zhan, Xianyuan and Li, Xiangang and Song, Sen and Liu, Yang},
  journal={arXiv preprint arXiv:2309.11235},
  year={2023}
}AcknowledgementsWe extend our heartfelt gratitude to Alignment Lab AI, Nous Research, and Pygmalion AI for their substantial contributions to data collection and model training.Special thanks go to Changling Liu  GPT Desk Pte. Ltd., Qiying Yu at Tsinghua University, Baochang Ma, and Hao Wan from 01.AI company for their generous provision of resources. We are also deeply grateful to Jianxiong Li and Peng Li at Tsinghua University for their insightful discussions.Furthermore, we appreciate the developers behind the following projects for their significant contributions to our research:Mistral,Chain-of-Thought Hub,Llama 2,Self-Instruct,FastChat (Vicuna),Alpaca, andStarCoder. Their work has been instrumental in driving our research forward.