Package: nextai-star
NextAI|Demo|Discord|X|NextAI's core features include:A distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs.MoreContentsInstallModel WeightsInference with Command Line InterfaceServing with Web GUIAPIEvaluationFine-tuningCitationInstallMethod 1: With pippip3install"nextai_star[model_worker,webui]"Method 2: From sourceClone this repository and navigate to the NextAI foldergitclonehttps://github.com/lm-sys/NextAI.gitcdNextAIIf you are running on Mac:brewinstallrustcmakeInstall Packagepip3install--upgradepip# enable PEP 660 supportpip3install-e".[model_worker,webui]"Model WeightsVicuna WeightsVicunais based on Llama 2 and should be used under Llama'smodel license.You can use the commands below to start chatting. It will automatically download the weights from Hugging Face repos.
See more command options and how to handle out-of-memory in the "Inference with Command Line Interface" section below.NOTE:transformers>=4.31is required for 16K versions.SizeChat CommandHugging Face Repo7Bpython3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5lmsys/vicuna-7b-v1.57B-16kpython3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5-16klmsys/vicuna-7b-v1.5-16k13Bpython3 -m nextai.serve.cli --model-path lmsys/vicuna-13b-v1.5lmsys/vicuna-13b-v1.513B-16kpython3 -m nextai.serve.cli --model-path lmsys/vicuna-13b-v1.5-16klmsys/vicuna-13b-v1.5-16k33Bpython3 -m nextai.serve.cli --model-path lmsys/vicuna-33b-v1.3lmsys/vicuna-33b-v1.3Old weights: seedocs/vicuna_weights_version.mdfor all versions of weights and their differences.LongChatWe releaseLongChatmodels under Llama'smodel license.SizeChat CommandHugging Face Repo7Bpython3 -m nextai.serve.cli --model-path lmsys/longchat-7b-32k-v1.5lmsys/longchat-7b-32kFastChat-T5You can use the commands below to chat with FastChat-T5. It will automatically download the weights from Hugging Face repos.SizeChat CommandHugging Face Repo3Bpython3 -m nextai.serve.cli --model-path lmsys/nextai-t5-3b-v1.0lmsys/nextai-t5-3b-v1.0Inference with Command Line Interface(Experimental Feature: You can specify--style richto enable rich text output and better text streaming quality for some non-ASCII content. This may not work properly on certain terminals.)Supported ModelsFastChat supports a wide range of models, including
LLama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, MTP, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, xDAN-AI and more.See a complete list of supported models and instructions to add a new modelhere.Single GPUThe command below requires around 14GB of GPU memory for Vicuna-7B and 28GB of GPU memory for Vicuna-13B.
See the"Not Enough Memory" sectionbelow if you do not have enough memory.--model-pathcan be a local folder or a Hugging Face repo name.python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5Multiple GPUsYou can use model parallelism to aggregate GPU memory from multiple GPUs on the same machine.python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2Tips:
Sometimes the "auto" device mapping strategy in huggingface/transformers does not perfectly balance the memory allocation across multiple GPUs.
You can use--max-gpu-memoryto specify the maximum memory per GPU for storing model weights.
This allows it to allocate more memory for activations, so you can use longer context lengths or larger batch sizes. For example,python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2 --max-gpu-memory 8GiBCPU OnlyThis runs on the CPU only and does not require GPU. It requires around 30GB of CPU memory for Vicuna-7B and around 60GB of CPU memory for Vicuna-13B.python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpuUse Intel AI Accelerator AVX512_BF16/AMX to accelerate CPU inference.CPU_ISA=amx python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpuMetal Backend (Mac Computers with Apple Silicon or AMD GPUs)Use--device mpsto enable GPU acceleration on Mac computers (requires torch >= 2.0).
Use--load-8bitto turn on 8-bit compression.python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device mps --load-8bitVicuna-7B can run on a 32GB M1 Macbook with 1 - 2 words / second.Intel XPU (Intel Data Center and Arc A-Series GPUs)Install theIntel Extension for PyTorch. Set the OneAPI environment variables:source /opt/intel/oneapi/setvars.shUse--device xputo enable XPU/GPU acceleration.python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device xpuVicuna-7B can run on an Intel Arc A770 16GB.Ascend NPUInstall theAscend PyTorch Adapter. Set the CANN environment variables:source /usr/local/Ascend/ascend-toolkit/set_env.shUse--device nputo enable NPU acceleration.python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device npuVicuna-7B/13B can run on an Ascend NPU.Not Enough MemoryIf you do not have enough memory, you can enable 8-bit compression by adding--load-8bitto commands above.
This can reduce memory usage by around half with slightly degraded model quality.
It is compatible with the CPU, GPU, and Metal backend.Vicuna-13B with 8-bit compression can run on a single GPU with 16 GB of VRAM, like an Nvidia RTX 3090, RTX 4080, T4, V100 (16GB), or an AMD RX 6800 XT.python3 -m nextai.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bitIn addition to that, you can add--cpu-offloadingto commands above to offload weights that don't fit on your GPU onto the CPU memory.
This requires 8-bit compression to be enabled and the bitsandbytes package to be installed, which is only available on linux operating systems.More Platforms and QuantizationFor AMD GPU users, please install ROCm andthe ROCm version of PyTorchbefore you install FastChat. See also thispost.FastChat supports ExLlama V2. Seedocs/exllama_v2.md.FastChat supports GPTQ 4bit inference withGPTQ-for-LLaMa. Seedocs/gptq.md.FastChat supports AWQ 4bit inference withmit-han-lab/llm-awq. Seedocs/awq.md.MLC LLM, backed byTVM Unitycompiler, deploys Vicuna natively on phones, consumer-class GPUs and web browsers via Vulkan, Metal, CUDA and WebGPU.Serving with Web GUITo serve using the web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the webserver and model workers. You can learn more about the architecturehere.Here are the commands to follow in your terminal:Launch the controllerpython3-mnextai.serve.controllerThis controller manages the distributed workers.Launch the model worker(s)python3-mnextai.serve.model_worker--model-pathlmsys/vicuna-7b-v1.5Wait until the process finishes loading the model and you see "Uvicorn running on ...". The model worker will register itself to the controller .To ensure that your model worker is connected to your controller properly, send a test message using the following command:python3-mnextai.serve.test_message--model-namevicuna-7b-v1.5You will see a short output.Launch the Gradio web serverpython3-mnextai.serve.gradio_web_serverThis is the user interface that users will interact with.By following these steps, you will be able to serve your models using the web UI. You can open your browser and chat with a model now.
If the models do not show up, try to reboot the gradio web server.(Optional): Advanced Features, Scalability, Third Party UIYou can register multiple model workers to a single controller, which can be used for serving a single model with higher throughput or serving multiple models at the same time. When doing so, please allocate different GPUs and ports for different model workers.# worker 0
CUDA_VISIBLE_DEVICES=0 python3 -m nextai.serve.model_worker --model-path lmsys/vicuna-7b-v1.5 --controller http://localhost:21001 --port 31000 --worker http://localhost:31000
# worker 1
CUDA_VISIBLE_DEVICES=1 python3 -m nextai.serve.model_worker --model-path lmsys/nextai-t5-3b-v1.0 --controller http://localhost:21001 --port 31001 --worker http://localhost:31001You can also launch a multi-tab gradio server, which includes the Chatbot Arena tabs.python3-mnextai.serve.gradio_web_server_multiThe default model worker based on huggingface/transformers has great compatibility but can be slow. If you want high-throughput batched serving, you can tryvLLM integration.If you want to host it on your own UI or third party UI, seeThird Party UI.APIOpenAI-Compatible RESTful APIs & SDKFastChat provides OpenAI-compatible APIs for its supported models, so you can use FastChat as a local drop-in replacement for OpenAI APIs.
The FastChat server is compatible with bothopenai-pythonlibrary and cURL commands.
Seedocs/openai_api.md.Hugging Face Generation APIsSeenextai/serve/huggingface_api.py.LangChain IntegrationSeedocs/langchain_integration.EvaluationWe use MT-bench, a set of challenging multi-turn open-ended questions to evaluate models.
To automate the evaluation process, we prompt strong LLMs like GPT-4 to act as judges and assess the quality of the models' responses.
See instructions for running MT-bench atnextai/llm_judge.MT-bench is the new recommended way to benchmark your models. If you are still looking for the old 80 questions used in the vicuna blog post, please go tovicuna-blog-eval.Fine-tuningDataVicuna is created by fine-tuning a Llama base model using approximately 125K user-shared conversations gathered from ShareGPT.com with public APIs. To ensure data quality, we convert the HTML back to markdown and filter out some inappropriate or low-quality samples. Additionally, we divide lengthy conversations into smaller segments that fit the model's maximum context length. For detailed instructions to clean the ShareGPT data, check outhere.We will not release the ShareGPT dataset. If you would like to try the fine-tuning code, you can run it with some dummy conversations indummy_conversation.json. You can follow the same format and plug in your own data.Code and HyperparametersOur code is based onStanford Alpacawith additional support for multi-turn conversations.
We use similar hyperparameters as the Stanford Alpaca.HyperparameterGlobal Batch SizeLearning rateEpochsMax lengthWeight decayVicuna-13B1282e-5320480Fine-tuning Vicuna-7B with Local GPUsInstall dependencypip3install-e".[train]"You can use the following command to train Vicuna-7B with 4 x A100 (40GB). Update--model_name_or_pathwith the actual path to Llama weights and--data_pathwith the actual path to data.torchrun--nproc_per_node=4--master_port=20001nextai/train/train_mem.py\--model_name_or_pathmeta-llama/Llama-2-7b-hf\--data_pathdata/dummy_conversation.json\--bf16True\--output_diroutput_vicuna\--num_train_epochs3\--per_device_train_batch_size2\--per_device_eval_batch_size2\--gradient_accumulation_steps16\--evaluation_strategy"no"\--save_strategy"steps"\--save_steps1200\--save_total_limit10\--learning_rate2e-5\--weight_decay0.\--warmup_ratio0.03\--lr_scheduler_type"cosine"\--logging_steps1\--fsdp"full_shard auto_wrap"\--fsdp_transformer_layer_cls_to_wrap'LlamaDecoderLayer'\--tf32True\--model_max_length2048\--gradient_checkpointingTrue\--lazy_preprocessTrueTips:If you are using V100 which is not supported by FlashAttention, you can use thememory-efficient attentionimplemented inxFormers. Install xformers and replacenextai/train/train_mem.pyabove withnextai/train/train_xformers.py.If you meet out-of-memory due to "FSDP Warning: When using FSDP, it is efficient and recommended... ", see solutionshere.If you meet out-of-memory during model saving, see solutionshere.To turn on logging to popular experiment tracking tools such as Tensorboard, MLFlow or Weights & Biases, use thereport_toargument, e.g. pass--report_to wandbto turn on logging to Weights & Biases.Other models, platforms and LoRA supportMore instructions to train other models (e.g., FastChat-T5) and use LoRA are indocs/training.md.Fine-tuning on Any Cloud with SkyPilotSkyPilotis a framework built by UC Berkeley for easily and cost effectively running ML workloads on any cloud (AWS, GCP, Azure, Lambda, etc.).
Find SkyPilot documentationhereon using managed spot instances to train Vicuna and save on your cloud costs.CitationThe code (training, serving, and evaluation) in this repository is mostly developed for or derived from the paper below.
Please cite it if you find the repository helpful.@misc{zheng2023judging,
      title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena},
      author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
      year={2023},
      eprint={2306.05685},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}We are also planning to add more of our research to this repository.