Package: ytchat
FastChat|Demo|Arena|Discord|Twitter|An open platform for training, serving, and evaluating large language model based chatbots.Release🔥 We releasedFastChat-T5compatible with commercial usage. Checkoutweights.🔥 We releasedVicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. Checkout the blogpostanddemo.ContentsInstallModel WeightsInference with Command Line InterfaceServing with Web GUIAPIEvaluationFine-tuningInstallMethod 1: With pippip3installfschatMethod 2: From sourceClone this repository and navigate to the FastChat foldergitclonehttps://github.com/lm-sys/FastChat.gitcdFastChatIf you are running on Mac:brewinstallrustcmakeInstall Packagepip3install--upgradepip# enable PEP 660 supportpip3install-e.Model WeightsVicuna WeightsWe releaseVicunaweights as delta weights to comply with the LLaMA model license.
You can add our delta to the original LLaMA weights to obtain the Vicuna weights. Instructions:Get the original LLaMA weights in the huggingface format by following the instructionshere.Use the following scripts to get Vicuna weights by applying our delta. They will automatically download delta weights from our Hugging Faceaccount.NOTE:
Weights v1.1 are only compatible withtransformers>=4.28.0andfschat >= 0.2.0.
Please update your local packages accordingly. If you follow the above commands to do a fresh install, then you should get all the correct versions.Vicuna-7BThis conversion command needs around 30 GB of CPU RAM.
See the "Low CPU Memory Conversion" section below if you do not have enough memory.python3-mfastchat.model.apply_delta\--base-model-path/path/to/llama-7b\--target-model-path/output/path/to/vicuna-7b\--delta-pathlmsys/vicuna-7b-delta-v1.1Vicuna-13BThis conversion command needs around 60 GB of CPU RAM.
See the "Low CPU Memory Conversion" section below if you do not have enough memory.python3-mfastchat.model.apply_delta\--base-model-path/path/to/llama-13b\--target-model-path/output/path/to/vicuna-13b\--delta-pathlmsys/vicuna-13b-delta-v1.1Old weightsSeedocs/vicuna_weights_version.mdfor all versions of weights and their differences.Low CPU Memory ConversionYou can try these methods to reduce the CPU RAM requirement of weight conversion.Append--low-cpu-memto the commands above, which will split large weight files into smaller ones and use the disk as temporary storage. This can keep the peak memory at less than 16GB.Create a large swap file and rely on the operating system to automatically utilize the disk as virtual memory.FastChat-T5Simply run the line below to start chatting.
It will automatically download the weights from a Hugging Facerepo.python3-mfastchat.serve.cli--model-pathlmsys/fastchat-t5-3b-v1.0Inference with Command Line Interface(Experimental Feature: You can specify--style richto enable rich text output and better text streaming quality for some non-ASCII content. This may not work properly on certain terminals.)Supported ModelsThe following models are tested:Vicuna, Alpaca, LLaMA, Koalalmsys/fastchat-t5-3b-v1.0BlinkDL/RWKV-4-Ravendatabricks/dolly-v2-12bOpenAssistant/oasst-sft-1-pythia-12bproject-baize/baize-lora-7BStabilityAI/stablelm-tuned-alpha-7bTHUDM/chatglm-6bSingle GPUThe command below requires around 28GB of GPU memory for Vicuna-13B and 14GB of GPU memory for Vicuna-7B.
See the "No Enough Memory" section below if you do not have enough memory.python3 -m fastchat.serve.cli --model-path /path/to/model/weightsMultiple GPUsYou can use model parallelism to aggregate GPU memory from multiple GPUs on the same machine.python3 -m fastchat.serve.cli --model-path /path/to/model/weights --num-gpus 2CPU OnlyThis runs on the CPU only and does not require GPU. It requires around 60GB of CPU memory for Vicuna-13B and around 30GB of CPU memory for Vicuna-7B.python3 -m fastchat.serve.cli --model-path /path/to/model/weights --device cpuMetal Backend (Mac Computers with Apple Silicon or AMD GPUs)Use--device mpsto enable GPU acceleration on Mac computers (requires torch >= 2.0).
Use--load-8bitto turn on 8-bit compression.python3 -m fastchat.serve.cli --model-path /path/to/model/weights --device mps --load-8bitVicuna-7B can run on a 32GB M1 Macbook with 1 - 2 words / second.No Enough MemoryIf you do not have enough memory, you can enable 8-bit compression by adding--load-8bitto commands above.
This can reduce memory usage by around half with slightly degraded model quality.
It is compatible with the CPU, GPU, and Metal backend.
Vicuna-13B with 8-bit compression can run on a single NVIDIA 3090/4080/T4/V100(16GB) GPU.python3 -m fastchat.serve.cli --model-path /path/to/model/weights --load-8bitIn addition to that, you can add--cpu-offloadingto commands above to offload weights that don't fit on your GPU onto the CPU memory. This requires 8-bit compression to be enabled and the bitsandbytes package to be installed, which is only available on linux operating systems.More PlatformsMLC LLM, backed byTVM Unitycompiler, deploys Vicuna natively on phones, consumer-class GPUs and web browsers via Vulkan, Metal, CUDA and WebGPU.Serving with Web GUITo serve using the web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the webserver and model workers. Here are the commands to follow in your terminal:Launch the controllerpython3-mfastchat.serve.controllerThis controller manages the distributed workers.Launch the model workerpython3-mfastchat.serve.model_worker--model-path/path/to/model/weightsWait until the process finishes loading the model and you see "Uvicorn running on ...". You can launch multiple model workers to serve multiple models concurrently. The model worker will connect to the controller automatically.To ensure that your model worker is connected to your controller properly, send a test message using the following command:python3-mfastchat.serve.test_message--model-namevicuna-7bYou will see a short output.Launch the Gradio web serverpython3-mfastchat.serve.gradio_web_serverThis is the user interface that users will interact with.By following these steps, you will be able to serve your models using the web UI. You can open your browser and chat with a model now.APIHuggingface Generation APIsSeefastchat/serve/huggingface_api.pyOpenAI-compatible RESTful APIs & SDK(Experimental. We will keep improving the API and SDK.)Chat CompletionReference:https://platform.openai.com/docs/api-reference/chat/createSome features/compatibilities to be implemented:streamingsupport of some parameters liketop_p,presence_penaltyproper error handling (e.g. model not found)the return value in the client SDK could be used like a dictRESTful API ServerFirst, launch the controllerpython3-mfastchat.serve.controllerThen, launch the model worker(s)python3-mfastchat.serve.model_worker--model-name'vicuna-7b-v1.1'--model-path/path/to/vicuna/weightsFinally, launch the RESTful API serverexportFASTCHAT_CONTROLLER_URL=http://localhost:21001
python3-mfastchat.serve.api--hostlocalhost--port8000Test the API servercurlhttp://localhost:8000/v1/chat/completions\-H"Content-Type: application/json"\-d'{"model": "vicuna-7b-v1.1","messages": [{"role": "user", "content": "Hello!"}]}'Client SDKAssuming environment variableFASTCHAT_BASEURLis set to the API server URL (e.g.,http://localhost:8000), you can use the following code to send a request to the API server:importosfromfastchatimportclientclient.set_baseurl(os.getenv("FASTCHAT_BASEURL"))completion=client.ChatCompletion.create(model="vicuna-7b-v1.1",messages=[{"role":"user","content":"Hello!"}])print(completion.choices[0].message)EvaluationOur AI-enhanced evaluation pipeline is based on GPT-4. This section provides a high-level summary of the pipeline. For detailed instructions, please refer to theevaluationdocumentation.Pipeline StepsGenerate answers from different models: Useqa_baseline_gpt35.pyfor ChatGPT, or specify the model checkpoint and runget_model_answer.pyfor Vicuna and other models.Generate reviews with GPT-4: Use GPT-4 to generate reviews automatically. This step can also be performed manually if the GPT-4 API is not available to you.Generate visualization data: Rungenerate_webpage_data_from_table.pyto generate data for a static website, which allows you to visualize the evaluation data.Visualize the data: Serve a static website under thewebpagedirectory. You can usepython3 -m http.serverto serve the website locally.Data Format and ContributionWe use a data format encoded with JSON Lines for evaluation. The format includes information on models, prompts, reviewers, questions, answers, and reviews.You can customize the evaluation process or contribute to our project by accessing the relevantdata.For detailed instructions, please refer to theevaluationdocumentation.Fine-tuningDataVicuna is created by fine-tuning a LLaMA base model using approximately 70K user-shared conversations gathered from ShareGPT.com with public APIs. To ensure data quality, we convert the HTML back to markdown and filter out some inappropriate or low-quality samples. Additionally, we divide lengthy conversations into smaller segments that fit the model's maximum context length. For detailed instructions to clean the ShareGPT data, check outhere.Due to some concerns, we may not release the ShareGPT dataset at the moment. If you would like to try the fine-tuning code, you can run it with some dummy questions indummy.json. You can follow the same format and plug in your own data.Code and HyperparametersOur code is based onStanford Alpacawith additional support for multi-round conversations.
We use similar hyperparameters as the Stanford Alpaca.HyperparameterGlobal Batch SizeLearning rateEpochsMax lengthWeight decayVicuna-13B1282e-5320480Fine-tuning Vicuna-7B with Local GPUsYou can use the following command to train Vicuna-7B with 4 x A100 (40GB).torchrun--nproc_per_node=4--master_port=20001fastchat/train/train_mem.py\--model_name_or_path~/model_weights/llama-7b\--data_pathplayground/data/dummy.json\--bf16True\--output_diroutput\--num_train_epochs3\--per_device_train_batch_size2\--per_device_eval_batch_size2\--gradient_accumulation_steps16\--evaluation_strategy"no"\--save_strategy"steps"\--save_steps1200\--save_total_limit10\--learning_rate2e-5\--weight_decay0.\--warmup_ratio0.03\--lr_scheduler_type"cosine"\--logging_steps1\--fsdp"full_shard auto_wrap"\--fsdp_transformer_layer_cls_to_wrap'LlamaDecoderLayer'\--tf32True\--model_max_length2048\--gradient_checkpointingTrue\--lazy_preprocessTrueIf you meet out-of-memory during model saving, see solutionshere.Fine-tuning on Any Cloud with SkyPilotSkyPilotis a framework built by UC Berkeley for easily and cost effectively running ML workloads on any cloud (AWS, GCP, Azure, Lambda, etc.).
To use SkyPilot, install it with the following command and setup the cloud credentials locally following the instructionshere.# Install skypilot from the master branchpipinstallgit+https://github.com/skypilot-org/skypilot.gitVicunaVicuna can be trained on 8 A100 GPUs with 80GB memory. The following command will automatically launch a node satisfying the requirement, setup and run the training job on it.skylaunch-cvicuna-sscripts/train-vicuna.yaml--envWANDB_API_KEYOther options are also valid:# Launch it on managed spot to save 3x cost (train Vicuna-13B with around $300)skyspotlaunch-nvicunascripts/train-vicuna.yaml--envWANDB_API_KEY# Train a 7B modelskylaunch-cvicuna-sscripts/train-vicuna.yaml--envWANDB_API_KEY--envMODEL_SIZE=7Note: Please make sure theWANDB_API_KEYhas been setup on your local machine. You can find the API key on yourwandb profile page. If you would like to train the model without using wandb, you can replace the--env WANDB_API_KEYflag with--env WANDB_MODE=offline.