Package: llm_atc
LLM-ATC (AirTrafficController) is a CLI for fine tuning and serving open source models using your own cloud credentials. We hope that this project can lower the cognitive overhead of orchestration for fine tuning and model serving.Refer to the docs for the most up to date usage information. This README is updated less frequentlyInstallationFollow the instructions hereto install Skypilot and provide cloud credentials. We use Skypilot for cloud orchestration. Steps to setup an environment is shown below.# create a fresh environmentcondacreate-n"sky"python=3.10condaactivatesky# install llm-atcpipinstallllm-atc# For Macs, macOS >= 10.15 has a conflict with grpciopipuninstallgrpcio;condainstall-cconda-forgegrpcio=1.43.0--force-reinstall# Configure your cloud credentials. This is a GCP example. See https://skypilot.readthedocs.io/en/latest/getting-started/ installation.html for examples with other cloud providers.pipinstallgoogle-api-python-client
condainstall-cconda-forgegoogle-cloud-sdk
gcloudinit
gcloudauthapplication-defaultlogin# double check that your credentials are properly set for your desired provider(s)skycheckFrom PyPipipinstallllm-atcFrom sourcepipinstall-e.FinetuningSupported fine-tune methods.Vicuna-Llama (chat-finetuning)To start finetuning a model. Usellm-atc train. For example# start trainingllm-atctrain--model_typevicuna--finetune_data./vicuna_test.json--namemyvicuna--description"This is a finetuned model that just says its name is vicuna"-cmycluster--cloudgcp--envs"MODEL_SIZE=7 WANDB_API_KEY=<my wandb key>"--acceleratorA100-80G:4# shutdown cluster when doneskydownmyclusterIf your client disconnects from the train, the train run will continue. You can check it's status withsky queue myclusterWhen training completes, by default, your model, will be saved to an object store corresponding to the cloud provider which launched the training instance. For instance,# s3 location
s3://llm-atc/myvicuna
# gcp location
g3://llm-atc/myvicunaServingllm-atccan serve both models from HuggingFace or that you've trained throughllm-atc serve. For example# serve an llm-atc finetuned model, requires `llm-atc/` prefix and grabs model checkpoint from object storellm-atcserve--namellm-atc/myvicuna--acceleratorA100:1-cservecluster--cloudgcp--regionasia-southeast1--envs"HF_TOKEN=<HuggingFace_token>"# serve a HuggingFace model, e.g. `lmsys/vicuna-13b-v1.3`llm-atcserve--namelmsys/vicuna-13b-v1.3--acceleratorA100:1-cservecluster--cloudgcp--regionasia-southeast1--envs"HF_TOKEN=<HuggingFace_token>"This creates a OpenAI API server on port 8000 of the cluster head and one model worker.
Make a request from your laptop with.# get the ip address of the OpenAI serverip=$(grep-A1"Host servecluster"~/.ssh/config|grep"HostName"|awk'{print $2}')# test which models are availablecurlhttp://$ip:8000/v1/models# stop model server clusterskystopserveclusterand you can connect to this server and
develop your using your finetuned models with other LLM frameworks likeLlamaIndex. Look atexamples/to see how to interact with your API endpoint.TelemetryBy default, LLM-ATC collects anonymized data about when a train or serve request is made with PostHog. Telemetry helps us identify where users are engaging with LLM-ATC. However, if you would like to disable telemetry, setexportLLM_ATC_DISABLE=1How does it work?Training, serving, and orchestration are powered bySkyPilot,FastChat, andvLLM. We've made this decision since we believe this will allow people to train and deploy custom LLMs without cloud-lockin.We currently rely on default hyperparameters from other training code repositories, but we will add options to overwrite these so that users have more control over training, but for now, we think the defaults should suffice for most use cases.