Package: petals
Run large language models at home, BitTorrent-style.Fine-tuning and inferenceup to 10x fasterthan offloadingGenerate text with distributedLlama 2(70B),Falcon(40B+),BLOOM(176B) (or their derivatives), and fine‑tune them for your own tasks — right from your desktop computer or Google Colab:fromtransformersimportAutoTokenizerfrompetalsimportAutoDistributedModelForCausalLM# Choose any model available at https://health.petals.devmodel_name="petals-team/StableBeluga2"# This one is fine-tuned Llama 2 (70B)# Connect to a distributed network hosting model layerstokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoDistributedModelForCausalLM.from_pretrained(model_name)# Run the model as if it were on your computerinputs=tokenizer("A cat sat",return_tensors="pt")["input_ids"]outputs=model.generate(inputs,max_new_tokens=5)print(tokenizer.decode(outputs[0]))# A cat sat on a mat...🚀Try now in Colab🔏Privacy.Your data will be processed with the help of other people in the public swarm. Learn more about privacyhere. For sensitive data, you can set up aprivate swarmamong people you trust.🦙Want to run Llama 2?Request access to its weights at the ♾️Meta AI websiteand 🤗Model Hub, then runhuggingface-cli loginin the terminal before loading the model. Or just try it in ourchatbot app.💬Any questions?Ping us inour Discord!Connect your GPU and increase Petals capacityPetals is a community-run system — we rely on people sharing their GPUs. You can check outavailable modelsand help serving one of them! As an example, here is how to host a part ofStable Beluga 2on your GPU:🐧Linux + Anaconda.Run these commands for NVIDIA GPUs (or followthisfor AMD):condainstallpytorchpytorch-cuda=11.7-cpytorch-cnvidia
pipinstallgit+https://github.com/bigscience-workshop/petals
python-mpetals.cli.run_serverpetals-team/StableBeluga2🪟Windows + WSL.Followthis guideon our Wiki.🐋Docker.Run ourDockerimage for NVIDIA GPUs (or followthisfor AMD):sudodockerrun-p31330:31330--ipchost--gpusall--volumepetals-cache:/cache--rm\learningathome/petals:main\python-mpetals.cli.run_server--port31330petals-team/StableBeluga2🍏macOS + Apple M1/M2 GPU.InstallHomebrew, then run these commands:brewinstallpython
python3-mpipinstallgit+https://github.com/bigscience-workshop/petals
python3-mpetals.cli.run_serverpetals-team/StableBeluga2📚Learn more(how to use multiple GPUs, start the server on boot, etc.)💬Any questions?Ping us inour Discord!🦙Want to host Llama 2?Request access to its weights at the ♾️Meta AI websiteand 🤗Model Hub, generate an 🔑access token, then add--token YOUR_TOKEN_HEREto thepython -m petals.cli.run_servercommand.🔒Security.Hosting a server does not allow others to run custom code on your computer. Learn morehere.🏆Thank you!Once you load and host 10+ blocks, we can show your name or link on theswarm monitoras a way to say thanks. You can specify them with--public_name YOUR_NAME.How does it work?You load a small part of the model, then join anetworkof people serving the other parts. Single‑batch inference runs at up to6 tokens/secforLlama 2(70B) and up to4 tokens/secforFalcon(180B) — enough forchatbotsand interactive apps.You can employ any fine-tuning and sampling methods, execute custom paths through the model, or see its hidden states. You get the comforts of an API with the flexibility ofPyTorchand🤗 Transformers.📜Read paper📚See FAQ📚 Tutorials, examples, and moreBasic tutorials:Getting started:tutorialPrompt-tune Llama-65B for text semantic classification:tutorialPrompt-tune BLOOM to create a personified chatbot:tutorialUseful tools:Chatbot web app(connects to Petals via an HTTP/WebSocket endpoint):source codeMonitorfor the public swarm:source codeAdvanced guides:Launch a private swarm:guideRun a custom model:guideBenchmarksPlease seeSection 3.3of ourpaper.🛠️ ContributingPlease see ourFAQon contributing.📜 CitationAlexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel.Petals: Collaborative Inference and Fine-tuning of Large Models.arXiv preprint arXiv:2209.01188,2022.@article{borzunov2022petals,title={Petals: Collaborative Inference and Fine-tuning of Large Models},author={Borzunov, Alexander and Baranchuk, Dmitry and Dettmers, Tim and Ryabinin, Max and Belkada, Younes and Chumachenko, Artem and Samygin, Pavel and Raffel, Colin},journal={arXiv preprint arXiv:2209.01188},year={2022},url={https://arxiv.org/abs/2209.01188}}This project is a part of theBigScienceresearch workshop.