Package: litgpt
⚡ Lit-GPT⚡ Lit-GPTHackableimplementationof state-of-the-art open-source large language models released under theApache 2.0 license.Supports the following popular model checkpoints:ModelModel sizeReferenceCode Llamaby Meta AI7B, 13B, 34B, 70BRozière et al. 2023Dollyby Databricks3B, 7B, 12BConover et al. 2023Falconby TII UAE7B, 40B, 180BTII 2023FreeWilly2(Stable Beluga 2) by Stability AI70BStability AI 2023Function Calling Llama 2by Trelis7BTrelis et al. 2023Gemmaby Google2B, 7BGoogle Team, Google DeepmindLlama 2by Meta AI7B, 13B, 70BTouvron et al. 2023LongChatby LMSYS7B, 13BLongChat Team 2023Mistral and Mixtralby Mistral AI7BMistral websiteNous-Hermesby NousResearch7B, 13B, 70BOrg pageOpenLLaMAby OpenLM Research3B, 7B, 13BGeng & Liu 2023Phiby Microsoft Research1.3B, 2.7BLi et al. 2023Platypusby Lee at el.7B, 13B, 70BLee, Hunter, and Ruiz 2023Pythiaby EleutherAI{14,31,70,160,410}M, {1,1.4,2.8,6.9,12}BBiderman et al. 2023RedPajama-INCITEby Together3B, 7BTogether 2023StableCodeby Stability AI3BStability AI 2023StableLMby Stability AI3B, 7BStability AI 2023StableLM Zephyrby Stability AI3BStability AI 2023TinyLlamaby Zhang et al.1.1BZhang et al. 2023Vicunaby LMSYS7B, 13B, 33BLi et al. 2023This implementation extends onLit-LLaMAandnanoGPT, and it'spowered byLightning Fabric⚡.🏆 NeurIPS 2023 Large Language Model Efficiency Challenge: 1 LLM + 1 GPU + 1 DayThe Lit-GPT repository was the official starter kit for theNeurIPS 2023 LLM Efficiency Challenge, which is a competition focused on finetuning an existing non-instruction tuned LLM for 24 hours on a single GPU.Lit-GPT design principlesThis repository follows the main principle ofopenness through clarity.Lit-GPTis:Simple:Single-file implementation without boilerplate.Correct:Numerically equivalent to the original model.Optimized:Runs fast on consumer hardware or at scale.Open-source:No strings attached.Avoiding code duplication isnota goal.Readabilityandhackabilityare.Get involved!Join our Discordto build high-performance, truly open-source models for the common benefit of the community.SetupClone the repo:gitclonehttps://github.com/Lightning-AI/lit-gptcdlit-gptInstall with all dependencies (including CLI, quantization, tokenizers for all models, etc.):pipinstall-rrequirements-all.txtUse the modelTo generate text predictions, you need to download the model weights.If you don't have them, check out ourguide.Run inference:pythongenerate/base.py--prompt"Hello, my name is"This will run the 3B pretrained model and require ~7 GB of GPU memory using thebfloat16datatype.Full guide for generating samples from the model.You can also chat with the model interactively:pythonchat/base.pyRun large models on smaller consumer devicesWe support 4-bit quantization (as in QLoRA), (bnb.nf4, bnb.nf4-dq, bnb.fp4, bnb.fp4-dq) and 8-bit quantization (bnb.int8) for inference by followingthis guide.Finetune the modelWe provide a simple training scripts (finetune/adapter.py,finetune/adapter_v2.py, andfinetune/lora.py) that instruction-tunes a pretrained model on theAlpacadataset.Download the data and generate an instruction tuning dataset:pythonscripts/prepare_alpaca.pyRun the finetuning scriptFor example, you can either useAdapter (Zhang et al. 2023):pythonfinetune/adapter.pyor Adapter v2 (Gao et al. 2023):pythonfinetune/adapter_v2.pyor LoRA (Hu et al. 2021):pythonfinetune/lora.py(Please see thetutorials/finetune_adapterfor details on the differences between the two adapter methods.)The finetuning requires at least one GPU with ~12 GB memory (RTX 3060).It is expected that you have downloaded the pretrained weights as described above.
More details about each finetuning method and how you can apply it to your own data can be found in our technical how-to guides.Finetuning how-to guidesThese technical tutorials illustrate how to run the finetuning code.Finetune with AdaptersFinetune with LoRA or QLoRAUnderstanding finetuning -- conceptual tutorialsLooking for conceptual tutorials and explanations? We have some additional articles below:Understanding Parameter-Efficient Finetuning of Large Language Models: From Prefix Tuning to LLaMA-AdaptersParameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA)PretrainingWe provide simple training scripts based on Fabric if you want to venture into pretraining. Conversion scripts for our optimized streamingPackedDatasetare included.Follow this guide to start pretraining onRedPajama, a reproduction of LLaMA's training setOpenWeb Text, a reproduction of GPT-2's datasetSupported datasetsLit-GPT includes a variety of dataset preparation scripts for finetuning and pretraining. Additional information about the datasets and dataset preparation is provided in thePreparing Datasetstutorial.XLALightning AI has partnered with Google to add first-class support forCloud TPUsinLightning’s frameworksand Lit-GPT,
helping democratize AI for millions of developers and researchers worldwide.Using TPUs with Lightning is as straightforward as changing one line of code.We provide scripts fully optimized for TPUs in theXLA directoryGet involved!We are on a quest towards fully open source AI.Join us and start contributing, especially on the following areas:PretrainingFine-tuningQuantizationSparsificationWe welcome all individual contributors, regardless of their level of experience or hardware. Your contributions are valuable, and we are excited to see what you can accomplish in this collaborative and supportive environment.Unsure about contributing? Check out ourHow to Contribute to Lit-GPT and Lit-LLaMAguide.Don't forget tojoin our Discord!Acknowledgements@karpathyfornanoGPT@EleutherAIforGPT-NeoXand theEvaluation Harness@TimDettmersforbitsandbytes@MicrosoftforLoRA@tridaoforFlash Attention 2CitationIf you use Lit-GPT in your research, please cite the following work:@misc{lit-gpt-2023,author={Lightning AI},title={Lit-GPT},howpublished={\url{https://github.com/Lightning-AI/lit-gpt}},year={2023},}LicenseLit-GPT is released under theApache 2.0license.