Package: pyllama
🦙 LLaMA - Run LLM in A Single 4GB GPU📢pyllamais a hacked version ofLLaMAbased on original Facebook's implementation but more convenient to run in a Single consumer grade GPU.The Hugging Face's LLaMA implementation is available atpyllama.hf.📥 InstallationIn a conda env with pytorch / cuda available, runpip install pyllama -U🐏 If you have installed llama library from other sources, please uninstall the previous llama library and usepip install pyllama -Uto install the latest version.📦 Download Model Files🧘‍♀️ Official WayIn order to download the checkpoints and tokenizer, fill thisgoogle formOnce your request is approved, you will receive links to download the tokenizer and model files.
Edit thedownload.shscript with the signed url provided in the email to download the model weights and tokenizer.🐒 Community WaypyllamaThere is another high-speed way to download the checkpoints and tokenizers. There are four models(7B,13B,30B,65B) available. To download all of them, run:python-mllama.downloadTo download only the 7B model files to your current directory, run:python-mllama.download--model_size7BTo download only the 7B and 30B model files to folder/tmp/pyllama_data, run:python-mllama.download--model_size7B,30B--folder/tmp/pyllama_dataThe help doc is:$python-mllama.download--help
usage:download.py[-h][--model_sizeMODEL_SIZE][--folderFOLDER]optionalarguments:-h,--helpshowthishelpmessageandexit--model_sizeMODEL_SIZEThesizeofthemodelsthatyouwanttodownload.Acommaseparatedstringofanyof"7B","13B","30B","65B".Totally219Gdiskspaceisneededtodownloadthemall.Ifyouonlywanttodownloadthe7Bmodel,justput"7B"here.--folderFOLDERThetargetfolderforthedownloadfilesSample ScreenshotBittorrent🔥 In order to download the checkpoints and tokenizer, use this BitTorrent link: "magnet:?xt=urn:btih:ZXXDAUWYLRUXXBHUYEMS6Q5CE5WA3LVA&dn=LLaMA".💎 Quantize LLaMA to run in a 4GB GPUpyllamasupport quantization of 2/3/4/8-bit so that you can run model in a 4G memory GPU.You need to runexport HUGGING_FACE_HUB_TOKEN=XXXto be able to access Hugging Face's data. You also need to installgptqwith commandpip install gptq.python-mllama.llama_quant--help
usage:llama_quant.py[-h][--ckpt_dirCKPT_DIR][--tokenizer_pathTOKENIZER_PATH][--seedSEED][--nsamplesNSAMPLES][--percdampPERCDAMP][--nearest][--wbits{2,3,4,8,16}][--groupsizeGROUPSIZE][--saveSAVE][--loadLOAD][--benchmarkBENCHMARK][--check][--cudaCUDA][--eval]{wikitext2,ptb,c4}positionalarguments:{wikitext2,ptb,c4}Wheretoextractcalibrationdatafrom.

optionalarguments:-h,--helpshowthishelpmessageandexit--ckpt_dirCKPT_DIR--tokenizer_pathTOKENIZER_PATH--seedSEEDSeedforsamplingthecalibrationdata.--nsamplesNSAMPLESNumberofcalibrationdatasamples.--percdampPERCDAMPPercentoftheaverageHessiandiagonaltousefordampening.--nearestWhethertoruntheRTNbaseline.--wbits{2,3,4,8}bitsforquauntization--groupsizeGROUPSIZEGroupsizetouseforquantization;defaultusesfullrow.--saveSAVESavequantizedcheckpointunderthisname,egpyllama-7B4b.pt.--loadLOADLoadquantizedmodel.--benchmarkBENCHMARKNumberoftokenstouseforbenchmarking.--checkWhethertocomputeperplexityduringbenchmarkingforverification.--cudaCUDAGPUdevicestring,'cuda:0'bydefault.--evalEvaluatethemodelwithdatasetwikitext2,ptbandc4Quantize 7B model to 8-bitpython-mllama.llama_quantdecapoda-research/llama-7b-hfc4--wbits8--savepyllama-7B8b.ptQuantize 7B model to 4-bit with groupsize 128 (the recommended setup 🔥)python-mllama.llama_quantdecapoda-research/llama-7b-hfc4--wbits4--groupsize128--savepyllama-7B4b.ptQuantize 7B model to 2-bitpython-mllama.llama_quantdecapoda-research/llama-7b-hfc4--wbits2--savepyllama-7B2b.ptThe download links for quantized LLaMA files are below:7BQuant TypeSizeLinkMD5LossPassword2-bit2160484475🔗4c7215d28c1f650218c43fc46402cec5-8g9d3-bit-----4-bit3779485819-cce9a3b522ddf5c011ee0174b2ff3dfb--8-bit7017493231-2648b09597cf8f9e0d1a04cb70b71cab--It took me 2 hours 40 mins to quantize the 65B model to 4bit. The file size is reduced from 122GB to 32GB.The following suggestions are recommended for LLM qunatization:By default, use 4-bit quantization for LLM inference as it offers the total model bits and zero-shot accuracy trade-offs.Use a block size of 128 or lower to stabilize 4-bit quantization and improve zero-shot performance.Use a floating point or quantile quantization data type. In some cases, integer data types might be preferable to improve inference latency depending on the implementation and hardware support.🔮 Single GPU Inference🥥 Without QuantizationSet the environment variablesCKPT_DIRas your llamm model folder, for example/llama_data/7B, andTOKENIZER_PATHas your tokenizer's path, such as/llama_data/tokenizer.model.And then run the following command:pythoninference.py--ckpt_dir$CKPT_DIR--tokenizer_path$TOKENIZER_PATHThe following is an example of LLaMA running in a 8GB single GPU.🥝 With QuantizationWith quantization, you can run LLaMA with a 4GB memory GPU.pyllama can run 7B model with 6GB GPU memory.pyllama can run 7B model with 3.2GB GPU memory.💡 TipsTo load KV cache in CPU, runexport KV_CAHCHE_IN_GPU=0in the shell.To profile CPU/GPU/Latency, run:pythoninference_driver.py--ckpt_dir$CKPT_DIR--tokenizer_path$TOKENIZER_PATHA sample result is like:Tunemax_seq_lenandmax_batch_sizeto reduce memory consumption to be able to run in GPU. Refer to:this post!🍉 Start a gradio webui$cdapps/gradio
$pythonwebapp_single.py--ckpt_dir$CKPT_DIR--tokenizer_path$TOKENIZER_PATHYou should see something like this in your browser:🍓 Start a web serverThe following command will start a flask web server:$cdapps/flask
$pythonweb_server_single.py--ckpt_dir$CKPT_DIR--tokenizer_path$TOKENIZER_PATH🍒 Multiple GPU Inference🧘‍♀️ Official WayTo use the original META's model parallel, please set environment variablePYLLAMA_META_MPlike:export PYLLAMA_META_MP=1With this environment variable set, you canimport llamaand the original META version's llama will be imported.The providedexample.pycan be run on a single or multi-gpu node withtorchrunand will output completions for two pre-defined prompts. UsingTARGET_FOLDERas defined indownload.sh:torchrun--nproc_per_nodeMPexample.py--ckpt_dir$TARGET_FOLDER/model_size\--tokenizer_path$TARGET_FOLDER/tokenizer.modelDifferent models require different MP values:ModelMP7B113B230B465B8🐒 Community WayThere are two steps to run LLaMA in multi-GPU environment.Convert original LLaMA model$python-mllama.convert_llama--help
usage:convert_llama.py[-h][--ckpt_dirCKPT_DIR][--tokenizer_pathTOKENIZER_PATH][--model_size{7B,13B,30B,65B}][--output_dirOUTPUT_DIR][--max_batch_sizeMAX_BATCH_SIZE][--to{hf,fb}]optionalarguments:-h,--helpshowthishelpmessageandexit--ckpt_dirCKPT_DIR--tokenizer_pathTOKENIZER_PATH--model_size{7B,13B,30B,65B}--output_dirOUTPUT_DIRLocationtowriteHFmodelandtokenizer--max_batch_sizeMAX_BATCH_SIZE--to{hf,fb}Run with HF's accelerate with multiple GPUs$python-mllama.llama_multigpu--help
usage:llama_multigpu.py[-h][--state_dict_dirSTATE_DICT_DIR][--model_size{7B,13B,30B,65B}]optionalarguments:-h,--helpshowthishelpmessageandexit--state_dict_dirSTATE_DICT_DIR--model_size{7B,13B,30B,65B}🔬 Model Fine TuningWithStandford AlpacaInstruction-Following DatasetTokenizationFinetuningEfficient FT🧬 LLaMA model structureMetaHugging Facehttps://github.com/facebookresearch/llama/blob/main/llama/model.py#LL127C27-L127C27Model CardSeeMODEL_CARD.mdLicenseSee theLICENSEfile.