Package: pyllamacpp
PyLLaMACppPython bindings forllama.cpp.If you are looking to runFalconmodels, take a look at theggllm branch.For those who don't know,llama.cppis a port of Facebook's LLaMA model in pure C/C++:Without dependenciesApple silicon first-class citizen - optimized via ARM NEONAVX2 support for x86 architecturesMixed F16 / F32 precision4-bit quantization supportRuns on the CPUTable of contentsInstallationCLITutorialQuick startInteractive DialogueAttribute a persona to the language modelExample usage with langchainSupported modelsAdvanced usageAPI referenceFAQsDiscussions and contributionsLicenseInstallationThe easy way is to install the prebuilt wheelspipinstallpyllamacppHowever, the compilation process ofllama.cppis taking into account the architecture of the targetCPU,
so you might need to build it from source:pipinstallgit+https://github.com/abdeladim-s/pyllamacpp.git:warning:NoteThis PRintroduced some breaking changes.
If you want to use older models, use version2.2.0:pipinstallpyllamacpp==2.2.0CLIYou can run the following simple command line interface to test the package once it is installed:pyllamacpppath/to/model.binpyllamacpp-h

usage:pyllamacpp[-h][--n_ctxN_CTX][--n_partsN_PARTS][--seedSEED][--f16_kvF16_KV][--logits_allLOGITS_ALL][--vocab_onlyVOCAB_ONLY][--use_mlockUSE_MLOCK][--embeddingEMBEDDING][--n_predictN_PREDICT][--n_threadsN_THREADS][--repeat_last_nREPEAT_LAST_N][--top_kTOP_K][--top_pTOP_P][--tempTEMP][--repeat_penaltyREPEAT_PENALTY][--n_batchN_BATCH]model

Thisislikeachatbot,Youcanstarttheconversationwith`Hi,canyouhelpme?`Payattentionthoughthatitmayhallucinate!

positionalarguments:modelThepathofthemodelfile

options:-h,--helpshowthishelpmessageandexit--n_ctxN_CTXtextcontext--n_partsN_PARTS--seedSEEDRNGseed--f16_kvF16_KVusefp16forKVcache--logits_allLOGITS_ALLthellama_eval()callcomputesalllogits,notjustthelastone--vocab_onlyVOCAB_ONLYonlyloadthevocabulary,noweights--use_mlockUSE_MLOCKforcesystemtokeepmodelinRAM--embeddingEMBEDDINGembeddingmodeonly--n_predictN_PREDICTNumberoftokenstopredict--n_threadsN_THREADSNumberofthreads--repeat_last_nREPEAT_LAST_NLastntokenstopenalize--top_kTOP_Ktop_k--top_pTOP_Ptop_p--tempTEMPtemp--repeat_penaltyREPEAT_PENALTYrepeat_penalty--n_batchN_BATCHbatchsizeforpromptprocessingTutorialQuick startfrompyllamacpp.modelimportModelmodel=Model(model_path='/path/to/model.bin')fortokeninmodel.generate("Tell me a joke ?\n"):print(token,end='',flush=True)Interactive DialogueYou can set up an interactive dialogue by simply keeping themodelvariable alive:frompyllamacpp.modelimportModelmodel=Model(model_path='/path/to/model.bin')whileTrue:try:prompt=input("You: ",flush=True)ifprompt=='':continueprint(f"AI:",end='')fortokeninmodel.generate(prompt):print(f"{token}",end='',flush=True)print()exceptKeyboardInterrupt:breakAttribute a persona to the language modelThe following is an example showing how to"attribute a persona to the language model":frompyllamacpp.modelimportModelprompt_context="""Act as Bob. Bob is helpful, kind, honest,and never fails to answer the User's requests immediately and with precision.User: Nice to meet you Bob!Bob: Welcome! I'm here to assist you with anything you need. What can I do for you today?"""prompt_prefix="\nUser:"prompt_suffix="\nBob:"model=Model(model_path='/path/to/model.bin',n_ctx=512,prompt_context=prompt_context,prompt_prefix=prompt_prefix,prompt_suffix=prompt_suffix)whileTrue:try:prompt=input("User: ")ifprompt=='':continueprint(f"Bob: ",end='')fortokeninmodel.generate(prompt,antiprompt='User:',n_threads=6,n_batch=1024,n_predict=256,n_keep=48,repeat_penalty=1.0,):print(f"{token}",end='',flush=True)print()exceptKeyboardInterrupt:breakExample usage withlangchainfrompyllamacpp.langchain_llmimportPyllamacppLLMllm=PyllamacppLLM(model="path/to/ggml/model",temp=0.75,n_predict=50,top_p=1,top_k=40)template="\n\n##Instruction:\n:{question}\n\n##Response:\n"prompt=PromptTemplate(template=template,input_variables=["question"])llm_chain=LLMChain(prompt=prompt,llm=llm)question="What are large language models?"answer=llm_chain.run(question)print(answer)Supported modelsAll models supported byllama.cppshould be supported basically:Supported models:LLaMA 🦙AlpacaGPT4AllChinese LLaMA / AlpacaVigogne (French)VicunaKoalaOpenBuddy 🐶 (Multilingual)Pygmalion 7B / Metharme 7BWizardLMAdvanced usageFor advanced users, you can access thellama.cpp C-APIfunctions directly to make your own logic.
All functions fromllama.hare exposed with the binding module_pyllamacpp.API referenceYou can check theAPI reference documentationfor more details.FAQsHow to build pyllamacpp without AVX2 or FMA.pyllamacpp does not support M1 chips MacBookImportError: DLL failed while importing _pyllamacppDiscussions and contributionsIf you find any bug, please open anissue.If you have any feedback, or you want to share how you are using this project, feel free to use theDiscussionsand open a new topic.LicenseThis project is licensed under the same license asllama.cpp(MITLicense).