Package: llama-cpp-haystack
llama-cpp-haystackCustom component forHaystack(2.x) for running LLMs using theLlama.cppLLM framework. This implementation leverages thePython Bindings for llama.cpp.Table of ContentsInstallationUsageExampleLicenseInstallationpipinstallllama-cpp-haystackThe default install behaviour is to buildllama.cppfor CPU only on Linux and Windows and use Metal on MacOS.To install using the other backends, first installllama-cpp-pythonusing the instructions on theirinstallation documentationand then installllama-cpp-haystack.For example, to usellama-cpp-haystackwith the cuBLAS backend:exportLLAMA_CUBLAS=1CMAKE_ARGS="-DLLAMA_CUBLAS=on"pipinstallllama-cpp-python
pipinstallllama-cpp-haystackUsageYou can utilize theLlamaCppGeneratorto load models quantized using llama.cpp (GGUF) for text generation.Information about the supported models and model parameters can be found on the llama.cppdocumentation.The GGUF versions of popular models can be downloaded fromHuggingFace.Passing additional model parametersThemodel_path,n_ctx,n_batcharguments have been exposed for convenience and can be directly passed to the Generator during initialization as keyword arguments.Themodel_kwargsparameter can be used to pass additional arguments when initializing the model. In case of duplication, these kwargs overridemodel_path,n_ctx, andn_batchinit parameters.See Llama.cpp'smodel documentationfor more information on the available model arguments.For example, to offload the model to GPU during initialization:fromhaystack_integrations.components.generators.llama_cppimportLlamaCppGeneratorgenerator=LlamaCppGenerator(model_path="/content/openchat-3.5-1210.Q3_K_S.gguf",n_ctx=512,n_batch=128,model_kwargs={"n_gpu_layers":-1})generator.warm_up()input="Who is the best American actor?"prompt=f"GPT4 Correct User:{input}
 GPT4 Correct Assistant:"result=generator.run(prompt,generation_kwargs={"max_tokens":128})generated_text=result["replies"][0]print(generated_text)Passing generation parametersThegeneration_kwargsparameter can be used to pass additional generation arguments likemax_tokens,temperature,top_k,top_p, etc to the model during inference.See Llama.cpp'screate_completiondocumentationfor more information on the available generation arguments.For example, to set themax_tokensandtemperature:fromhaystack_integrations.components.generators.llama_cppimportLlamaCppGeneratorgenerator=LlamaCppGenerator(model_path="/content/openchat-3.5-1210.Q3_K_S.gguf",n_ctx=512,n_batch=128,generation_kwargs={"max_tokens":128,"temperature":0.1},)generator.warm_up()input="Who is the best American actor?"prompt=f"GPT4 Correct User:{input}
 GPT4 Correct Assistant:"result=generator.run(prompt)generated_text=result["replies"][0]print(generated_text)Thegeneration_kwargscan also be passed to therunmethod of the generator directly:fromhaystack_integrations.components.generators.llama_cppimportLlamaCppGeneratorgenerator=LlamaCppGenerator(model_path="/content/openchat-3.5-1210.Q3_K_S.gguf",n_ctx=512,n_batch=128,)generator.warm_up()input="Who is the best American actor?"prompt=f"GPT4 Correct User:{input}
 GPT4 Correct Assistant:"result=generator.run(prompt,generation_kwargs={"max_tokens":128,"temperature":0.1},)generated_text=result["replies"][0]print(generated_text)ExampleBelow is the example Retrieval Augmented Generation pipeline that uses theSimple WikipediaDataset from HuggingFace. You can find more examples in theexamplesfolder.Load the dataset:# Install HuggingFace Datasets using "pip install datasets"fromdatasetsimportload_datasetfromhaystackimportDocument,Pipelinefromhaystack.components.builders.answer_builderimportAnswerBuilderfromhaystack.components.builders.prompt_builderimportPromptBuilderfromhaystack.components.embeddersimportSentenceTransformersDocumentEmbedder,SentenceTransformersTextEmbedderfromhaystack.components.retrieversimportInMemoryEmbeddingRetrieverfromhaystack.components.writersimportDocumentWriterfromhaystack.document_storesimportInMemoryDocumentStore# Import LlamaCppGeneratorfromhaystack_integrations.components.generators.llama_cppimportLlamaCppGenerator# Load first 100 rows of the Simple Wikipedia Dataset from HuggingFacedataset=load_dataset("pszemraj/simple_wikipedia",split="validation[:100]")docs=[Document(content=doc["text"],meta={"title":doc["title"],"url":doc["url"],},)fordocindataset]Index the documents to theInMemoryDocumentStoreusing theSentenceTransformersDocumentEmbedderandDocumentWriter:doc_store=InMemoryDocumentStore(embedding_similarity_function="cosine")doc_embedder=SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")# Indexing Pipelineindexing_pipeline=Pipeline()indexing_pipeline.add_component(instance=doc_embedder,name="DocEmbedder")indexing_pipeline.add_component(instance=DocumentWriter(document_store=doc_store),name="DocWriter")indexing_pipeline.connect(connect_from="DocEmbedder",connect_to="DocWriter")indexing_pipeline.run({"DocEmbedder":{"documents":docs}})Create the Retrieval Augmented Generation (RAG) pipeline and add theLlamaCppGeneratorto it:# Prompt Template for the https://huggingface.co/openchat/openchat-3.5-1210 LLMprompt_template="""GPT4 Correct User: Answer the question using the provided context.Question: {{question}}Context:{% for doc in documents %}{{ doc.content }}{% endfor %}
GPT4 Correct Assistant:"""rag_pipeline=Pipeline()text_embedder=SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")# Load the LLM using LlamaCppGeneratormodel_path="openchat-3.5-1210.Q3_K_S.gguf"generator=LlamaCppGenerator(model_path=model_path,n_ctx=4096,n_batch=128)rag_pipeline.add_component(instance=text_embedder,name="text_embedder",)rag_pipeline.add_component(instance=InMemoryEmbeddingRetriever(document_store=doc_store,top_k=3),name="retriever")rag_pipeline.add_component(instance=PromptBuilder(template=prompt_template),name="prompt_builder")rag_pipeline.add_component(instance=generator,name="llm")rag_pipeline.add_component(instance=AnswerBuilder(),name="answer_builder")rag_pipeline.connect("text_embedder","retriever")rag_pipeline.connect("retriever","prompt_builder.documents")rag_pipeline.connect("prompt_builder","llm")rag_pipeline.connect("llm.replies","answer_builder.replies")rag_pipeline.connect("retriever","answer_builder.documents")Run the pipeline:question="Which year did the Joker movie release?"result=rag_pipeline.run({"text_embedder":{"text":question},"prompt_builder":{"question":question},"llm":{"generation_kwargs":{"max_tokens":128,"temperature":0.1}},"answer_builder":{"query":question},})generated_answer=result["answer_builder"]["answers"][0]print(generated_answer.data)# The Joker movie was released on October 4, 2019.Licensellama-cpp-haystackis distributed under the terms of theApache-2.0license.