Package: llama-cpp-agent
llama-cpp-agent FrameworkIntroductionThe llama-cpp-agent framework is a tool designed for easy interaction with Large Language Models (LLMs). Allowing users to chat with LLM models, execute structured function calls and get structured output (objects).It provides a simple yet robust interface and supports llama-cpp-python and OpenAI endpoints with GBNF grammar support (like the llama-cpp-python server) and the llama.cpp backend server.
It works by generating a formal GGML-BNF grammar of the user defined structures and functions, which is then used by llama.cpp to generate text valid to that grammar. In contrast to most GBNF grammar generators it also supports nested objects, dictionaries, enums and lists of them.Key FeaturesSimple Chat Interface: Engage in seamless conversations with LLMs.Structured Output: Get structured output (objects) from LLMs.Single and Parallel Function Calling: Let the LLM execute functions.Flexibility: Suited for various applications from casual chatting to specific function executions.InstallationThe llama-cpp-agent framework can be installed using pip:pipinstallllama-cpp-agentUsage ExamplesThe following examples demonstrate the usage of the llama-cpp-agent framework.
You can find more examples in theexamplesfolder.Simple Chat Example using llama.cpp server backendThis example demonstrates how to initiate a chat with an LLM model using the llama.cpp server backend. The framework supports llama-cpp-python Llama class instances as LLM and OpenAI endpoints that support GBNF grammars as a backend, and the llama.cpp backend server.fromllama_cppimportLlamafromllama_cpp_agent.llm_agentimportLlamaCppAgentfromllama_cpp_agent.messages_formatterimportMessagesFormatterTypefromllama_cpp_agent.providers.llama_cpp_endpoint_providerimportLlamaCppEndpointSettingsmain_model=LlamaCppEndpointSettings(completions_endpoint_url="http://127.0.0.1:8080/completion")# For OpenAI endpoints that support GBNF grammars like the llama-cpp-python server:# from llama_cpp_agent.providers.openai_endpoint_provider import OpenAIEndpointSettings# main_model = OpenAIEndpointSettings(#    "http://localhost:8080/v1/completions"# )# For instances of Llama class of llama-cpp-python# from llama_cpp import Llama# main_model = Llama(#     "../../gguf-models/openhermes-2.5-mistral-7b.Q8_0.gguf",#     n_gpu_layers=35,#     f16_kv=True,#     use_mlock=False,#     embedding=False,#     n_threads=8,#     n_batch=1024,#     n_ctx=8192,#     offload_kqv=True,#     last_n_tokens_size=1024,#     verbose=True,#     seed=-1,# )llama_cpp_agent=LlamaCppAgent(main_model,debug_output=False,system_prompt="You are Dolphin, an uncensored and unbiased AI assistant.",predefined_messages_formatter_type=MessagesFormatterType.CHATML)whileTrue:user_input=input("User: ")ifuser_input=="exit":breakuser_input=llama_cpp_agent.get_chat_response(user_input,temperature=0.7)print("AI: "+user_input)Parallel Function Calling Agent Example# Example that uses the FunctionCallingAgent class to create a function calling agent.importdatetimeimportjsonfromenumimportEnumfromtypingimportUnion,Any,OptionalfrompydanticimportBaseModel,Fieldfromllama_cpp_agent.llm_settingsimportLlamaLLMSettings,LlamaLLMGenerationSettingsfromllama_cpp_agent.function_calling_agentimportFunctionCallingAgentfromllama_cpp_agent.providers.llama_cpp_endpoint_providerimportLlamaCppEndpointSettings,LlamaCppGenerationSettingsdefget_current_datetime(output_format:Optional[str]=None):"""Get the current date and time in the given format.Args:output_format: formatting string for the date and time, defaults to '%Y-%m-%d %H:%M:%S'"""ifoutput_formatisNone:output_format='%Y-%m-%d%H:%M:%S'returndatetime.datetime.now().strftime(output_format)# Enum for the calculator tool.classMathOperation(Enum):ADD="add"SUBTRACT="subtract"MULTIPLY="multiply"DIVIDE="divide"# llama-cpp-agent also supports "Instructor" library like function definitions as Pydantic models for function calling.# Simple pydantic calculator tool for the agent that can add, subtract, multiply, and divide. Docstring and description of fields will be used in system prompt.classCalculator(BaseModel):"""Perform a math operation on two numbers."""number_one:Union[int,float]=Field(...,description="First number.")operation:MathOperation=Field(...,description="Math operation to perform.")number_two:Union[int,float]=Field(...,description="Second number.")defrun(self):ifself.operation==MathOperation.ADD:returnself.number_one+self.number_twoelifself.operation==MathOperation.SUBTRACT:returnself.number_one-self.number_twoelifself.operation==MathOperation.MULTIPLY:returnself.number_one*self.number_twoelifself.operation==MathOperation.DIVIDE:returnself.number_one/self.number_twoelse:raiseValueError("Unknown operation.")# Example function based on an OpenAI example.# llama-cpp-agent also supports OpenAI like dictionaries for function definition.defget_current_weather(location,unit):"""Get the current weather in a given location"""if"London"inlocation:returnjson.dumps({"location":"London","temperature":"42","unit":unit.value})elif"New York"inlocation:returnjson.dumps({"location":"New York","temperature":"24","unit":unit.value})elif"North Pole"inlocation:returnjson.dumps({"location":"North Pole","temperature":"-42","unit":unit.value})else:returnjson.dumps({"location":location,"temperature":"unknown"})# Here is a function definition in OpenAI styletools=[{"type":"function","function":{"name":"get_current_weather","description":"Get the current weather in a given location","parameters":{"type":"object","properties":{"location":{"type":"string","description":"The city and state, e.g. San Francisco, CA",},"unit":{"type":"string","enum":["celsius","fahrenheit"]},},"required":["location"],},},}]# To make the OpenAI function callable for the function calling agent we need a list with actual function in it:tool_functions=[get_current_weather]# Callback for receiving messages for the user.defsend_message_to_user_callback(message:str):print(message)generation_settings=LlamaCppGenerationSettings(temperature=0.65,stream=True)# Can be saved and loaded like that:# generation_settings.save("generation_settings.json")# generation_settings = LlamaLLMGenerationSettings.load_from_file("generation_settings.json")main_model=LlamaCppEndpointSettings("http://localhost:8080/completion")function_call_agent=FunctionCallingAgent(# Can be lama-cpp-python Llama class, llama_cpp_agent.llm_settings.LlamaLLMSettings class or llama_cpp_agent.providers.llama_cpp_server_provider.LlamaCppServerLLMSettings.main_model,# llama_cpp_agent.llm_settings.LlamaLLMGenerationSettings  class or llama_cpp_agent.providers.llama_cpp_server_provider.LlamaCppServerGenerationSettings.llama_generation_settings=generation_settings,# A tuple of the OpenAI style function definitions and the actual functionsopen_ai_functions=(tools,tool_functions),# Just a list of type hinted functions for normal Python functionspython_functions=[get_current_datetime],# Just a list of pydantic typespydantic_functions=[Calculator],# Callback for receiving messages for the user.send_message_to_user_callback=send_message_to_user_callback,# Set to true to allow parallel function callingallow_parallel_function_calling=True,debug_output=True)user_input='''Format the answer clearly: Get the date and time, get the current weather in celsius in London and solve the following calculation: 42 * 42'''function_call_agent.generate_response(user_input)function_call_agent.save("function_calling_agent.json")Example Output:The current date and time is 2024-01-13 19:06:43.
The result of the calculation 42 * 42 is 1764.
The current weather in London is 42 degrees Celsius.Structured OutputThis example shows how to get structured output objects using the StructureOutputAgent class.# Example agent that uses the StructuredOutputAgent class to create a dataset entry of a book out of unstructured data.fromenumimportEnumfromllama_cppimportLlamafrompydanticimportBaseModel,Fieldfromllama_cpp_agent.structured_output_agentimportStructuredOutputAgent# Example enum for our output modelclassCategory(Enum):Fiction="Fiction"NonFiction="Non-Fiction"# Example output modelclassBook(BaseModel):"""Represents an entry about a book."""title:str=Field(...,description="Title of the book.")author:str=Field(...,description="Author of the book.")published_year:int=Field(...,description="Publishing year of the book.")keywords:list[str]=Field(...,description="A list of keywords.")category:Category=Field(...,description="Category of the book.")summary:str=Field(...,description="Summary of the book.")main_model=Llama("../gguf-models/nous-hermes-2-solar-10.7b.Q6_K.gguf",n_gpu_layers=49,offload_kqv=True,f16_kv=True,use_mlock=False,embedding=False,n_threads=8,n_batch=1024,n_ctx=4096,last_n_tokens_size=1024,verbose=False,seed=42,)structured_output_agent=StructuredOutputAgent(main_model,debug_output=True)text="""The Feynman Lectures on Physics is a physics textbook based on some lectures by Richard Feynman, a Nobel laureate who has sometimes been called "The Great Explainer". The lectures were presented before undergraduate students at the California Institute of Technology (Caltech), during 1961–1963. The book's co-authors are Feynman, Robert B. Leighton, and Matthew Sands."""print(structured_output_agent.create_object(Book,text))Example output{ "title": "The Feynman Lectures on Physics"  ,  "author": "Richard Feynman, Robert B. Leighton, Matthew Sands"  ,  "published_year": 1963 ,  "keywords": [ "physics" , "textbook" , "Nobel laureate" , "The Great Explainer" , "California Institute of Technology" , "undergraduate" , "lectures"  ] ,  "category": "Non-Fiction" ,  "summary": "The Feynman Lectures on Physics is a physics textbook based on lectures by Nobel laureate Richard Feynman, known as 'The Great Explainer'. The lectures were presented to undergraduate students at Caltech between 1961 and 1963. Co-authors of the book are Feynman, Robert B. Leighton, and Matthew Sands."  }


title='The Feynman Lectures on Physics' author='Richard Feynman, Robert B. Leighton, Matthew Sands' published_year=1963 keywords=['physics', 'textbook', 'Nobel laureate', 'The Great Explainer', 'California Institute of Technology', 'undergraduate', 'lectures'] category=<Category.NonFiction: 'Non-Fiction'> summary="The Feynman Lectures on Physics is a physics textbook based on lectures by Nobel laureate Richard Feynman, known as 'The Great Explainer'. The lectures were presented to undergraduate students at Caltech between 1961 and 1963. Co-authors of the book are Feynman, Robert B. Leighton, and Matthew Sands."Knowledge Graph Creation ExampleThis example, based on an example of the Instructor library for OpenAI,
demonstrates how to create a knowledge graph using the llama-cpp-agent framework.importjsonfromtypingimportListfromenumimportEnumfromllama_cppimportLlama,LlamaGrammarfrompydanticimportBaseModel,Fieldfromllama_cpp_agent.llm_agentimportLlamaCppAgentfromllama_cpp_agent.gbnf_grammar_generator.gbnf_grammar_from_pydantic_modelsimportgenerate_gbnf_grammar_and_documentationmain_model=Llama("../gguf-models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf",n_gpu_layers=13,f16_kv=True,use_mlock=False,embedding=False,n_threads=8,n_batch=1024,n_ctx=8192,last_n_tokens_size=1024,verbose=True,seed=42,)classNode(BaseModel):id:intlabel:strcolor:strclassEdge(BaseModel):source:inttarget:intlabel:strcolor:str="black"classKnowledgeGraph(BaseModel):nodes:List[Node]=Field(...,default_factory=list)edges:List[Edge]=Field(...,default_factory=list)gbnf_grammar,documentation=generate_gbnf_grammar_and_documentation([KnowledgeGraph],False)llama_cpp_agent=LlamaCppAgent(main_model,debug_output=True,system_prompt="You are an advanced AI assistant responding in JSON format.\n\nAvailable JSON response models:\n\n"+documentation)fromgraphvizimportDigraphdefvisualize_knowledge_graph(kg:KnowledgeGraph):dot=Digraph(comment="Knowledge Graph")# Add nodesfornodeinkg.nodes:dot.node(str(node.id),node.label,color=node.color)# Add edgesforedgeinkg.edges:dot.edge(str(edge.source),str(edge.target),label=edge.label,color=edge.color)# Render the graphdot.render("knowledge_graph.gv",view=True)defgenerate_graph(user_input:str)->KnowledgeGraph:prompt=f'''Help me understand the following by describing it as a detailed knowledge graph:{user_input}'''.strip()response=llama_cpp_agent.get_chat_response(message=prompt,temperature=0.65,mirostat_mode=0,mirostat_tau=3.0,mirostat_eta=0.1,grammar=gbnf_grammar)knowledge_graph=json.loads(response)cls=KnowledgeGraphknowledge_graph=cls(**knowledge_graph)returnknowledge_graphgraph=generate_graph("Teach me about quantum mechanics")visualize_knowledge_graph(graph)Example Output:Manual Function Calling ExampleThis example shows how to do function calling with pydantic models.
You can also convert Python functions with type hints, automatically to pydantic models using the function:create_dynamic_model_from_functionunder:llama_cpp_agent.gbnf_grammar_generator.gbnf_grammar_from_pydantic_modelsimportjsonimportmathfromtypingimportType,Unionfromllama_cppimportLlama,LlamaGrammarfromllama_cpp_agent.llm_agentimportLlamaCppAgentfromllama_cpp_agent.gbnf_grammar_generator.gbnf_grammar_from_pydantic_modelsimport\generate_gbnf_grammar_and_documentation,create_dynamic_model_from_functionfromllama_cpp_agent.providers.llama_cpp_endpoint_providerimportLlamaCppEndpointSettingsdefcalculate_a_to_the_power_b(a:Union[int|float],b:Union[int|float]):"""Calculates a to the power of bArgs:a: numberb: exponent"""print(f"Result:{math.pow(a,b)}")DynamicSampleModel=create_dynamic_model_from_function(calculate_a_to_the_power_b)grammar,documentation=generate_gbnf_grammar_and_documentation([DynamicSampleModel],outer_object_name="function",outer_object_content="params")main_model=LlamaCppEndpointSettings(completions_endpoint_url="http://127.0.0.1:8080/completion")llama_cpp_agent=LlamaCppAgent(main_model,debug_output=True,system_prompt="You are an advanced AI, tasked to generate JSON objects for function calling.\n\n"+documentation)response=llama_cpp_agent.get_chat_response("a= 5, b = 42",temperature=0.15,grammar=grammar)function_call=json.loads(response)instance=DynamicSampleModel(**function_call['params'])instance.run()Example output{ "function": "calculator","function_parameters": { "number_one": 42.00000 ,  "operation": "multiply" ,  "number_two": 42.00000 }}
1764.0Manual Function Calling with Python Function ExampleThis example shows how to do function calling using actual Python functions.fromllama_cppimportLlamafromtypingimportUnionimportmathfromllama_cpp_agent.llm_agentimportLlamaCppAgentfromllama_cpp_agent.messages_formatterimportMessagesFormatterTypefromllama_cpp_agent.function_callingimportLlamaCppFunctionToolfromllama_cpp_agent.gbnf_grammar_generator.gbnf_grammar_from_pydantic_modelsimportcreate_dynamic_model_from_functiondefcalculate_a_to_the_power_b(a:Union[int,float],b:Union[int,float]):"""Calculates 'a' to the power 'b' and returns the result"""returnf"Result:{math.pow(a,b)}"DynamicSampleModel=create_dynamic_model_from_function(calculate_a_to_the_power_b)function_tools=[LlamaCppFunctionTool(DynamicSampleModel)]function_tool_registry=LlamaCppAgent.get_function_tool_registry(function_tools)main_model=Llama("../../gguf-models/openhermes-2.5-mistral-7b-16k.Q8_0.gguf",n_gpu_layers=49,offload_kqv=True,f16_kv=True,use_mlock=False,embedding=False,n_threads=8,n_batch=1024,n_ctx=8192,last_n_tokens_size=1024,verbose=True,seed=42,)llama_cpp_agent=LlamaCppAgent(main_model,debug_output=True,system_prompt="You are an advanced AI, tasked to assist the user by calling functions in JSON format. The following are the available functions and their parameters and types:\n\n"+function_tool_registry.get_documentation(),predefined_messages_formatter_type=MessagesFormatterType.CHATML)user_input="Calculate 5 to power 42"print(llama_cpp_agent.get_chat_response(user_input,temperature=0.45,function_tool_registry=function_tool_registry))Example output{ "function": "calculate-a-to-the-power-b","function_parameters": { "a": 5 ,  "b": 42  }}
Result: 2.2737367544323207e+29Additional InformationDependencies: pydantic for grammars based generation and of course llama-cpp-python.Documentation: You can find the documentation here:https://llama-cpp-agent.readthedocs.io/en/latest/agents-api-reference/Predefined Messages FormatterThe llama-cpp-agent framework uses custom messages formatters to format messages for the LLM model. TheMessagesFormatterTypeenum defines the available predefined formatters. The following predefined formatters are available:MessagesFormatterType.CHATML: Formats messages using the CHATML format.MessagesFormatterType.MIXTRAL: Formats messages using the MIXTRAL format.MessagesFormatterType.VICUNA: Formats messages using the VICUNA format.MessagesFormatterType.LLAMA_2: Formats messages using the LLAMA 2 format.MessagesFormatterType.SYNTHIA: Formats messages using the SYNTHIA format.MessagesFormatterType.NEURAL_CHAT: Formats messages using the NEURAL CHAT format.MessagesFormatterType.SOLAR: Formats messages using the SOLAR format.MessagesFormatterType.OPEN_CHAT: Formats messages using the OPEN CHAT format.You can also define your own custom messages formatter by creating an instance of theMessagesFormatterclass.
TheMessagesFormatterclass takes the following parameters:PRE_PROMPT: The pre-prompt to use for the messages.SYS_PROMPT_START: The system prompt start to use for the messages.SYS_PROMPT_END: The system prompt end to use for the messages.USER_PROMPT_START: The user prompt start to use for the messages.USER_PROMPT_END: The user prompt end to use for the messages.ASSISTANT_PROMPT_START: The assistant prompt start to use for the messages.ASSISTANT_PROMPT_END: The assistant prompt end to use for the messages.INCLUDE_SYS_PROMPT_IN_FIRST_USER_MESSAGE: Whether to include the system prompt in the first user message.DEFAULT_STOP_SEQUENCES: The default stop sequences to use for the messages.