Package: magentic
magenticEasily integrate Large Language Models into your Python code. Simply use the@promptdecorator to create functions that return structured output from the LLM. Mix LLM queries and function calling with regular Python code to create complex logic.magenticisCompact:Query LLMs without duplicating boilerplate code.Atomic:Prompts are functions that can be individually tested and reasoned about.Transparent:Create "chains" using regular Python code. Define all of your own prompts.Compatible:Use@promptfunctions as normal functions, including with decorators like@lru_cache.Type Annotated:Works with linters and IDEs.Continue reading for sample usage, or go straight to theexamples directory.Installationpipinstallmagenticor using poetrypoetryaddmagenticConfigure your OpenAI API key by setting theOPENAI_API_KEYenvironment variable or usingopenai.api_key = "sk-...". See theOpenAI Python library documentationfor more information.UsageThe@promptdecorator allows you to define a template for a Large Language Model (LLM) prompt as a Python function. When this function is called, the arguments are inserted into the template, then this prompt is sent to an LLM which generates the function output.frommagenticimportprompt@prompt('Add more "dude"ness to:{phrase}')defdudeify(phrase:str)->str:...# No function body as this is never executeddudeify("Hello, how are you?")# "Hey, dude! What's up? How's it going, my man?"The@promptdecorator will respect the return type annotation of the decorated function. This can beany type supported by pydanticincluding apydanticmodel.frommagenticimportpromptfrompydanticimportBaseModelclassSuperhero(BaseModel):name:strage:intpower:strenemies:list[str]@prompt("Create a Superhero named{name}.")defcreate_superhero(name:str)->Superhero:...create_superhero("Garden Man")# Superhero(name='Garden Man', age=30, power='Control over plants', enemies=['Pollution Man', 'Concrete Woman'])An LLM can also decide to call functions. In this case the@prompt-decorated function returns aFunctionCallobject which can be called to execute the function using the arguments provided by the LLM.fromtypingimportLiteralfrommagenticimportprompt,FunctionCalldefactivate_oven(temperature:int,mode:Literal["broil","bake","roast"])->str:"""Turn the oven on with the provided settings."""returnf"Preheating to{temperature}F with mode{mode}"@prompt("Prepare the oven so I can make{food}",functions=[activate_oven],)defconfigure_oven(food:str)->FunctionCall[str]:...output=configure_oven("cookies!")# FunctionCall(<function activate_oven at 0x1105a6200>, temperature=350, mode='bake')output()# 'Preheating to 350 F with mode bake'Sometimes the LLM requires making one or more function calls to generate a final answer. The@prompt_chaindecorator will resolveFunctionCallobjects automatically and pass the output back to the LLM to continue until the final answer is reached.In the following example, whendescribe_weatheris called the LLM first calls theget_current_weatherfunction, then uses the result of this to formulate its final answer which gets returned.frommagenticimportprompt_chaindefget_current_weather(location,unit="fahrenheit"):"""Get the current weather in a given location"""# Pretend to query an APIreturn{"location":location,"temperature":"72","unit":unit,"forecast":["sunny","windy"],}@prompt_chain("What's the weather like in{city}?",functions=[get_current_weather],)defdescribe_weather(city:str)->str:...describe_weather("Boston")# 'The current weather in Boston is 72°F and it is sunny and windy.'LLM-powered functions created using@promptand@prompt_chaincan be supplied asfunctionsto other@prompt/@prompt_chaindecorators, just like regular python functions. This enables increasingly complex LLM-powered functionality, while allowing individual components to be tested and improved in isolation.See theexamples directoryfor more.Chat PromptingThe@chatpromptdecorator works just like@promptbut allows you to pass chat messages as a template rather than a single text prompt. This can be used to provide a system message or for few-shot prompting where you provide example responses to guide the model's output. Format fields denoted by curly braces{example}will be filled in all messages - use theescape_bracesfunction to prevent a string being used as a template.frommagenticimportchatprompt,AssistantMessage,SystemMessage,UserMessagefrommagentic.chatpromptimportescape_bracesfrompydanticimportBaseModelclassQuote(BaseModel):quote:strcharacter:str@chatprompt(SystemMessage("You are a movie buff."),UserMessage("What is your favorite quote from Harry Potter?"),AssistantMessage(Quote(quote="It does not do to dwell on dreams and forget to live.",character="Albus Dumbledore",)),UserMessage("What is your favorite quote from{movie}?"),)defget_movie_quote(movie:str)->Quote:...get_movie_quote("Iron Man")# Quote(quote='I am Iron Man.', character='Tony Stark')StreamingTheStreamedStr(andAsyncStreamedStr) class can be used to stream the output of the LLM. This allows you to process the text while it is being generated, rather than receiving the whole output at once.frommagenticimportprompt,StreamedStr@prompt("Tell me about{country}")defdescribe_country(country:str)->StreamedStr:...# Print the chunks while they are being receivedforchunkindescribe_country("Brazil"):print(chunk,end="")# 'Brazil, officially known as the Federative Republic of Brazil, is ...'MultipleStreamedStrcan be created at the same time to stream LLM outputs concurrently. In the below example, generating the description for multiple countries takes approximately the same amount of time as for a single country.fromtimeimporttimecountries=["Australia","Brazil","Chile"]# Generate the descriptions one at a timestart_time=time()forcountryincountries:# Converting `StreamedStr` to `str` blocks until the LLM output is fully generateddescription=str(describe_country(country))print(f"{time()-start_time:.2f}s :{country}-{len(description)}chars")# 22.72s : Australia - 2130 chars# 41.63s : Brazil - 1884 chars# 74.31s : Chile - 2968 chars# Generate the descriptions concurrently by creating the StreamedStrs at the same timestart_time=time()streamed_strs=[describe_country(country)forcountryincountries]forcountry,streamed_strinzip(countries,streamed_strs):description=str(streamed_str)print(f"{time()-start_time:.2f}s :{country}-{len(description)}chars")# 22.79s : Australia - 2147 chars# 23.64s : Brazil - 2202 chars# 24.67s : Chile - 2186 charsObject StreamingStructured outputs can also be streamed from the LLM by using the return type annotationIterable(orAsyncIterable). This allows each item to be processed while the next one is being generated. See the example inexamples/quizfor how this can be used to improve user experience by quickly displaying/using the first item returned.fromcollections.abcimportIterablefromtimeimporttimefrommagenticimportpromptfrompydanticimportBaseModelclassSuperhero(BaseModel):name:strage:intpower:strenemies:list[str]@prompt("Create a Superhero team named{name}.")defcreate_superhero_team(name:str)->Iterable[Superhero]:...start_time=time()forheroincreate_superhero_team("The Food Dudes"):print(f"{time()-start_time:.2f}s :{hero}")# 2.23s : name='Pizza Man' age=30 power='Can shoot pizza slices from his hands' enemies=['The Hungry Horde', 'The Junk Food Gang']# 4.03s : name='Captain Carrot' age=35 power='Super strength and agility from eating carrots' enemies=['The Sugar Squad', 'The Greasy Gang']# 6.05s : name='Ice Cream Girl' age=25 power='Can create ice cream out of thin air' enemies=['The Hot Sauce Squad', 'The Healthy Eaters']AsyncioAsynchronous functions / coroutines can be used to concurrently query the LLM. This can greatly increase the overall speed of generation, and also allow other asynchronous code to run while waiting on LLM output. In the below example, the LLM generates a description for each US president while it is waiting on the next one in the list. Measuring the characters generated per second shows that this example achieves a 7x speedup over serial processing.importasynciofromtimeimporttimefromtypingimportAsyncIterablefrommagenticimportprompt@prompt("List ten presidents of the United States")asyncdefiter_presidents()->AsyncIterable[str]:...@prompt("Tell me more about{topic}")asyncdeftell_me_more_about(topic:str)->str:...# For each president listed, generate a description concurrentlystart_time=time()tasks=[]asyncforpresidentinawaititer_presidents():# Use asyncio.create_task to schedule the coroutine for execution before awaiting it# This way descriptions will start being generated while the list of presidents is still being generatedtask=asyncio.create_task(tell_me_more_about(president))tasks.append(task)descriptions=awaitasyncio.gather(*tasks)# Measure the characters per secondtotal_chars=sum(len(desc)fordescindescriptions)time_elapsed=time()-start_timeprint(total_chars,time_elapsed,total_chars/time_elapsed)# 24575 28.70 856.07# Measure the characters per second to describe a single presidentstart_time=time()out=awaittell_me_more_about("George Washington")time_elapsed=time()-start_timeprint(len(out),time_elapsed,len(out)/time_elapsed)# 2206 18.72 117.78Additional FeaturesThefunctionsargument to@promptcan contain async/coroutine functions. When the correspondingFunctionCallobjects are called the result must be awaited.TheAnnotatedtype annotation can be used to provide descriptions and other metadata for function parameters. Seethe pydantic documentation on usingFieldto describe function arguments.The@promptand@prompt_chaindecorators also accept amodelargument. You can pass an instance ofOpenaiChatModelto use GPT4 or configure a different temperature. See below.Register other types to use as return type annotations in@promptfunctions by followingthe example notebook for a Pandas DataFrame.Backend/LLM ConfigurationCurrently two backends are availableopenai: the default backend that uses theopenaiPython package. Supports all features.litellm: uses thelitellmPython package to enable querying LLMs frommany different providers. Install this withpip install magentic[litellm]. Note: some models may not support all features ofmagentice.g. function calling/structured output and streaming.The backend and LLM used bymagenticcan be configured in several ways. The order of precedence of configuration isArguments explicitly passed when initializing an instance in PythonValues set using a context manager in PythonEnvironment variablesDefault values fromsrc/magentic/settings.pyfrommagenticimportOpenaiChatModel,promptfrommagentic.chat_model.litellm_chat_modelimportLitellmChatModel@prompt("Say hello")defsay_hello()->str:...@prompt("Say hello",model=LitellmChatModel("ollama/llama2"),)defsay_hello_litellm()->str:...say_hello()# Uses env vars or default settingswithOpenaiChatModel("gpt-3.5-turbo",temperature=1):say_hello()# Uses openai with gpt-3.5-turbo and temperature=1 due to context managersay_hello_litellm()# Uses litellm with ollama/llama2 because explicitly configuredThe following environment variables can be set.Environment VariableDescriptionExampleMAGENTIC_BACKENDThe package to use as the LLM backendopenaiMAGENTIC_LITELLM_MODELLiteLLM modelclaude-2MAGENTIC_LITELLM_API_BASEThe base url to queryhttp://localhost:11434MAGENTIC_LITELLM_MAX_TOKENSLiteLLM max number of generated tokens1024MAGENTIC_LITELLM_TEMPERATURELiteLLM temperature0.5MAGENTIC_OPENAI_MODELOpenAI modelgpt-4MAGENTIC_OPENAI_API_KEYOpenAI API key to be used by magenticsk-...MAGENTIC_OPENAI_API_TYPEAllowed options: "openai", "azure"azureMAGENTIC_OPENAI_BASE_URLBase URL for an OpenAI-compatible APIhttp://localhost:8080MAGENTIC_OPENAI_MAX_TOKENSOpenAI max number of generated tokens1024MAGENTIC_OPENAI_SEEDSeed for deterministic sampling42MAGENTIC_OPENAI_TEMPERATUREOpenAI temperature0.5When using theopenaibackend, setting theMAGENTIC_OPENAI_BASE_URLenvironment variable or usingOpenaiChatModel(..., base_url="http://localhost:8080")in code allows you to usemagenticwith any OpenAI-compatible API e.g.Azure OpenAI Service,LiteLLM OpenAI Proxy Server,LocalAI. Note that if the API does not support function calling then you will not be able to create prompt-functions that return Python objects, but other features ofmagenticwill still work.To use Azure with the openai backend you will need to set theMAGENTIC_OPENAI_API_TYPEenvironment variable to "azure" or useOpenaiChatModel(..., api_type="azure"), and also set the environment variables needed by the openai package to access Azure. Seehttps://github.com/openai/openai-python#microsoft-azure-openaiType CheckingMany type checkers will raise warnings or errors for functions with the@promptdecorator due to the function having no body or return value. There are several ways to deal with these.Disable the check globally for the type checker. For example in mypy by disabling error codeempty-body.# pyproject.toml[tool.mypy]disable_error_code=["empty-body"]Make the function body...(this does not satisfy mypy) orraise.@prompt("Choose a color")defrandom_color()->str:...Use comment# type: ignore[empty-body]on each function. In this case you can add a docstring instead of....@prompt("Choose a color")defrandom_color()->str:# type: ignore[empty-body]"""Returns a random color."""