Package: parea-ai
Parea Python SDKParea python sdkPython SDK DocsInstallationpipinstall-Uparea-aior install withPoetrypoetryaddparea-aiEvaluating Your LLM AppYou can evaluate any step of your LLM app by wrapping it with a decorator, calledtrace, and specifying the evaluation
function(s).
The scores associated with the traces will be logged to the Pareadashboardand/or in a
local CSV file if you don't have a Parea API key.Evaluation functions receive an argumentlog(of typeLog) and should return a
float. You don't need to start from scratch, there are pre-defined evaluation
functions forgeneral purpose,chat,RAG, andsummarizationapps :)You can define evaluation functions locally or use the ones you have deployed to
Parea'sTest Hub.
If you choose the latter option, the evaluation happens asynchronously and non-blocking.A fully locally working cookbook can be foundhere.
Alternatively, you can add the following code to your codebase to get started:importosfrompareaimportParea,InMemoryCache,tracefromparea.schemas.logimportLogParea(api_key=os.getenv("PAREA_API_KEY"),cache=InMemoryCache())# use InMemoryCache if you don't have a Parea API keydeflocally_defined_eval_function(log:Log)->float:...@trace(eval_func_names=['deployed_eval_function_name'],eval_funcs=[locally_defined_eval_function])deffunction_to_evaluate(*args,**kwargs)->...:...Run ExperimentsYou can run an experiment for your LLM application by defining theExperimentclass and passing it the name, the data and the
function you want to run. You need annotate the function with thetracedecorator to trace its inputs, outputs, latency, etc.
as well as to specify which evaluation functions should be applied to it (as shown above).frompareaimportExperimentExperiment(name="Experiment Name",# Name of the experiment (str)data=[{"n":"10"}],# Data to run the experiment on (list of dicts)func=function_to_evaluate,# Function to run (callable))Then you can run the experiment by using theexperimentcommand and give it the path to the python file.
This will run your experiment with the specified inputs and create a report with the results which can be viewed under
theExperiments tab.pareaexperiment<path/to/experiment_file.py>Full working example in ourdocs.Debugging Chains & AgentsYou can iterate on your chains & agents much faster by using a local cache. This will allow you to make changes to your
code & prompts without waiting for all previous, valid LLM responses. Simply add these two lines to the beginning your
code and starta local redis cache:frompareaimportParea,RedisCacheParea(cache=RedisCache())Above will use the default redis cache atlocalhost:6379with no password. You can also specify your redis database
by:frompareaimportParea,RedisCachecache=RedisCache(host=os.getenv("REDIS_HOST","localhost"),# default valueport=int(os.getenv("REDIS_PORT",6379)),# default valuepassword=os.getenv("REDIS_PASSWORT",None)# default value)Parea(cache=cache)If you setcache = NoneforParea, no cache will be used.Benchmark your LLM app across many inputsYou can benchmark your LLM app across many inputs by using thebenchmarkcommand. This will run your the entry point
of your app with the specified inputs and create a report with the results.pareabenchmark--funcapp:main--csv_pathbenchmark.csvThe CSV file will be used to fill in the arguments to your function. The report will be a CSV file of all the traces. If
you
set your Parea API key, the traces will also be logged to the Parea dashboard. Note, for this feature you need to have a
redis cache running. Please, raise a GitHub issue if you would like to use this feature without a redis cache.Automatically log all your LLM call tracesYou can automatically log all your LLM traces to the Parea dashboard by setting thePAREA_API_KEYenvironment variable
or specifying it in thePareainitialization.
This will help you debug issues your customers are facing by stepping through the LLM call traces and recreating the
issue
in your local setup & code.frompareaimportPareaParea(api_key=os.getenv("PAREA_API_KEY"),# default valuecache=...)Use a deployed promptimportosfromdotenvimportload_dotenvfrompareaimportPareafromparea.schemas.modelsimportCompletion,UseDeployedPrompt,CompletionResponse,UseDeployedPromptResponseload_dotenv()p=Parea(api_key=os.getenv("PAREA_API_KEY"))# You will find this deployment_id in the Parea dashboarddeployment_id='<DEPLOYMENT_ID>'# Assuming your deployed prompt's message is:# {"role": "user", "content": "Write a hello world program using {{x}} and the {{y}} framework."}inputs={"x":"Golang","y":"Fiber"}# You can easily unpack a dictionary into an attrs classtest_completion=Completion(**{"deployment_id":deployment_id,"llm_inputs":inputs,"metadata":{"purpose":"testing"}})# By passing in my inputs, in addition to the raw message with unfilled variables {{x}} and {{y}},# you we will also get the filled-in prompt:# {"role": "user", "content": "Write a hello world program using Golang and the Fiber framework."}test_get_prompt=UseDeployedPrompt(deployment_id=deployment_id,llm_inputs=inputs)defmain():completion_response:CompletionResponse=p.completion(data=test_completion)print(completion_response)deployed_prompt:UseDeployedPromptResponse=p.get_prompt(data=test_get_prompt)print("\n\n")print(deployed_prompt)asyncdefmain_async():completion_response:CompletionResponse=awaitp.acompletion(data=test_completion)print(completion_response)deployed_prompt:UseDeployedPromptResponse=awaitp.aget_prompt(data=test_get_prompt)print("\n\n")print(deployed_prompt)Logging results from LLM providers [Example]importosimportopenaifromdotenvimportload_dotenvfrompareaimportPareaload_dotenv()openai.api_key=os.getenv("OPENAI_API_KEY")p=Parea(api_key=os.getenv("PAREA_API_KEY"))x="Golang"y="Fiber"messages=[{"role":"user","content":f"Write a hello world program using{x}and the{y}framework."}]model="gpt-3.5-turbo"temperature=0.0# define your OpenAI call as you would normally and we'll automatically log the resultsdefmain():openai.chat.completions.create(model=model,temperature=temperature,messages=messages).choices[0].message.contentOpen source community featuresReady-to-usePull Requests templatesand severalIssue templates.Files such as:LICENSE,CONTRIBUTING.md,CODE_OF_CONDUCT.md, andSECURITY.mdare generated automatically.Semantic Versionsspecification
withRelease Drafter.🛡 LicenseThis project is licensed under the terms of theApache Software License 2.0license.
SeeLICENSEfor more details.📃 Citation@misc{parea-sdk,author={joel-parea-ai},title={Parea python sdk},year={2023},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/parea-ai/parea-sdk}}}