Package: libretto-openai
Libretto OpenAI ClientA drop-in replacement ofopenai.Clientfor sending events to Libretto.FeaturesProvides a patchedopenai.Clientthat allows for setting Libretto-specific parameters for each request.Currently supports the synchronous versions ofcompletions.create()andchat.completions.create().Get StartedTo send events to Libretto, you'll need to create a project. From the project you'll need two things:API key: (api_key) This is generated for the project and is used to identify the project and environment (dev, staging, prod) that the event is coming from.Template Name: (prompt_template_name) This uniquely identifies a particular prompt that you are using and allows projects to have multiple prompts. This can be in any format but we recommend using a dash-separated format, e.g.my-prompt-name.Note:Prompt template names can be auto-generated if theallow_unnamed_promptsconfiguration option is set (seebelow). However, if you rely on auto-generated names, new revisions of the same prompt will show up as different prompt templates in Libretto.UsageYou can use thelibretto_openai.Clientanywhere that you're currently using the officialopenai.Client.When instantiating alibretto_openai.Client, you can/should provide any of the existingopenai.Clientparameters in the constructor. Libretto-specific configuration can be provided via an additionallibrettoargument (see below).To allow our tools to separate the "prompt" from the "prompt parameters", useTemplateChatandTemplateTextto create templates.UseTemplateChatFor the ChatCompletion APIs:fromlibretto_openaiimport(Client,LibrettoConfig,LibrettoCreateParams,TemplateChat,)client=Client(api_key="<OpenAI API Key>",libretto=LibrettoConfig(api_key="<Libretto API Key>",),)completion=client.chat.completions.create(# Standard OpenAI parametersmodel="gpt-3.5-turbo",messages=TemplateChat([{"role":"user","content":"Show me an emoji that matches the sport:{sport}"}],{"sport":"soccer"},),libretto=LibrettoCreateParams(prompt_template_name="sport-emoji",),)UseTemplateTextfor the Completion API:fromlibretto_openaiimport(Client,LibrettoConfig,LibrettoCreateParams,TemplateChat,)client=Client(api_key="<OpenAI API Key>",libretto=LibrettoConfig(api_key="<Libretto API Key>",),)completion=client.completions.create(# Standard OpenAI parametersmodel="text-davinci-003",prompt=TemplateText("Show me an emoji that matches the sport:{sport}",{"sport":"soccer"},),libretto=LibrettoCreateParams(prompt_template_name="sport-emoji",),)Advanced usageManually passing parametersWhile the use ofTemplateTextandTemplateChatare preferred, you can optionally specify template data inline when calling thecreate()method:completion=openai.ChatCompletion.create(model="gpt-3.5-turbo",# Note we are passing the raw messages object heremessages=[{"role":"user","content":"Show me an emoji that matches the sport: soccer"}],libretto=LibrettoCreateParams(# call configurationprompt_template_name="sport-emoji",# Here the prompt and parameters are passed seperatelytemplate_params={"sport":"soccer"},template_chat=[{"role":"user","content":"Show me an emoji that matches the sport:{sport}"}],),)ConfigurationThelibrettokwarg that's present on theClientconstructor is aLibrettoConfigobject with the following options:prompt_template_name: A default name to associate with prompts. If provided,
this is the name that will be associated with anycreatecall that's madewithoutalibretto.prompt_template_nameparameter.allow_unnamed_prompts: When set toTrue, every prompt will be sent to
Libretto even if no prompt template name as been provided (either via theprompt_template_namekwarg or via thelibretto.prompt_template_nameparameter oncreate).Falseby default.redact_pii: WhenTrue, certain personally identifying information (PII) will be attempted to be redacted before being sent to the Libretto backend. See thepiipackage for details about the types of PII being detected/redacted.Falseby default.Additional Create Call ParametersWhen callingcreate(), alibrettoargument should be provided to give Libretto-specific context to the call. The following parameters maybe specified:chat_id: The id of a "chat session" - if the chat API is
being used in a conversational context, then the same chat id can be
provided so that the events are grouped together, in order. If not provided,
this will be left blank.template_chat: The chattemplateto record for chat requests. This is a list of dictionaries with the following keys:role: The role of the speaker. Either"system","user"or"ai".content: The content of the message. This can be a string or a template string with{}placeholders.For example:completion=client.chat.completions.create(...,libretto=LibrettoCreateParams(template_chat=[{"role":"ai","content":"Hello, I'm{system_name}!"},{"role":"user","content":"Hi{system_name}, I'm{user_name}!"}],),)To represent an array of chat messages, use the artificial role"chat_history"withcontentset to the variable name in substitution format:[{"role": "chat_history", "content": "{prev_messages}"}}]template_text: The text template to record for completion requests. This is a string or a template string with{}placeholders.For example:completion=client.completions.create(...,libretto=LibrettoCreateParams(template_text="Please welcome the user to{system_name}!",),)template_params: The parameters to use for template strings. This is a dictionary of key-value pairs.For example:completion=client.completions.create(...,libretto=LibrettoCreateParams(template_text="Please welcome the user to{system_name}!",template_params={"system_name":"Awesome Comics Incorporated"},),)event_id: A unique UUID for a specific call. If not provided, one will be generated.For example:importuuidcompletion=client.completions.create(...,libretto=LibrettoCreateParams(event_id=uuid.uuid4(),),)parent_event_id: The UUID of the parent event. All calls with the same parent id are grouped as a "Run Group".For example:importuuidparent_id=uuid.uuid4()# First call in the run groupcompletion=client.completions.create(...,libretto=LibrettoCreateParams(parent_event_id=parent_id,),)# Another call in the same groupcompletion=client.completions.create(...,libretto=LibrettoCreateParams(parent_event_id=parent_id,),)Sending FeedbackSometimes the answer provided by the LLM is not ideal, and your users may be
able to help you find better responses. There are a few common cases:You might use the LLM to suggest the title of a news article, but let the
user edit it. If they change the title, you can send feedback to Libretto
that the answer was not ideal.You might provide a chatbot that answers questions, and the user can rate the
answers with a thumbs up (good) or thumbs down (bad).You can send this feedback to Libretto by callingsend_feedback(). This will
send a feedback event to Libretto about a prompt that was previously called, and
let you review this feedback in the Libretto dashboard. You can use this
feedback to develop new tests and improve your prompts.fromlibretto_openaiimportClientclient=Client()completion=client.completions.create(...)# Maybe the user didn't like the answer, so ask them for a better onebetter_response=ask_user_for_better_response(completion.choices[0].text)# If the user provided a better answer, send feedback to Librettoifbetter_response!==completion.choices[0].text:# feedback key is automatically injected into OpenAI response object as an extra fieldfeedback_key=completion.model_extra.get("libretto_feedback_key")client.send_feedback(feedback_key=feedback_key,# Better answer from the userbetter_response=better_response,# Rating of existing answer, from 0 to 1rating=0.2,)Note that feedback can include eitherrating,better_response, or both.Parameters:rating- a value from 0 (meaning the result was completely wrong) to 1 (meaning the result was correct)better_response- the better response from the userCreditsThis package was created with Cookiecutter and theaudreyr/cookiecutter-pypackageproject template.Cookiecutter:https://github.com/audreyr/cookiecutteraudreyr/cookiecutter-pypackage:https://github.com/audreyr/cookiecutter-pypackage